VereskLearn

Module 4 · Retrieval

Lesson 4.11

Multimodal retrieval

Multimodal retrieval searches images by text, using a caption or an image vector for each picture.

Lesson 30 of 622 min read

The problem

Why it matters

Without it

Text search cannot see a photo. A question about a red light finds nothing when the manual has only a picture.

With it

Each image gets a caption or an image vector. The question is matched against those, so the picture can be returned.

How it works

  • Each picture gets a caption

    A vision model reads each picture once, when the manual is loaded, and writes a caption. The caption then goes through the same search as a text chunk. Some systems embed the pixels directly, which can match pictures without a caption, but that costs more.

    FileCaption
    front.pngFront panel with a red light.
    back.pngBack view with two cables.
    side.pngSide view with a power button.
    Three pictures from a device manual, with their captions.
    // Each picture has a file name and a caption.const images = [  { file: "front.png", caption: "Front panel with a red light." },  { file: "back.png", caption: "Back view with two cables." },];for (const i of images) console.log(i.file, i.caption);

    Output

    front.png Front panel with a red light.
back.png Back view with two cables.

  • Counting words gives numbers

    The fake embedding counts how often the words light, cable, and red appear in a caption. The front panel caption has light once and red once, so its vector is 1, 0, 1. The back view has none of the three words, so its vector is 0, 0, 0. The side view is also 0, 0, 0.

    Captionlightcablered
    Front panel with a red light.101
    Back view with two cables.000
    Side view with a power button.000
    The vector for each caption. The columns count light, cable, and red.
    const keys = ["light", "cable", "red"];const embed = (text: string) => {  const words = text.toLowerCase().replace(".", "").split(" ");  return keys.map((k) => words.filter((w) => w === k).length);};console.log(embed("Front panel with a red light."));console.log(embed("Back view with two cables."));

    Output

    [ 1, 0, 1 ]
[ 0, 0, 0 ]

  • The question gets the same counts

    The question "blinking red light" goes through the same count, which gives 1, 0, 1. Each caption vector is then compared with it by a dot product. The front panel scores 2, and the back and side views score 0.

    Dot product of the question vector with each caption vector.
    const keys = ["light", "cable", "red"];const embed = (text: string) => {  const words = text.toLowerCase().replace(".", "").split(" ");  return keys.map((k) => words.filter((w) => w === k).length);};const dot = (a: number[], b: number[]) =>  a.reduce((sum, v, i) => sum + v * b[i], 0);const query = embed("blinking red light");console.log(dot(query, embed("Front panel with a red light.")));

    Output

    2

  • Ranking returns the image file

    Each image is scored with the dot product, and the list is sorted from high to low. The top file, front.png, is returned with its caption, so the reader can see what the picture shows. Small text inside a picture needs OCR, a method that reads printed text from an image.

    • front.pngscore 2, returned
    • back.pngscore 0
    • side.pngscore 0
    Images ranked for "blinking red light".

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Captioning only the title. A caption must describe what is visible, or the search cannot find it.
  • Searching images with keyword counts and expecting meaning. Use a real embedding model for meaning.
  • Returning the image without its page or source. A person needs the surrounding text to check it.

Use it when and skip it when

Is it the right tool?

Use it when

  • The manual or catalog depends on diagrams, photos, or screenshots.
  • Users ask about something visible, such as a light color or a panel layout.

Skip it when

  • The documents are text only. Use text chunks.
  • The picture has small text that a caption cannot hold. Run OCR first.

Words to know

Terms in this lesson

Caption
A short text that describes an image.
Vision model
A model that reads an image and produces text or numbers.
OCR
Optical character recognition, a method that reads printed text from an image.
Image vector
A list of numbers that stands for the meaning of a picture.
Full code