VereskLearn

Module 4 · Retrieval

Lesson 4.2

Metadata filters

A metadata filter removes chunks by their labels before the search scores them.

Lesson 21 of 622 min read

The problem

Why it matters

Without it

The search ranks an old 3-second reset article above the current 10-second one. Both chunks match the question well.

With it

Each chunk carries labels, such as its version. The search skips chunks whose labels do not match, so only current answers come back.

How it works

  • Each chunk is stored with labels

    Metadata is a set of labels saved with each chunk when it is loaded. The version label says which edition of the manual the text came from. Each label must be set before any question arrives.

    ChunkVersionVector
    Hold 10 seconds.20.9, 0.1
    Hold 3 seconds.10.95, 0.05
    Hold the power key.20.8, 0.2
    Three chunks from a device manual. Two editions are mixed in one set.
    // Each chunk carries its text and a version label.const chunks = [  { text: "Hold 10 seconds.", version: 2 },  { text: "Hold 3 seconds.", version: 1 },  { text: "Hold the power key.", version: 2 },];for (const c of chunks) console.log(c.version, c.text);

    Output

    2 Hold 10 seconds.
1 Hold 3 seconds.
2 Hold the power key.

  • A filter drops chunks before scoring

    A filter is a test on a label, such as version equal to 2. Chunks that fail the test are removed before any scoring happens. The version 1 chunk fails, so it is never scored and cannot rank first.

    • Hold 10 seconds.version 2, kept
    • Hold 3 seconds.version 1, dropped
    • Hold the power key.version 2, kept
    The three chunks after the version 2 filter. The version 1 chunk is dropped.
    // Keep only the chunks whose version label is 2.const chunks = [  { text: "Hold 10 seconds.", version: 2 },  { text: "Hold 3 seconds.", version: 1 },  { text: "Hold the power key.", version: 2 },];const kept = chunks.filter((c) => c.version === 2);console.log(kept.length, "of", chunks.length, "kept");

    Output

    2 of 3 kept

  • Scores decide the order of what is left

    Each kept chunk is scored with a dot product against the question vector, 0.9 and 0.1. Without the filter, the old 3-second chunk scores 0.86 and ranks first. With the filter, the current 10-second chunk scores 0.82 and ranks first.

    No filter

    • Hold 3 seconds. (0.86)
    • Hold 10 seconds. (0.82)
    • Hold the power key. (0.74)

    Version 2 only

    • Hold 10 seconds. (0.82)
    • Hold the power key. (0.74)
    The ranking for the same question, with and without the version 2 filter.
    // Dot product of the question with a chunk vector.const dot = (a: number[], b: number[]) => a[0] * b[0] + a[1] * b[1];const question = [0.9, 0.1];const old = dot(question, [0.95, 0.05]).toFixed(2);console.log("Old 3 seconds:", old);const current = dot(question, [0.9, 0.1]).toFixed(2);console.log("Current 10 seconds:", current);

    Output

    Old 3 seconds: 0.86
Current 10 seconds: 0.82

  • Labels must match exactly

    The filter compares labels with exact equality. A label typed as "Version 2" fails a test for the number 2. A chunk with no version label fails every version test, so check labels when the documents are loaded.

    Passes

    • version = 2, a number
    • Label set by the loader

    Fails

    • version = "Version 2", text not a number
    • No version label at all
    Which labels pass a version 2 test and which fail.
    // A chunk loaded with no version label at all.const chunk: { text: string; version?: number } = {  text: "Hold the power key.",};console.log("Passes version 2:", chunk.version === 2);

    Output

    Passes version 2: false

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Filtering on a label that was never set. Every chunk fails the filter, and the answer is empty.
  • Filtering too hard. A question that names no version still needs the chunks of every version.
  • Writing the label inside the text instead of in the metadata. The filter cannot read a label inside a sentence.

Use it when and skip it when

Is it the right tool?

Use it when

  • Two chunks say the same thing, but one of them is outdated.
  • Answers depend on a field, such as a product line, a region, or a date.

Skip it when

  • The filter would remove the right answer. Test the filter with a question first.
  • The label is not known when the question arrives. A filter needs a value to test.

Words to know

Terms in this lesson

Metadata
Labels stored with a chunk, such as its version or source.
Filter
A test that keeps or drops each chunk before the search runs.
Equality
A test that passes only when two values are the same.
Edition
One version of a document, such as the 2019 or the 2024 release.
Full code