VereskLearn

Module 3 · Preparing documents

Lesson 3.5

Metadata

Metadata is extra data stored with each chunk, such as its source and year, so answers can be filtered.

Lesson 18 of 622 min read

The problem

Why it matters

Without it

Chunks are only text. An answer cannot say which manual or version it came from, so the reader cannot check it.

With it

Each chunk carries its source, version, and year. Answers name the source, and old versions can be filtered out.

How it works

  • Metadata sits beside the text

    Metadata is extra data stored beside each chunk, such as its source, version, and year. The text is what gets embedded, and the fields stay in their own columns. For a manual, the version might be "2.1".

    Chunk textsrcyear
    Refunds take 5 to 7 days.2.12024
    Refunds take 3 days.1.02019
    Two chunks, each with its text and two metadata fields.
    type Chunk = { text: string; src: string; year: number };const chunk: Chunk = {  text: "Refunds take 5 to 7 days.", src: "2.1", year: 2024,};console.log("Year ->", chunk.year);

    Output

    Year -> 2024

  • Fields come from the document

    Every chunk record gets the fields of the document it came from. A chunk from the 2024 manual stores the year 2024. A wrong year in the record gives a wrong filter result, so check the fields when a document is added.

    Document

    • Manual 2.1
    • Year 2024

    Chunk records

    • Chunk 1: src 2.1, year 2024
    • Chunk 2: src 2.1, year 2024
    One document and the records of its chunks.

  • Use one name for each field

    Use one name for each field in every document. "version" should not be "ver" in one file and "release" in another. A filter on "version" misses any chunk stored under a different name.

    • versionmatched by the filter
    • vermissed
    • releasemissed
    Three names for one field. Only the first one matches the filter.
    const fields = ["version", "ver", "release"];const filterKey = "version";const matches = fields.filter((f) => f === filterKey);console.log("Matches ->", matches.length, "of", fields.length);

    Output

    Matches -> 1 of 3

  • A filter drops chunks before ranking

    A filter keeps only the chunks that match a condition, such as "year is 2020 or later". It is a plain comparison, so it needs no model call and gives an exact result. In the example, the 2019 chunk is dropped and one chunk remains. Lesson 4.2 covers filters in search.

    Chunks before and after the year filter.
    const years = [2024, 2019];const kept = years.filter((y) => y >= 2020);console.log("Kept ->", kept.join(", "));

    Output

    Kept -> 2024

  • Build a citation from the fields

    A citation is built from the fields, such as "Manual 2.1 (2024)". A reader can then find the passage in the named source. An answer with no citation gives the reader nothing to check.

    1. Chunk text
    2. Fields: src 2.1, year 2024
    3. Citation: Manual 2.1 (2024)
    The fields of one chunk become its citation.
    const src = "2.1";const year = 2024;console.log("Manual " + src + " (" + year + ")");

    Output

    Manual 2.1 (2024)

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Storing the source only in the file name. A name is hard to filter reliably, so store it as a field.
  • Using two names for one field. A filter on "version" then misses chunks stored as "ver".
  • Adding fields that nobody uses. Each field costs storage and review time, so keep the list short.

Use it when and skip it when

Is it the right tool?

Use it when

  • Answers must name the source or the version.
  • Old and new versions of a document both exist in the store.

Skip it when

  • Every document is the same kind and version, so nothing needs a filter.
  • The answer never needs a source, such as a brainstorming tool.

Words to know

Terms in this lesson

Metadata
Extra fields stored with a chunk, such as its source or year.
Field
One named value in a record, such as year or src.
Filter
A test that keeps only the records that match a condition.
Citation
A reference that names the source of an answer.
Full code