VereskLearn

Module 3 · Preparing documents

Lesson 3.6

Embed and store

Embed and store turns each chunk into a vector and saves it, so a repeated load does not add duplicates.

Lesson 19 of 622 min read

The problem

Why it matters

Without it

Each run of the loader adds the chunks again. The store fills with copies, and search returns the same passage many times.

With it

Each chunk gets a stable id from its source and position. A second run finds the same id and replaces the row.

How it works

  • Embedding turns text into a list

    The chunk text goes to an embedding model, which returns a list of numbers. Real models return hundreds of numbers. This lesson uses a fake embedding that counts the letters a, e, and o. Changing the model changes every vector, so the whole store must be rebuilt.

    • a2
    • e2
    • o1
    The fake embedding of "Refunds take 5 to 7 days.", one count per letter.
    const text = "Refunds take 5 to 7 days.".toLowerCase();const counts = ["a", "e", "o"].map((l) => text.split(l).length - 1);console.log("Vector ->", JSON.stringify(counts));

    Output

    Vector -> [2,2,1]

  • Stable ids come from the location

    The id is built from the source and the position, such as "manual-2.1-chunk-1". The same chunk always gets the same id, on every run. A random id would be new each time, so each run would add a new row.

    Id typeExampleResult on a second load
    Stablemanual-2.1-chunk-1Replaces the same row
    Randoma new value each runAdds a new row
    What a second load does with each kind of id.
    const stableId = (src: string, pos: number) =>  "manual-" + src + "-chunk-" + pos;console.log(stableId("2.1", 1));

    Output

    manual-2.1-chunk-1

  • An upsert replaces a repeated row

    An upsert inserts a row, or updates it when the id already exists. A second load of "manual-2.1-chunk-1" replaces its row and adds nothing. After two loads of the two chunks, the store still holds two rows.

    Rows in the store after each load of the same two chunks.
    const store = new Map<string, string>();const load = (id: string, text: string) => store.set(id, text);load("manual-2.1-chunk-1", "Refunds take 5 to 7 days.");load("manual-2.1-chunk-1", "Refunds take 5 to 7 days.");console.log("Rows ->", store.size);

    Output

    Rows -> 1

  • Each row holds text, metadata, and a vector

    The table holds the id, the text, the metadata, and the vector. In a real project the table is often PostgreSQL with pgvector, an extension for vector columns. This lesson uses a Map, so the code runs without a database.

    ColumnValue
    idmanual-2.1-chunk-1
    textRefunds take 5 to 7 days.
    src2.1
    vector[2, 2, 1]
    One row for the first chunk.

  • An index keeps search fast

    Without an index, search compares the query with every row. An HNSW index links each vector to nearby vectors, so search can skip most rows. With 1,000,000 rows, search checks a small share of them, at a small cost in accuracy.

    • Without an indexevery row
    • With an HNSW indexa small share of rows
    How much of the store a search compares.

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Using random ids. Every run then adds new rows, and the store fills with copies.
  • Mixing embedding models. Vectors from two models cannot be compared, so rebuild the whole store.
  • Skipping the index on a large store. Search then compares every row, which gets slow.

Use it when and skip it when

Is it the right tool?

Use it when

  • You load documents more than once, such as after each update.
  • You search by meaning and need a store that compares vectors quickly.

Skip it when

  • The collection is a few dozen chunks. A simple list is enough.
  • You never search by similarity, such as when every lookup uses an id.

Words to know

Terms in this lesson

Upsert
Insert a row, or update it if its id already exists.
Stable id
An id built from the content location, so it is the same on every run.
pgvector
An extension that lets PostgreSQL store and compare vectors.
HNSW index
A graph index that finds close vectors without comparing every row.
Full code