VereskLearn

Module 1 · Talking to a model

Lesson 1.4

Embeddings: text as numbers

An embedding is a list of numbers that stands for the meaning of a piece of text.

Lesson 6 of 622 min read

The problem

Why it matters

Without it

A computer compares text by letters. The phrases "refund" and "money back" share no word, so an exact match misses the link.

With it

Each text becomes a list of numbers. Texts with similar meaning get similar lists, so the computer can compare them.

How it works

  • A list of numbers stands for a text

    An embedding is a list of numbers that stands for the meaning of a text. A real model makes the list by reading the text. This lesson uses a hand-made table, so every list is known in advance.

    SentenceEmbedding
    How do I get a refund?[0.9, 0.1, 0]
    Can I get my money back?[0.8, 0.2, 0.1]
    How do I reset my password?[0, 0.9, 0.2]
    Three FAQ questions and their embeddings. Each list has three numbers.
    // A Map stores a value under a key, here the sentence.const embeddingTable = new Map<string, number[]>([  ["How do I get a refund?", [0.9, 0.1, 0]],  ["Can I get my money back?", [0.8, 0.2, 0.1]],]);const numbers = embeddingTable.get("How do I get a refund?");console.log(JSON.stringify(numbers));

    Output

    [0.9,0.1,0]

  • Every list has the same length

    The number of values in one list is its dimension. Every embedding in this table has three values, so its dimension is 3. Real models use far more values, often hundreds, but the comparison works the same way.

    • 0.9axis 1
    • 0.1axis 2
    • 0axis 3
    The three values of the refund embedding. Each value is a position on its own axis.
    const refundNumbers = [0.9, 0.1, 0];console.log("Dimension ->", refundNumbers.length);

    Output

    Dimension -> 3

  • Similar meanings get nearby lists

    Texts with a similar meaning get similar lists. "Can I get my money back?" has 0.8, 0.2, and 0.1, close to the refund list. The password question has 0, 0.9, and 0.2, far from both. This picture uses only the first two numbers of each list.

    RefundMoney backReset password
    The sentences placed by their first two numbers. The refund and money-back points sit close together.

  • A lookup has no list for new text

    A lookup returns nothing for a sentence that is not stored. "Do you sell gift cards?" is not in the table, so the lookup returns undefined. A real model makes a list for any text, which is why the table is only a teaching stand-in.

    1. Any sentence
    2. Model
    3. List of numbers
    A real model turns any sentence into a list of numbers.
    const embeddingTable = new Map<string, number[]>([  ["How do I get a refund?", [0.9, 0.1, 0]],]);console.log(embeddingTable.get("Do you sell gift cards?"));

    Output

    undefined

  • Arithmetic compares two lists

    A program compares two embeddings with arithmetic, not by matching letters. It multiplies each pair of values and adds the products, which gives the dot product. The refund and money-back lists have a dot product of 0.74. The refund and password lists have 0.09, far lower.

    Dot products of the refund list with two others. The money-back list scores much higher.
    const refund = [0.9, 0.1, 0];const moneyBack = [0.8, 0.2, 0.1];// Multiply each pair of values, then add the products.let dotProduct = 0;for (let i = 0; i < refund.length; i++) {  dotProduct += refund[i] * moneyBack[i];}console.log("Dot product ->", dotProduct.toFixed(2));

    Output

    Dot product -> 0.74

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Reading one number as a meaning. The whole list carries the meaning, not one value.
  • Comparing embeddings from different models. Each model makes its own numbers, so the values do not line up.
  • Expecting a lookup table to handle new text. A real model is needed for any text that was not added in advance.

Use it when and skip it when

Is it the right tool?

Use it when

  • You need to find texts with similar meaning, not only the same words.
  • You want to group, sort, or compare texts with arithmetic.

Skip it when

  • The question has one exact answer, such as an order number. Use an exact lookup.
  • The text must match letter for letter, such as a product code.

Words to know

Terms in this lesson

Embedding
A list of numbers that stands for the meaning of a text.
Vector
Another name for a list of numbers with a fixed length. This course uses the two words the same way.
Dimension
The number of values in one embedding.
Map
A container that stores a value under a key, so you can look it up later.
Full code