VereskLearn

Module 2 · Measure first: evals

Lesson 2.2

Retrieval metrics

Retrieval metrics check whether search finds the right chunks before the model writes anything.

Lesson 9 of 622 min read

The problem

Why it matters

Without it

A bad answer could come from search or from the model. With no score for search, the cause stays unclear.

With it

Search has its own score. A poor answer can be traced to missing chunks or to the model.

How it works

  • Each question lists its right chunks

    Each question gets the ids of the chunks that hold its answer. The refund answer sits in chunk "c2". The address question needs two chunks, "c5" and "c9", so a question can have more than one right chunk.

    QuestionRight chunks
    How long does a refund take?c2
    How do I change my address?c5, c9
    When will my order ship?c8
    The right chunk ids for three questions from the help center.
    const relevant: Record<string, string[]> = {  refund: ["c2"],  address: ["c5", "c9"],};const list = relevant.address.join(", ");console.log("Address ->", list);

    Output

    Address -> c5, c9

  • Search returns a ranked list

    Search returns chunks in order, with the best match first. The letter k is how many results you check. For the address question, the ranked list is c3, c9, c4, so the top 2 are c3 and c9.

    • c3rank 1
    • c9rank 2, right chunk
    • c4rank 3, not checked
    The ranked search results for the address question. The top 2 are checked.
    const ranked = ["c3", "c9", "c4"];const k = 2;const top = ranked.slice(0, k);console.log("Top", k, "->", top.join(", "));

    Output

    Top 2 -> c3, c9

  • Hit rate counts questions with a right chunk

    Hit rate at k is the share of questions with at least one right chunk in the top k. The refund and address questions both hit. The shipping question misses, because c8 is not in its top 2. Two hits out of three questions give 0.67.

    QuestionRight chunk in top 2Hit
    Refundc21
    Addressc91
    Shippingnone0
    Hit or miss for each question, with the top 2 results checked.
    const hits = [1, 1, 0];const hitRate = hits.reduce((s, h) => s + h, 0) / hits.length;console.log("Hit rate at 2 ->", hitRate.toFixed(2));

    Output

    Hit rate at 2 -> 0.67

  • Recall counts the right chunks found

    Recall at k is the share of the right chunks found in the top k. The address question has 2 right chunks, and only c9 is in its top 2, so its recall is 0.5. Averaged over the three questions, recall at 2 is 0.5.

    Recall at 2 for each question. The average is 0.5.
    const relevant = ["c5", "c9"];const top = ["c3", "c9"];const found = relevant.filter((id) => top.includes(id)).length;console.log("Recall at 2 ->", (found / relevant.length).toFixed(1));

    Output

    Recall at 2 -> 0.5

  • Reciprocal rank rewards an early hit

    Reciprocal rank is one divided by the position of the first right chunk. The refund answer is in first place, so it scores 1. The address answer is in second place, so it scores 0.5, and the shipping question scores 0. The mean over the three questions is 0.5.

    • Position 11
    • Position 20.5
    • Position 30.33
    • Not found0
    Reciprocal rank for the first right chunk at each position.
    const ranked = ["c3", "c9", "c4"];const relevant = ["c5", "c9"];const position = ranked.findIndex((id) => relevant.includes(id));console.log("Position ->", position + 1);console.log("Reciprocal rank ->", 1 / (position + 1));

    Output

    Position -> 2
Reciprocal rank -> 0.5

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Scoring search by comparing answer wording. The check then depends on wording, not on whether search worked.
  • Choosing k after seeing the scores. Pick k first, then compare settings at the same k.
  • Averaging over too few questions. Two questions can move the score by a large amount.

Use it when and skip it when

Is it the right tool?

Use it when

  • You change search settings, such as the chunk size or the number of results.
  • Answers are wrong, and you need to know whether search or the model caused it.

Skip it when

  • Your set has no chunk ids for the answers. Add them first, or use the answer checks in lesson 2.3.
  • The system has no search step, such as a single lookup by id.

Words to know

Terms in this lesson

Chunk id
A label that names one stored chunk, such as "c2".
Top k
The first k results returned by search, where k is a number you choose.
Recall
The share of the right items that search returned.
Reciprocal rank
One divided by the position of the first right item.
Full code