VereskLearn

Module 4 · Retrieval

Lesson 4.8

Corrective retrieval

Corrective retrieval grades each result and rewrites the query when nothing good comes back.

Lesson 27 of 622 min read

The problem

Why it matters

Without it

The system sends weak chunks to the model anyway. The model then writes an answer from text that does not answer the question.

With it

A grader checks each chunk first. When nothing passes, the system rewrites the query and searches again. It can also say the documents do not cover the question.

How it works

  • The first search finds nothing useful

    Users write "how long until my money is back". The words do not match the documents, which say "refund". The first search therefore returns 0 chunks from the two-chunk library.

    QueryChunks found
    how long until my money is back0
    refund1
    Results for each query over the two-chunk library.
    // Fake search: chunks that share a whole word with the query.const library = [  "Gift card balances are on the account page",  "Refund takes 5 to 7 days",];const words = "how long until my money is back".split(" ");const hits = library.filter((c) => {  return words.some((w) => c.toLowerCase().split(" ").includes(w));});console.log(hits.length);

    Output

    0

  • A grader keeps only chunks that answer

    A grader is a check that decides whether a chunk answers the question. In this example a fake grader accepts only the refund chunk. A real grader is a model call for each chunk, so systems grade only the top few results.

    • Refund takes 5 to 7 dayskept
    • Gift card balances are on the account pagerejected
    The two library chunks after the grader has read them.
    // Fake grader: accept only the chunk that answers the question.const answering = new Set(["Refund takes 5 to 7 days"]);const grade = (chunk: string) => answering.has(chunk);console.log(grade("Refund takes 5 to 7 days"));console.log(grade("Gift card balances are on the account page"));

    Output

    true
false

  • A rewrite uses the document words

    The rewriter replaces "money back" with "refund", a word the refund chunk uses. A plain search for "refund" finds that chunk, and the grader keeps it, so the second search keeps 1 chunk. Matching is by whole word, so the plural "Refunds" would not match.

    Chunks kept after grading, for each search.
    // The library and the rewritten query.const library = [  "Gift card balances are on the account page",  "Refund takes 5 to 7 days",];const rewritten = "refund";const hits = library.filter((chunk) => {  return chunk.toLowerCase().split(" ").includes(rewritten);});console.log(hits);

    Output

    [ 'Refund takes 5 to 7 days' ]

  • Refusing when nothing passes

    Each round is one search, one grading pass, and one rewrite. If a round keeps nothing, the system tries again. It stops after a set number of rounds, such as two. If nothing still passes, it says the documents do not cover the question instead of guessing.

    A chunk passed

    • Answer from the kept chunk
    • Show its source

    Nothing passed

    • Say the documents do not cover it
    • Do not guess an answer
    What the system returns after the last round, in two cases.

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Grading with the same signal that found the chunk. A grader that only repeats the search adds no check.
  • Rewriting without a limit. Set a maximum number of rounds, such as two.
  • Answering when the grader keeps nothing. The model must say the documents do not cover the question.

Use it when and skip it when

Is it the right tool?

Use it when

  • Users ask in words the documents do not use, and first searches often return nothing useful.
  • A wrong answer costs more than a refusal, such as in a billing answer.

Skip it when

  • The first search already passes your test questions. The extra step only adds delay.
  • Each round must finish inside a tight time limit.

Words to know

Terms in this lesson

Grader
A check that decides whether a chunk answers the question.
Rewrite
A new query written to find better chunks.
Round
One cycle of search, grade, and rewrite.
Refusal
An answer that says the documents do not cover the question.
Full code