What retrieval cannot fix

Retrieval finds the closest chunks, but it cannot add missing facts or pick the newest edition.

Lesson 31 of 622 min read

The problem

Why it matters

Without it

A user expects the search to know the answer. It returns the closest chunks anyway, and the closest chunk can be outdated or unrelated.

With it

Each failure has a known cause and a known fix. Name the failure, then choose the fix from the lessons in this module.

How it works

Missing content cannot be ranked into view
If no chunk holds the answer, no search can return it. Ranking only puts existing chunks in order, so it cannot add a missing fact. The fix is to add the document, or to say the answer is not in the documents.
Two editions can tie
The 2019 and 2024 refund chunks have the same vector, so both score 0.82 for "refund time". A tie like this is a warning sign, so check the labels first. The fix is a metadata filter on the edition label (lesson 4.2).
2019 refund0.82
2024 refund0.82
Office closed0.18
Scores for "refund time". The two editions tie at the top.
```
const dot = (a: number[], b: number[]) => a[0] * b[0] + a[1] * b[1];const query = [0.9, 0.1];console.log("2019:", dot(query, [0.9, 0.1]).toFixed(2));console.log("2024:", dot(query, [0.9, 0.1]).toFixed(2));
```
Output
```
2019: 0.82
2024: 0.82
```
Unrelated questions still get results
The question "do you sell gift cards" has no matching chunk, yet search returns 2 anyway. The top score is only 0.29, for a chunk about office holidays, and scores are relative. No cutoff is safe until tests show where it belongs. A grader that can say no is the fix (lesson 4.8).
- Our office is closed on holidays.score 0.29, unrelated
- Refunds take 5 days. (2019)score 0.21, unrelated
The two results for "do you sell gift cards", with their scores.

A table of causes and fixes

Four failures explain most wrong answers from retrieval. Each has a cause that ranking cannot fix on its own. Test each one on purpose by adding no-answer questions and two-edition questions to the golden set.

Failure	Sign	Fix
Missing content	Top results do not answer	Add the document, or say not covered
Two editions	Equal top scores	Metadata filter (lesson 4.2)
Unrelated results	Low top score, still returned	Grader that can say no (lesson 4.8)
Vague wording	Zero hits for the question	Query rewriting (lesson 4.6)

Four failures, the sign that shows each one, and the fix.

Full code

Read the program

See the full code

Common mistakes

What goes wrong

Tuning the ranking for a missing answer. Ranking cannot create a document that is not there.
Trusting a high score as proof. An outdated chunk can score the highest.
Testing only questions that have answers. Add no-answer questions to the test set.

Use it when and skip it when

Is it the right tool?

Use it when

You need a checklist before blaming the model for a wrong answer.
You plan a document set with several editions of the same content.

Skip it when

The search already returns the right chunk for every golden question.
You are not yet measuring retrieval. Start with lesson 2.2 first.

Words to know

Terms in this lesson

Edition: One version of a document, such as the 2019 or the 2024 release.
Tie: Two items with the same score.
Golden set: A fixed list of questions with known good answers.
Missing content: An answer that no document contains.

What retrieval cannot fix

Missing content cannot be ranked into view

Two editions can tie

Unrelated questions still get results

A table of causes and fixes