Context, retrieval, or fine-tuning

Pick the cheapest option that fixes the problem: better context first, then retrieval, then fine-tuning.

Lesson 60 of 622 min read

The problem

Why it matters

Without it

A team trains a custom model for every complaint about answers. Each training run is slow and costly, and the facts still go stale.

With it

A short list of questions picks the option. Most problems are fixed by better context or retrieval before any training.

How it works

Missing facts go into the context
The context is all the text sent to the model in one request. If an answer needs a fact the model lacks, add that fact to the context. A price list of 30 items fits in one request, so the fix is to paste it in.
Case Missing fact Too many docs Bad format Failed tries
Price list, 30 items yes no no no
Manual, 2,000 pages yes yes no no
Format after many tries no no yes yes
The three cases from the code, and the flags that describe each one.
Too many documents means retrieval
Retrieval searches the documents and adds only the few chunks that match each question. A manual of 2,000 pages cannot fit in one request, and documents that change each week are hard to paste. The code sends a case to retrieval only when the missing-fact flag and the too-many flag are both true.
1. Question
2. Search picks a few chunks
3. Chunks go into the request
The retrieval path for the 2,000-page manual, left to right.
```
// Retrieval only when a fact is missing and the docs are too many.const choose = (missing: boolean, tooMany: boolean) => {  return missing && tooMany ? "retrieval" : "better context";};console.log(choose(true, true));
```
Output
```
retrieval
```
Format problems start with instructions
If replies have the wrong shape, improve the instructions first. Fine-tuning comes later, when the instructions still fail, such as when 3 of 10 replies still add extra text. The code chooses fine-tuning only when the format flag and the failed-tries flag are both true.
- Better instructions3 of 10 still add extra text
- Fine-tuningused only when instructions fail
The format path, in the order the lesson recommends.
```
// Fine-tuning only when the format is bad and the tries failed.const chooseFormat = (badFormat: boolean, failed: boolean) => {  if (badFormat && failed) return "fine-tuning";  return "better instructions";};console.log(chooseFormat(true, true));
```
Output
```
fine-tuning
```
Measure each option on the same eval set
Run the same eval set for each option, and compare the scores. Suppose the price-list fix raises the score from 36 to 44 out of 50. Keep the cheapest option that gives the best score. A rule like this one is only a starting point, and the eval set decides.
Before fix36
Price list in context44
Eval score out of 50, before and after the price-list fix.
Record the reason and the order
Record why the option was chosen, such as "answers missed the 2024 price list". The record stops the team from retraining for the same reason later. The order matters. A longer prompt is cheap to try, retrieval needs a store, and training needs data and compute.

Case	Missing fact	Too many docs	Bad format	Failed tries
Price list, 30 items	yes	no	no	no
Manual, 2,000 pages	yes	yes	no	no
Format after many tries	no	no	yes	yes

ProblemContextRetrievalFine-tuning

Full code

Read the program

See the full code

Common mistakes

What goes wrong

Fine-tuning to add facts. The facts go stale, and the model may still answer from memory.
Choosing without an eval. Without a score, the choice rests on opinion.
Building retrieval for a few short facts. Pasting them into the request is simpler and exact.

Use it when and skip it when

Is it the right tool?

Use it when

A team must decide which option to build before writing code.
Answers keep failing, and the cause is not yet known.

Skip it when

The task already passes the eval set with the current setup.
You need a different model, not a better setup. Lesson 10.3 covers that case.

Words to know

Terms in this lesson

Context: All the text sent to the model in one request.
Retrieval store: A database of chunks that search reads from.
Fine-tuning: Extra training on example pairs that changes the model weights.
Weights: The numbers inside a model that set how it responds.

Context, retrieval, or fine-tuning

Missing facts go into the context

Too many documents means retrieval

Format problems start with instructions

Measure each option on the same eval set

Record the reason and the order