Drift

Drift is a slow change in answer quality, found by running the same fixed test set over time.

Lesson 58 of 622 min read

The problem

Why it matters

Without it

Quality falls by a few points each week. Nobody notices until users complain.

With it

A weekly run on the same test set shows the trend. A drop past a set threshold raises a flag.

How it works

A pass rate is a share of checks
Each weekly run asks the same questions and checks the same expected sources. The pass rate is the passed checks divided by all checks. A rate of 0.92 means that 92 of 100 checks passed.
Week Checks passed Pass rate
1 92 of 100 0.92
5 84 of 100 0.84
Two weeks of the fixed test set, as checks passed.
The baseline averages the first four weeks
The baseline is the average pass rate of the first four weeks. The four rates add up to 3.66, and dividing by four gives a baseline of 0.915. Later weeks are compared with that line.
- Pass rate
Pass rate by week, from the sample runs.
```
const weeks = [0.92, 0.91, 0.93, 0.9];const sum = weeks.reduce((total, rate) => total + rate, 0);console.log("Baseline: " + (sum / weeks.length).toFixed(3));
```
Output
```
Baseline: 0.915
```
The threshold flags a drop of more than 0.05
A week is flagged when its rate is more than 0.05 below the baseline. Weeks 1 to 4 are within that range. Week 5 is 0.075 below the baseline, so it is flagged.
Week Rate Drop Flag
1 0.92 -0.005 ok
2 0.91 0.005 ok
3 0.93 -0.015 ok
4 0.90 0.015 ok
5 0.84 0.075 DRIFT
Each week, its drop below the baseline of 0.915, and its flag.
A flag starts a check, not a fix
A flag tells you that quality dropped, but not why. The next step is to find the changed document, prompt, or model. Only then does a fix follow.
```
const baseline = 0.915;const rate = 0.84;const drop = baseline - rate;console.log(drop > 0.05 ? "DRIFT" : "ok");
```
Output
```
DRIFT
```

Week	Checks passed	Pass rate
1	92 of 100	0.92
5	84 of 100	0.84

Week	Rate	Drop	Flag
1	0.92	-0.005	ok
2	0.91	0.005	ok
3	0.93	-0.015	ok
4	0.90	0.015	ok
5	0.84	0.075	DRIFT

Full code

Read the program

See the full code

Common mistakes

What goes wrong

Changing the test set each week. The scores then measure the new set, not the drift.
Using one run as the baseline. One unusual run sets the wrong reference line.
Setting the threshold once and never checking it. Test it against real past changes.

Use it when and skip it when

Is it the right tool?

Use it when

The system runs for months while documents or models change.
Users report that quality is falling, with no clear change to blame.

Skip it when

The system is new and changes often, so a baseline has no meaning yet.

Words to know

Terms in this lesson

Drift: A slow change in quality over time.
Baseline: A reference score that new runs are compared with.
Threshold: The size of a change that raises a flag.
Pass rate: The share of checks that passed in one run.

Drift

A pass rate is a share of checks

The baseline averages the first four weeks

The threshold flags a drop of more than 0.05

A flag starts a check, not a fix