VereskLearn

Module 2 · Measure first: evals

Lesson 2.5

Baselines, diffs, and regressions

A baseline is a saved result from a known version, and a diff compares a new result with it.

Lesson 12 of 622 min read

The problem

Why it matters

Without it

Each run is compared with memory. Small drops go unnoticed, and nobody knows which version scored best.

With it

The saved baseline shows each case that changed. A case that passed before and fails now is a regression.

How it works

  • Save a baseline from a trusted run

    After a run you trust, save the result of each case under a version name, such as "v1.3". That saved file is the baseline. A diff is only as good as its baseline, so use a run you trust.

    Casev1.3 result
    refundpass
    passwordpass
    shipfail
    The saved baseline for version v1.3.
    const baseline: Record<string, boolean> = {  refund: true, password: true, ship: false,};console.log("Saved cases ->", Object.keys(baseline).length);

    Output

    Saved cases -> 3

  • A new run uses the same cases

    Run the same cases with the same settings and save the result as "v1.4". If the cases differ, the diff mixes two kinds of change. The version name also links each result to the code and settings that made it.

    v1.3 baseline

    • refund: pass
    • password: pass
    • ship: fail

    v1.4 new run

    • refund: fail
    • password: pass
    • ship: pass
    The two saved versions, case by case.

  • Each case gets one label

    Each case gets one label: same, fixed, or regression. The password case passed in both versions, so it is labeled same. The ship case failed before and passes now, so it is fixed.

    • refundregression
    • passwordsame
    • shipfixed
    The label for each case in the diff.
    const before = { refund: true, password: true, ship: false };const after = { refund: false, password: true, ship: true };for (const id of Object.keys(before)) {  let label = "same";  if (before[id] && !after[id]) label = "regression";  if (!before[id] && after[id]) label = "fixed";  console.log(id, label);}

    Output

    refund regression
password same
ship fixed

  • A total can hide a regression

    A regression is a case that passed in the baseline and fails now. The refund case passed in v1.3 and fails in v1.4. The ship fix offsets it, so both totals are 2 of 3. Only the case list shows the regression.

    The passes in each version. The totals match, so the case labels matter.
    const baselineTotal = [true, true, false].filter(Boolean).length;const newTotal = [false, true, true].filter(Boolean).length;console.log("Totals ->", baselineTotal, newTotal);

    Output

    Totals -> 2 2

  • Fix or explain a regression before shipping

    A regression blocks the change until it is fixed or explained in writing. The refund case needs a fix or a written reason before v1.4 ships. Never update the baseline to match the new result, because that hides the regression.

    1. Regression found
    2. Fix or written reason
    3. Run again
    4. Ship v1.4
    The path for a regression, from detection to release.

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Comparing against a baseline with different cases. The diff then mixes two kinds of change.
  • Updating the baseline to match the new result. That hides the regression instead of fixing it.
  • Reading only the totals. A flat total can hide one regression and one fix.

Use it when and skip it when

Is it the right tool?

Use it when

  • You change prompts, settings, or documents and want to see what changed.
  • You need a record of which version scored best.

Skip it when

  • The set is so small that you can read every answer by hand.
  • No version history is kept, so a baseline has no clear meaning.

Words to know

Terms in this lesson

Baseline
A saved result from a known version, used as the reference.
Diff
A comparison of two results, case by case.
Regression
A case that passed before and fails now.
Version name
A label, such as "v1.3", that identifies a saved result.
Full code