VereskLearn

Module 2 · Measure first: evals

Lesson 2.4

Repeat runs and flaky results

A flaky result passes on some runs and fails on others, so each case runs more than once.

Lesson 11 of 622 min read

The problem

Why it matters

Without it

Each case runs once. A pass or fail may be luck, and a real change can look like noise.

With it

Each case runs five times. The pass count shows how stable the answer is.

How it works

  • One run can mislead

    The model may give a different reply to the same question, because of the sampling described in lesson 1.3. One run can pass by luck or fail by bad luck. Running the case five times shows how often it passes.

    • Run 1pass
    • Run 2fail
    • Run 3pass
    • Run 4pass
    • Run 5fail
    Five runs of the password question. Two runs fail.
    const runs = [true, false, true, true, false];const passes = runs.filter((ok) => ok).length;console.log("Passes ->", passes + " of " + runs.length);

    Output

    Passes -> 3 of 5

  • Stable and flaky cases

    A case that gives the same outcome on every run is stable. The refund question passes 5 times out of 5, so it is stable. The password question passes 3 times out of 5, so it is flaky.

    Passes out of 5 runs for three cases.
    const runs = [true, false, true, true, false];const stable = runs.every((ok) => ok === runs[0]);console.log("Stable ->", stable);

    Output

    Stable -> false

  • Choose the pass rule before the run

    A case passes when at least 4 of its 5 runs pass. Decide this rule before the run and keep it for later comparisons. At least 4 of 5 is strict, and at least 3 of 5 is lenient, so write the choice down.

    Runs passedVerdict
    5pass
    4pass
    3fail
    0fail
    The verdict for each count of passes under the rule of 4 of 5.
    const PASS_RULE = 4;const passes = 3;console.log("Verdict ->", passes >= PASS_RULE ? "pass" : "fail");

    Output

    Verdict -> fail

  • Report flaky cases on their own line

    A flaky case gets its own line in the report, so it does not hide inside a plain fail. The count, such as 3/5, appears next to the label. Save every run, not only the summary, so you can see which runs failed.

    • refund-15/5 stable, pass
    • password-13/5 flaky, fail
    • ship-10/5 stable, fail
    The report lines for three cases.
    const runs = [true, false, true, true, false];const passes = runs.filter((ok) => ok).length;const allSame = passes === 0 || passes === runs.length;const kind = allSame ? "stable" : "flaky";console.log("password-1", passes + "/" + runs.length, kind);

    Output

    password-1 3/5 flaky

  • Compare totals with care

    Compare the total pass count of two versions, not one run. A change of one case out of 50 can be noise, so repeat the run before deciding. Five runs of 50 cases means 250 model calls, so set that budget first.

    SettingValue
    Runs per case5
    Cases50
    Model calls250
    The size of one full run with five repeats.
    const runsPerCase = 5;const cases = 50;console.log("Model calls ->", runsPerCase * cases);

    Output

    Model calls -> 250

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Running each case once and calling it stable. One pass says little about the next run.
  • Changing the pass rule after seeing the results. Set the rule first, then keep it.
  • Hiding flaky cases inside the average. Report them on their own line.

Use it when and skip it when

Is it the right tool?

Use it when

  • The model gives different replies to the same question.
  • You compare two versions, and the difference is only a few cases.

Skip it when

  • The system always returns the same result, such as a lookup table.
  • Each run is very costly, and you accept the risk of a single run.

Words to know

Terms in this lesson

Flaky
A case that passes some runs and fails others.
Repeat run
One more run of the same case with the same settings.
Pass rule
The number of passing runs a case needs to count as a pass.
Stable
A case that gives the same outcome on every run.
Full code