VereskLearn

Module 2 · Measure first: evals

Lesson 2.1

The golden set

A golden set is a fixed list of questions, each with the fact a correct answer must contain.

Lesson 8 of 622 min read

The problem

Why it matters

Without it

Changes are tested by trying a few questions by hand. Each run tests different questions, so results cannot be compared.

With it

The same questions run after every change. A drop in the score shows that the change made answers worse.

How it works

  • Start with questions users asked

    A golden set starts with questions that real users asked, such as "How long does a refund take?" Each question is written down once and then kept fixed. A first set of 50 questions is a good start for a small help center.

    IdQuestion
    r1How long does a refund take?
    p1How do I reset it?
    The two cases in this lesson. Each has an id and a question.
    type GoldenCase = { id: string; q: string; fact: string };const goldenSet: GoldenCase[] = [  { id: "r1", q: "How long does a refund take?", fact: "5 to 7" },  { id: "p1", q: "How do I reset it?", fact: "Forgot password" },];console.log("Cases ->", goldenSet.length);

    Output

    Cases -> 2

  • Each case has a required fact

    For each question, the set stores the fact a correct answer must contain. The refund answer must contain "5 to 7". Each case also names its source, such as "refund-guide, section 2", which lesson 2.2 uses to score search.

    Question

    • How long does a refund take?

    Required fact

    • 5 to 7
    • Source: refund-guide, section 2
    Each case pairs a question with one required fact.
    const fact = "5 to 7";const answer = "Refunds take 5 to 7 business days.";// The check is a plain text search, so the wording can vary.console.log("Fact found ->", answer.includes(fact));

    Output

    Fact found -> true

  • The set lives in a file the team shares

    Keep the set in a file under version control, such as "golden.json". Each case has an id, a question, and its required fact. The team agrees on the set before any change, so everyone uses the same meaning of a correct answer.

    1. Write the cases
    2. Save golden.json
    3. Run after each change
    4. Compare scores
    The set is written once, saved, and reused for every change.

  • Each run scores the set

    The run gives each question to the assistant and checks its answer for the required fact. A case passes when the fact appears. In this set, the refund answer passes and the password answer fails, so the score is 1 of 2.

    • r1: refund timePASS
    • p1: password resetFAIL
    The two cases after one run.
    const refundAnswer = "Refunds take 5 to 7 business days.";const passwordAnswer = "Contact support for help.";const passes = [  refundAnswer.includes("5 to 7"),  passwordAnswer.includes("Forgot password"),];console.log("Pass flags ->", passes.join(", "));const score = passes.filter(Boolean).length;console.log("Score ->", score + " of " + passes.length);

    Output

    Pass flags -> true, false
Score -> 1 of 2

  • Compare the score before and after a change

    Run every case after each change and count the passes. A score of 42 out of 50 before a change and 45 after it shows the change helped. The cases stay the same, so the two scores can be compared directly.

    Passing cases out of 50, before and after one change.

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Writing the cases after seeing the answers. The set then measures the old system, not what users need.
  • Keeping the set too small. Five cases cannot show a small change in quality.
  • Changing a case when its run fails. Fix the system first, then keep the case unchanged for comparison.

Use it when and skip it when

Is it the right tool?

Use it when

  • You change prompts, search settings, or documents and need to know the effect.
  • Your team needs one shared definition of a correct answer.

Skip it when

  • The assistant is a throwaway demo that nobody will change.
  • Each answer is judged only once by a person. A fixed list cannot cover that work.

Words to know

Terms in this lesson

Golden set
A fixed list of cases used to test every change the same way.
Case
One question in the set, with the fact a correct answer must contain.
Required fact
A piece of information that a correct answer must include.
Score
The number of passing cases, or their share of the total.
Full code