VereskLearn

Module 2 · Measure first: evals

Lesson 2.3

Checking answers and LLM-as-judge

An answer check decides whether a reply is correct, using rules or a second model as a judge.

Lesson 10 of 622 min read

The problem

Why it matters

Without it

An exact match fails a correct answer with other words, such as "about a week" for "5 to 7 days".

With it

A rule catches the plain cases. A judge model grades the answers that use different words.

How it works

  • A rule looks for an accepted form

    The refund fact has two accepted forms: "5 to 7" and "five to seven". The rule lowercases the answer, then checks whether any accepted form appears in it. The check runs in microseconds and costs nothing per answer.

    AnswerRule result
    Refunds take 5 to 7 business days.pass
    Refunds take about a week.fail
    Refunds take 2 days.fail
    The rule result for three answers to the refund question.
    const accepted = ["5 to 7", "five to seven"];const answer = "Refunds take 5 to 7 business days.";const lower = answer.toLowerCase();const pass = accepted.some((f) => lower.includes(f));console.log("Rule ->", pass ? "pass" : "fail");

    Output

    Rule -> pass

  • A rule misses other wording

    The rule fails "Refunds take about a week." because no accepted form appears in it. That answer is correct, so the rule is too strict for it. Rules are cheap, so they run before the judge. A judge model checks only the answers the rule fails.

    1. Answer
    2. Rule check
    3. Judge check, if the rule fails
    4. Pass or fail
    Rules run first. The judge checks the answers that the rule fails.

  • A judge model reads the answer

    A judge is a second model call. It receives the question, the required fact, and the answer, then replies yes or no. The code uses a fake judge that says yes for "about a week". A real judge reads the answer and decides.

    • Questioninput
    • Required factinput
    • Answerinput
    • yes or nooutput
    What the judge receives, and what it returns.
    // A fake judge. A real one is a second model call.function fakeJudge(answer: string): string {  return answer.includes("about a week") ? "yes" : "no";}console.log("Judge ->", fakeJudge("Refunds take about a week."));

    Output

    Judge -> yes

  • Test the judge before trusting it

    A judge can be wrong, so test it first. A person grades about 20 answers, then the same answers go to the judge. If the two disagree often, fix the judge instructions before using them. The instructions must name the fact and the accepted wording.

    Person grades

    • About 20 answers
    • Pass or fail for each

    Judge grades

    • The same 20 answers
    • Count the matches
    The same answers are graded twice, then the results are compared.
    const person = [true, true, false, true];const judge = [true, false, false, true];const same = person.filter((p, i) => p === judge[i]).length;console.log("Agreement ->", same + " of " + person.length);

    Output

    Agreement -> 3 of 4

  • Record a reason with each result

    Store each pass or fail with a reason, such as "missing fact: 5 to 7". The reason shows what to fix, so a failure is not only a count. Ask the judge a narrow question, such as "Does the answer state the required fact?"

    AnswerResultReason
    Refunds take 5 to 7 business days.passfact found
    Refunds take 2 days.failmissing fact: 5 to 7
    Each result keeps its reason, so the fix is clear.
    const answer = "Refunds take 2 days.";const ok = answer.includes("5 to 7");const reason = ok ? "fact found" : "missing fact: 5 to 7";console.log(ok ? "pass" : "fail", "-", reason);

    Output

    fail - missing fact: 5 to 7

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Trusting a judge without testing it. A judge can pass wrong answers, so grade a sample by hand first.
  • Asking a vague question, such as "Is this good?" Name the fact to check instead.
  • Using the same model as both assistant and judge with no check. Its errors can hide its own mistakes.

Use it when and skip it when

Is it the right tool?

Use it when

  • Answers must contain a fact, but the wording varies from answer to answer.
  • You need a check on meaning that a keyword cannot give.

Skip it when

  • The answer is one exact value, such as an order number. A plain equality check is enough.
  • You have no hand-graded answers to test the judge against.

Words to know

Terms in this lesson

Rule check
A check written in code, such as a search for an accepted form.
Judge model
A model that grades another answer, usually with yes or no.
Accepted form
One wording of a fact that counts as correct.
Hand-graded
Checked by a person, and used as the reference for tests.
Full code