VereskLearn

Module 9 · Running it for real

Lesson 9.6

Debugging a bad answer

Debug a bad answer by checking the stages in order and naming the first stage that failed.

Lesson 59 of 622 min read

The problem

Why it matters

Without it

The answer is wrong, so a fix is guessed. A prompt change is tried, and the answer still fails for another reason.

With it

A logged request shows what each stage produced. The first failing stage points to the fix.

How it works

  • Stages run in a fixed order

    A request passes through retrieval, context, and generation, in that order. Retrieval finds the source chunks, context builds the text the model reads, and generation writes the answer. Each stage can fail on its own.

    1. Retrieval
    2. Context
    3. Generation
    4. Answer
    The stages of one request, from left to right.

  • Retrieval checks the found ids

    The expected source is faq-12. In run A, retrieval found only faq-40, so faq-12 never reached the context. A prompt change cannot fix that run, so the search must change.

    RunFoundSentAnswer rightFirst failed stage
    Afaq-40faq-40noretrieval
    Bfaq-12, faq-40faq-40nocontext
    Cfaq-12faq-12nogeneration
    Dfaq-12faq-12yesnone
    Four logged runs. Each should have used faq-12.
    const found = ["faq-40"];if (!found.includes("faq-12")) console.log("retrieval");

    Output

    retrieval

  • Context checks the sent ids

    Run B found faq-12, but the context sent only faq-40 to the model. The context builder dropped the source. A size limit or a sorting bug is a likely cause.

    const found = ["faq-12", "faq-40"];const sent = ["faq-40"];if (found.includes("faq-12") && !sent.includes("faq-12")) {  console.log("context");}

    Output

    context

  • Generation fails last

    Run C sent faq-12, and the answer was still wrong. The prompt, the rules, or the model is the cause now. Change the prompt, then run the eval set again to confirm the fix.

    • Aretrieval
    • Bcontext
    • Cgeneration
    • Dno failure
    The stage that failed for each logged run.
    const sent = ["faq-12"];const answerRight = false;const sentSource = sent.includes("faq-12");if (sentSource && !answerRight) console.log("generation");

    Output

    generation

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Changing the prompt first. Many bad answers start at retrieval or context, not at the wording.
  • Judging a stage without its log. A guess skips the evidence that the log holds.
  • Stopping at the first wrong-looking answer. Check every logged request that shares the failure.

Use it when and skip it when

Is it the right tool?

Use it when

  • A user reports a wrong answer, and you need the cause.
  • A change made the eval scores drop.

Skip it when

  • The answer is right, and you only want to change its wording.

Words to know

Terms in this lesson

Stage
One step of the pipeline, such as retrieval or generation.
Retrieval
Finding the source chunks for a question.
Log entry
The record kept for one request.
Failed stage
The first stage that went wrong, which explains the rest.
Full code