Severity buckets and control cases

Severity buckets sort failures by harm, and control cases check that the assistant refuses what it cannot answer.

Lesson 13 of 622 min read

The problem

Why it matters

Without it

Every failure counts the same. A wrong refund time and an awkward sentence look equally bad in the score.

With it

Failures sit in buckets. A critical wrong fact blocks a release, while a minor wording issue does not.

How it works

Three buckets sort failures by harm
Each failure goes in one bucket: critical, major, or minor. Critical means a wrong fact that a reader would act on, such as "refund in 2 days". Define the buckets before the run, so the labels do not drift toward the result you hoped for.
Bucket Example Effect on release
Critical Refund in 2 days Blocks the release
Major Reply with no source Reviewed before release
Minor Awkward wording, right fact Does not block
The three buckets, with an example and its effect on the release.
```
type Failure = { id: string; bucket: string };const failures: Failure[] = [  { id: "refund", bucket: "critical" },  { id: "ship", bucket: "minor" },];console.log("Buckets ->", failures.map((f) => f.bucket).join(", "));
```
Output
```
Buckets -> critical, minor
```
Sorting the failures
A wrong refund time, such as "2 days", is critical. A reply with no source is major, because the reader cannot check it. Awkward wording with the right fact is minor.
- Refund in 2 dayscritical
- No source givenmajor
- Awkward wordingminor
Three failures, each in its bucket.
```
const failures = [  { id: "refund", bucket: "critical" },  { id: "password", bucket: "major" },  { id: "ship", bucket: "minor" },];const count = (b: string) =>  failures.filter((f) => f.bucket === b).length;console.log("Critical ->", count("critical"));
```
Output
```
Critical -> 1
```
Control cases check for refusal
A control case asks about something the documents do not cover, such as "Do you sell gift cards for kids?" The right reply says the answer is not in the documents. A plain score does not reward that reply, so the control must be counted on purpose.
Right reply
- The answer is not in the documents.
Wrong reply
- Gift cards cost 25 dollars.
A right reply and a wrong reply to the same control question.
```
const refuses = (reply: string) =>  reply.includes("not in the documents");console.log(refuses("The answer is not in the documents."));console.log(refuses("Gift cards cost 25 dollars."));
```
Output
```
true
false
```
Scoring the controls
A control passes only when the assistant refuses or escalates. An invented answer, such as "Gift cards cost 25 dollars.", is a critical failure. Keep control cases in the same set as the other cases, so every run checks both kinds of behavior.
- Refusalpass
- Invented pricecritical failure
The two control replies after one run.
```
const controlReplies = [  "The answer is not in the documents.",  "Gift cards cost 25 dollars.",];const phrase = "not in the documents";const passes = controlReplies.filter((r) => r.includes(phrase));const total = controlReplies.length;console.log("Control passes ->", passes.length, "of", total);
```
Output
```
Control passes -> 1 of 2
```
The release rule
A change ships only when it has zero critical failures and every control case passes. One critical failure blocks the release. Here the refund failure and the invented price both block it.
1. Zero critical failures
2. Every control passes
3. Release allowed
Both conditions must hold before a release is allowed.
```
const critical = 1;const controlsPassed = 1;const controlsTotal = 2;const ok = critical === 0 && controlsPassed === controlsTotal;console.log("Release ->", ok ? "allowed" : "blocked");
```
Output
```
Release -> blocked
```

Bucket	Example	Effect on release
Critical	Refund in 2 days	Blocks the release
Major	Reply with no source	Reviewed before release
Minor	Awkward wording, right fact	Does not block

FailureBucketControl casesRelease rule

Full code

Read the program

See the full code

Common mistakes

What goes wrong

Putting every failure in the major bucket. The buckets then stop separating harm from annoyance.
Writing only questions that have answers. A system that always answers then looks perfect.
Deciding the bucket after the run. Set the bucket definitions first, then label the failures.

Use it when and skip it when

Is it the right tool?

Use it when

Some wrong answers cause more harm than others, such as a wrong price.
The assistant must say it does not know when the documents do not cover a question.

Skip it when

Every wrong answer has the same cost, such as a demo for one small team.
The assistant only picks from a fixed list and never writes a free-form answer.

Words to know

Terms in this lesson

Severity: How much harm a failure can cause.
Bucket: A named group of failures, such as critical, major, or minor.
Critical failure: A wrong fact that a reader could act on.
Control case: A question with no answer in the documents, where the right reply is a refusal.

Severity buckets and control cases

Three buckets sort failures by harm

Sorting the failures

Control cases check for refusal

Scoring the controls

The release rule