VereskLearn

Module 5 · Writing the answer

Lesson 5.6

Long context and attention

A longer context costs more and can hide the answer, so send only the chunks that help.

Lesson 37 of 622 min read

The problem

Why it matters

Without it

Every chunk is sent to be safe. The request costs more, and the best passage can sit in the middle of the text.

With it

A short, ordered set of sources is sent. The best passage sits first, and the request stays small.

How it works

  • Tokens measure the size

    A token is a small piece of text, roughly four characters of English. This sample estimates tokens as characters divided by four, rounded up. A 25 character sentence is about 7 tokens.

    function estimateTokens(text: string): number {  return Math.ceil(text.length / 4);}console.log(estimateTokens("Refunds take 5 to 7 days."));

    Output

    7

  • Fewer chunks can drop the answer

    The sample has ten chunks, and the answer sits in position 8. Sending only the first three chunks is smaller, but the answer is not in them. The size falls from 81 to 25 tokens, and the answer goes with it.

    SentTokensAnswer found
    All ten81yes
    First three25no
    First three, best moved first23yes
    Ten chunks in search order, and three ways to send them.

  • Order can keep the answer

    Moving the best chunk to the front puts it inside the first three. The same three slots now hold the answer, at 23 tokens. Order can matter as much as size, so both need a check.

    const chunks = [  "Filler one.",  "Filler two.",  "Refunds take 5 to 7 days.",];const best = chunks[2];const others = chunks.filter((chunk) => chunk !== best);const reordered = [best, ...others];console.log(reordered.slice(0, 2).join(" | "));

    Output

    Refunds take 5 to 7 days. | Filler one.

  • Attention weighs the input

    Attention is the part of a model that decides which parts of the input to weigh. A very long input spreads that weight over more text. The eval set shows whether a fact placed in the middle is still used.

    Near the start

    • Usually read with a strong weight
    • Answer found in the tests

    Deep in the middle

    • May get less weight
    • May be missed more often
    Where a fact sits in a long input.

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Sending everything for safety. More text costs more and can bury the answer.
  • Cutting by chunk count only. Chunks differ in length, so count tokens.
  • Changing the order without a test. Run the eval set after each change to the order.

Use it when and skip it when

Is it the right tool?

Use it when

  • Requests are slow or costly because the context is large.
  • The answer exists in the sources, but the model misses it.

Skip it when

  • The whole source text is short enough to send safely.

Words to know

Terms in this lesson

Context window
The limit on how much text one request can hold.
Token
A small piece of text, the unit that a model counts and bills.
Attention
The model mechanism that weighs parts of the input against each other.
Full code