VereskLearn

Module 9 · Running it for real

Lesson 9.4

Caching

A cache saves an answer under a key, so a repeat question gets the saved answer with no model call.

Lesson 57 of 622 min read

The problem

Why it matters

Without it

Each repeat question calls the model again. The same answer costs time and money every time.

With it

A repeat question finds its saved answer by key. The model runs only for questions it has not answered.

How it works

  • Normalize before hashing

    The question is trimmed and lowercased before it is hashed. Two questions that differ only in spacing or case then give the same key. The sample turns the question into one plain form.

    const raw = "  how long do REFUNDS take? ";const normal = raw.trim().toLowerCase();console.log(JSON.stringify(normal));

    Output

    "how long do refunds take?"

  • The key is a SHA-256 hash

    SHA-256 is a hash function, which turns any text into a fixed-length code. The key hashes the model name and the normalized question together. The same text always gives the same code, so it works as a lookup key.

    import { createHash } from "node:crypto";const text = "model-a|how long do refunds take?";const key = createHash("sha256").update(text).digest("hex");console.log("Key starts with " + key.slice(0, 12));console.log("Key length: " + key.length);

    Output

    Key starts with 27384c6a4ec6
Key length: 64

  • A hit skips the model

    A hit finds the saved answer under the key and makes no model call. A miss calls the model once and saves the answer under the key. The table shows three requests, and only the first and third miss.

    RequestKey foundModel call
    How long do refunds take? (model-a)noyes
    Same question, extra spaces (model-a)yesno
    Same question (model-b)noyes
    Three requests, and whether each one reaches the model.

  • Counting calls shows the saving

    The modelCalls counter counts how many times the model runs. The sample makes three requests, and the model runs only twice. The cache holds two entries, one for each model.

    Requests made, and model calls actually made.

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Leaving the model name out of the key. An answer from one model is then served for another.
  • Leaving the sources or the prompt version out of the key. Old answers then survive a change to the documents.
  • Caching a refused or failed answer. Save only answers that passed the checks.

Use it when and skip it when

Is it the right tool?

Use it when

  • Many users ask the same questions.
  • Answers are slow or costly, and they do not change between requests.

Skip it when

  • Each question is unique, so there are no repeats to save.
  • Answers must reflect the newest data on every request.

Words to know

Terms in this lesson

Cache
A store that keeps results so they can be reused.
Key
The value that finds a saved item in a cache.
Hash
A fixed-length code computed from text, the same for the same text.
Cache hit
A request answered from the cache, with no model call.
Full code