VereskLearn

Module 1 · Talking to a model

Lesson 1.3

Temperature and why answers vary

Temperature sets how much the next-word choice can vary, so one prompt can give different replies.

Lesson 5 of 622 min read

The problem

Why it matters

Without it

The same question gets different wording on each run. Tests fail at random, and nobody can tell whether a change helped.

With it

A low temperature makes the choice nearly fixed. The same question then gets the same reply, so test runs can be compared.

How it works

  • The model gives each next word a chance

    After the words "Refunds take", the model gives each possible next word a chance, written as a percent here. This example uses 50 percent for "5", 30 percent for "7", and 20 percent for "10". The chances add up to 100 percent.

    The chance of each next word after "Refunds take", at temperature 1.
    // The chances the model gives after "Refunds take".const words = ["5", "7", "10"];const chances = [0.5, 0.3, 0.2];const total = chances.reduce((sum, c) => sum + c, 0);console.log("Words ->", words.join(", "));console.log("Total chance ->", total.toFixed(1));

    Output

    Words -> 5, 7, 10
Total chance -> 1.0

  • Temperature reshapes the chances

    Temperature changes the chances before one word is picked. The code raises each chance to the power 1 divided by the temperature, then rescales the results to total 1. Many systems divide each raw score by the temperature, which gives the same result.

    • 5
    • 7
    • 10
    percent0204060800.512temperature
    The chance of each word at three temperatures. A low temperature favors the top word.
    // Raise each chance to the power 1 / temperature, then rescale.function adjust(chances: number[], temperature: number): number[] {  const powered = chances.map((c) => Math.pow(c, 1 / temperature));  const total = powered.reduce((sum, p) => sum + p, 0);  return powered.map((p) => p / total);}const chances = [0.5, 0.3, 0.2];const fmt = (w: number[]) => w.map((c) => c.toFixed(2)).join(" ");console.log("At 0.5 ->", fmt(adjust(chances, 0.5)));console.log("At 2 ->", fmt(adjust(chances, 2)));

    Output

    At 0.5 -> 0.66 0.24 0.11
At 2 -> 0.42 0.32 0.26

  • A random roll picks one word

    The program picks one word with a random number from 0 to 1, called a roll. It adds the chances in order and picks the first word where the running total passes the roll. A roll of 0.55 picks "5" at temperature 0.5, but "7" at temperature 1 and 2.

    TemperatureTotal after "5"Total after "7"Roll 0.55 picks
    0.50.660.905
    10.500.807
    20.420.747
    Running totals for a roll of 0.55. Only the lowest temperature changes the pick.
    const words = ["5", "7", "10"];// Pick the first word whose running total passes the roll.function pick(weights: number[], roll: number): string {  let total = 0;  const i = weights.findIndex((w) => (total += w) > roll);  return words[i < 0 ? words.length - 1 : i];}const pickA = pick([0.66, 0.24, 0.11], 0.55);const pickB = pick([0.5, 0.3, 0.2], 0.55);console.log(pickA, pickB);

    Output

    5 7

  • The loop repeats for each word

    The chosen word joins the reply, so the reply becomes "Refunds take 5". The model then gives new chances for the next word, such as "days", and the program picks again. The loop stops when the reply ends or reaches a length limit. Temperature does not make a reply true, only more or less varied.

    1. Refunds take
    2. Chances for "5", "7", "10"
    3. Pick "5"
    4. Chances for the next word
    One pass of the loop. The chosen word feeds the next round of chances.

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Setting temperature to 0 and expecting correct answers. It only makes the choice repeatable, and the top word can still be wrong.
  • Comparing two runs made at different temperatures. The difference comes from the setting, not from your code change.
  • Using a high temperature for facts. Higher values pick less likely words, which raises the chance of a wrong detail.

Use it when and skip it when

Is it the right tool?

Use it when

  • You want the same reply each time, such as in a repeatable test run.
  • You want varied wording, such as several sample titles for one product.

Skip it when

  • The output must follow a fixed format. Check the format in code, because temperature does not guarantee it.
  • You need one exact fact. A lookup returns the same value on every call.

Words to know

Terms in this lesson

Temperature
A setting that controls how much the next-word choice can vary.
Chance
A number from 0 to 1 that says how likely an outcome is.
Sampling
Picking one outcome at random, using the chances.
Roll
A random number between 0 and 1 used to pick an outcome.
Full code