VereskLearn

Module 9 · Running it for real

Lesson 9.1

Streaming answers with citations

Streaming sends the answer piece by piece as it is written, and sends the citations last.

Lesson 54 of 622 min read

The problem

Why it matters

Without it

The reader waits in silence until the whole answer is ready. A long answer can take many seconds.

With it

Text appears as soon as the first piece arrives. The citations follow at the end, once they are known.

How it works

  • Text arrives one piece at a time

    The model writes its answer in small pieces, and each piece is sent as soon as it exists. The sample answer arrives as six pieces, each with a trailing space. The screen can show each piece at once.

    • Refunds
    • take
    • 5
    • to
    • 7
    • days. last text piece
    The six pieces of the sample answer, in arrival order.

  • A generator yields each piece

    An async generator is a function that yields values over time. Each yield hands one piece to the caller, and the caller reads it with for await. The sample yields two pieces to show the pattern.

    async function* streamWords() {  for (const word of ["Refunds", "take"]) {    yield word + " ";  }}(async () => {  for await (const piece of streamWords()) {    console.log(JSON.stringify(piece));  }})();

    Output

    "Refunds "
"take "

  • Citations come last

    The stream carries two kinds of event. Text events hold one piece of the answer each. The citations event holds the source ids, and it arrives only after the last text piece.

    EventHoldsWhen it arrives
    textone piece of the answeras each piece is written
    citationsthe source ids, such as faq-12after the last text piece
    The two event kinds in the sample stream.

  • The screen joins the pieces

    Each piece keeps its own trailing space, so the pieces join with the right gaps. The full answer is trimmed once at the end, which removes the last space. The sample joins six pieces into one sentence.

    1. Model
    2. Text events
    3. Screen
    4. Citations
    The path of one answer from the model to the screen.
    const pieces = ["Refunds ", "take ", "5 to 7 days."];let shown = "";for (const piece of pieces) {  shown += piece;}console.log(shown.trim());

    Output

    Refunds take 5 to 7 days.

Full code

Read the program

See the full code

Common mistakes

What goes wrong

  • Showing citations before the answer ends. The list can change, so wait for the citations event.
  • Losing the text when the stream fails halfway. Show an error and keep what arrived, marked as partial.
  • Dropping the spaces between pieces. Keep each piece exactly as sent, or the words run together.

Use it when and skip it when

Is it the right tool?

Use it when

  • Answers are long, and the reader would otherwise wait many seconds.
  • Each answer must end with its sources.

Skip it when

  • Answers are short and return in well under a second.
  • The reader only sees the final text, such as in an email.

Words to know

Terms in this lesson

Stream
A reply sent in pieces over time, not all at once.
Event
One item in a stream, such as a text piece or the citations.
Async generator
A function that yields values over time instead of returning one value.
Partial answer
An answer that is not finished yet.
Full code