Answers that arrive well

Streaming a model's answer is a latency fix: the first words in half a second instead of the whole reply in eight. It works. But most chat interfaces draw each piece the moment it lands, which hands the network's rhythm straight to the reader. Three words, a stall, a sentence at once, a word drawn half written, a pair of asterisks that turns into bold a second later. The answer starts sooner and reads worse.

The fix is one idea: arrival and display run on two clocks. The network decides what's available. The interface decides when it's shown. Everything below is what goes between them.

Streaming text in the Lab plays one answer six ways side by side, from each piece as it lands to the version described here. This is the reasoning behind it, and the code.

Never draw what you'll redraw

Whatever goes on screen should stay as it was drawn. That rules out two common things. A half word: stre followed by streaming a frame later reads like a typo being corrected. And raw markup: an opening ** drawn as asterisks until its pair arrives, then the whole run jumping to bold.

Words are the easy part. Show a word only once the space after it has arrived:

/** The first `count` characters, cut back to the last whole word. */
function wholeWords(text: string, count: number, done = false) {
  const end = Math.floor(count);
  if (end >= text.length) {
    return done ? text : text.slice(0, text.search(/\S*$/));
  }
  if (/\s/.test(text[end])) return text.slice(0, end);
  return text.slice(0, end).replace(/\S*$/, "");
}

Markup takes a parser that reads the end of the text as what it's becoming. Asterisks or backticks that might open a pair are held back until it's clear. An open bold is bold to the end of what's there. Three backticks open a code block straight away, so its first line lands in the box rather than in a paragraph that becomes one later. The test that keeps it honest parses the answer at every length and checks that each step only adds to the one before.

Let the backlog set the pace

The usual answer to bursts is a typewriter: a fixed number of letters a second. It's smooth, and it's wrong as soon as the model is quick, because it ignores the stream. A fast answer has finished arriving while the typewriter is still on its second paragraph, and the reader waits on an animation.

Set the pace from the backlog instead. Whatever is waiting should be out in about a third of a second, so the rate is the backlog divided by that. The rate eases toward its target rather than jumping to it, which turns a burst of forty characters into a quick run of words rather than a block. A floor keeps a trickle at a comfortable reading speed, and once the stream ends the rest goes three times as fast: there's nothing left to wait for.

type Paced = { shown: number; rate: number };

const LAG = 0.35; // a backlog drains in about this many seconds
const FLOOR = 30; // the slowest it goes, in characters a second
const SETTLE = 0.2; // how quickly the rate follows the backlog
const FINISH = 3; // once the stream has ended, this much faster

function pace(
  { shown, rate }: Paced,
  received: number,
  dt: number,
  done: boolean,
): Paced {
  const backlog = received - shown;
  if (backlog <= 0) return { shown: received, rate: 0 };
  const speed = backlog / LAG;
  const target = Math.max(FLOOR, done ? speed * FINISH : speed);
  const next = rate + (target - rate) * Math.min(1, dt / SETTLE);
  const step = Math.max(next, FLOOR) * dt;
  return { shown: Math.min(received, shown + step), rate: next };
}

It's pure, so it's tested without a browser, and the component around it is a frame loop that runs only while there's something to let out. Two details matter there. Clamp the frame time: a tab coming back from the background has seconds of backlog, and it should drain quickly rather than land all at once. And only pace an answer the reader watched arrive. One read back from earlier, or finished before it was drawn, shows whole. Pacing it again is just a wait.

A word appears, nothing moves

The last choice is how a new word comes in. A caret is the obvious one, and it adds motion: it moves with every word and blinks through every pause. Fading each word in over 200 ms does the opposite. Nothing on screen moves, and nothing appears at full strength in a single frame. It's opacity alone.

The catch is React. Render the answer as one string and there's nothing to fade; split it into words keyed by index and a word can change keys when the markup around it settles, which fades it in again. Key each word by where it starts in the answer. That never changes once the word is drawn, so React keeps the node and the fade runs once.

/** Each word with the space after it, keyed by where it starts. */
const wordsOf = (text: string) =>
  [...text.matchAll(/\s*\S+\s*|\s+/g)].map((match) => ({
    text: match[0],
    at: match.index,
  }));

export function Answer({ text }: { text: string }) {
  return wordsOf(text).map((word) => (
    <span key={word.at} className="stream-word">
      {word.text}
    </span>
  ));
}

.stream-word {
  animation: stream-in 200ms var(--ease-out-quint) both;
}
@keyframes stream-in {
  from {
    opacity: 0;
  }
}
@media (prefers-reduced-motion: reduce) {
  .stream-word {
    animation: none;
  }
}

Before the first word, and after

Two moments sit outside the stream, and both get forgotten. Before the first word there's a wait of a second or more, and it needs a sign that something is happening: a spinner and a word, in the place the answer will appear, so the eye is already there when it starts.

Then the reader may want it to stop. While an answer is coming, make the send button the stop button: it's where the hand already is. Stopping keeps what arrived rather than clearing it, says so when nothing had, and gives the keyboard back to the field so the next question can be typed straight away.

The cloud on the homepage answers this way. To check your own, throttle the network and read along. If your eye is chasing the text, it's drawn as it arrives. If you can read at your own speed and it's never far behind, it's paced.