Streaming text

The stream

As it arrives
Streaming text is easy to get wrong. Three things help:
- Pace the words, so bursts don't jolt.
- Close open markup, so nothing flips.
- Keep the layout still as it grows.
```
show(wholeWords(text, n))
```
The rest is taste.
Streaming text is easy to get wrong. Three things help:
- Pace the words, so bursts don't jolt.
- Close open markup, so nothing flips.
- Keep the layout still as it grows.
```
show(wholeWords(text, n))
```
The rest is taste.
Finished 0.0s after the stream
- Keeps up: yes
- Steady: no
- Still: no
- Soft: no
Each piece drawn the moment it lands. Bold shows its asterisks until they close, and the code is a paragraph until its fence does.
Settled markup
Streaming text is easy to get wrong. Three things help:
- Pace the words, so bursts don't jolt.
- Close open markup, so nothing flips.
- Keep the layout still as it grows.
```
show(wholeWords(text, n))
```
The rest is taste.
Streaming text is easy to get wrong. Three things help:
- Pace the words, so bursts don't jolt.
- Close open markup, so nothing flips.
- Keep the layout still as it grows.
```
show(wholeWords(text, n))
```
The rest is taste.
Finished 0.0s after the stream
- Keeps up: yes
- Steady: no
- Still: yes
- Soft: no
Markup still arriving is held back, and open markup is drawn as what it's becoming, so nothing flips. It still comes in bursts.
Typewriter
Streaming text is easy to get wrong. Three things help:
- Pace the words, so bursts don't jolt.
- Close open markup, so nothing flips.
- Keep the layout still as it grows.
```
show(wholeWords(text, n))
```
The rest is taste.
Streaming text is easy to get wrong. Three things help:
- Pace the words, so bursts don't jolt.
- Close open markup, so nothing flips.
- Keep the layout still as it grows.
```
show(wholeWords(text, n))
```
The rest is taste.
Finished 3.9s after the stream
- Keeps up: no
- Steady: yes
- Still: yes
- Soft: partly
A steady 34 letters a second, whatever the stream does: smooth, but it falls behind and ends long after.
Paced
Streaming text is easy to get wrong. Three things help:
- Pace the words, so bursts don't jolt.
- Close open markup, so nothing flips.
- Keep the layout still as it grows.
```
show(wholeWords(text, n))
```
The rest is taste.
Streaming text is easy to get wrong. Three things help:
- Pace the words, so bursts don't jolt.
- Close open markup, so nothing flips.
- Keep the layout still as it grows.
```
show(wholeWords(text, n))
```
The rest is taste.
Finished 0.2s after the stream
- Keeps up: yes
- Steady: yes
- Still: yes
- Soft: no
Lets out what's waiting at a pace set by how much is waiting, in whole words: steady, and never far behind.
Paced, with a caret
Streaming text is easy to get wrong. Three things help:
- Pace the words, so bursts don't jolt.
- Close open markup, so nothing flips.
- Keep the layout still as it grows.
```
show(wholeWords(text, n))
```
The rest is taste.
Streaming text is easy to get wrong. Three things help:
- Pace the words, so bursts don't jolt.
- Close open markup, so nothing flips.
- Keep the layout still as it grows.
```
show(wholeWords(text, n))
```
The rest is taste.
Finished 0.2s after the stream
- Keeps up: yes
- Steady: yes
- Still: partly
- Soft: partly
A caret where the next word lands shows where to look, but it jumps with every word and blinks through every pause.
Paced, fading in
Streaming text is easy to get wrong. Three things help:
- Pace the words, so bursts don't jolt.
- Close open markup, so nothing flips.
- Keep the layout still as it grows.
```
show(wholeWords(text, n))
```
The rest is taste.
Streaming text is easy to get wrong. Three things help:
- Pace the words, so bursts don't jolt.
- Close open markup, so nothing flips.
- Keep the layout still as it grows.
```
show(wholeWords(text, n))
```
The rest is taste.
Finished 0.2s after the stream
- Keeps up: yes
- Steady: yes
- Still: yes
- Soft: yes
Each word fades in over a fifth of a second, so none appears all at once. This is how the cloud answers.

The answer arrived in 8 bursts, over 3.0s. Paced, it finished 0.2s after the stream; typed, 3.9s after.

Network

7.5s

// What's arrived is let out at a pace set by how much is waiting: a backlog
// drains in about LAG seconds, so it's never far behind, and the rate eases
// toward that pace rather than jumping, so a burst becomes a run of words.
export function pace({ shown, rate }: Paced, received: number, dt: number, done: boolean): Paced {
  const backlog = received - shown;
  if (backlog <= 0) return { shown: received, rate: 0 };
  const target = Math.max(FLOOR, (backlog / LAG) * (done ? FINISH : 1));
  const next = rate + (target - rate) * Math.min(1, dt / SETTLE);
  return { shown: Math.min(received, shown + Math.max(next, FLOOR) * dt), rate: next };
}

// Shown in whole words: a word appears once the space after it has arrived.
export function wholeWords(text: string, count: number, done = false) {
  const end = Math.floor(count);
  if (end >= text.length) return done ? text : text.slice(0, text.search(/\S*$/));
  if (/\s/.test(text[end])) return text.slice(0, end);
  return text.slice(0, end).replace(/\S*$/, "");
}

// Settled: markup still arriving is held back, so "**" or a half-written
// fence never shows, and open markup is drawn as what it's becoming.
export function settleTail(text: string) {
  const lineStart = text.lastIndexOf("\n") + 1;
  const line = text.slice(lineStart);
  if (/^(?:-\s?|`{1,3}\w*)$/.test(line)) return text.slice(0, lineStart);
  return text.replace(/[*`]+$/, "");
}

// Each word is keyed by where it starts in the answer, so a word already
// drawn is never drawn again, and fades in exactly once.
wordsOf(span.text, span.at).map((word) => (
  <span key={word.at} className="stream-word">{word.text}</span> // @keyframes: from { opacity: 0 }
));

It plays the same answer on a loop. Try a bursty or a stalling network.

The stream

A model's reply reaches the browser a few characters at a time, but not evenly. There's a wait for the first word, then pieces arrive whenever the network hands them over: one at a time, or held back by a proxy and let go a dozen at once, or not at all for a second. The ticks along the top are those arrivals, the same for every panel; the network choice changes how they come.

Four questions

Each way is judged on four things. Does it keep up, finishing close to when the stream does? Is it steady, words arriving at an even pace rather than in jolts? Does it keep still, never changing or moving what it has already drawn? And is it soft, with no word landing all at once? Each panel changes one thing from the one before it, so you can see what each change buys.

Markup that doesn't flip

Answers carry a little Markdown, and drawn as it stands it flips: bold shows its asterisks until the closing pair arrives, and a code block is a paragraph that starts with three backticks until its fence closes, when it jumps into a box. Settling reads the end of the text as what it's becoming. Asterisks or backticks that may be the start of a pair are held back, open bold is bold to the end, and an open fence is a code block straight away. Nothing already drawn changes, which a test checks at every length of the answer.

Pacing

A typewriter is the usual answer to bursts: a steady number of letters a second. It's smooth, but it ignores the stream, so it falls behind when the answer comes quickly and finishes seconds after it. Pacing sets the speed from the backlog instead. Whatever is waiting goes out in about a third of a second, at a rate that eases toward that pace rather than jumping, so a burst of forty characters becomes a quick run of words, and it's never far behind. Words go out whole, each once the space after it has arrived, so none is drawn half written.

Coming in softly

A caret shows where the next word will land, but it moves with every word and blinks through every pause: more motion, not less. Fading each word in over a fifth of a second does the opposite. Nothing moves, and nothing appears all at once. It's opacity alone, and each word is keyed by where it starts in the answer, so it fades in once. That's how the cloud on the homepage answers now, through the same pacing code. Under reduced motion words appear without fading, and the stream waits, finished, until Replay.