Yichen Jiang
96c44c83d4
The TTFT and tok/s readings divide by measured wall time, so they are not
reproducible: the same replayed scenario yielded 69 and 70 tok/s on
consecutive local runs, and a 3 ms replayed stream reads 26333 tok/s. Baking
those into committed goldens made the Web lane flaky by construction, and the
goldens for the readings themselves were never refreshed.
Three fixes, then a refresh:
The footer's decorative dots are `aria-hidden`, so the readings concatenated
into one accessible string — `Ran for 13sTTFT 0.2s12 tok/s`. That is a real
defect on its own (a reader hears one run-on instead of three facts) and it
also denied `{{duration}}` the word boundary it matches on, so even the
previously-stable `Ran for` duration started leaking raw. The separators now
carry flanking spaces.
`normalizeAria` gains `{{throughput}}` beside `{{duration}}`, and its duration
alternation accepts the stats line's compact `2m42s` as well as the
message-chrome template's `2m 42s` — the compact form had no pattern at all,
which is why `LLM 382m39s` survived the first refresh.
Refreshed 17 goldens. They also record that the stats line's `LLM` group now
renders at all: it folds assistant `timing`, which the live transcript adapter
only began attaching in this branch, so the group was previously dead in Chat.
Verified by running the lane in replay three times after the refresh: 41/41
files green each time, goldens untouched. Before this change two consecutive
runs disagreed on both the values and the failure count.
4.3 KiB
Agent Note: Web turn and window latency/throughput metrics
Status: implemented
English | 中文
Problem
The Web chat records per-step LLM timing (stepStartTime / firstTokenTime / completedTime) and per-step usage, and the trajectory view exposes them per step, but the chat surface answers neither "how responsive was this turn" nor "how fast is this session going": the assistant footer shows only the turn wall time, and the stats line folds only wall-time totals.
Decision
A package-local fold, ui-conversation's chat/turn-metrics.ts, is the single derivation from assistant nodes to latency/throughput readings. assistantStepReading turns one node into a step reading: TTFT needs both stepStartTime and firstTokenTime, decode span needs firstTokenTime, negative spans clamp to zero, and output tokens come from the untrusted usage value only when they are finite and non-negative. deriveTurnMetrics folds readings per turn: the lowest-numbered step owns the turn's TTFT slot, and throughput divides the summed output tokens by the summed decode spans over exactly the steps carrying both, so an unsampled step drops out instead of skewing the ratio; a turn with neither figure emits no entry.
The assistant footer appends the readings to the existing hover-revealed time chrome after Ran for, as TTFT {s}s · {tps} tok/s, each omitted independently when unrecorded. ChatView shows a turn's readings only when that turn's turnTimings entry has an endTime: the loaded window is a contiguous log suffix, so an in-window settled turn carries every one of its steps and the first-step TTFT is genuine rather than a window artifact. formatLatencySeconds is unit-less so each locale template owns its second suffix (TTFT {seconds}s / 首 token {seconds}秒).
The stats line reuses the same step reading in its window fold: deriveStats accumulates TTFT sum/count and decode span/tokens, rendering a TTFT avg … · … tok/s group beside the LLM/tool wall times. Like those wall times the group is window-scoped and folds no billing; token accounting stays on the token-meter projections.
Alternatives considered
A durable session projection (token-meter shape). A ProjectionDefinition folding step timings host-side would survive compaction and window paging and cover the whole log. Deferred, not rejected: projection state must stay O(1) (averages, not percentiles), it needs a host change plus a schema, and the chat stats line is already documented as window-scoped for its duration facts — the new group joins that scope. A later PR can add the durable projection without moving these readings.
Per-step footer chrome. Showing each assistant message its own TTFT would attach chrome to mid-turn narration nodes, which the footer design deliberately keeps chrome-free; the trajectory view already exposes per-step timing detail.
Gating footer metrics on node presence instead of turn/end timing. Rendering whatever steps happen to be loaded would show a plausible-looking TTFT that is actually the first loaded step after paging. The endTime gate plus the suffix-window invariant makes the displayed figure the turn's true first-step latency or nothing.
Consequences
A settled in-window turn's footer reveals TTFT/tok/s on hover after the wall time, and the stats line shows window-average latency and throughput beside its wall times, all without new session events or host changes. Metrics degrade by omission: providers or steps without timing or usage samples drop individual figures rather than rendering zeros. Older history outside the loaded window stays uncounted, recorded in the package README's stats-line limitation.
Both readings divide by measured wall time, so neither is reproducible: the same replayed scenario yielded 69 and 70 tok/s on consecutive local runs, and a 3 ms replayed stream reads 26333 tok/s. The Web aria goldens therefore normalize throughput to {{throughput}} beside the existing {{duration}}, and the footer's decorative separators gained flanking spaces — without them the readings concatenate into one accessible string (Ran for 13sTTFT 0.2s12 tok/s), which both loses the reading boundaries a screen reader needs and denies {{duration}} the word boundary it matches on.