Pine/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 1 Wiki Activity

feat(llm-replay): indexed override patches for error injection

Browse Source 
The override sidecar now accepts { patches: [{ at, entry }] } alongside the
legacy whole-script ReplayEntry[] replacement: the JSONL-derived script is
kept and only the named call indexes are swapped (at == length appends, for
a retry attempt following an injected transient throw). Out-of-range or
non-integer indexes fail loud with the derived length in the diagnostic.
This is the mock-LLM error capability the web e2e scenarios drive: 'call N
throws AUTH/SERVER, everything else replays as recorded'.
This commit is contained in:
Tianyi Cui
2026-07-26 03:26:41 +08:00
parent ee25ccf38b
commit 2e52c0670c
4 changed files with 97 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/support/llm-replay/README.md
+1 -1
View File
@@ -8,7 +8,7 @@ Its consumers are the ACP snapshot harness in `examples/acp-agent` and the `stre
The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`", done by the snapshot harness — this plugin does not record. A fixture may carry its `request/header` content tokenized to `{{system}}`/`{{tools}}` (the harness pins that content in one scenario and scrubs the rest); replay is indifferent — derivation reads only `assistant/chunk` events and the line-0 session header.
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`: a `ReplayEntry[]`) that REPLACES the derived script. A `hang` entry may name `readyFile`; replay writes that empty marker after its prefix chunks reach the loop and before it waits for cancellation, so an external driver can cancel deterministically without observing a presentation update.
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`) that either REPLACES the derived script (a bare `ReplayEntry[]`) or AUGMENTS it (`{ patches: [{ at, entry }] }`: keep every JSONL-derived call, swap only the named 0-based call indexes; `at` equal to the derived length appends — the slot for the retry attempt that follows an injected transient throw). A `hang` entry may name `readyFile`; replay writes that empty marker after its prefix chunks reach the loop and before it waits for cancellation, so an external driver can cancel deterministically without observing a presentation update.
## Nested agents: per-session keying