test(examples): snapshot headless one-shot stream

This commit is contained in:
Tianyi Cui
2026-07-15 22:43:03 +08:00
parent 7278fc21d8
commit 5e3da065db
13 changed files with 251 additions and 13 deletions
+2 -2
View File
@@ -7,7 +7,7 @@ How this repo tests, tier by tier, and the rules that keep a green suite meaning
- **Unit** (`pnpm run test`): vitest over `packages|examples/*/tests/**/*.spec.ts`, colocated with what they test. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Prefer edge cases, error paths, event ordering, concurrency races, and permanent contract regressions (see `packages/core/agent-loop/tests/contract-regressions.spec.ts`).
- **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped.
- **Real-API e2e** (`pnpm run test:e2e`): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (`EXA_API_KEY`, `PERPLEXITY_API_KEY`, …); each suite self-skips without its key so keyless CI stays green ([real-API e2e RFC](rfc/implemented/testing/2026-06-19-real-api-e2e-ci.md)).
- **Snapshot** (`pnpm run test:snapshot`): boots the real example subprocess, replays a recorded session keyless, diffs normalized stdout + the re-persisted log against committed goldens ([snapshot RFC](rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md)). Use `pnpm run test:snapshot:record` when the model transcript should change; use `pnpm run test:snapshot:refresh` when the committed transcript is still the right mock LLM input and replay goldens need keyless rewrite. Review the golden diff. System-prompt/tool-schema content is pinned by ONE scenario (`text-turn`) and tokenized in every other fixture, so a prompt or schema edit churns one committed line ([pinned-header RFC](rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **Snapshot** (`pnpm run test:snapshot`): real example subprocesses replay recorded model sessions keylessly and compare normalized stdout plus re-persisted logs ([snapshot RFC](rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md)). The primary suite pins ACP JSON-RPC; the headless projection reuses `advanced-toolchain` for `stream-json`. Use `pnpm run test:snapshot:record` when the model transcript changes and `pnpm run test:snapshot:refresh` when only replay outputs change; review the golden diff. One scenario per header class pins system-prompt/tool-schema content; other fixtures tokenize it ([pinned-header RFC](rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
## The with-key policy: inference is cheap here
@@ -30,4 +30,4 @@ An e2e assertion re-runs the command or re-reads the file externally; a keyword
## When a snapshot test is required
Any change affecting the editor-facing transcript or end-to-end agent UX — the ACP bridge, the loop's observable output, tool presentation — adds or updates a scenario in the owning example's snapshot suite (`examples/<name>/tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/support/acp-snapshot/README.md) suite factory; `examples/acp-agent` is the primary suite), or states in the PR why none applies. New capability seams, lifecycle shapes, or transcript surfaces name their coverage at every tier at plan time and verify the harness can express it — a harness gap is scheduled work, not a mid-build surprise.
A change affecting an editor transcript, headless event stream, or agent UX adds or updates the owning `examples/<name>/tests/snapshots/` scenario, or explains its omission in the PR. `examples/acp-agent` hosts the primary [`dsh-acp-snapshot`](../packages/support/acp-snapshot/README.md) table and the headless `stream-json` projection. Plans for new capability seams, lifecycle shapes, or transcript surfaces identify every test tier and any required harness work before implementation.