test(acp-example): five snapshot scenarios + cancel/error input ops

Adds the first cut of snapshot scenarios, each asserting a normalized stdout
transcript golden and (for model turns) a re-persisted session-log golden:

- text-turn, tool-call-turn, multi-turn: RECORDED against the real API — the
  committed session.jsonl is a genuine harvested log; replay derives the model
  script from it and reproduces deterministically with no key. tool-call-turn
  exercises the real bash executor (echo SNAPSHOT_OK → tool/call + tool/result
  + a post-tool answer step).
- error-finish, cancel: AUTHORED via a replay.override.json sidecar (the live
  API can't be coaxed into a deterministic 401 or mid-stream cancel). error-
  finish replays a {kind:throw} 401 → the bridge answers the prompt with a
  JSON-RPC error and the log records turn/end{kind:error}; cancel replays a
  {kind:hang} → stopReason:cancelled.

Two input-DSL ops support these: promptExpectError (awaits the prompt, asserts
it rejects — the editor's view of a failed turn — and swallows it) and
promptAndCancel (dispatches the prompt unawaited, waits until the client
OBSERVES the streamed agent_message_chunk, then cancels — pinning frame order
so the cancel transcript is deterministic; fixes a flake Codex caught where the
late chunk and the cancelled response could interleave either way).

Scenarios carry a `recorded` flag so test:snapshot:record only re-runs the
live-API ones. reasoning/max-tokens scenarios are deferred (hard to force
deterministically from the live model). Per docs/rfc/implemented/2026-06-19.
This commit is contained in:
Tianyi Cui
2026-06-19 04:10:49 +08:00
parent 81d434896d
commit c94f1563f5
24 changed files with 5173 additions and 9 deletions
@@ -0,0 +1,7 @@
{
"steps": [
{ "op": "initialize" },
{ "op": "newSession" },
{ "op": "promptAndCancel", "text": "Start a long task; this turn will be cancelled mid-stream." }
]
}
@@ -0,0 +1,3 @@
[
{ "kind": "hang" }
]
@@ -0,0 +1,95 @@
{
"type": "session",
"version": 1,
"id": "{{sessionId}}",
"createdAt": 0,
"cwd": "{{cwd}}"
}
{
"type": "turn/start",
"seq": 0,
"time": 0,
"data": {
"turn": 1,
"trigger": {
"kind": "message",
"source": {
"kind": "user"
}
}
}
}
{
"type": "user/message",
"seq": 1,
"time": 0,
"data": {
"content": [
{
"type": "text",
"text": "Start a long task; this turn will be cancelled mid-stream."
}
],
"source": {
"kind": "user"
}
}
}
{
"type": "step/start",
"seq": 2,
"time": 0,
"data": {
"turn": 1,
"step": 1
}
}
{
"type": "assistant/chunk",
"seq": 3,
"time": 0,
"data": {
"turn": 1,
"step": 1,
"chunk": {
"type": "block-start",
"index": 0,
"blockType": "text"
}
}
}
{
"type": "assistant/chunk",
"seq": 4,
"time": 0,
"data": {
"turn": 1,
"step": 1,
"chunk": {
"type": "text-delta",
"index": 0,
"text": "partial"
}
}
}
{
"type": "step/end",
"seq": 5,
"time": 0,
"data": {
"turn": 1,
"step": 1
}
}
{
"type": "turn/end",
"seq": 6,
"time": 0,
"data": {
"turn": 1,
"reason": {
"kind": "aborted",
"reason": "session/cancel"
}
}
}
@@ -0,0 +1 @@
{"type":"session","version":1,"id":"00000000-0000-0000-0000-000000000000","createdAt":0}
@@ -0,0 +1,62 @@
{
"jsonrpc": "2.0",
"id": 1,
"result": {
"protocolVersion": 1,
"agentInfo": {
"name": "deepseek-harness-acp",
"version": "0.0.1"
},
"agentCapabilities": {
"loadSession": true,
"promptCapabilities": {
"image": false,
"audio": false,
"embeddedContext": false
}
},
"authMethods": []
}
}
{
"jsonrpc": "2.0",
"id": 2,
"result": {
"sessionId": "{{sessionId}}"
}
}
{
"jsonrpc": "2.0",
"method": "session/update",
"params": {
"sessionId": "{{sessionId}}",
"update": {
"sessionUpdate": "user_message_chunk",
"content": {
"type": "text",
"text": "Start a long task; this turn will be cancelled mid-stream."
}
}
}
}
{
"jsonrpc": "2.0",
"method": "session/update",
"params": {
"sessionId": "{{sessionId}}",
"update": {
"sessionUpdate": "agent_message_chunk",
"content": {
"type": "text",
"text": "partial"
}
}
}
}
{
"jsonrpc": "2.0",
"id": 3,
"result": {
"stopReason": "cancelled"
}
}