deepseek-harness/docs/adr/0018-session-persistence.md

ADR 0018: Session persistence as an abstract service over the existing SessionEvent

Status: accepted (2026-06-15)

Context

Sessions lived only in memory. The example session-jsonl.ts plugin (duplicated byte-for-byte in both examples) was write-only telemetry: it buffered session/event and appended JSON lines, with no read/replay path, no crash-safety (no fsync, no atomic write, a fire-and-forget dispose drain), no listing, and no format versioning. Nothing could rehydrate a past session from disk into a live agent, so durable resume ("continue yesterday's task"), durable forking, and the ACP session/load method (RFC 010) were all impossible.

The event-sourced model makes the append-only log the single source of truth and derives LLM history from it. Persistence had to stay faithful to that: persist the existing SessionEvent directly, with no parallel "persisted message" type that the log is converted to and from. The backend also had to be swappable — a file store now, a database store later — behind one interface.

Decision

Persistence is an abstract capability seam (ADR 0009, the dsh-bash template), not loop or core logic:

1. Interface (dsh-session-persistence, ctx.sessionPersistence) — an abstract SessionPersistence service: create/append/load/list/has/delete/update. Its persisted unit IS the existing SessionEvent ({ type, seq, time, data }), reused verbatim — no conversion type.
2. Implementation (dsh-session-persistence-jsonl) — an append-only JSONL log per session (a SessionHeader line then one SessionEvent per line, verbatim including assistant/chunk) plus an atomic .summary.json sidecar for the mutable SessionSummary.

Key choices recorded here because they are durable, contested, and surprising:

• The canonical durable log persists every SessionEvent verbatim, including assistant/chunk. deriveMessages() skips chunks, and a chunk-filtered rollout (Codex's policy.rs) is tempting — but seq = log.length and the load-validation events[i].seq === i require a contiguous log; filtering chunks out would leave holes and break both the contract and resume. A chunk-filtered projection is possible later as a derived view with its own renumbering, but it is NOT the canonical log.
• Append-only; a crashed turn is closed, never truncated. Committed events — those at or below a flushed turn/end — are never rewritten. The loop only flushes at turn/end, so a crash can leave a durable log whose final turn never closed: real, fully-written events sit after the last turn/end. A single turn can be huge in a long-horizon task (many steps, large tool output spanning a long autonomous run), so discarding the interrupted turn would silently destroy a large amount of real work — truncating a turn is wrong. Instead, on reload load PRESERVES those events and CLOSES the orphaned turn by durably appending the minimal synthetic boundary events: an error tool/result for every tool-call the crash left unanswered, then a step/end if a step was still open, then a turn/end carrying the merge-extensible { kind: 'interrupted' } reason (a marker that records the turn was cut short by a crash, not completed by the model — no loop ever emits it). The synthetic tool results matter for resume correctness: the loop logs the assistant/message (carrying the tool-call blocks) BEFORE running the tools, so a crash mid-tool leaves calls without results; deriveMessages() would then replay a dangling assistant tool-call, which every provider rejects as an invalid transcript on the next request. Answering each orphaned call with an error result keeps the rehydrated history valid. load returns the balanced log, so a resumed session is immediately usable. The ONLY thing discarded is a never-fully-written torn tail fragment — a final record whose bytes (JSONL) or row were never completely flushed; that fragment is not a valid event and is dropped before the synthetic closers are written. A parse error or seq gap in the COMMITTED region (at or before the last real turn/end) is genuine corruption and makes the session unloadable.
• File backend canonical, DB backend a drop-in. SessionEvent maps 1:1 onto a row (session_id, seq, type, time, data) — append is INSERT (in a transaction asserting the contiguous-seq contract), load is SELECT … ORDER BY seq. A future dsh-session-persistence-sqlite is a SessionPersistence subclass with no interface change (opencode runs this exact shape on SQLite/WAL).
• Metadata is out-of-log. Format version, cwd, and lineage are storage concerns, not replayable conversation state, so they live in a SessionMeta (SessionHeader & SessionSummary) owned by dsh-session and attached to a Session via a new readonly session.header — never in SessionEventMap, never reaching deriveMessages(). The alternative (a merge-extensible session/meta event as log line 0) was rejected: an in-log event would ride along with a seeded/forked session for free, but metadata is not replayable state, so the explicit out-of-log header seam is the cleaner cost.
• load returns a resumable event log, not just bytes. load(sessionId) yields the SessionMeta plus the committed SessionEvent[] (through the last complete turn/end), shaped so a caller can reconstruct a live session with the loaded events as seed (so lastTurnNumber/deriveMessages continue) on the SAME session id. The agent-facing create/resume factory that consumes this is a separate seam (a follow-up on ctx.agents); the persistence layer deliberately stops at the load primitive and does NOT reach into the loop. The agent-loop does NOT hard-inject sessionPersistence (that would pend non-persistent demos forever), so any resume path built on this rejects with a clear error when the backend is absent.

Format versioning: the header carries a version; load rejects an unknown version (no v1 migration). Stated honestly: append-only + flush is robust to partial trailing writes (tolerated on load) but not to fsync-less power loss mid-line; a DB/WAL backend is the stronger option later.

Consequences

Two new packages and the metadata seam in dsh-session (session.header, the create(id?, options?) signature). Bought: durable resume/fork, a read/replay path, crash tolerance, and the foundation RFC 010's session/load needs — all over the existing event-sourced log, with the backend swappable behind one interface. The reusable runPersistenceContract suite holds every backend to the same append-only / contiguous-seq / lazy-materialization / serializability semantics. This completes ADR 0003's deferred "real persistence backend" and resolves its TODO(review) on the event vocabulary: persisting the log freezes its shape, and the assistant/chunk fidelity question is answered above (persist verbatim).