From 9126697d87624c9db711073253bfe3e982e0e18f Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Mon, 15 Jun 2026 21:45:21 +0800 Subject: [PATCH] feat(session-persistence-sqlite): second backend validating the abstraction MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a SQLite SessionPersistence backend (node:sqlite), a SECOND implementation built to prove the abstract seam + the shared runPersistenceContract suite are genuinely backend-agnostic. Each SessionEvent maps 1:1 onto an events row (session_id, seq, type, time, data); append is an INSERT inside a transaction asserting the contiguous-seq contract; the mutable SessionSummary lives in the sessions metadata row. It satisfies the SAME contract semantics as the JSONL backend, expressed over rows instead of file bytes: - Lazy materialization: create() records intent in memory; no row until the first append (a never-appended session is absent from has()/list() via a materialized flag set inside the first append transaction). - Crash-tail-on-load: load() returns events only through the last complete turn/end and deletes the uncommitted tail; a seq gap in the committed region makes the session unloadable. - Transactional append: a mid-batch failure (a UNIQUE seq collision from a concurrent writer) rolls back entirely, keeping the cursor truthful. Like the JSONL backend it is also the write-path plugin (session/event → buffer → session/flush drain, onCreated seed/adopt/collision handling, HMR seeding, dispose-to-quiescence). The package runs the shared runPersistenceContract suite plus SQLite-specific tests (transaction rollback, crash-tail cut, schema version, HMR adoption). Docs flip every "SQLite is future/deferred" reference (ADR 0016, architecture.md, the persistence module doc + README) to "implemented; the contract holds both backends to identical semantics". --- docs/adr/0016-session-persistence.md | 2 +- docs/architecture.md | 3 +- packages/session-persistence-sqlite/README.md | 27 + .../session-persistence-sqlite/package.json | 35 ++ .../session-persistence-sqlite/src/index.ts | 482 ++++++++++++++ .../session-persistence-sqlite/src/schema.ts | 135 ++++ .../tests/sqlite.spec.ts | 595 ++++++++++++++++++ .../session-persistence-sqlite/tsconfig.json | 15 + packages/session-persistence/README.md | 2 +- packages/session-persistence/src/index.ts | 9 +- scripts/publint-all.ts | 1 + tsconfig.base.json | 1 + tsconfig.build.json | 1 + tsconfig.typecheck.json | 1 + yarn.lock | 15 + 15 files changed, 1317 insertions(+), 7 deletions(-) create mode 100644 packages/session-persistence-sqlite/README.md create mode 100644 packages/session-persistence-sqlite/package.json create mode 100644 packages/session-persistence-sqlite/src/index.ts create mode 100644 packages/session-persistence-sqlite/src/schema.ts create mode 100644 packages/session-persistence-sqlite/tests/sqlite.spec.ts create mode 100644 packages/session-persistence-sqlite/tsconfig.json diff --git a/docs/adr/0016-session-persistence.md b/docs/adr/0016-session-persistence.md index 4ab7ad6e37..d4ac30df40 100644 --- a/docs/adr/0016-session-persistence.md +++ b/docs/adr/0016-session-persistence.md @@ -19,7 +19,7 @@ Key choices recorded here because they are durable, contested, and surprising: - **The canonical durable log persists every `SessionEvent` verbatim, including `assistant/chunk`.** `deriveMessages()` skips chunks, and a chunk-filtered rollout (Codex's `policy.rs`) is tempting — but `seq = log.length` and the load-validation `events[i].seq === i` require a *contiguous* log; filtering chunks out would leave holes and break both the contract and resume. A chunk-filtered projection is possible later as a derived view with its own renumbering, but it is NOT the canonical log. - **Append-only with a single exception.** Committed events — those at or below a flushed `turn/end` — are never rewritten. The loop only flushes at `turn/end`, so a crash can leave a half-written final turn below the last checkpoint; `load` returns events only up to the **last complete `turn/end`**, and the first post-load `append` runs a one-time **truncation-repair** (`ftruncate` + `fsync`) that physically discards only that never-committed crash tail before writing. -- **File backend canonical, DB backend a drop-in.** `SessionEvent` maps 1:1 onto a row `(session_id, seq, type, time, data)` — `append` is INSERT (in a transaction asserting the contiguous-seq contract), `load` is SELECT … ORDER BY seq. A future `dsh-session-persistence-sqlite` is a `SessionPersistence` subclass with no interface change (opencode runs this exact shape on SQLite/WAL). +- **File backend canonical, DB backend a proven drop-in.** `SessionEvent` maps 1:1 onto a row `(session_id, seq, type, time, data)` — `append` is INSERT (in a transaction asserting the contiguous-seq contract), `load` is SELECT … ORDER BY seq. `dsh-session-persistence-sqlite` is exactly this: a `SessionPersistence` subclass with no interface change (opencode runs this exact shape on SQLite/WAL), and it passes the same `runPersistenceContract` suite as the JSONL backend — so the contract holds both backends to identical semantics (lazy materialization, crash-tail-on-load, contiguous-seq), expressed once over file bytes and once over rows. - **Metadata is out-of-log.** Format version, cwd, and lineage are storage concerns, not replayable conversation state, so they live in a `SessionMeta` (`SessionHeader & SessionSummary`) owned by `dsh-session` and attached to a `Session` via a new readonly `session.header` — never in `SessionEventMap`, never reaching `deriveMessages()`. The alternative (a merge-extensible `session/meta` event as log line 0) was rejected: an in-log event would ride along with a seeded/forked session for free, but metadata is not replayable state, so the explicit out-of-log header seam is the cleaner cost. - **Resume is an async factory, not a change to synchronous create.** `ctx.agents.resume({ resumeSessionId })` awaits `ctx.sessionPersistence.load`, recreates the live session with the loaded events (so `lastTurnNumber`/`deriveMessages` continue), and starts a fresh agent on the resumed id (NOT `${agentId}-session`). The agent-loop does NOT hard-inject `sessionPersistence` (that would pend non-persistent demos forever); `resume` rejects with a typed error when it is absent. diff --git a/docs/architecture.md b/docs/architecture.md index 85398a9f7c..829283b729 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -95,7 +95,7 @@ A `Session` is an append-only log of typed `SessionEvent`s — the single source Replay/fork = `ctx.sessions.create(id, { seed: seedEvents })`. Trace/telemetry = listen to `session/event`. -**Durability seam**: `session/event` is a synchronous notification; persistence plugins buffer (write-behind) and drain at the awaited `session/flush` checkpoint the loop fires at every turn end. The durable backend is a real **capability seam**: the abstract `SessionPersistence` service (`dsh-session-persistence`, `ctx.sessionPersistence`) defines create/append/load/list/update over the existing `SessionEvent` (no parallel persisted type), and `dsh-session-persistence-jsonl` is the first implementation — an append-only JSONL log per session with crash-safe atomic writes, truncation-repair of a never-committed crash tail, and a read/replay path. Session metadata (format version, cwd, lineage) travels separately as `SessionMeta`, attached to a `Session` via `session.header`. Resuming a persisted session into a live agent is `ctx.agents.resume({ resumeSessionId })`. A SQLite/WAL backend is a future drop-in `SessionPersistence` subclass (the row shape `(session_id, seq, type, time, data)` maps 1:1 onto `SessionEvent`). +**Durability seam**: `session/event` is a synchronous notification; persistence plugins buffer (write-behind) and drain at the awaited `session/flush` checkpoint the loop fires at every turn end. The durable backend is a real **capability seam**: the abstract `SessionPersistence` service (`dsh-session-persistence`, `ctx.sessionPersistence`) defines create/append/load/list/update over the existing `SessionEvent` (no parallel persisted type), and `dsh-session-persistence-jsonl` is the first implementation — an append-only JSONL log per session with crash-safe atomic writes, truncation-repair of a never-committed crash tail, and a read/replay path. Session metadata (format version, cwd, lineage) travels separately as `SessionMeta`, attached to a `Session` via `session.header`. Resuming a persisted session into a live agent is `ctx.agents.resume({ resumeSessionId })`. A second backend, `dsh-session-persistence-sqlite` (`node:sqlite`, one row per `SessionEvent` — the row shape `(session_id, seq, type, time, data)` maps 1:1 onto it), passes the same `runPersistenceContract` suite, proving the seam is genuinely backend-agnostic. ## Prompt assembly (dsh-system-prompt) @@ -241,7 +241,6 @@ Code skeletons for the three plugin shapes (tool, hook/permission-gate, UI) and Tracked here deliberately — each is designed-for but not implemented: - **Sub-agent spawn/fork semantics** (seam: `AgentLoop.create()`); inter-agent channels beyond `send`/`steer`/events. -- **SQLite/WAL persistence backend** — a drop-in `SessionPersistence` subclass (the abstract seam + the JSONL backend landed; see the durability-seam paragraph). - **Compaction implementation** (auto thresholds, summarization prompts) on the `agent/request` seam, with its session-event types added by declaration merging. - **Parallel tool execution** (concurrency-safety hints on ToolDefinition). - **Session branching/tree** (pi-style entry tree) if needed beyond seed-based forking. diff --git a/packages/session-persistence-sqlite/README.md b/packages/session-persistence-sqlite/README.md new file mode 100644 index 0000000000..5af385e63e --- /dev/null +++ b/packages/session-persistence-sqlite/README.md @@ -0,0 +1,27 @@ +# @deepseek-ai/dsh-session-persistence-sqlite + +A SQLite durable session-persistence backend — a second `SessionPersistence` implementation ([ADR 0016](../../docs/adr/0016-session-persistence.md)), built to validate that the abstract seam and the shared `runPersistenceContract` suite are genuinely backend-agnostic. It satisfies the SAME contract as `dsh-session-persistence-jsonl` (append-only, contiguous-seq, lazy materialization, crash-tail-on-load), expressed over `node:sqlite` rows instead of file bytes. + +## Storage model + +Each `SessionEvent` maps 1:1 onto a row in an `events` table `(session_id, seq, type, time, data)` — `data` is the event payload as JSON text, so the row shape is the event verbatim (including `assistant/chunk`, keeping `seq` contiguous). Out-of-log metadata (`SessionMeta`) lives in a `sessions` row, including the mutable `SessionSummary` fields (`updatedAt`, `title`, `firstPrompt`) that `update()` rewrites without touching the event log. + +`node:sqlite` requires Node ≥ 22.5 (this repo runs Node ≥ 24); the database opens with `foreign_keys = ON` (so `ON DELETE CASCADE` drops a session's events with its row) and `journal_mode = WAL`. + +## Contract semantics over rows + +- **Append = a transaction.** `append` runs `BEGIN`/`COMMIT` around the batch: it materializes the `sessions` row (if still lazy) and INSERTs every event, asserting the contiguous-seq contract first (the first event's `seq` must equal the stored next-seq). A mid-batch failure (a UNIQUE violation on a duplicated seq) rolls back entirely, so the stored log and the in-memory cursor stay consistent. +- **Lazy materialization.** `create()` records intent in memory only — no row is written until the first `append`. A created-but-never-appended session is absent from `has()`/`list()` (a `materialized` flag on the row, set inside the first append transaction; `has`/`list` filter to materialized rows). +- **Crash-tail-on-load.** `load()` reads every stored event ordered by `seq` and returns only the prefix through the **last complete `turn/end`** (the `SessionPersistence.load` contract). A batch that landed without its closing `turn/end` (a process killed mid-turn) is an uncommitted tail and is deleted on load; a `seq` gap inside the committed region makes the session unloadable. + +## Configuration (schemastery) + +```ts +interface Config { + path: string // SQLite database file path, or ':memory:' for an in-process DB +} +``` + +## Write path + +Like the JSONL backend, the plugin also installs the `session/event` → buffer → `session/flush` drain: it snapshots each event when buffered (the live `session.events` object is mutable), persists a fork's seed once on `session/created`, keeps a per-session write cursor so a resumed session never re-appends stored events, and seeds existing live sessions on apply (HMR does not replay `session/created`). Dispose awaits every in-flight init + final drain and then closes the database, so no write lands after teardown. diff --git a/packages/session-persistence-sqlite/package.json b/packages/session-persistence-sqlite/package.json new file mode 100644 index 0000000000..0cbefc4f00 --- /dev/null +++ b/packages/session-persistence-sqlite/package.json @@ -0,0 +1,35 @@ +{ + "name": "@deepseek-ai/dsh-session-persistence-sqlite", + "description": "SQLite durable session persistence backend for the DeepSeek Harness", + "version": "0.0.1", + "private": true, + "type": "module", + "main": "lib/index.js", + "types": "lib/index.d.ts", + "exports": { + ".": { + "types": "./lib/index.d.ts", + "default": "./lib/index.js" + }, + "./src/*": "./src/*", + "./package.json": "./package.json" + }, + "files": [ + "lib", + "src" + ], + "license": "BSD-3-Clause", + "peerDependencies": { + "@deepseek-ai/dsh-session": "^0.0.1", + "@deepseek-ai/dsh-session-persistence": "^0.0.1", + "cordis": "^4.0.0-rc.6" + }, + "dependencies": { + "schemastery": "^3.18.0" + }, + "devDependencies": { + "@deepseek-ai/dsh-session": "^0.0.1", + "@deepseek-ai/dsh-session-persistence": "^0.0.1", + "cordis": "^4.0.0-rc.6" + } +} diff --git a/packages/session-persistence-sqlite/src/index.ts b/packages/session-persistence-sqlite/src/index.ts new file mode 100644 index 0000000000..f279e4684e --- /dev/null +++ b/packages/session-persistence-sqlite/src/index.ts @@ -0,0 +1,482 @@ +/** + * SQLite durable session-persistence backend (`@deepseek-ai/dsh-session-persistence-sqlite`). + * + * A SECOND {@link SessionPersistence} implementation, built to validate that + * the abstract seam + the shared `runPersistenceContract` suite are genuinely + * backend-agnostic: the same append-only / contiguous-seq / lazy-materialization + * / crash-tail-on-load semantics the JSONL backend expresses over file bytes, + * expressed here over `node:sqlite` rows. Each `SessionEvent` maps 1:1 onto a + * row `(session_id, seq, type, time, data)`; `append` is an INSERT inside a + * transaction that asserts the contiguous-seq contract; the mutable + * `SessionSummary` lives in the `sessions` metadata row. + * + * Like the JSONL backend it is also the write-path plugin: it installs the + * `session/event` → buffer → `session/flush` drain, persists a fork's seed once + * on `session/created`, keeps a per-session write cursor so a resumed session + * never re-appends stored events, and seeds existing live sessions on apply + * (HMR does not replay `session/created`). + * + * @module @deepseek-ai/dsh-session-persistence-sqlite + */ + +import { Context } from 'cordis' +import z from 'schemastery' +import { DatabaseSync } from 'node:sqlite' +import { mkdir } from 'node:fs/promises' +import { dirname, resolve } from 'node:path' +import { SessionPersistence } from '@deepseek-ai/dsh-session-persistence' +import { isJsonValue } from '@deepseek-ai/dsh-session' +import type { Session, SessionEvent, SessionId, SessionMeta, SessionSummary } from '@deepseek-ai/dsh-session' +import { + cutAtLastTurnEnd, openDatabase, rowToEvent, rowToMeta, type EventRow, type SessionRow, +} from './schema.ts' + +export { SCHEMA_VERSION } from './schema.ts' + +/** Plugin configuration. */ +export interface Config { + /** + * Filesystem path to the SQLite database file. The special value `:memory:` + * opens an in-process database (tests); a file path is created (with parent + * dirs) on construction. + */ + path: string +} + +/** Backend bookkeeping for a session id (NOT the live Session object). */ +interface SessionState { + meta: SessionMeta + /** Next seq to write — equals the number of committed events. */ + cursor: number + /** Whether the session has at least one persisted event (materialized). */ + materialized: boolean + /** The live Session that owns this state (collision detection); see onCreated. */ + owner?: Session +} + +/** + * Whether a live session's `seed` reproduces a persisted `prefix` exactly (the + * prefix is no longer than the seed and each event DEEP-equals the seed event + * at the same index). Distinguishes a session legitimately continuing a + * persisted log (HMR re-seeing its own session, or a resume) from a different + * session that merely reuses the id. Mirrors the JSONL backend's check; both + * sides are JSON-serializable by contract, so `JSON.stringify` is a sound + * canonical form. + */ +function seedCoversPrefix(seed: readonly SessionEvent[], prefix: readonly SessionEvent[]): boolean { + return prefix.length <= seed.length + && prefix.every((e, i) => { + const s = seed[i] + return s !== undefined && JSON.stringify(s) === JSON.stringify(e) + }) +} + +/** Reject non-JSON-serializable `event.data`, naming the offending type. */ +function assertSerializable(events: readonly SessionEvent[]): void { + for (const event of events) { + if (!isJsonValue(event.data)) { + throw new Error(`event "${event.type}" carries non-JSON-serializable data (seq ${event.seq})`) + } + } +} + +/** + * The SQLite persistence backend. Load as a plugin; it registers as + * `ctx.sessionPersistence` and installs the write-path listeners. + */ +export class SessionPersistenceSqlite extends SessionPersistence { + static inject = ['sessions'] + + static Config: z = z.object({ + path: z.string().required(), + }) + + private db!: DatabaseSync + private ready: Promise + /** Backend bookkeeping keyed by session id (NOT the live Session object). */ + private states = new Map() + /** Write-behind buffers keyed by the live Session (write path). */ + private buffers = new Map() + /** Per-session serialization chain (keyed by session id). */ + private chains = new Map>() + /** Per-session init promise (onCreated), keyed by the LIVE Session object. */ + private inits = new Map>() + + constructor(ctx: Context, public config: Config) { + super(ctx) + // Open the database asynchronously (the parent directory may need creating); + // every backend op awaits `ready` first. Opening synchronously in the ctor + // would force a sync mkdir and block plugin apply. + this.ready = this.openDb(config.path) + this.installWritePath() + } + + private async openDb(path: string): Promise { + if (path !== ':memory:') { + const abs = resolve(path) + await mkdir(dirname(abs), { recursive: true, mode: 0o700 }) + this.db = openDatabase(abs) + } else { + this.db = openDatabase(path) + } + } + + // --- SessionPersistence backend surface (all serialized per session id) --- + + create(meta: SessionMeta): Promise { + const snapshot: SessionMeta = { ...meta } + return this.serialize(snapshot.id, () => this.createCore(snapshot)) + } + + private async createCore(meta: SessionMeta): Promise { + await this.ready + if (this.states.has(meta.id)) { + throw new Error(`session "${meta.id}" already exists in this backend`) + } + if (this.rowFor(meta.id) !== undefined) { + throw new Error(`session "${meta.id}" already has a persisted row; load/resume it instead of creating`) + } + // Lazy: record intent in memory only. No row until the first append, so an + // abandoned (never-appended) session leaves nothing behind and stays absent + // from has()/list(). + this.states.set(meta.id, { meta, cursor: 0, materialized: false }) + } + + append(id: SessionId, events: readonly SessionEvent[]): Promise { + return this.serialize(id, () => this.appendCore(id, events)) + } + + private async appendCore(id: SessionId, events: readonly SessionEvent[]): Promise { + await this.ready + if (events.length === 0) return + // Validate serializability up front so a bad event surfaces the typed error + // (rather than failing later inside the INSERT loop, mid-transaction). + assertSerializable(events) + let state = this.states.get(id) + if (state === undefined) state = await this.adopt(id) + + // Contiguity contract: each event's seq must continue the stored log. + for (const [i, event] of events.entries()) { + if (event.seq !== state.cursor + i) { + throw new Error(`append seq mismatch for "${id}": expected ${state.cursor + i} at index ${i}, got ${event.seq}`) + } + } + + // The transaction is the durability + atomicity boundary: materialize the + // sessions row (if lazy) and INSERT every event, or roll back entirely. A + // BEGIN/COMMIT around the batch means a mid-batch failure (a UNIQUE + // violation on a duplicated seq from a concurrent writer) leaves the stored + // log untouched, so the cursor stays truthful and a retry is clean. + const insertEvent = this.db.prepare( + 'INSERT INTO events (session_id, seq, type, time, data) VALUES (?, ?, ?, ?, ?)', + ) + this.db.exec('BEGIN') + try { + if (!state.materialized) this.writeRow(state.meta) + for (const event of events) { + insertEvent.run(id, event.seq, event.type, event.time, JSON.stringify(event.data)) + } + // Bump updatedAt on every append (the mutable summary lives in the row). + const updatedAt = Date.now() + this.db.prepare('UPDATE sessions SET updated_at = ? WHERE id = ?').run(updatedAt, id) + this.db.exec('COMMIT') + state.meta = { ...state.meta, updatedAt } + } catch (error) { + this.db.exec('ROLLBACK') + throw error + } + state.materialized = true + state.cursor += events.length + } + + load(id: SessionId): Promise<{ meta: SessionMeta; events: SessionEvent[] }> { + return this.serialize(id, () => this.loadCore(id)) + } + + private async loadCore(id: SessionId): Promise<{ meta: SessionMeta; events: SessionEvent[] }> { + await this.ready + const row = this.rowFor(id) + if (row === undefined) throw new Error(`session "${id}" not found`) + const meta = rowToMeta(row) + this.assertVersion(meta) + + // Read every stored event ordered by seq, then cut at the last complete + // turn/end — the same crash-tail semantics as the JSONL backend. A row that + // landed without its closing turn/end (process killed mid-turn) is an + // uncommitted tail and is excluded; a seq gap inside the committed region + // makes the session unloadable (cutAtLastTurnEnd throws). + const eventRows = this.db + .prepare('SELECT seq, type, time, data FROM events WHERE session_id = ? ORDER BY seq') + .all(id) as unknown as EventRow[] + const all = eventRows.map(rowToEvent) + const { committed, cutTail } = cutAtLastTurnEnd(all) + + // Physically discard the crash tail so the stored log matches what load + // returned (the next append continues at the committed length). Mirrors the + // JSONL truncation-repair, but done eagerly here (a DELETE is transactional; + // there is no half-written-line hazard to defer past). + if (cutTail) { + this.db.prepare('DELETE FROM events WHERE session_id = ? AND seq >= ?').run(id, committed.length) + } + + // Record state so a later append continues at the committed length. The + // state keeps its OWN copy of the meta; the returned value is separate so a + // consumer mutating loaded.meta cannot corrupt the backend's row metadata. + this.states.set(id, { meta: { ...meta }, cursor: committed.length, materialized: committed.length > 0 }) + return { meta, events: committed } + } + + async list(): Promise { + await this.ready + // Materialized rows only: a created-but-never-appended (lazy) session has no + // row at all, and a load that cut every event back to zero leaves + // materialized = 0. Both are excluded, matching has(). + const rows = this.db + .prepare('SELECT * FROM sessions WHERE materialized = 1') + .all() as unknown as SessionRow[] + return rows.map(rowToMeta) + } + + async has(id: SessionId): Promise { + await this.ready + const state = this.states.get(id) + if (state?.materialized) return true + const row = this.rowFor(id) + return row !== undefined && row.materialized === 1 + } + + delete(id: SessionId): Promise { + return this.serialize(id, () => this.deleteCore(id)) + } + + private async deleteCore(id: SessionId): Promise { + await this.ready + // ON DELETE CASCADE drops the session's events with its row. + this.db.prepare('DELETE FROM sessions WHERE id = ?').run(id) + this.states.delete(id) + } + + update(id: SessionId, summary: Partial): Promise { + return this.serialize(id, () => this.updateCore(id, summary)) + } + + private async updateCore(id: SessionId, summary: Partial): Promise { + await this.ready + let state = this.states.get(id) + if (state === undefined) state = await this.adopt(id) + const nextMeta: SessionMeta = { ...state.meta, ...summary } + // update's only durable effect is the summary fields; the event log is + // untouched. If the row is not materialized yet (a lazy session updated + // before its first append) there is nothing to write — keep the pending + // summary in memory so the materializing append carries it. + if (state.materialized) this.writeRow(nextMeta) + state.meta = nextMeta + } + + // --- row helpers --- + + /** Fetch a session's row, or undefined if absent. */ + private rowFor(id: SessionId): SessionRow | undefined { + const row = this.db.prepare('SELECT * FROM sessions WHERE id = ?').get(id) as unknown as SessionRow | undefined + return row + } + + /** + * Insert-or-replace a session's metadata row, marked materialized. The only + * callers are the first materializing `append` and a post-materialization + * `update` — a row is written only once a session has durable events, so + * `materialized` is always 1 (a never-appended session has no row at all). + */ + private writeRow(meta: SessionMeta): void { + this.db.prepare(` + INSERT INTO sessions (id, version, created_at, cwd, parent_session, updated_at, title, first_prompt, materialized) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, 1) + ON CONFLICT(id) DO UPDATE SET + version = excluded.version, + created_at = excluded.created_at, + cwd = excluded.cwd, + parent_session = excluded.parent_session, + updated_at = excluded.updated_at, + title = excluded.title, + first_prompt = excluded.first_prompt, + materialized = excluded.materialized + `).run( + meta.id, + meta.version, + meta.createdAt, + meta.cwd ?? null, + meta.parentSession ?? null, + meta.updatedAt, + meta.title ?? null, + meta.firstPrompt ?? null, + ) + } + + /** Build a state for a session present in the DB but not yet in memory. */ + private async adopt(id: SessionId): Promise { + await this.loadCore(id) // sets the state; load (serialized) would deadlock + const state = this.states.get(id) + /* v8 ignore next -- loadCore always sets the state for the id */ + if (!state) throw new Error(`failed to adopt session "${id}"`) + return state + } + + private assertVersion(meta: SessionMeta): void { + if (meta.version !== 1) { + throw new Error(`unsupported session format version ${meta.version} for "${meta.id}" (only v1 is supported)`) + } + } + + /** + * Run `op` after any in-flight operation for the same session id, so writes + * for one session never interleave. Errors do not poison the chain. NOTE: + * serialized public methods must NOT call each other (deadlock); they call + * the unserialized `*Core` helpers instead. + */ + private serialize(id: SessionId, op: () => Promise): Promise { + const prior = this.chains.get(id) ?? Promise.resolve() + const next = prior.then(op, op) + this.chains.set(id, next.then(() => undefined, () => undefined)) + return next + } + + // --- write path (session/event → flush drain) --- + + private installWritePath(): void { + const ctx = this.ctx + + ctx.on('session/created', (session) => { void this.initFor(session) }) + + // Snapshot + buffer every event (the live object is mutable; clone so a + // later in-place mutation cannot rewrite a buffered event). Serializability + // is guaranteed at the source (Session.append), so structuredClone is safe. + ctx.on('session/event', (session, event) => { + let buffer = this.buffers.get(session) + if (!buffer) this.buffers.set(session, buffer = []) + buffer.push(structuredClone(event)) + }) + + ctx.on('session/flush', session => this.flush(session)) + + // Dispose must reach quiescence: await every init + final drain, then close + // the database, BEFORE returning, so no write lands after teardown. + ctx.effect(() => async () => { + await Promise.allSettled([...this.inits.values()]) + await Promise.allSettled([...this.buffers.keys()].map(s => this.flush(s))) + await Promise.allSettled([...this.chains.values()]) + await this.ready + this.db.close() + }, 'session-persistence-sqlite write path') + + // HMR: a hot reload does not replay session/created, so seed existing live + // sessions (mirrors dsh-invariants and the JSONL backend). + for (const session of ctx.sessions.list()) void this.initFor(session) + } + + /** Start (once) the async init for a session and remember its promise. */ + private initFor(session: Session): Promise { + const existing = this.inits.get(session) + if (existing) return existing + const seed = session.events.map(e => structuredClone(e)) + const p = this.onCreated(session, seed) + p.catch(() => { /* observed by flush/dispose via the stored promise */ }) + this.inits.set(session, p) + return p + } + + /** + * On session/created: sync the backend's state to a live Session. Cases + * mirror the JSONL backend: + * 1. Already tracked → no-op (or claim ownerless state if the seed matches). + * 2. A row EXISTS and is a seq-aligned PREFIX of the live events → adopt + * (HMR/resume), persisting any live suffix beyond the stored prefix. + * 3. A row EXISTS but is NOT a prefix → reject (id collision). + * 4. No row → a genuinely new session: register meta (lazy) + persist seed. + */ + private async onCreated(session: Session, seed: readonly SessionEvent[]): Promise { + await this.ready + const id = session.header.id + const tracked = this.states.get(id) + if (tracked !== undefined) { + /* v8 ignore next -- initFor dedupes per session object; same-object re-entry can't occur */ + if (tracked.owner === session) return + if (tracked.owner === undefined) { + // Ownerless state from a public create()/load(). The first live session + // claims it ONLY if its seed reproduces the persisted prefix. + if (!await this.seedMatchesPersisted(id, seed, tracked.cursor)) { + throw new Error(`session "${id}" is already persisted with ${tracked.cursor} event(s) that do not match this live session (id collision)`) + } + tracked.owner = session + const suffix = seed.slice(tracked.cursor) + if (suffix.length > 0) await this.append(id, suffix) + return + } + // Owned by a DIFFERENT live session. Reclaim ONLY a truly-abandoned id + // (never materialized, no pending buffer); else it is a real collision. + const ownerBuffer = this.buffers.get(tracked.owner) + if (!tracked.materialized && !ownerBuffer?.length) { + this.states.delete(id) + } else { + throw new Error(`session "${id}" is already bound to a different live session in this backend (id collision)`) + } + } + + const row = this.rowFor(id) + if (row !== undefined && row.materialized === 1) { + const stored = this.eventsFor(id) + if (!seedCoversPrefix(seed, stored)) { + throw new Error(`session "${id}" already has a persisted log that does not match this live session (id collision)`) + } + await this.serialize(id, () => this.loadCore(id)) + const adopted = this.states.get(id) + /* v8 ignore next -- loadCore always sets the state for the id */ + if (adopted !== undefined) adopted.owner = session + const suffix = seed.slice(stored.length) + if (suffix.length > 0) await this.append(id, suffix) + return + } + + // case 4: a genuinely new session. + const meta: SessionMeta = { ...session.header, updatedAt: Date.now() } + await this.create(meta) + const created = this.states.get(id) + /* v8 ignore next -- create() always sets the state for the id */ + if (created !== undefined) created.owner = session + if (seed.length > 0) await this.append(id, seed) + } + + /** The committed events for a session id (last-turn/end cut applied). */ + private eventsFor(id: SessionId): SessionEvent[] { + const rows = this.db + .prepare('SELECT seq, type, time, data FROM events WHERE session_id = ? ORDER BY seq') + .all(id) as unknown as EventRow[] + return cutAtLastTurnEnd(rows.map(rowToEvent)).committed + } + + /** Whether a live session's seed reproduces the first `cursor` stored events. */ + private async seedMatchesPersisted(id: SessionId, seed: readonly SessionEvent[], cursor: number): Promise { + await this.ready + if (cursor === 0) return true + return seedCoversPrefix(seed, this.eventsFor(id).slice(0, cursor)) + } + + private async flush(session: Session): Promise { + await this.inits.get(session) + await this.serialize(session.header.id, () => this.drain(session)) + } + + /** Drain a session's write buffer to the database. Caller serializes per id. */ + private async drain(session: Session): Promise { + const buffer = this.buffers.get(session) + if (!buffer?.length) return + const batch = buffer.slice() + const state = this.states.get(session.header.id) + /* v8 ignore next -- state is always set by the awaited init before flush */ + const cursor = state?.cursor ?? 0 + const fresh = batch.filter(e => e.seq >= cursor) + if (fresh.length > 0) await this.appendCore(session.header.id, fresh) + buffer.splice(0, batch.length) + } +} + +export default SessionPersistenceSqlite diff --git a/packages/session-persistence-sqlite/src/schema.ts b/packages/session-persistence-sqlite/src/schema.ts new file mode 100644 index 0000000000..5f20b5c08b --- /dev/null +++ b/packages/session-persistence-sqlite/src/schema.ts @@ -0,0 +1,135 @@ +/** + * Schema + load-time helpers for the SQLite session-persistence backend: the + * DDL (a `sessions` metadata table and a 1:1 `events` row per `SessionEvent`), + * the database open/configure step, and the last-`turn/end` cut that gives the + * SQLite backend the SAME crash-tail-on-load semantics as the JSONL backend. + * + * @module dsh-session-persistence-sqlite/schema + */ + +import { DatabaseSync } from 'node:sqlite' +import type { SessionEvent, SessionId, SessionMeta } from '@deepseek-ai/dsh-session' + +/** + * The on-disk schema version. Bumped only on a breaking change to the table + * layout; orthogonal to a session's own `version` (which versions the EVENT + * vocabulary, stored per session in the `sessions` row). + */ +export const SCHEMA_VERSION = 1 + +/** + * A row of the `sessions` table — the out-of-log metadata (`SessionMeta`) plus + * the `materialized` flag that implements lazy materialization (a created-but- + * never-appended session has `materialized = 0` and is excluded from + * `has`/`list`, mirroring the JSONL backend's "no file until first append"). + */ +export interface SessionRow { + id: string + version: number + created_at: number + cwd: string | null + parent_session: string | null + updated_at: number + title: string | null + first_prompt: string | null + materialized: number +} + +/** An `events` table row: one `SessionEvent` mapped 1:1 (`data` is JSON text). */ +export interface EventRow { + seq: number + type: string + time: number + data: string +} + +/** + * Open the database at `path` and apply the schema + pragmas. `foreign_keys` + * makes `ON DELETE CASCADE` drop a session's events with its row; `journal_mode + * = WAL` matches the durability model the ADR records (the row shape maps 1:1 + * onto `SessionEvent`; opencode runs this exact shape on SQLite/WAL). + */ +export function openDatabase(path: string): DatabaseSync { + const db = new DatabaseSync(path) + db.exec('PRAGMA foreign_keys = ON') + db.exec('PRAGMA journal_mode = WAL') + db.exec(` + CREATE TABLE IF NOT EXISTS sessions ( + id TEXT PRIMARY KEY, + version INTEGER NOT NULL, + created_at INTEGER NOT NULL, + cwd TEXT, + parent_session TEXT, + updated_at INTEGER NOT NULL, + title TEXT, + first_prompt TEXT, + materialized INTEGER NOT NULL DEFAULT 0 + ) STRICT + `) + db.exec(` + CREATE TABLE IF NOT EXISTS events ( + session_id TEXT NOT NULL REFERENCES sessions(id) ON DELETE CASCADE, + seq INTEGER NOT NULL, + type TEXT NOT NULL, + time INTEGER NOT NULL, + data TEXT NOT NULL, + PRIMARY KEY (session_id, seq) + ) STRICT + `) + return db +} + +/** Reconstruct the full {@link SessionMeta} from a `sessions` row. */ +export function rowToMeta(row: SessionRow): SessionMeta { + return { + version: row.version, + id: row.id as SessionId, + createdAt: row.created_at, + updatedAt: row.updated_at, + ...row.cwd !== null ? { cwd: row.cwd } : {}, + ...row.parent_session !== null ? { parentSession: row.parent_session as SessionId } : {}, + ...row.title !== null ? { title: row.title } : {}, + ...row.first_prompt !== null ? { firstPrompt: row.first_prompt } : {}, + } +} + +/** Reconstruct a {@link SessionEvent} from an `events` row (parses `data`). */ +export function rowToEvent(row: EventRow): SessionEvent { + return { + type: row.type, + seq: row.seq, + time: row.time, + data: JSON.parse(row.data) as SessionEvent['data'], + } as SessionEvent +} + +/** + * The committed prefix of an ordered event list: everything up to and including + * the LAST `turn/end`, plus whether a crash tail (events after it) was cut. + * + * The loop only flushes at `turn/end`, so the last `turn/end` is the last + * durable boundary; anything after it is a never-committed crash tail (a batch + * that landed without its closing `turn/end`, e.g. a process killed mid-turn). + * This is the SQLite analogue of the JSONL backend's `scanLog` truncation point + * — the SAME contract (`SessionPersistence.load`), expressed over rows rather + * than file bytes. The committed region MUST be contiguous (`events[i].seq === + * i`); a gap there means committed data was lost and the session is unloadable. + */ +export function cutAtLastTurnEnd(events: readonly SessionEvent[]): { committed: SessionEvent[]; cutTail: boolean } { + let lastTurnEnd = -1 + events.forEach((event, i) => { + if (event.type === 'turn/end') lastTurnEnd = i + }) + // No committed turn/end anywhere: the whole list is an uncommitted first-turn + // tail. Nothing is committed (mirrors scanLog returning zero events). + if (lastTurnEnd < 0) { + return { committed: [], cutTail: events.length > 0 } + } + const committed = events.slice(0, lastTurnEnd + 1) + committed.forEach((event, i) => { + if (event.seq !== i) { + throw new Error(`corrupt session log: seq gap in committed region at index ${i} (expected ${i}, got ${event.seq})`) + } + }) + return { committed, cutTail: lastTurnEnd < events.length - 1 } +} diff --git a/packages/session-persistence-sqlite/tests/sqlite.spec.ts b/packages/session-persistence-sqlite/tests/sqlite.spec.ts new file mode 100644 index 0000000000..854013e03a --- /dev/null +++ b/packages/session-persistence-sqlite/tests/sqlite.spec.ts @@ -0,0 +1,595 @@ +import { afterEach, describe, expect, it } from 'vitest' +import { Context } from 'cordis' +import { mkdtemp, rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import SessionStore, { SessionId } from '@deepseek-ai/dsh-session' +import type { Session, SessionEvent, SessionMeta } from '@deepseek-ai/dsh-session' +import SessionPersistenceSqlite, { SCHEMA_VERSION } from '@deepseek-ai/dsh-session-persistence-sqlite' +import { cutAtLastTurnEnd, openDatabase } from '../src/schema.ts' +import { runPersistenceContract, meta, oneTurnLog } from '../../session-persistence/tests/contract.ts' + +const dirs: string[] = [] +afterEach(async () => { for (const d of dirs.splice(0)) await rm(d, { recursive: true, force: true }) }) + +async function freshDbPath(): Promise { + const dir = await mkdtemp(join(tmpdir(), 'dsh-sqlite-')) + dirs.push(dir) + return join(dir, 'sessions.db') +} + +// The payoff: the SAME backend-agnostic contract the JSONL backend runs, now +// proving the SQLite backend satisfies identical semantics. +runPersistenceContract('sqlite', async () => { + const ctx = new Context() + await ctx.plugin(SessionStore) + const fiber = await ctx.plugin(SessionPersistenceSqlite, { path: ':memory:' }) + return { + persistence: ctx.sessionPersistence, + dispose: async () => { await fiber.dispose() }, + } +}) + +describe('cutAtLastTurnEnd', () => { + it('returns the prefix through the last complete turn/end and flags a cut tail', () => { + const log = oneTurnLog() + const withTail: SessionEvent[] = [ + ...log, + { type: 'turn/start', seq: 6, time: 7, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } }, + { type: 'user/message', seq: 7, time: 8, data: { content: [{ type: 'text', text: 'q2' }], source: { kind: 'user' } } }, + ] + const { committed, cutTail } = cutAtLastTurnEnd(withTail) + expect(committed).toEqual(log) + expect(cutTail).toBe(true) + }) + + it('treats a log with no turn/end as fully uncommitted', () => { + const partial: SessionEvent[] = [ + { type: 'turn/start', seq: 0, time: 1, data: { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } } }, + { type: 'user/message', seq: 1, time: 2, data: { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } } }, + ] + expect(cutAtLastTurnEnd(partial)).toEqual({ committed: [], cutTail: true }) + }) + + it('reports no cut when the log ends exactly on a turn/end', () => { + const { committed, cutTail } = cutAtLastTurnEnd(oneTurnLog()) + expect(committed).toEqual(oneTurnLog()) + expect(cutTail).toBe(false) + }) + + it('an empty log is committed-empty with no tail', () => { + expect(cutAtLastTurnEnd([])).toEqual({ committed: [], cutTail: false }) + }) + + it('throws on a seq gap inside the committed region', () => { + const gapped: SessionEvent[] = [ + { type: 'turn/start', seq: 0, time: 1, data: { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } } }, + { type: 'step/start', seq: 2, time: 2, data: { turn: 1, step: 1 } }, // seq 1 missing + { type: 'turn/end', seq: 3, time: 3, data: { turn: 1, reason: { kind: 'completed' } } }, + ] + expect(() => cutAtLastTurnEnd(gapped)).toThrow(/seq gap in committed region/) + }) +}) + +describe('SessionPersistenceSqlite: durability and crash semantics', () => { + it('a crash tail (rows after the last turn/end) is excluded and deleted on load', async () => { + const path = await freshDbPath() + const m = meta('crash') + // Run 1: persist a complete turn, then a half-written second turn (no turn/end). + const ctx1 = new Context() + await ctx1.plugin(SessionStore) + const fiber1 = await ctx1.plugin(SessionPersistenceSqlite, { path }) + await ctx1.sessionPersistence.create(m) + await ctx1.sessionPersistence.append(m.id, oneTurnLog()) + await ctx1.sessionPersistence.append(m.id, [ + { type: 'turn/start', seq: 6, time: 7, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } }, + { type: 'user/message', seq: 7, time: 8, data: { content: [{ type: 'text', text: 'q2' }], source: { kind: 'user' } } }, + ]) + await fiber1.dispose() + + // Run 2: load returns only the committed first turn; the tail is gone. + const ctx2 = new Context() + await ctx2.plugin(SessionStore) + const fiber2 = await ctx2.plugin(SessionPersistenceSqlite, { path }) + const loaded = await ctx2.sessionPersistence.load(m.id) + expect(loaded.events).toEqual(oneTurnLog()) + + // The next append continues at seq 6 (the committed length) and the cut + // tail was physically deleted, so there is no UNIQUE collision. + await ctx2.sessionPersistence.append(m.id, [ + { type: 'turn/start', seq: 6, time: 9, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } }, + { type: 'turn/end', seq: 7, time: 10, data: { turn: 2, reason: { kind: 'completed' } } }, + ]) + const reloaded = await ctx2.sessionPersistence.load(m.id) + expect(reloaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7]) + await fiber2.dispose() + }) + + it('append rolls back the whole batch on a mid-batch seq collision (transaction)', async () => { + const ctx = new Context() + await ctx.plugin(SessionStore) + const fiber = await ctx.plugin(SessionPersistenceSqlite, { path: ':memory:' }) + const m = meta('rollback') + await ctx.sessionPersistence.create(m) + await ctx.sessionPersistence.append(m.id, oneTurnLog()) // seqs 0..5 + + // A batch that re-states an already-stored seq must be rejected and leave + // the stored log unchanged (the UNIQUE (session_id, seq) constraint fires + // inside the transaction → ROLLBACK). + await expect(ctx.sessionPersistence.append(m.id, oneTurnLog())).rejects.toThrow() + const loaded = await ctx.sessionPersistence.load(m.id) + expect(loaded.events).toEqual(oneTurnLog()) // unchanged + await fiber.dispose() + }) + + it('persists across separate backend instances over the same file', async () => { + const path = await freshDbPath() + const m = meta('persist', '/proj') + const ctx1 = new Context() + await ctx1.plugin(SessionStore) + const fiber1 = await ctx1.plugin(SessionPersistenceSqlite, { path }) + await ctx1.sessionPersistence.create(m) + await ctx1.sessionPersistence.append(m.id, oneTurnLog()) + await ctx1.sessionPersistence.update(m.id, { title: 'T', firstPrompt: 'hi' }) + await fiber1.dispose() + + const ctx2 = new Context() + await ctx2.plugin(SessionStore) + const fiber2 = await ctx2.plugin(SessionPersistenceSqlite, { path }) + expect((await ctx2.sessionPersistence.list()).map(x => x.id)).toContain(m.id) + const loaded = await ctx2.sessionPersistence.load(m.id) + expect(loaded.meta).toMatchObject({ id: m.id, cwd: '/proj', title: 'T', firstPrompt: 'hi' }) + expect(loaded.events).toEqual(oneTurnLog()) + await fiber2.dispose() + }) + + it('rejects an unknown format version on load', async () => { + const path = await freshDbPath() + // Materialize a row with version 2 directly via the real schema. + const db = openDatabase(path) + db.prepare('INSERT INTO sessions (id, version, created_at, updated_at, materialized) VALUES (?, ?, ?, ?, 1)') + .run('v2', 2, 1, 1) + db.close() + + const ctx = new Context() + await ctx.plugin(SessionStore) + const fiber = await ctx.plugin(SessionPersistenceSqlite, { path }) + await expect(ctx.sessionPersistence.load(SessionId('v2'))).rejects.toThrow(/version 2/) + await fiber.dispose() + }) + + it('create rejects a duplicate id (in memory and on a persisted row)', async () => { + const path = await freshDbPath() + const m = meta('dup') + const ctx = new Context() + await ctx.plugin(SessionStore) + const fiber = await ctx.plugin(SessionPersistenceSqlite, { path }) + await ctx.sessionPersistence.create(m) + // Same in-memory state. + await expect(ctx.sessionPersistence.create(m)).rejects.toThrow(/already exists/) + await ctx.sessionPersistence.append(m.id, oneTurnLog()) + await fiber.dispose() + + // A fresh instance over the same file sees the persisted row. + const ctx2 = new Context() + await ctx2.plugin(SessionStore) + const fiber2 = await ctx2.plugin(SessionPersistenceSqlite, { path }) + await expect(ctx2.sessionPersistence.create(m)).rejects.toThrow(/already has a persisted row/) + await fiber2.dispose() + }) + + it('exposes the schema version constant', () => { + expect(SCHEMA_VERSION).toBe(1) + }) +}) + +describe('SessionPersistenceSqlite: write path (session/event → flush)', () => { + function send(session: Session, events: SessionEvent[]): void { + for (const e of events) session.append(e.type, e.data) + } + + it('persists a turn appended through the live session on flush', async () => { + const ctx = new Context() + await ctx.plugin(SessionStore) + const fiber = await ctx.plugin(SessionPersistenceSqlite, { path: ':memory:' }) + const session = ctx.sessions.create('w1') + send(session, oneTurnLog()) + await ctx.parallel('session/flush', session) + const loaded = await ctx.sessionPersistence.load(SessionId('w1')) + expect(loaded.events.map(e => e.type)).toEqual(oneTurnLog().map(e => e.type)) + await fiber.dispose() + }) + + it('a resumed session does not re-append its seed', async () => { + const path = await freshDbPath() + // Run 1: persist a full turn through the live session. + const ctx1 = new Context() + await ctx1.plugin(SessionStore) + const fiber1 = await ctx1.plugin(SessionPersistenceSqlite, { path }) + const s1 = ctx1.sessions.create('resume') + for (const e of oneTurnLog()) s1.append(e.type, e.data) + await ctx1.parallel('session/flush', s1) + await fiber1.dispose() + + // Run 2: reconstruct the live session from the loaded log (seed), then add a + // second turn. The seed must NOT be re-appended (no UNIQUE collision), and + // the second turn continues the seq. + const ctx2 = new Context() + await ctx2.plugin(SessionStore) + const fiber2 = await ctx2.plugin(SessionPersistenceSqlite, { path }) + const { events } = await ctx2.sessionPersistence.load(SessionId('resume')) + const s2 = ctx2.sessions.create('resume', { seed: events }) + s2.append('turn/start', { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } }) + s2.append('turn/end', { turn: 2, reason: { kind: 'completed' } }) + await ctx2.parallel('session/flush', s2) + const reloaded = await ctx2.sessionPersistence.load(SessionId('resume')) + expect(reloaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7]) + await fiber2.dispose() + }) + + it('HMR: applying the plugin seeds existing live sessions', async () => { + const ctx = new Context() + await ctx.plugin(SessionStore) + const session = ctx.sessions.create('hmr') + for (const e of oneTurnLog()) session.append(e.type, e.data) + // Plugin applied AFTER the session already has events. + const fiber = await ctx.plugin(SessionPersistenceSqlite, { path: ':memory:' }) + await ctx.parallel('session/flush', session) + expect(await ctx.sessionPersistence.has(SessionId('hmr'))).toBe(true) + await fiber.dispose() + }) + + it('dispose drains a pending buffer before closing the database', async () => { + const path = await freshDbPath() + const ctx = new Context() + await ctx.plugin(SessionStore) + const fiber = await ctx.plugin(SessionPersistenceSqlite, { path }) + const session = ctx.sessions.create('drain') + for (const e of oneTurnLog()) session.append(e.type, e.data) + // No explicit flush — dispose must drain the buffer. + await fiber.dispose() + + const ctx2 = new Context() + await ctx2.plugin(SessionStore) + const fiber2 = await ctx2.plugin(SessionPersistenceSqlite, { path }) + expect(await ctx2.sessionPersistence.has(SessionId('drain'))).toBe(true) + await fiber2.dispose() + }) + + it('rejects a different live session colliding on a persisted id', async () => { + const path = await freshDbPath() + const ctx1 = new Context() + await ctx1.plugin(SessionStore) + const fiber1 = await ctx1.plugin(SessionPersistenceSqlite, { path }) + const s1 = ctx1.sessions.create('collide') + for (const e of oneTurnLog()) s1.append(e.type, e.data) + await ctx1.parallel('session/flush', s1) + await fiber1.dispose() + + // A fresh, unrelated session reusing the id (no seed) must be rejected. + const ctx2 = new Context() + await ctx2.plugin(SessionStore) + const fiber2 = await ctx2.plugin(SessionPersistenceSqlite, { path }) + const s2 = ctx2.sessions.create('collide') + s2.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } }) + await expect(ctx2.parallel('session/flush', s2)).rejects.toThrow(/id collision/) + await fiber2.dispose() + }) + + it('update before the first append keeps the summary in memory and the session lazy', async () => { + const ctx = new Context() + await ctx.plugin(SessionStore) + const fiber = await ctx.plugin(SessionPersistenceSqlite, { path: ':memory:' }) + const m = meta('lazy-update') + await ctx.sessionPersistence.create(m) + await ctx.sessionPersistence.update(m.id, { title: 'pending' }) + // Still lazy: no materialized row yet. + expect(await ctx.sessionPersistence.has(m.id)).toBe(false) + // The first append materializes and carries the pending title. + await ctx.sessionPersistence.append(m.id, oneTurnLog()) + const loaded = await ctx.sessionPersistence.load(m.id) + expect(loaded.meta.title).toBe('pending') + await fiber.dispose() + }) +}) + +describe('SessionPersistenceSqlite: edge cases', () => { + async function backend(path = ':memory:'): Promise<{ ctx: Context; dispose: () => Promise }> { + const ctx = new Context() + await ctx.plugin(SessionStore) + const fiber = await ctx.plugin(SessionPersistenceSqlite, { path }) + return { ctx, dispose: () => fiber.dispose() } + } + + it('append of an empty batch is a no-op', async () => { + const { ctx, dispose } = await backend() + const m = meta('empty-batch') + await ctx.sessionPersistence.create(m) + await ctx.sessionPersistence.append(m.id, []) + expect(await ctx.sessionPersistence.has(m.id)).toBe(false) // still lazy + await dispose() + }) + + it('load rejects a missing session', async () => { + const { ctx, dispose } = await backend() + await expect(ctx.sessionPersistence.load(SessionId('nope'))).rejects.toThrow(/not found/) + await dispose() + }) + + it('delete of a non-existent session is a no-op', async () => { + const { ctx, dispose } = await backend() + await ctx.sessionPersistence.delete(SessionId('ghost')) + expect(await ctx.sessionPersistence.has(SessionId('ghost'))).toBe(false) + await dispose() + }) + + it('append adopts a session that exists only in the DB (fresh instance)', async () => { + const path = await freshDbPath() + const m = meta('adopt-append') + const b1 = await backend(path) + await b1.ctx.sessionPersistence.create(m) + await b1.ctx.sessionPersistence.append(m.id, oneTurnLog()) + await b1.dispose() + + // A fresh instance appends a second turn WITHOUT a prior create/load: append + // must adopt the on-disk row (cursor = stored length) and continue the seq. + const b2 = await backend(path) + await b2.ctx.sessionPersistence.append(m.id, [ + { type: 'turn/start', seq: 6, time: 7, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } }, + { type: 'turn/end', seq: 7, time: 8, data: { turn: 2, reason: { kind: 'completed' } } }, + ]) + const loaded = await b2.ctx.sessionPersistence.load(m.id) + expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7]) + await b2.dispose() + }) + + it('update adopts a session that exists only in the DB (fresh instance)', async () => { + const path = await freshDbPath() + const m = meta('adopt-update') + const b1 = await backend(path) + await b1.ctx.sessionPersistence.create(m) + await b1.ctx.sessionPersistence.append(m.id, oneTurnLog()) + await b1.dispose() + + const b2 = await backend(path) + await b2.ctx.sessionPersistence.update(m.id, { title: 'after restart' }) + const loaded = await b2.ctx.sessionPersistence.load(m.id) + expect(loaded.meta.title).toBe('after restart') + await b2.dispose() + }) + + it('append rolls back and rethrows when an event INSERT fails inside the transaction', async () => { + const path = await freshDbPath() + const m = meta('rollback-insert') + const b1 = await backend(path) + await b1.ctx.sessionPersistence.create(m) + await b1.ctx.sessionPersistence.append(m.id, oneTurnLog()) + + // A SECOND backend over the same file loads the session first, so it adopts + // cursor 6 (the committed length) into its OWN in-memory state. + const b2 = await backend(path) + await b2.ctx.sessionPersistence.load(m.id) // cursor 6 in b2 + const turn2: SessionEvent[] = [ + { type: 'turn/start', seq: 6, time: 7, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } }, + { type: 'turn/end', seq: 7, time: 8, data: { turn: 2, reason: { kind: 'completed' } } }, + ] + // b1 commits seq 6..7 first. + await b1.ctx.sessionPersistence.append(m.id, turn2) + // b2 still thinks its cursor is 6, so this batch passes the contiguity check + // but its INSERT of seq 6 hits the UNIQUE (session_id, seq) constraint + // mid-transaction → ROLLBACK + rethrow. + await expect(b2.ctx.sessionPersistence.append(m.id, turn2)).rejects.toThrow(/UNIQUE/) + // b1's turn is intact; b2's rolled-back attempt left nothing extra. + const loaded = await b1.ctx.sessionPersistence.load(m.id) + expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7]) + await b1.dispose() + await b2.dispose() + }) + + it('round-trips a header with parentSession (fork lineage)', async () => { + const { ctx, dispose } = await backend() + const m: SessionMeta = { ...meta('child'), parentSession: SessionId('parent') } + await ctx.sessionPersistence.create(m) + await ctx.sessionPersistence.append(m.id, oneTurnLog()) + const loaded = await ctx.sessionPersistence.load(m.id) + expect(loaded.meta.parentSession).toBe(SessionId('parent')) + await dispose() + }) + + it('a fresh live session reusing a previously-loaded id is rejected (ownerless guard)', async () => { + const path = await freshDbPath() + const m = meta('ownerless') + const b1 = await backend(path) + await b1.ctx.sessionPersistence.create(m) + await b1.ctx.sessionPersistence.append(m.id, oneTurnLog()) + await b1.dispose() + + const b2 = await backend(path) + // load() leaves ownerless state with cursor 6. + await b2.ctx.sessionPersistence.load(m.id) + // A fresh, unrelated live session reusing the id has a shorter/non-matching + // seed → its onCreated must reject rather than graft onto the loaded prefix. + const s = b2.ctx.sessions.create('ownerless') + s.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } }) + await expect(b2.ctx.parallel('session/flush', s)).rejects.toThrow(/id collision/) + await b2.dispose() + }) + + it('a live session whose seed matches the loaded prefix claims ownerless state and persists the suffix', async () => { + const path = await freshDbPath() + const m = meta('claim') + const b1 = await backend(path) + await b1.ctx.sessionPersistence.create(m) + await b1.ctx.sessionPersistence.append(m.id, oneTurnLog()) + await b1.dispose() + + const b2 = await backend(path) + const { events } = await b2.ctx.sessionPersistence.load(m.id) // ownerless, cursor 6 + // A live session seeded with the loaded log PLUS a new turn claims the state + // and persists only the suffix. + const s = b2.ctx.sessions.create('claim', { seed: [ + ...events, + { type: 'turn/start', seq: 6, time: 7, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } }, + { type: 'turn/end', seq: 7, time: 8, data: { turn: 2, reason: { kind: 'completed' } } }, + ] }) + await b2.ctx.parallel('session/flush', s) + const loaded = await b2.ctx.sessionPersistence.load(m.id) + expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7]) + await b2.dispose() + }) + + it('an abandoned lazy session (never materialized) releases its id for reuse', async () => { + const { ctx, dispose } = await backend() + const inits = (ctx.sessionPersistence as unknown as { inits: Map> }).inits + let first!: Session + const firstFiber = await ctx.plugin(Object.assign((inner: Context) => { + first = inner.sessions.create('reuse') + }, { inject: ['sessions'] })) + await inits.get(first) // let the lazy create register the state + await firstFiber.dispose() // disposed before any append → never materialized + + let reuse!: Session + await ctx.plugin(Object.assign((inner: Context) => { + reuse = inner.sessions.create('reuse') + }, { inject: ['sessions'] })) + await expect(inits.get(reuse)).resolves.toBeUndefined() + reuse.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } }) + reuse.append('turn/end', { turn: 1, reason: { kind: 'completed' } }) + await ctx.parallel('session/flush', reuse) + expect(await ctx.sessionPersistence.has(SessionId('reuse'))).toBe(true) + await dispose() + }) + + it('does NOT reclaim an id whose abandoned owner still has buffered (unflushed) events', async () => { + const { ctx, dispose } = await backend() + const inits = (ctx.sessionPersistence as unknown as { inits: Map> }).inits + let first!: Session + const firstFiber = await ctx.plugin(Object.assign((inner: Context) => { + first = inner.sessions.create('buffered') + }, { inject: ['sessions'] })) + await inits.get(first) + // Append a turn but do NOT flush — events sit in the write-behind buffer. + first.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } }) + first.append('turn/end', { turn: 1, reason: { kind: 'completed' } }) + await firstFiber.dispose() // disposed before flush; not materialized, buffer pending + + let reuse!: Session + await ctx.plugin(Object.assign((inner: Context) => { + reuse = inner.sessions.create('buffered') + }, { inject: ['sessions'] })) + await expect(inits.get(reuse)).rejects.toThrow(/already bound to a different live session/) + await dispose() + }) + + it('initFor is idempotent: re-emitting session/created does not re-initialize', async () => { + const { ctx, dispose } = await backend() + const session = ctx.sessions.create('idem') + ctx.emit('session/created', session) // second create event for the same object + session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } }) + session.append('turn/end', { turn: 1, reason: { kind: 'completed' } }) + await ctx.parallel('session/flush', session) + expect(await ctx.sessionPersistence.has(SessionId('idem'))).toBe(true) + await dispose() + }) + + it('a live session claims cursor-0 ownerless state created via the public API and persists its seed', async () => { + const { ctx, dispose } = await backend() + // create() registers ownerless state with cursor 0 (no events yet). + await ctx.sessionPersistence.create(meta('cursor0')) + // A live session reusing that id, seeded with a turn, claims the ownerless + // state (cursor 0 trivially matches any seed) and persists the whole seed. + const s = ctx.sessions.create('cursor0', { seed: [ + { type: 'turn/start', seq: 0, time: 1, data: { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } } }, + { type: 'turn/end', seq: 1, time: 2, data: { turn: 1, reason: { kind: 'completed' } } }, + ] }) + await ctx.parallel('session/flush', s) + const loaded = await ctx.sessionPersistence.load(SessionId('cursor0')) + expect(loaded.events.map(e => e.seq)).toEqual([0, 1]) + await dispose() + }) + + it('HMR: reloading the backend adopts a still-live, already-materialized session', async () => { + const path = await freshDbPath() + const ctx = new Context() + await ctx.plugin(SessionStore) + // The session lives in its OWN fiber so it survives the backend reload. + let session!: Session + await ctx.plugin(Object.assign((inner: Context) => { + session = inner.sessions.create('hmr-adopt') + }, { inject: ['sessions'] })) + + // Backend instance 1 materializes the session on disk. + const backend1 = await ctx.plugin(SessionPersistenceSqlite, { path }) + session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } }) + session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }) + session.append('turn/end', { turn: 1, reason: { kind: 'completed' } }) + await ctx.parallel('session/flush', session) + + // Hot-reload: dispose instance 1, plug in instance 2 over the SAME file + // while the session stays live. Instance 2 has an empty states map but the + // row is materialized on disk and is a prefix of the live events — it must + // ADOPT (not reject), and a second turn then persists. + await backend1.dispose() + await ctx.plugin(SessionPersistenceSqlite, { path }) + session.append('turn/start', { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } }) + session.append('turn/end', { turn: 2, reason: { kind: 'completed' } }) + await expect(ctx.parallel('session/flush', session)).resolves.not.toThrow() + + const loaded = await ctx.sessionPersistence.load(SessionId('hmr-adopt')) + expect(loaded.events.filter(e => e.type === 'turn/start')).toHaveLength(2) + await ctx.fiber.dispose() + }) + + it('HMR: adoption persists the live SUFFIX that was ahead of the on-disk prefix', async () => { + const path = await freshDbPath() + const ctx = new Context() + await ctx.plugin(SessionStore) + let session!: Session + await ctx.plugin(Object.assign((inner: Context) => { + session = inner.sessions.create('hmr-suffix') + }, { inject: ['sessions'] })) + + const backend1 = await ctx.plugin(SessionPersistenceSqlite, { path }) + session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } }) + session.append('turn/end', { turn: 1, reason: { kind: 'completed' } }) + await ctx.parallel('session/flush', session) + + // Append turn 2 to the LIVE session, then dispose instance 1 WITHOUT + // flushing turn 2: it is now ONLY in the live session's events. + await backend1.dispose() + session.append('turn/start', { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } }) + session.append('turn/end', { turn: 2, reason: { kind: 'completed' } }) + + // Instance 2 adopts the on-disk prefix (turn 1) and MUST persist the live + // suffix (turn 2) carried in the session's events. + await ctx.plugin(SessionPersistenceSqlite, { path }) + await ctx.parallel('session/flush', session) + const loaded = await ctx.sessionPersistence.load(SessionId('hmr-suffix')) + expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3]) + expect(loaded.events.filter(e => e.type === 'turn/start')).toHaveLength(2) + await ctx.fiber.dispose() + }) + + it('HMR: a DIFFERENT session colliding with a materialized on-disk id is rejected', async () => { + const path = await freshDbPath() + // Instance 1 materializes a session and disposes. + const b1 = await backend(path) + const s1 = b1.ctx.sessions.create('hmr-collide') + for (const e of oneTurnLog()) s1.append(e.type, e.data) + await b1.ctx.parallel('session/flush', s1) + await b1.dispose() + + // A fresh context with an UNRELATED live session reusing the id meets a + // materialized row that is NOT a prefix of its events → reject. + const ctx = new Context() + await ctx.plugin(SessionStore) + let session!: Session + await ctx.plugin(Object.assign((inner: Context) => { + session = inner.sessions.create('hmr-collide') + }, { inject: ['sessions'] })) + session.append('turn/start', { turn: 9, trigger: { kind: 'message', source: { kind: 'user' } } }) + await ctx.plugin(SessionPersistenceSqlite, { path }) + await expect(ctx.parallel('session/flush', session)).rejects.toThrow(/id collision/) + await ctx.fiber.dispose() + }) +}) diff --git a/packages/session-persistence-sqlite/tsconfig.json b/packages/session-persistence-sqlite/tsconfig.json new file mode 100644 index 0000000000..3595f989bd --- /dev/null +++ b/packages/session-persistence-sqlite/tsconfig.json @@ -0,0 +1,15 @@ +{ + "extends": "../../tsconfig.base.json", + "compilerOptions": { + "rootDir": "src", + "outDir": "lib" + }, + "include": ["src"], + "references": [ + { "path": "../../vendor/cosmokit" }, + { "path": "../../vendor/cordis" }, + { "path": "../../vendor/schemastery" }, + { "path": "../session" }, + { "path": "../session-persistence" } + ] +} diff --git a/packages/session-persistence/README.md b/packages/session-persistence/README.md index aa30d7c028..77ee9a4e2d 100644 --- a/packages/session-persistence/README.md +++ b/packages/session-persistence/README.md @@ -26,7 +26,7 @@ The persisted unit IS the existing `SessionEvent` (event-sourced model — the l Import `runPersistenceContract` from `tests/contract.ts` and call it with a factory that yields a fresh, empty backend plus a teardown. Every backend is held to the same append-only / contiguous-seq / lazy-materialization / serializability semantics; a backend's own spec adds implementation-specific tests (crash repair, path sanitization) on top. -> **TODO (validate the abstraction with a second backend):** `dsh-session-persistence-jsonl` is currently the only implementation, so the interface and `runPersistenceContract` are only proven against one storage model. A second backend — a SQLite implementation (`dsh-session-persistence-sqlite`), where each `SessionEvent` maps 1:1 onto a row `(session_id, seq, type, time, data)` — would run the SAME `runPersistenceContract` suite and so prove the seam is genuinely backend-agnostic (lazy materialization, crash-tail-on-load, contiguous-seq all expressed against a transactional store rather than an append-only file). +Two backends run this suite: `dsh-session-persistence-jsonl` (append-only file log) and `dsh-session-persistence-sqlite` (`node:sqlite`, each `SessionEvent` one row `(session_id, seq, type, time, data)`). Both passing the same contract is the proof that the seam is genuinely backend-agnostic — lazy materialization, crash-tail-on-load, and contiguous-seq hold identically over file bytes and over a transactional store. ## Metadata types diff --git a/packages/session-persistence/src/index.ts b/packages/session-persistence/src/index.ts index 8a7d1da992..f08ac2e1f0 100644 --- a/packages/session-persistence/src/index.ts +++ b/packages/session-persistence/src/index.ts @@ -4,9 +4,12 @@ * list, and update sessions — without saying HOW. Implementations subclass * {@link SessionPersistence} and register themselves as the * `sessionPersistence` service; `@deepseek-ai/dsh-session-persistence-jsonl` - * (an append-only JSONL log per session) is the first. Future backends swap in - * SQLite/WAL, an object store, or a remote service without touching the - * consumers (the write-path plugin, the agent-loop resume seam). + * (an append-only JSONL log per session) is the first and + * `@deepseek-ai/dsh-session-persistence-sqlite` (`node:sqlite`, one row per + * event) is a second that validates the seam is backend-agnostic by passing + * the same `runPersistenceContract` suite. Further backends swap in an object + * store or a remote service without touching the consumers (the write-path + * plugin, the agent-loop resume seam). * * The persisted unit IS the existing {@link SessionEvent} — there is no * parallel "persisted message" type the log must be converted to and from diff --git a/scripts/publint-all.ts b/scripts/publint-all.ts index 8cd274647a..5b84790263 100644 --- a/scripts/publint-all.ts +++ b/scripts/publint-all.ts @@ -8,6 +8,7 @@ const packages = [ 'packages/session', 'packages/session-persistence', 'packages/session-persistence-jsonl', + 'packages/session-persistence-sqlite', 'packages/system-prompt', 'packages/tools', 'packages/agent', diff --git a/tsconfig.base.json b/tsconfig.base.json index d9010a3c6b..9d2e8bd18d 100644 --- a/tsconfig.base.json +++ b/tsconfig.base.json @@ -38,6 +38,7 @@ "@deepseek-ai/dsh-session": ["./packages/session/src"], "@deepseek-ai/dsh-session-persistence": ["./packages/session-persistence/src"], "@deepseek-ai/dsh-session-persistence-jsonl": ["./packages/session-persistence-jsonl/src"], + "@deepseek-ai/dsh-session-persistence-sqlite": ["./packages/session-persistence-sqlite/src"], "@deepseek-ai/dsh-system-prompt": ["./packages/system-prompt/src"], "@deepseek-ai/dsh-tools": ["./packages/tools/src"], "@deepseek-ai/dsh-agent": ["./packages/agent/src"], diff --git a/tsconfig.build.json b/tsconfig.build.json index 92e10bca73..874351ad84 100644 --- a/tsconfig.build.json +++ b/tsconfig.build.json @@ -14,6 +14,7 @@ { "path": "./packages/session" }, { "path": "./packages/session-persistence" }, { "path": "./packages/session-persistence-jsonl" }, + { "path": "./packages/session-persistence-sqlite" }, { "path": "./packages/system-prompt" }, { "path": "./packages/agent" }, { "path": "./packages/tools" }, diff --git a/tsconfig.typecheck.json b/tsconfig.typecheck.json index da1d1ba7b6..6051a68410 100644 --- a/tsconfig.typecheck.json +++ b/tsconfig.typecheck.json @@ -20,6 +20,7 @@ "@deepseek-ai/dsh-session": ["./packages/session/src"], "@deepseek-ai/dsh-session-persistence": ["./packages/session-persistence/src"], "@deepseek-ai/dsh-session-persistence-jsonl": ["./packages/session-persistence-jsonl/src"], + "@deepseek-ai/dsh-session-persistence-sqlite": ["./packages/session-persistence-sqlite/src"], "@deepseek-ai/dsh-system-prompt": ["./packages/system-prompt/src"], "@deepseek-ai/dsh-tools": ["./packages/tools/src"], "@deepseek-ai/dsh-agent": ["./packages/agent/src"], diff --git a/yarn.lock b/yarn.lock index 8b2339874e..328a61174c 100644 --- a/yarn.lock +++ b/yarn.lock @@ -706,6 +706,21 @@ __metadata: languageName: unknown linkType: soft +"@deepseek-ai/dsh-session-persistence-sqlite@workspace:packages/session-persistence-sqlite": + version: 0.0.0-use.local + resolution: "@deepseek-ai/dsh-session-persistence-sqlite@workspace:packages/session-persistence-sqlite" + dependencies: + "@deepseek-ai/dsh-session": "npm:^0.0.1" + "@deepseek-ai/dsh-session-persistence": "npm:^0.0.1" + cordis: "npm:^4.0.0-rc.6" + schemastery: "npm:^3.18.0" + peerDependencies: + "@deepseek-ai/dsh-session": ^0.0.1 + "@deepseek-ai/dsh-session-persistence": ^0.0.1 + cordis: ^4.0.0-rc.6 + languageName: unknown + linkType: soft + "@deepseek-ai/dsh-session-persistence@npm:^0.0.1, @deepseek-ai/dsh-session-persistence@workspace:packages/session-persistence": version: 0.0.0-use.local resolution: "@deepseek-ai/dsh-session-persistence@workspace:packages/session-persistence"