Pine/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 1 Wiki Activity

feat(session): opt-in packed chunk rows in the JSONL log

Browse Source 
Providers stream token-sized deltas, so a session log stores hundreds of
near-identical assistant/chunk lines whose JSON envelopes dwarf their
payloads (~56x measured on a real DeepSeek session, 73% of file bytes).

Add a lossless storage codec to dsh-session: packChunkRuns() folds each
run of >=3 consecutive same-block delta chunks into one storage row --
text-chunks / reasoning-chunks / tool-call-chunks, bare slash-less tags
like the header line's 'session' so rows cannot be confused with session
events -- and decodeStorageRecord() expands rows back to the exact
original events (seq0/time0 + dt gap array reconstruct every member's
seq/time; tool-call rows carry the run-constant id/name). The encoder
whitelists exact shapes and stores anything unrecognized verbatim; the
decoder validates row-tagged values and fails loud on malformation.

The JSONL backend gains a packChunks config (default false). Writing
packs only when enabled -- default-off output stays byte-identical to
the previous layout, so snapshot goldens are untouched. Reading is
layout-blind: scanLog always decodes rows and now checks seq contiguity
with a cursor instead of the line index, so packed, unpacked, and mixed
files all load identically. Fixture readers (llm-replay parseSessionLog,
acp-snapshot normalizeSessionLog) share the codec; the normalizer zeroes
a row's time0/dt exactly like an event's time. The two demo bundles
plumb packChunks from cordis.yml to the backend.

Measured on a real coding session: 105 KB -> 42 KB (-60%), 475 lines ->
74, with reasoning/tool-call heavy sessions saving the most. Covered by
example + fast-check round-trip codec tests, backend packed/mixed/torn-
tail specs, and an end-to-end demo run loading a packed log through a
default-config backend.
This commit is contained in:
kingwl
2026-07-15 21:26:36 +08:00
parent b045b553a9
commit 7ef21239ca
21 changed files with 816 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/session-persistence/session-persistence-jsonl/README.md
+4 -2
View File
@@ -7,10 +7,11 @@ The JSONL durable session-persistence backend — a concrete `SessionPersistence
```
<root>/
cwd-<sha256(cwd)[:12]>/ # per-project bucket (or _no-cwd/ when no cwd)
<encoded-id>.jsonl # header line + one SessionEvent per line (verbatim)
<encoded-id>.jsonl # header line + one storage record per line
```
- The first `.jsonl` line is the immutable `SessionHeader` tagged `{ type: 'session', version, id, cwd?, createdAt, parentSession?, seedLength? }`; every subsequent line is one `SessionEvent` JSON, **verbatim including `assistant/chunk`** so `seq` stays contiguous (`events[i].seq === i`).
- The first `.jsonl` line is the immutable `SessionHeader` tagged `{ type: 'session', version, id, cwd?, createdAt, parentSession?, seedLength? }`; every subsequent line is one storage record. `assistant/chunk` events are never dropped, and `seq` stays contiguous across the decoded log.
- A storage record is a `SessionEvent` JSON verbatim, or — written only under `packChunks` — a **packed chunk row** (`text-chunks` / `reasoning-chunks` / `tool-call-chunks`; bare slash-less tags like the header's `session`, so row tags cannot be confused with event types): one line holding a run of ≥3 consecutive same-block `assistant/chunk` delta events, `seq0`/`time0` plus per-member `dt` gaps reconstructing every member's `seq`/`time` exactly. The lossless codec lives in `@deepseek-ai/dsh-session` (`packChunkRuns`/`decodeStorageRecord`) and whitelists exact shapes — anything unrecognized stores verbatim. Reading is layout-blind: `load` always decodes rows, so packed, unpacked, and mixed files load identically.
- Session ids are unvalidated branded strings, so they are percent-encoded to a single safe path segment before use (no traversal, no collision).
## Config
@@ -18,6 +19,7 @@ The JSONL durable session-persistence backend — a concrete `SessionPersistence
| Key | Type | Notes |
|---|---|---|
| `root` | `string` (required) | Root directory for all session files. **No default** — a `process.cwd()` default would scatter files as the process's cwd changes (bash calls, subprocesses). |
| `packChunks` | `boolean` (default `false`) | Write delta-chunk runs as packed rows (~60% smaller logs measured on a real coding session). Off, the written layout is byte-identical to the pre-packing format; reading packed rows works regardless of this switch. Off by default while the snapshot goldens stay one-event-per-line — recording with packing on rewrites every fixture `session.jsonl`. |
## Durability and crash semantics