docs(rfc): classify RFCs by kind via path-encoded subdirectories
Add a second axis to every RFC — its class (feature, bug-fix,
simplification, architecture, process, testing) — encoded in the path
as docs/rfc/{lifecycle}/{class}/file.md. The folder is the label, so
the closed set is enforced by structure rather than a parsed field.
Two new doc-sync gates back it:
- verify-rfc-classification: every RFC sits in a valid class folder and
the README index lists it under the matching lifecycle→class heading.
- verify-doc-refs: every docs/*.md path cited in a packages|examples TS
comment resolves — closes a drift class verify-md-links can't see, and
catches the four comment refs this reorg moved.
The README gains a Classification section explaining the taxonomy and
per-class index sub-sections. A self-referential process RFC records why
the scheme is path-encoded and gated.
This commit is contained in:
Tianyi Cui
@@ -0,0 +1,67 @@
|
||||
# RFC: Runtime schemas for the event vocabulary (Zod vs the merge-extensible-map pattern)
|
||||
|
||||
Status: proposed
|
||||
|
||||
<!-- XXX: legacy ADR/RFC body format, not yet normalized to a unified RFC template. -->
|
||||
|
||||
## Problem
|
||||
|
||||
The harness models its core vocabulary — content blocks, message sources, finish reasons, turn triggers, turn-end reasons, and session events — as **merge-extensible maps**: a TypeScript `interface` (e.g. `SessionEventMap`, `ContentBlockMap`) that plugins augment via declaration merging, with the public union derived as `Map[keyof Map]`. This is the repo's universal extension pattern, documented in [docs/architecture.md](../../../architecture.md) ("The same merge-extensible-map pattern is used for `MessageSource`, `FinishReason`, `TurnTrigger`, and `TurnEndReason`") and relied on by the `defineTool` `InferArgs` DSL and the `assertNever` exhaustiveness convention.
|
||||
|
||||
The pattern is **compile-time only**. The types vanish at runtime: there is no schema object to validate an incoming value against, parse untrusted input with, or enumerate at runtime. Two concrete consequences surfaced in review of [the session-persistence work](../../implemented/architecture/2026-06-14-session-persistence.md) (#33):
|
||||
|
||||
1. **Persistence treats `event.data` as opaque JSON.** The JSONL/SQLite backends `JSON.stringify`/`JSON.parse` each event verbatim; the only runtime guard is `isJsonValue` (round-trip serializability — rejects BigInt, functions, cycles, non-finite numbers, …), NOT structural validation. A corrupted-but-still-JSON event datum (wrong field types, missing fields) round-trips silently and is only caught later, if at all, by a consumer's `switch`.
|
||||
2. **No runtime contract for plugin-added variants.** A plugin that declaration-merges a new `SessionEventMap` key gets compile-time typing for its own code, but nothing validates that the values it produces match the shape it declared — at the producer, at the persistence boundary, or on reload.
|
||||
|
||||
A reviewer asked whether the project should move "all the JSON serialization/deserialization" — and ultimately the event vocabulary itself — to **Zod** (or a similar runtime-schema library), so the durable boundary and the plugin extension points are backed by runtime schemas rather than erased types.
|
||||
|
||||
This RFC scopes that question. It does **not** propose an implementation; it records the tradeoff so the decision is made deliberately rather than incrementally inside a persistence PR.
|
||||
|
||||
## Why this is not a persistence change
|
||||
|
||||
It is tempting to read "use Zod for serialization" as a local change to `dsh-session-persistence-jsonl/src/format.ts`. It is not, for one structural reason: **a plugin cannot declaration-merge a Zod schema.** Declaration merging is a TypeScript compile-time mechanism; a Zod schema is a runtime value. To validate events with Zod you need a **runtime registry** that every event-producing package contributes its schema to (e.g. `ctx.sessionEvents.register('compaction/marker', z.object({…}))`), and every consumer reads from. That registry — not the persistence backend — becomes the source of truth for the vocabulary, replacing the merge-extensible interface.
|
||||
|
||||
So the real proposal is: **replace the compile-time merge-extensible-map pattern with a runtime schema registry, repo-wide.** That is a core-vocabulary redesign.
|
||||
|
||||
## Blast radius (measured)
|
||||
|
||||
A migration of the event/vocabulary surface to runtime schemas touches, at minimum:
|
||||
|
||||
- **Six merge-extensible maps** (~370 LOC of core types): `ContentBlockMap`, `MessageSourceMap`, `FinishReasonMap` (in `dsh-llm`); `TurnTriggerMap`, `TurnEndReasonMap`, `SessionEventMap` (in `dsh-session`).
|
||||
- **~10 `declare module` augmentation sites** across `dsh-agent`, `dsh-agent-loop`, `dsh-bash`, `dsh-llm`, `dsh-session`, `dsh-session-persistence`, `dsh-system-prompt`, `dsh-tools` — each would move from declaration merging to a runtime `register()` call.
|
||||
- **The event producers** — 16 `session.append(...)` call sites in the loop — unchanged in shape but now validated at the boundary.
|
||||
- **~7 switch-consumers** that branch on these unions: `deriveMessages` (`dsh-session`), `BlockAssembler` (`dsh-llm`), the `dsh-invariants` plugin, both LLM adapters (`dsh-llm-deepseek`, `dsh-llm-pi-ai`), and the tool schema layer (`dsh-tools`). The `assertNever`-on-closed-unions vs fall-through-on-extensible-unions convention (a documented lint rule) would need rethinking — runtime variants are not statically exhaustive.
|
||||
- **The `defineTool` `InferArgs` DSL** (`dsh-tools`), which derives zero-cast `execute` arg types from a compile-time schema spec — the showcase of the current approach.
|
||||
- **Docs**: architecture.md (the pattern is described as foundational), [dev-mode invariants](../../implemented/architecture/2026-06-11-dev-invariants-over-deep-readonly.md), and any RFC that references the pattern.
|
||||
|
||||
This is a HUGE change. It is not in scope for the RFC-009 session-persistence work and must not be smuggled in through it.
|
||||
|
||||
## Options
|
||||
|
||||
### A. Status quo — merge-extensible types + `isJsonValue` at the durable boundary
|
||||
Keep the compile-time pattern. Persistence stays opaque-JSON + serializability guard. Plugins extend via declaration merging; correctness of event *shape* is the producer's responsibility, enforced by TypeScript at compile time and by the `dsh-invariants` plugin's structural checks in dev.
|
||||
|
||||
- **Pros**: zero churn; plugin extension is a one-line `interface` augmentation with full type inference and no runtime registration ceremony; no new runtime dependency; the `defineTool` DSL and `assertNever` exhaustiveness keep working.
|
||||
- **Cons**: no runtime structural validation at the persistence boundary or at plugin seams; a malformed-but-JSON datum is caught late.
|
||||
|
||||
### B. Header/closed-shape validation only (schemastery), events stay opaque
|
||||
Tighten only the genuinely-closed shapes that already have hand-rolled type guards — e.g. the JSONL `HeaderLine` guard (`isHeaderLine`) — using **schemastery** (the repo's existing schema library, already used for every plugin `static Config`). Leave the merge-extensible event union as-is.
|
||||
|
||||
- **Pros**: small, fits the existing convention (schemastery, not a new lib); replaces hand-rolled guards on closed shapes with declarative schemas; no core redesign.
|
||||
- **Cons**: does not address event-data validation (the thing the reviewer actually asked about); only helps the fixed metadata records.
|
||||
|
||||
### C. Runtime schema registry for the whole vocabulary (Zod or schemastery)
|
||||
Replace the merge-extensible maps with a runtime registry the producers contribute to and the persistence/consumer paths validate against.
|
||||
|
||||
- **Pros**: real runtime validation at the durable boundary and at plugin seams; one source of truth; enables generic tooling (auto-generated docs, fuzzing, wire-format checks).
|
||||
- **Cons**: the full blast radius above; **Zod is not currently a direct dependency** (only a transitive dep of `@earendil-works/pi-ai`) and the repo's chosen schema lib is **schemastery** — adopting Zod broadly is itself a dependency decision; declaration-merge ergonomics (one-line plugin extension, full inference) are replaced by runtime registration + manual type wiring; the `assertNever` exhaustiveness guarantee weakens (runtime variants aren't statically exhaustive).
|
||||
|
||||
## Recommendation
|
||||
|
||||
Defer. Do **not** change #33. If runtime validation is wanted at the durable boundary in the near term, **Option B** (schemastery on the closed header/metadata shapes) is the proportionate step and stays within the existing convention. **Option C** is a genuine architecture decision that should be evaluated on its own merits — including whether the chosen library is Zod or schemastery — and, if accepted, land as its own change with its own RFC, not as a side effect of persistence serialization.
|
||||
|
||||
## Open questions
|
||||
|
||||
- If a registry is adopted, is the library **schemastery** (already in the tree, already the config schema lib) or **Zod** (richer ecosystem, currently only transitive)? Adopting two schema libraries is a cost in itself.
|
||||
- Can a hybrid keep compile-time inference (so `defineTool` and plugin DX survive) while adding an *optional* runtime schema per variant, validated only at the persistence/wire boundary rather than on every in-process append?
|
||||
- Does the `dsh-invariants` plugin already cover enough of the runtime-shape gap in dev that boundary validation is only needed for genuinely untrusted input (reload of an externally-modified log)?
|
||||
@@ -0,0 +1,35 @@
|
||||
# RFC: Extract a generic long-running tool runtime
|
||||
|
||||
Status: proposed
|
||||
|
||||
## Problem
|
||||
|
||||
The bash capability seam supports both foreground commands and long-running background tasks. Background support is large: the abstract executor exposes `start`, `get`, `ownerOf`, `list`, `readOutput`, `kill`, and `onTaskDone`; the local executor tracks tasks, incremental reads, owner tokens, process cleanup, and completion listeners; the model sees three tools (`bash`, `bash_output`, `bash_kill`); the tool plugin injects completion notices back into the owning agent's session. The local executor fences task access behind owner tokens because predictable global task ids are a cross-session read/kill hazard.
|
||||
|
||||
The [tool cookbook](../../../cookbook/adding-a-tool.md) already points at the real design smell: background bash is really generic long-running-tool infrastructure living inside one tool. If future tools need background execution, polling, kill, ownership, and completion notices, those semantics should not be hidden in `dsh-bash`.
|
||||
|
||||
## Proposal
|
||||
|
||||
Move long-running task semantics above bash into a tool-agnostic runtime. Bash remains able to run background commands, but it stops owning the general concepts of task ids, ownership tokens, polling, cancellation, completion notifications, and model-facing "read/kill this task" commands.
|
||||
|
||||
The runtime should own:
|
||||
|
||||
- Stable task ids and owner tokens keyed to the calling session/agent.
|
||||
- Registration of a long-running task with a producer for incremental output and a completion promise.
|
||||
- Generic read/cancel/list operations with the same cross-session authorization rule for every tool.
|
||||
- Completion notification injection into the owning session.
|
||||
- Presentation hooks for pending/running/completed task state, with bash supplying only command-specific labels and output formatting.
|
||||
|
||||
`dsh-bash` then keeps the bash-specific execution contract: resolve a request into a command spec, run a foreground command, or start a process and hand its streams/process handle to the generic runtime. `dsh-tool-bash` keeps the model-facing command tool, but the follow-up operations become generic long-running-tool operations or a shared utility that bash registers with, rather than bespoke `bash_output`/`bash_kill` plumbing.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- The bash-specific packages no longer define the generic task registry, owner-token authorization, polling, cancellation, or completion-notification machinery.
|
||||
- A shared long-running-task service or tool layer owns those semantics and is documented as the path for any future background-capable tool.
|
||||
- Bash background behavior remains available through the shared layer, with tests proving cross-session isolation still holds.
|
||||
- ACP and snapshot fixtures render background bash through the shared task vocabulary, not through bash-only lifecycle semantics.
|
||||
- The [tool cookbook](../../../cookbook/adding-a-tool.md) points long-running tools at the shared runtime instead of telling each tool to invent its own task protocol.
|
||||
|
||||
## What we give up
|
||||
|
||||
The bash package loses local ownership of an already-working background-task implementation, and the implementing PR may temporarily churn model-facing tool names or transcript presentation. That churn is worthwhile if it leaves one background-task contract instead of making every future long-running tool clone bash's private protocol.
|
||||
@@ -0,0 +1,59 @@
|
||||
# RFC: Reorganize packages into a modular hierarchy
|
||||
|
||||
Status: proposed
|
||||
|
||||
## Problem
|
||||
|
||||
`packages/` is flat. Core product packages, provider integrations, capability seams, example UI support, and snapshot-only replay support all sit at the same level and look equally foundational. The [package README](../../../../packages/README.md) already has a `FIXME(package-hierarchy)` noting that `ui-stdio` and `llm-replay` were extracted from examples mostly for reuse and coverage. The flat layout makes support packages appear more product-shaped than they are and forces publish/lint/doc scripts to encode intent through comments or static lists.
|
||||
|
||||
This is not just cosmetic. A package's location currently says little about whether it is core API, a swappable capability, an adapter integration, an example harness helper, or test infrastructure. That makes future removal harder because every top-level package looks like part of the same public surface.
|
||||
|
||||
## Proposal
|
||||
|
||||
Move packages into a deliberate hierarchy under `packages/`. The exact layout is deferred to the implementing PR, but it should group packages by modular role rather than keep every package at one flat level.
|
||||
|
||||
One plausible shape:
|
||||
|
||||
```text
|
||||
packages/
|
||||
core/
|
||||
session/
|
||||
system-prompt/
|
||||
tools/
|
||||
agent/
|
||||
agent-loop/
|
||||
invariants/
|
||||
llm/
|
||||
llm/
|
||||
adapters/
|
||||
llm-deepseek/
|
||||
llm-pi-ai/
|
||||
bash/
|
||||
bash/
|
||||
bash-local/
|
||||
tool-bash/
|
||||
session-persistence/
|
||||
session-persistence/
|
||||
session-persistence-jsonl/
|
||||
session-persistence-sqlite/
|
||||
acp/
|
||||
support/
|
||||
ui-stdio/
|
||||
llm-replay/
|
||||
```
|
||||
|
||||
The final implementation may choose different names or groupings, but it should keep the same intent: core APIs, package families such as LLM/bash/session persistence, standalone integrations such as ACP, and support/test/example packages are distinguishable from the filesystem alone. Npm package names can stay `@deepseek-ai/dsh-*`; the hierarchy is about repo structure and maintenance policy, not public package renaming.
|
||||
|
||||
This proposal does not delete `llm-replay` or `ui-stdio` by itself. It makes their status honest: either they graduate into product packages with documented consumers, or they live under a support/testing/example classification where release and compatibility expectations are lower.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Packages move from the flat `packages/<name>/` layout into a documented modular hierarchy.
|
||||
- The implementing PR chooses the exact hierarchy and updates workspace globs, TypeScript paths, package docs, generated module graphs, `cordis.yml` package paths, build scripts, and publish/lint scripts in one coordinated move.
|
||||
- Scripts that publish, lint publishability, or generate package inventories use the hierarchy instead of an ad hoc static list where the hierarchy is enough to express the policy.
|
||||
- Docs explain which package groups are part of the product API and which groups are support/test/example infrastructure.
|
||||
- New package guidance tells authors where to place a package and discourages new one-off top-level groups.
|
||||
|
||||
## What we give up
|
||||
|
||||
The restructure churns imports, workspace globs, docs links, and package paths. That churn is acceptable pre-release if it prevents the flat layout from fossilizing support packages as product contracts.
|
||||
@@ -0,0 +1,27 @@
|
||||
# RFC: Make the shared example base providerless
|
||||
|
||||
Status: proposed
|
||||
|
||||
## Problem
|
||||
|
||||
The examples have two shared base files: [examples/base-core.yml](../../../../examples/base-core.yml) is providerless, while [examples/base.yml](../../../../examples/base.yml) includes that core plus the real `llm-deepseek` adapter. Snapshot replay needs the providerless core with `llm-replay`, because loading the real adapter without a key throws. The normal demos need the real adapter. The result is a naming inversion: the file named `base.yml` is not the reusable base for all examples, while the true base is `base-core.yml`.
|
||||
|
||||
The split is understandable, but it makes every config explanation longer. It also leads to awkward test setup like a keyless smoke test carrying a dummy API key so an adapter can boot even though the model is not called.
|
||||
|
||||
## Proposal
|
||||
|
||||
Rename the providerless core to [examples/base.yml](../../../../examples/base.yml) and make adapter selection explicit in each concrete example. The coding and ACP real configs add a tiny `llm-deepseek` include or local block; snapshot config adds `llm-replay`. Delete [examples/base-core.yml](../../../../examples/base-core.yml).
|
||||
|
||||
The shared base should contain only provider-neutral services and tools: `llm`, sessions, system prompt, tools, agents, invariants, bash executor, and bash tool schemas. Anything that chooses a model provider belongs at the leaf config.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- [examples/base.yml](../../../../examples/base.yml) is providerless.
|
||||
- [examples/base-core.yml](../../../../examples/base-core.yml) is deleted.
|
||||
- Real demo configs explicitly add the DeepSeek adapter.
|
||||
- Snapshot replay config includes the same providerless base and its replay adapter.
|
||||
- The [examples README](../../../../examples/README.md), example-specific READMEs, and RFC references stop explaining "base = base-core plus adapter".
|
||||
|
||||
## What we give up
|
||||
|
||||
Real demos lose one layer of convenience: each must opt into the adapter. That is the right default for examples, because adapter choice is the variable part and providerless wiring is the shared product core.
|
||||
Reference in New Issue
Block a user