refactor(token-meter): fold the surface once for the meter and the projection

`contextBreakdown.messageTokens` and `measure().surfaceTokens` answer the same
question in the same heuristic vocabulary, and the panel's composition rows are
only honest while they agree. Each owner carried its own copy of the positional
fold — same pricing, same `{seq, tokens}` node list, same replace-range lookup
and guard, differing only in mutable versus immutable application — so an edit
to either one would have moved the panel away from `measure()` with both sides
still green. The duplication gate caught the shared 62 tokens.

`src/surface-fold.ts` now owns `foldSurfaceTokens`: total, allocation-fresh,
returning the event's price, the next surface, and the signed total delta. The
service assigns that result where it used to prepare a commit closure, which
keeps its validate-before-mutate replay transaction intact — the fold throws
before any state is touched, so a malformed event still fails identically on
every retry. `_prepareSurfaceMutation` and `_estimateSurfaceEvent` go away with
it, and the projection's apply drops to one call.

Covers the identity with a session that appends and then compacts, asserting
the projection figure equals the service surface at each boundary; the test
fails when either side of the fold is perturbed.
This commit is contained in:
Yichen Jiang
2026-08-05 16:29:51 +08:00
parent 46562b2c30
commit e62cbe12e4
12 changed files with 138 additions and 100 deletions
@@ -0,0 +1,63 @@
/**
* The one positional surface fold, shared by the measurement service's replay
* state and the pure `contextBreakdown` projection. Both answer "what does the
* current model-visible conversation cost", so they MUST price and place every
* node identically: a private copy in either owner would let the panel's
* message figure drift away from `measure().surfaceTokens` with both sides
* still passing their own tests.
*
* @module @deepseek-ai/dsh-token-meter/surface-fold
*/
import { deriveEventMessage } from '@deepseek-ai/dsh-session'
import type { SurfaceEvent } from '@deepseek-ai/dsh-session'
import type { TokenSurfaceNode } from './types.ts'
import { estimateMessage } from './estimate.ts'
/** One surface event's placement and cost against the surface preceding it. */
export interface SurfaceTokenFold {
/** Heuristic price of the event's own message; 0 when it derives none. */
readonly tokens: number
/** The surface after the event, detached from the input. */
readonly nodes: TokenSurfaceNode[]
/** Signed change in the surface total: `tokens` minus anything shadowed. */
readonly deltaTokens: number
}
/**
* Fold one surface event onto a priced surface.
*
* Total and allocation-fresh: the caller assigns the result rather than
* mutating in place, so a throw here leaves the caller's state untouched and
* the same malformed event fails identically on every retry.
* @param nodes - the priced surface preceding this event, in model-visible order.
* @param event - the surface event to place.
* @returns the event's price, the next surface, and the signed total delta.
* @throws when a replacement names a range absent from `nodes` — committed
* logs are surface-validated at append time, so an unresolvable range is log
* corruption and must fail loud rather than skip the event.
*/
export function foldSurfaceTokens(
nodes: readonly TokenSurfaceNode[],
event: SurfaceEvent,
): SurfaceTokenFold {
const message = deriveEventMessage(event)
const tokens = message === null ? 0 : estimateMessage(message)
const op = event.surfaceOp
if (op === 'append') {
return { tokens, nodes: [...nodes, { seq: event.seq, tokens }], deltaTokens: tokens }
}
const startIdx = nodes.findIndex(node => node.seq === op.start)
const endIdx = nodes.findIndex(node => node.seq === op.end)
if (startIdx === -1 || endIdx === -1 || startIdx > endIdx) {
throw new Error(
`token surface: replace at seq ${event.seq} has invalid current range ${op.start}-${op.end}`,
)
}
const removed = nodes
.slice(startIdx, endIdx + 1)
.reduce((total, node) => total + node.tokens, 0)
const next = [...nodes]
next.splice(startIdx, endIdx - startIdx + 1, { seq: event.seq, tokens })
return { tokens, nodes: next, deltaTokens: tokens - removed }
}