refactor(token-meter): fold the surface once for the meter and the projection
`contextBreakdown.messageTokens` and `measure().surfaceTokens` answer the same
question in the same heuristic vocabulary, and the panel's composition rows are
only honest while they agree. Each owner carried its own copy of the positional
fold — same pricing, same `{seq, tokens}` node list, same replace-range lookup
and guard, differing only in mutable versus immutable application — so an edit
to either one would have moved the panel away from `measure()` with both sides
still green. The duplication gate caught the shared 62 tokens.
`src/surface-fold.ts` now owns `foldSurfaceTokens`: total, allocation-fresh,
returning the event's price, the next surface, and the signed total delta. The
service assigns that result where it used to prepare a commit closure, which
keeps its validate-before-mutate replay transaction intact — the fold throws
before any state is touched, so a malformed event still fails identically on
every retry. `_prepareSurfaceMutation` and `_estimateSurfaceEvent` go away with
it, and the projection's apply drops to one call.
Covers the identity with a session that appends and then compacts, asserting
the projection figure equals the service surface at each boundary; the test
fails when either side of the fold is perturbed.
This commit is contained in:
@@ -0,0 +1,63 @@
|
||||
/**
|
||||
* The one positional surface fold, shared by the measurement service's replay
|
||||
* state and the pure `contextBreakdown` projection. Both answer "what does the
|
||||
* current model-visible conversation cost", so they MUST price and place every
|
||||
* node identically: a private copy in either owner would let the panel's
|
||||
* message figure drift away from `measure().surfaceTokens` with both sides
|
||||
* still passing their own tests.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-token-meter/surface-fold
|
||||
*/
|
||||
|
||||
import { deriveEventMessage } from '@deepseek-ai/dsh-session'
|
||||
import type { SurfaceEvent } from '@deepseek-ai/dsh-session'
|
||||
import type { TokenSurfaceNode } from './types.ts'
|
||||
import { estimateMessage } from './estimate.ts'
|
||||
|
||||
/** One surface event's placement and cost against the surface preceding it. */
|
||||
export interface SurfaceTokenFold {
|
||||
/** Heuristic price of the event's own message; 0 when it derives none. */
|
||||
readonly tokens: number
|
||||
/** The surface after the event, detached from the input. */
|
||||
readonly nodes: TokenSurfaceNode[]
|
||||
/** Signed change in the surface total: `tokens` minus anything shadowed. */
|
||||
readonly deltaTokens: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Fold one surface event onto a priced surface.
|
||||
*
|
||||
* Total and allocation-fresh: the caller assigns the result rather than
|
||||
* mutating in place, so a throw here leaves the caller's state untouched and
|
||||
* the same malformed event fails identically on every retry.
|
||||
* @param nodes - the priced surface preceding this event, in model-visible order.
|
||||
* @param event - the surface event to place.
|
||||
* @returns the event's price, the next surface, and the signed total delta.
|
||||
* @throws when a replacement names a range absent from `nodes` — committed
|
||||
* logs are surface-validated at append time, so an unresolvable range is log
|
||||
* corruption and must fail loud rather than skip the event.
|
||||
*/
|
||||
export function foldSurfaceTokens(
|
||||
nodes: readonly TokenSurfaceNode[],
|
||||
event: SurfaceEvent,
|
||||
): SurfaceTokenFold {
|
||||
const message = deriveEventMessage(event)
|
||||
const tokens = message === null ? 0 : estimateMessage(message)
|
||||
const op = event.surfaceOp
|
||||
if (op === 'append') {
|
||||
return { tokens, nodes: [...nodes, { seq: event.seq, tokens }], deltaTokens: tokens }
|
||||
}
|
||||
const startIdx = nodes.findIndex(node => node.seq === op.start)
|
||||
const endIdx = nodes.findIndex(node => node.seq === op.end)
|
||||
if (startIdx === -1 || endIdx === -1 || startIdx > endIdx) {
|
||||
throw new Error(
|
||||
`token surface: replace at seq ${event.seq} has invalid current range ${op.start}-${op.end}`,
|
||||
)
|
||||
}
|
||||
const removed = nodes
|
||||
.slice(startIdx, endIdx + 1)
|
||||
.reduce((total, node) => total + node.tokens, 0)
|
||||
const next = [...nodes]
|
||||
next.splice(startIdx, endIdx - startIdx + 1, { seq: event.seq, tokens })
|
||||
return { tokens, nodes: next, deltaTokens: tokens - removed }
|
||||
}
|
||||
Reference in New Issue
Block a user