feat(llm): add per-provider retry policies
This commit is contained in:
Turtle
@@ -1,41 +1,50 @@
|
||||
# `@deepseek-ai/dsh-llm-retry`
|
||||
|
||||
Function plugin that retries selected transient model-request failures on the agent loop's closed-step recovery seam. It does not wrap `ctx.llm.stream()`: every adapter call remains one provider attempt, and every retry opens a fresh numbered step.
|
||||
Function plugin that applies exact-provider retry policy on the agent loop's closed-step recovery seam. It does not wrap `ctx.llm.stream()`: every adapter call remains one provider attempt, and every retry opens a fresh numbered step.
|
||||
|
||||
The default policy permits two retries for `RATE_LIMIT`, `SERVER`, `TIMEOUT`, and `TRANSPORT`, using bounded exponential backoff from 500 ms to 10 seconds with 10 percent jitter. Delay bounds must fit Node's supported timer range. A valid `providerRetryAfterMs` replaces local backoff when it is within the configured cap; an over-cap instruction delegates to the next recovery policy instead.
|
||||
Each provider adapter owns an optional nested `retryPolicy`, captured when its route registers on `ctx.llm`. Omission uses normal mode: two retries for `RATE_LIMIT`, `SERVER`, `TIMEOUT`, and `TRANSPORT`. A normal policy can change its finite budget, eligible codes, and backoff. Always mode asks downstream recovery first, then retries every model-request failure without an attempt limit; success, cancellation, or plugin disposal stops it.
|
||||
|
||||
Before waiting, the plugin appends a non-surface `llm/retry` event with the failure and scheduled delay. Cancellation and plugin disposal abort the wait; disposal drains the plugin's active backoffs, and a callback captured before disposal fails closed if invoked afterward.
|
||||
Both modes use bounded exponential backoff with symmetric jitter. A valid `providerRetryAfterMs` at or below `maxDelayMs` replaces local backoff without jitter. An over-cap provider delay makes normal mode delegate, while always mode uses its configured local backoff so it cannot terminate on that instruction.
|
||||
|
||||
The separately published `./invariant` companion checks that every retry record names the current open turn and its latest closed step, has a unique step record and increasing retry number, and carries a positive bounded retry budget and non-negative bounded timer delay. Full jitter may schedule zero milliseconds at its lower boundary.
|
||||
Before waiting, the plugin appends a non-surface `llm/retry` event with the provider, mode, failure, and scheduled delay. Normal events include the finite maximum; always events omit it, and UIs render `∞`. Cancellation and plugin disposal abort the wait; disposal drains active backoffs, and a callback captured before disposal fails closed.
|
||||
|
||||
The separately published `./invariant` companion checks that every retry record names the current open turn and latest closed step, matches the failed request's durable provider, has a unique step record and correct provider-policy retry number, and carries a valid mode-specific budget and bounded timer delay. Full jitter may schedule zero milliseconds at its lower boundary.
|
||||
|
||||
```yaml
|
||||
- name: '@deepseek-ai/dsh-llm-retry'
|
||||
- name: '@deepseek-ai/dsh-llm-deepseek'
|
||||
config:
|
||||
maxTransientRetries: 2
|
||||
initialDelayMs: 500
|
||||
maxDelayMs: 10000
|
||||
jitterRatio: 0.1
|
||||
retryableCodes: [RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT]
|
||||
apiKey: !!js process.env.DEEPSEEK_API_KEY
|
||||
retryPolicy:
|
||||
mode: always
|
||||
backoff:
|
||||
initialDelayMs: 1000
|
||||
maxDelayMs: 30000
|
||||
jitterRatio: 0.2
|
||||
|
||||
- name: '@deepseek-ai/dsh-llm-retry'
|
||||
```
|
||||
|
||||
The executor has no policy config. Multi-provider adapters such as `dsh-llm-pi-ai` place `retryPolicy` inside each provider profile, avoiding a second provider-name list.
|
||||
|
||||
## Model Experience
|
||||
|
||||
### Transient request recovery
|
||||
### Model-request recovery
|
||||
|
||||
#### What the model sees
|
||||
|
||||
No retry event, delay, or failure prose is model-visible. After a retry, the next numbered step reconstructs the same explicit provider/model request from durable session history; failed chunks never enter derived messages.
|
||||
No retry event, delay, provider error, or failed partial output is model-visible. The next numbered step reconstructs the same explicit provider/model request from durable surface history unless a downstream recovery policy deliberately changes that surface.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Each retry is a new provider request and may repeat input-token billing. The finite budget caps attempts; `llm/retry` itself contributes no tokens.
|
||||
Each retry is a new provider request and may repeat input-token billing. Normal mode has a finite budget; always mode can consume unbounded requests until success or cancellation. `llm/retry` itself contributes no tokens.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
The reconstructed request preserves the prior prefix and is eligible for provider cache reuse under that provider's rules. The non-surface status event does not change cache identity.
|
||||
The reconstructed request preserves the prior prefix and is eligible for provider cache reuse under that provider's rules. The non-surface retry event does not change cache identity.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **Agent steps are the only retry boundary** — direct `ctx.llm.stream()` consumers remain single-attempt because a raw stream cannot separate already-emitted chunks durably.
|
||||
- **Finite plugin budgets add** — this policy counts only configured transient codes; context-overflow compaction counts only its own code. A future policy with overlapping codes must document and test registration-order behavior.
|
||||
- **Always mode retries permanent failures** — authentication, quota, invalid-request, protocol, and unrecoverable context errors continue until success, cancellation, or disposal; deployments own provider-specific cost and latency controls.
|
||||
- **Recovery policies compose by waterfall order** — always mode accepts a downstream retry before applying its fallback. A later policy that never settles also prevents the fallback from running.
|
||||
- **`llm/retry` records scheduling, not completion** — later step and turn events establish success, exhaustion, or cancellation.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-llm-retry",
|
||||
"description": "Bounded transient LLM request retry policy for the DeepSeek Harness",
|
||||
"description": "Provider-routed LLM request retry policy for the DeepSeek Harness",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
/** Durable request-route lookup for one closed model step. @module @deepseek-ai/dsh-llm-retry/history */
|
||||
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
|
||||
/**
|
||||
* Find the provider in force when one step closed, excluding later recovery mutations.
|
||||
* A preceding retry is also a route marker because every provider change
|
||||
* requires a newer full request-header snapshot.
|
||||
* @param events - session events containing the closed step.
|
||||
* @param turn - turn that owns the failed step.
|
||||
* @param step - failed step whose provider is required.
|
||||
* @returns the provider from the request header in force at that step boundary.
|
||||
*/
|
||||
export function providerForClosedStep(
|
||||
events: readonly SessionEvent[],
|
||||
turn: number,
|
||||
step: number,
|
||||
): string | undefined {
|
||||
const stepEndIndex = events.findLastIndex(event =>
|
||||
event.type === 'step/end'
|
||||
&& event.data.turn === turn
|
||||
&& event.data.step === step,
|
||||
)
|
||||
if (stepEndIndex < 0) return undefined
|
||||
for (let index = stepEndIndex; index >= 0; index -= 1) {
|
||||
// The loop bounds prove this indexed read exists.
|
||||
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
|
||||
const event = events[index]!
|
||||
if (event.type === 'request/header') return event.data.header.config.provider
|
||||
if (event.type === 'llm/retry'
|
||||
&& event.data.turn === turn
|
||||
&& event.data.step < step) {
|
||||
return event.data.provider
|
||||
}
|
||||
if (event.type === 'turn/start' || event.type === 'turn/end') return undefined
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
@@ -1,5 +1,5 @@
|
||||
/**
|
||||
* Bounded transient model-request retry policy on the agent loop's closed-step
|
||||
* Provider-routed model-request retry policy on the agent loop's closed-step
|
||||
* recovery seam. Each scheduled retry is durable before its cancellable wait.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-llm-retry
|
||||
@@ -8,103 +8,50 @@
|
||||
import type { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import type { Agent, RequestError, RequestErrorDecision } from '@deepseek-ai/dsh-agent'
|
||||
import type { LlmFailure } from '@deepseek-ai/dsh-llm'
|
||||
import type {} from '@deepseek-ai/dsh-session'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
import type { LlmFailure, ResolvedRetryPolicy } from '@deepseek-ai/dsh-llm'
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import { providerForClosedStep } from './history.ts'
|
||||
|
||||
declare module '@deepseek-ai/dsh-session' {
|
||||
interface SessionEventMap {
|
||||
/** Durable, non-surface record of one transient retry scheduled after a closed failed step. */
|
||||
/** Durable, non-surface record of one provider-routed retry scheduled after a closed failed step. */
|
||||
'llm/retry': {
|
||||
turn: number
|
||||
step: number
|
||||
provider: string
|
||||
mode: 'normal'
|
||||
retry: number
|
||||
maxRetries: number
|
||||
delayMs: number
|
||||
failure: LlmFailure
|
||||
} | {
|
||||
turn: number
|
||||
step: number
|
||||
provider: string
|
||||
mode: 'always'
|
||||
retry: number
|
||||
delayMs: number
|
||||
failure: LlmFailure
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export const name = 'llm-retry'
|
||||
export const inject = ['agents']
|
||||
export const inject = ['agents', 'llm']
|
||||
|
||||
const DEFAULT_MAX_TRANSIENT_RETRIES = 2
|
||||
const DEFAULT_INITIAL_DELAY_MS = 500
|
||||
const DEFAULT_MAX_DELAY_MS = 10_000
|
||||
const DEFAULT_JITTER_RATIO = 0.1
|
||||
const DEFAULT_RETRYABLE_CODES = Object.freeze(['RATE_LIMIT', 'SERVER', 'TIMEOUT', 'TRANSPORT'])
|
||||
|
||||
/** Deployment-owned limits and classification for transient request recovery. */
|
||||
export interface Config {
|
||||
/** Maximum transient retries after the first request (default 2). */
|
||||
maxTransientRetries?: number
|
||||
/** Initial local exponential-backoff delay in milliseconds (default 500). */
|
||||
initialDelayMs?: number
|
||||
/** Maximum accepted or locally scheduled delay in milliseconds (default 10000). */
|
||||
maxDelayMs?: number
|
||||
/** Symmetric random multiplier range around one (default 0.1). */
|
||||
jitterRatio?: number
|
||||
/** Stable failure codes eligible for this policy. */
|
||||
retryableCodes?: string[]
|
||||
}
|
||||
/** This policy executor has no config; providers own `retryPolicy`. */
|
||||
export type Config = Readonly<Record<never, never>>
|
||||
|
||||
/** Runtime schema for {@link Config}. */
|
||||
export const Config: z<Config> = z.object({
|
||||
maxTransientRetries: z.number().step(1).min(0).default(DEFAULT_MAX_TRANSIENT_RETRIES),
|
||||
initialDelayMs: z.number().max(MAX_TIMER_DELAY_MS).default(DEFAULT_INITIAL_DELAY_MS),
|
||||
maxDelayMs: z.number().max(MAX_TIMER_DELAY_MS).default(DEFAULT_MAX_DELAY_MS),
|
||||
jitterRatio: z.number().min(0).max(1).default(DEFAULT_JITTER_RATIO),
|
||||
retryableCodes: z.array(z.string()).default([...DEFAULT_RETRYABLE_CODES]),
|
||||
})
|
||||
export const Config: z<Config> = z.object({})
|
||||
|
||||
interface ResolvedConfig {
|
||||
readonly maxTransientRetries: number
|
||||
readonly initialDelayMs: number
|
||||
readonly maxDelayMs: number
|
||||
readonly jitterRatio: number
|
||||
readonly retryableCodes: ReadonlySet<string>
|
||||
}
|
||||
|
||||
function resolveConfig(config: Config): ResolvedConfig {
|
||||
const maxTransientRetries = config.maxTransientRetries ?? DEFAULT_MAX_TRANSIENT_RETRIES
|
||||
const initialDelayMs = config.initialDelayMs ?? DEFAULT_INITIAL_DELAY_MS
|
||||
const maxDelayMs = config.maxDelayMs ?? DEFAULT_MAX_DELAY_MS
|
||||
const jitterRatio = config.jitterRatio ?? DEFAULT_JITTER_RATIO
|
||||
const codes = config.retryableCodes ?? [...DEFAULT_RETRYABLE_CODES]
|
||||
|
||||
if (!Number.isInteger(maxTransientRetries) || maxTransientRetries < 0) {
|
||||
throw new Error('llm-retry: maxTransientRetries must be a non-negative integer')
|
||||
function validateConfig(config: Config): void {
|
||||
const [key] = Object.keys(config)
|
||||
if (key === undefined) return
|
||||
if (key === 'retryPolicy') {
|
||||
throw new Error('llm-retry: retryPolicy belongs under each provider configuration')
|
||||
}
|
||||
if (!Number.isFinite(initialDelayMs) || initialDelayMs <= 0 || initialDelayMs > MAX_TIMER_DELAY_MS) {
|
||||
throw new Error(`llm-retry: initialDelayMs must be a positive finite number no greater than ${MAX_TIMER_DELAY_MS}`)
|
||||
}
|
||||
if (!Number.isFinite(maxDelayMs) || maxDelayMs <= 0 || maxDelayMs > MAX_TIMER_DELAY_MS) {
|
||||
throw new Error(`llm-retry: maxDelayMs must be a positive finite number no greater than ${MAX_TIMER_DELAY_MS}`)
|
||||
}
|
||||
if (initialDelayMs > maxDelayMs) {
|
||||
throw new Error('llm-retry: initialDelayMs must be less than or equal to maxDelayMs')
|
||||
}
|
||||
if (!Number.isFinite(jitterRatio) || jitterRatio < 0 || jitterRatio > 1) {
|
||||
throw new Error('llm-retry: jitterRatio must be between 0 and 1')
|
||||
}
|
||||
if (codes.length === 0) {
|
||||
throw new Error('llm-retry: retryableCodes must not be empty')
|
||||
}
|
||||
if (codes.some(code => code.length === 0)) {
|
||||
throw new Error('llm-retry: retryableCodes must contain only non-empty strings')
|
||||
}
|
||||
if (new Set(codes).size !== codes.length) {
|
||||
throw new Error('llm-retry: retryableCodes must not contain duplicates')
|
||||
}
|
||||
|
||||
return Object.freeze({
|
||||
maxTransientRetries,
|
||||
initialDelayMs,
|
||||
maxDelayMs,
|
||||
jitterRatio,
|
||||
retryableCodes: new Set(codes),
|
||||
})
|
||||
throw new Error(`llm-retry: unknown key "${key}"`)
|
||||
}
|
||||
|
||||
/** Non-serializable seams used to make timing policy deterministic in tests. */
|
||||
@@ -113,7 +60,38 @@ export interface RetryInternals {
|
||||
random?: () => number
|
||||
}
|
||||
|
||||
function localDelay(config: ResolvedConfig, retry: number, random: () => number): number {
|
||||
type DownstreamOutcome =
|
||||
| { readonly type: 'decision'; readonly decision: RequestErrorDecision }
|
||||
| { readonly type: 'error'; readonly error: unknown }
|
||||
| { readonly type: 'aborted' }
|
||||
|
||||
function downstreamUntilAbort(
|
||||
next: () => Promise<RequestErrorDecision>,
|
||||
signal: AbortSignal,
|
||||
): Promise<DownstreamOutcome> {
|
||||
if (signal.aborted) return Promise.resolve({ type: 'aborted' })
|
||||
return new Promise((resolve) => {
|
||||
const finish = (outcome: DownstreamOutcome): void => {
|
||||
signal.removeEventListener('abort', onAbort)
|
||||
resolve(outcome)
|
||||
}
|
||||
const onAbort = (): void => { finish({ type: 'aborted' }) }
|
||||
signal.addEventListener('abort', onAbort, { once: true })
|
||||
let downstream: Promise<RequestErrorDecision>
|
||||
try {
|
||||
downstream = next()
|
||||
} catch (error: unknown) {
|
||||
finish({ type: 'error', error })
|
||||
return
|
||||
}
|
||||
void downstream.then(
|
||||
(decision) => { finish({ type: 'decision', decision }) },
|
||||
(error: unknown) => { finish({ type: 'error', error }) },
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
function localDelay(config: ResolvedRetryPolicy, retry: number, random: () => number): number {
|
||||
const exponent = Math.min(retry - 1, 1024)
|
||||
const exponential = Math.min(config.initialDelayMs * 2 ** exponent, config.maxDelayMs)
|
||||
const jitter = 1 - config.jitterRatio + 2 * config.jitterRatio * random()
|
||||
@@ -136,13 +114,13 @@ function cancellableDelay(delayMs: number, signal: AbortSignal): Promise<boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Install bounded transient request recovery.
|
||||
* Install provider-routed normal or unbounded request recovery.
|
||||
* @param ctx - plugin context that owns the listener and active waits.
|
||||
* @param config - retry budget, delay bounds, jitter, and eligible codes.
|
||||
* @param config - empty executor config; provider registrations own policy.
|
||||
* @param internals - non-serializable deterministic seams for tests.
|
||||
*/
|
||||
export function apply(ctx: Context, config: Config = {}, internals: RetryInternals = {}): void {
|
||||
const resolved = resolveConfig(config)
|
||||
validateConfig(config)
|
||||
const random = internals.random ?? Math.random
|
||||
const lifetime = new AbortController()
|
||||
const active = new Set<Promise<RequestErrorDecision>>()
|
||||
@@ -152,25 +130,40 @@ export function apply(ctx: Context, config: Config = {}, internals: RetryInterna
|
||||
turn: number,
|
||||
step: number,
|
||||
failure: LlmFailure,
|
||||
provider: string,
|
||||
policy: ResolvedRetryPolicy,
|
||||
retry: number,
|
||||
delayMs: number,
|
||||
signal: AbortSignal,
|
||||
): Promise<RequestErrorDecision> {
|
||||
const fusedSignal = AbortSignal.any([signal, lifetime.signal])
|
||||
if (fusedSignal.aborted) return { action: 'fail' }
|
||||
agent.session.append('llm/retry', {
|
||||
turn,
|
||||
step,
|
||||
retry,
|
||||
maxRetries: resolved.maxTransientRetries,
|
||||
delayMs,
|
||||
failure,
|
||||
})
|
||||
const eventData = policy.mode === 'normal'
|
||||
? {
|
||||
turn,
|
||||
step,
|
||||
provider,
|
||||
mode: policy.mode,
|
||||
retry,
|
||||
maxRetries: policy.maxRetries,
|
||||
delayMs,
|
||||
failure,
|
||||
}
|
||||
: {
|
||||
turn,
|
||||
step,
|
||||
provider,
|
||||
mode: policy.mode,
|
||||
retry,
|
||||
delayMs,
|
||||
failure,
|
||||
}
|
||||
agent.session.append('llm/retry', eventData)
|
||||
if (!await cancellableDelay(delayMs, fusedSignal)) return { action: 'fail' }
|
||||
return { action: 'retry' }
|
||||
}
|
||||
|
||||
const disposeListener = ctx.on('agent/request-error', (
|
||||
const disposeListener = ctx.on('agent/request-error', async (
|
||||
agent: Agent,
|
||||
turn: number,
|
||||
step: number,
|
||||
@@ -184,22 +177,61 @@ export function apply(ctx: Context, config: Config = {}, internals: RetryInterna
|
||||
// removed. Lifetime cancellation must prevent that stale callback from
|
||||
// entering a downstream policy after disposal.
|
||||
if (lifetime.signal.aborted) return Promise.resolve<RequestErrorDecision>({ action: 'fail' })
|
||||
if (!resolved.retryableCodes.has(failure.code)) return next()
|
||||
const priorTransientFailures = priorFailures.filter(item => resolved.retryableCodes.has(item.code)).length
|
||||
if (priorTransientFailures >= resolved.maxTransientRetries) return next()
|
||||
// Bind policy to the header in force when this step closed. Downstream
|
||||
// recovery may append later state before an always fallback runs.
|
||||
const provider = providerForClosedStep(agent.session.events, turn, step)
|
||||
/* v8 ignore next 3 -- agent-loop closes only steps whose request header was recorded */
|
||||
if (provider === undefined) {
|
||||
throw new Error(`llm-retry: no request provider for closed turn ${turn}/step ${step}`)
|
||||
}
|
||||
const policy = ctx.llm.providerRetryPolicy(provider)
|
||||
|
||||
const retry = priorTransientFailures + 1
|
||||
if (policy.mode === 'always') {
|
||||
const downstream = await downstreamUntilAbort(
|
||||
next,
|
||||
AbortSignal.any([signal, lifetime.signal]),
|
||||
)
|
||||
if (downstream.type === 'aborted') return { action: 'fail' }
|
||||
if (downstream.type === 'error') {
|
||||
ctx.logger.warn(
|
||||
`llm-retry: provider "${provider}" always policy ignored a downstream recovery failure: %o`,
|
||||
downstream.error,
|
||||
)
|
||||
}
|
||||
if (downstream.type === 'decision' && downstream.decision.action === 'retry') {
|
||||
return downstream.decision
|
||||
}
|
||||
} else if (!policy.retryableCodes.includes(failure.code)) {
|
||||
return next()
|
||||
}
|
||||
|
||||
const firstPriorStep = step - priorFailures.length
|
||||
const priorPolicyRetry = agent.session.events.findLast((event): event is SessionEvent<'llm/retry'> =>
|
||||
event.type === 'llm/retry'
|
||||
&& event.data.turn === turn
|
||||
&& event.data.step >= firstPriorStep
|
||||
&& event.data.step < step
|
||||
&& event.data.provider === provider
|
||||
&& event.data.mode === policy.mode,
|
||||
)
|
||||
const previousRetry = priorPolicyRetry?.data.retry ?? 0
|
||||
if (policy.mode === 'normal' && previousRetry >= policy.maxRetries) return next()
|
||||
const retry = previousRetry + 1
|
||||
let delayMs: number
|
||||
if (failure.providerRetryAfterMs !== undefined
|
||||
&& Number.isFinite(failure.providerRetryAfterMs)
|
||||
&& failure.providerRetryAfterMs > 0) {
|
||||
if (failure.providerRetryAfterMs > resolved.maxDelayMs) return next()
|
||||
delayMs = failure.providerRetryAfterMs
|
||||
if (failure.providerRetryAfterMs > policy.maxDelayMs) {
|
||||
if (policy.mode === 'normal') return next()
|
||||
delayMs = localDelay(policy, retry, random)
|
||||
} else {
|
||||
delayMs = failure.providerRetryAfterMs
|
||||
}
|
||||
} else {
|
||||
delayMs = localDelay(resolved, retry, random)
|
||||
delayMs = localDelay(policy, retry, random)
|
||||
}
|
||||
|
||||
const tracked = backoff(agent, turn, step, failure, retry, delayMs, signal)
|
||||
const tracked = backoff(agent, turn, step, failure, provider, policy, retry, delayMs, signal)
|
||||
.finally(() => active.delete(tracked))
|
||||
active.add(tracked)
|
||||
return tracked
|
||||
|
||||
@@ -4,6 +4,7 @@ import type { Context } from 'cordis'
|
||||
import type { Session, SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
import type { InvariantFailure, InvariantInstaller } from '@deepseek-ai/dsh-invariants'
|
||||
import { providerForClosedStep } from './history.ts'
|
||||
import type {} from './index.ts'
|
||||
|
||||
const PACKAGE_NAME = '@deepseek-ai/dsh-llm-retry'
|
||||
@@ -19,34 +20,46 @@ function validateRetry(
|
||||
event: SessionEvent<'llm/retry'>,
|
||||
fail: InvariantFailure,
|
||||
): void {
|
||||
const { turn, step, retry, maxRetries, delayMs } = event.data
|
||||
const { turn, step, provider, mode, retry, delayMs } = event.data
|
||||
if (!Number.isSafeInteger(retry) || retry < 1) {
|
||||
fail('llm/retry retry must be a positive safe integer')
|
||||
}
|
||||
if (!Number.isSafeInteger(maxRetries) || maxRetries < 1 || retry > maxRetries) {
|
||||
fail(`llm/retry retry ${retry} must not exceed a positive safe maxRetries ${maxRetries}`)
|
||||
if (typeof provider !== 'string' || provider.length === 0) {
|
||||
fail('llm/retry provider must be non-empty string')
|
||||
}
|
||||
if (!(delayMs >= 0 && delayMs <= MAX_TIMER_DELAY_MS)) {
|
||||
fail(`llm/retry delayMs must be within 0..${MAX_TIMER_DELAY_MS}`)
|
||||
}
|
||||
|
||||
const currentTurnEvents: SessionEvent[] = []
|
||||
let openTurn: number | undefined
|
||||
for (const prior of history.slice().reverse()) {
|
||||
if (prior.type === 'turn/end') fail('llm/retry must be appended inside an open turn')
|
||||
if (prior.type === 'turn/start') {
|
||||
openTurn = prior.data.turn
|
||||
switch (mode) {
|
||||
case 'normal': {
|
||||
const { maxRetries } = event.data
|
||||
if (!Number.isSafeInteger(maxRetries) || maxRetries < 1 || retry > maxRetries) {
|
||||
fail(`llm/retry retry ${retry} must not exceed a positive safe maxRetries ${maxRetries}`)
|
||||
}
|
||||
break
|
||||
}
|
||||
currentTurnEvents.push(prior)
|
||||
case 'always':
|
||||
if ('maxRetries' in event.data) fail('llm/retry always mode must omit maxRetries')
|
||||
break
|
||||
default:
|
||||
fail(`llm/retry mode must be normal or always, got ${String(mode)}`)
|
||||
}
|
||||
if (openTurn === undefined) fail('llm/retry must be appended inside an open turn')
|
||||
if (typeof delayMs !== 'number' || !Number.isFinite(delayMs)
|
||||
|| delayMs < 0 || delayMs > MAX_TIMER_DELAY_MS) {
|
||||
fail(`llm/retry delayMs must be a finite number within 0..${MAX_TIMER_DELAY_MS}`)
|
||||
}
|
||||
|
||||
const turnStartIndex = history.findLastIndex(prior =>
|
||||
prior.type === 'turn/start' || prior.type === 'turn/end')
|
||||
const turnBoundary = history[turnStartIndex]
|
||||
if (turnBoundary?.type !== 'turn/start') {
|
||||
fail('llm/retry must be appended inside an open turn')
|
||||
}
|
||||
const openTurn = turnBoundary.data.turn
|
||||
if (turn !== openTurn) {
|
||||
fail(`llm/retry names turn ${turn}, but the open turn is ${openTurn}`)
|
||||
}
|
||||
|
||||
const currentTurnEvents = history.slice(turnStartIndex + 1)
|
||||
let closedStep: number | undefined
|
||||
for (const prior of currentTurnEvents) {
|
||||
for (const prior of currentTurnEvents.slice().reverse()) {
|
||||
if (prior.type === 'step/start') {
|
||||
fail(`llm/retry must follow step/end, but step ${prior.data.step} is still open`)
|
||||
}
|
||||
@@ -58,15 +71,26 @@ function validateRetry(
|
||||
if (closedStep === undefined || step !== closedStep) {
|
||||
fail(`llm/retry names step ${step}, but the latest closed step is ${String(closedStep)}`)
|
||||
}
|
||||
const routedProvider = providerForClosedStep(history, turn, step)
|
||||
if (routedProvider !== provider) {
|
||||
fail(`llm/retry provider ${provider} does not match the failed request provider ${String(routedProvider)}`)
|
||||
}
|
||||
|
||||
const priorRetries = currentTurnEvents
|
||||
.filter((prior): prior is SessionEvent<'llm/retry'> => prior.type === 'llm/retry')
|
||||
if (priorRetries.some(prior => prior.data.step === step)) {
|
||||
fail(`llm/retry duplicates the retry record for turn ${turn}/step ${step}`)
|
||||
}
|
||||
const priorRetry = priorRetries[0]
|
||||
if (priorRetry !== undefined && retry <= priorRetry.data.retry) {
|
||||
fail(`llm/retry retry ${retry} must increase after retry ${priorRetry.data.retry}`)
|
||||
const lastSuccessIndex = currentTurnEvents.findLastIndex(prior => prior.type === 'assistant/message')
|
||||
const priorPolicyRetry = currentTurnEvents.findLast((prior, index): prior is SessionEvent<'llm/retry'> => (
|
||||
index > lastSuccessIndex
|
||||
&& prior.type === 'llm/retry'
|
||||
&& prior.data.provider === provider
|
||||
&& prior.data.mode === mode
|
||||
))
|
||||
const expectedRetry = (priorPolicyRetry?.data.retry ?? 0) + 1
|
||||
if (retry !== expectedRetry) {
|
||||
fail(`llm/retry retry ${retry} must equal provider policy retry ${expectedRetry}`)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@ import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
import InvariantService from '@deepseek-ai/dsh-invariants'
|
||||
import * as RetryInvariant from '@deepseek-ai/dsh-llm-retry/invariant'
|
||||
import { providerForClosedStep } from '../src/history.ts'
|
||||
|
||||
async function setup(): Promise<Context> {
|
||||
const ctx = new Context()
|
||||
@@ -17,33 +18,116 @@ function closeStep(ctx: Context, id: string, turn = 1, step = 1) {
|
||||
const session = ctx.sessions.create(SessionId(id))
|
||||
session.append('turn/start', { turn, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('step/start', { turn, step })
|
||||
session.append('request/header', {
|
||||
header: { config: { provider: 'mock', model: 'mock' } },
|
||||
reason: 'initial',
|
||||
})
|
||||
session.append('step/end', { turn, step })
|
||||
return session
|
||||
}
|
||||
|
||||
const failure = { message: 'provider busy', code: 'RATE_LIMIT', status: 429 }
|
||||
const normal = { provider: 'mock', mode: 'normal' as const }
|
||||
|
||||
describe('llm-retry invariants', () => {
|
||||
it('has no provider without the requested closed step', () => {
|
||||
expect(providerForClosedStep([], 1, 1)).toBeUndefined()
|
||||
expect(providerForClosedStep([{
|
||||
type: 'step/end',
|
||||
data: { turn: 1, step: 1 },
|
||||
}] as never, 1, 1)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('does not inherit a provider across a turn boundary', () => {
|
||||
expect(providerForClosedStep([
|
||||
{ type: 'turn/start', data: { turn: 1 } },
|
||||
{
|
||||
type: 'request/header',
|
||||
data: { header: { config: { provider: 'prior' } } },
|
||||
},
|
||||
{ type: 'turn/end', data: { turn: 1 } },
|
||||
{ type: 'turn/start', data: { turn: 2 } },
|
||||
{ type: 'step/end', data: { turn: 2, step: 1 } },
|
||||
] as never, 2, 1)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('accepts increasing retry records for successive closed steps and ignores unrelated events', async () => {
|
||||
const ctx = await setup()
|
||||
const session = closeStep(ctx, 'retry-invariant-valid')
|
||||
expect(() => {
|
||||
session.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 2, delayMs: 500, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 2, delayMs: 500, failure,
|
||||
})
|
||||
session.append('step/start', { turn: 1, step: 2 })
|
||||
session.append('step/end', { turn: 1, step: 2 })
|
||||
session.append('llm/retry', {
|
||||
turn: 1, step: 2, retry: 2, maxRetries: 2, delayMs: 1_000, failure,
|
||||
turn: 1, step: 2, ...normal, retry: 2, maxRetries: 2, delayMs: 1_000, failure,
|
||||
})
|
||||
const zeroDelay = closeStep(ctx, 'retry-invariant-zero-delay')
|
||||
zeroDelay.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 1, delayMs: 0, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 1, delayMs: 0, failure,
|
||||
})
|
||||
}).not.toThrow()
|
||||
expect(() => { ctx.emit('tools/change') }).not.toThrow()
|
||||
})
|
||||
|
||||
it('accepts unbounded always records without serializing an infinite maximum', async () => {
|
||||
const ctx = await setup()
|
||||
const session = closeStep(ctx, 'retry-invariant-always')
|
||||
expect(() => {
|
||||
session.append('llm/retry', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
provider: 'mock',
|
||||
mode: 'always',
|
||||
retry: 1,
|
||||
delayMs: 500,
|
||||
failure,
|
||||
})
|
||||
}).not.toThrow()
|
||||
expect(() => {
|
||||
session.append('llm/retry', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
provider: 'mock',
|
||||
mode: 'always',
|
||||
retry: 1,
|
||||
maxRetries: 2,
|
||||
delayMs: 500,
|
||||
failure,
|
||||
} as never)
|
||||
}).toThrow(/always mode must omit maxRetries/)
|
||||
})
|
||||
|
||||
it('rejects empty providers and unknown modes from hostile durable input', async () => {
|
||||
const ctx = await setup()
|
||||
const emptyProvider = closeStep(ctx, 'retry-invariant-empty-provider')
|
||||
expect(() => {
|
||||
emptyProvider.append('llm/retry', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
provider: '',
|
||||
mode: 'always',
|
||||
retry: 1,
|
||||
delayMs: 1,
|
||||
failure,
|
||||
})
|
||||
}).toThrow(/provider must be non-empty/)
|
||||
|
||||
const unknownMode = closeStep(ctx, 'retry-invariant-unknown-mode')
|
||||
expect(() => {
|
||||
unknownMode.append('llm/retry', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
provider: 'mock',
|
||||
mode: 'sometimes',
|
||||
retry: 1,
|
||||
delayMs: 1,
|
||||
failure,
|
||||
} as never)
|
||||
}).toThrow(/mode must be normal or always/)
|
||||
})
|
||||
|
||||
it.each([
|
||||
[{ retry: 0, maxRetries: 2, delayMs: 1 }, /positive safe integer/],
|
||||
[{ retry: 1.5, maxRetries: 2, delayMs: 1 }, /positive safe integer/],
|
||||
@@ -56,7 +140,7 @@ describe('llm-retry invariants', () => {
|
||||
const ctx = await setup()
|
||||
const session = closeStep(ctx, `retry-invariant-bounds-${data.retry}-${data.maxRetries}-${data.delayMs}`)
|
||||
expect(() => {
|
||||
session.append('llm/retry', { turn: 1, step: 1, ...data, failure })
|
||||
session.append('llm/retry', { turn: 1, step: 1, ...normal, ...data, failure })
|
||||
}).toThrow(message)
|
||||
})
|
||||
|
||||
@@ -65,14 +149,14 @@ describe('llm-retry invariants', () => {
|
||||
const absent = ctx.sessions.create(SessionId('retry-invariant-no-turn'))
|
||||
expect(() => {
|
||||
absent.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
})
|
||||
}).toThrow(/inside an open turn/)
|
||||
|
||||
const wrongTurn = closeStep(ctx, 'retry-invariant-wrong-turn')
|
||||
expect(() => {
|
||||
wrongTurn.append('llm/retry', {
|
||||
turn: 2, step: 1, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
turn: 2, step: 1, ...normal, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
})
|
||||
}).toThrow(/open turn is 1/)
|
||||
|
||||
@@ -81,7 +165,7 @@ describe('llm-retry invariants', () => {
|
||||
openStep.append('step/start', { turn: 1, step: 1 })
|
||||
expect(() => {
|
||||
openStep.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
})
|
||||
}).toThrow(/step 1 is still open/)
|
||||
|
||||
@@ -89,14 +173,14 @@ describe('llm-retry invariants', () => {
|
||||
noStep.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
expect(() => {
|
||||
noStep.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
})
|
||||
}).toThrow(/latest closed step is undefined/)
|
||||
|
||||
const wrongStep = closeStep(ctx, 'retry-invariant-wrong-step')
|
||||
expect(() => {
|
||||
wrongStep.append('llm/retry', {
|
||||
turn: 1, step: 2, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
turn: 1, step: 2, ...normal, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
})
|
||||
}).toThrow(/latest closed step is 1/)
|
||||
|
||||
@@ -104,34 +188,97 @@ describe('llm-retry invariants', () => {
|
||||
closedTurn.append('turn/end', { turn: 1, reason: { kind: 'aborted' } })
|
||||
expect(() => {
|
||||
closedTurn.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
})
|
||||
}).toThrow(/inside an open turn/)
|
||||
})
|
||||
|
||||
it('binds the policy provider to the failed step rather than a later header', async () => {
|
||||
const ctx = await setup()
|
||||
const session = closeStep(ctx, 'retry-invariant-provider')
|
||||
session.append('request/header', {
|
||||
header: { config: { provider: 'other', model: 'mock' } },
|
||||
reason: 'change',
|
||||
})
|
||||
expect(() => {
|
||||
session.append('llm/retry', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
provider: 'mock',
|
||||
mode: 'always',
|
||||
retry: 1,
|
||||
delayMs: 1,
|
||||
failure,
|
||||
})
|
||||
}).not.toThrow()
|
||||
|
||||
const mismatch = closeStep(ctx, 'retry-invariant-provider-mismatch')
|
||||
expect(() => {
|
||||
mismatch.append('llm/retry', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
provider: 'other',
|
||||
mode: 'always',
|
||||
retry: 1,
|
||||
delayMs: 1,
|
||||
failure,
|
||||
})
|
||||
}).toThrow(/does not match the failed request provider mock/)
|
||||
})
|
||||
|
||||
it('rejects a current-turn retry without a current-turn provider route', async () => {
|
||||
const ctx = await setup()
|
||||
const session = closeStep(ctx, 'retry-invariant-prior-route')
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
session.append('turn/start', { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('step/start', { turn: 2, step: 1 })
|
||||
session.append('step/end', { turn: 2, step: 1 })
|
||||
expect(() => {
|
||||
session.append('llm/retry', {
|
||||
turn: 2,
|
||||
step: 1,
|
||||
provider: 'mock',
|
||||
mode: 'always',
|
||||
retry: 1,
|
||||
delayMs: 1,
|
||||
failure,
|
||||
})
|
||||
}).toThrow(/does not match the failed request provider undefined/)
|
||||
})
|
||||
|
||||
it('rejects non-numeric durable delays', async () => {
|
||||
const ctx = await setup()
|
||||
const session = closeStep(ctx, 'retry-invariant-delay-type')
|
||||
expect(() => {
|
||||
session.append('llm/retry', {
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 2, delayMs: '1', failure,
|
||||
} as never)
|
||||
}).toThrow(/delayMs must be a finite number/)
|
||||
})
|
||||
|
||||
it('rejects duplicate and non-increasing retry records', async () => {
|
||||
const ctx = await setup()
|
||||
const duplicate = closeStep(ctx, 'retry-invariant-duplicate')
|
||||
duplicate.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 3, delayMs: 1, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 3, delayMs: 1, failure,
|
||||
})
|
||||
expect(() => {
|
||||
duplicate.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 2, maxRetries: 3, delayMs: 1, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 2, maxRetries: 3, delayMs: 1, failure,
|
||||
})
|
||||
}).toThrow(/duplicates the retry record/)
|
||||
|
||||
const nonIncreasing = closeStep(ctx, 'retry-invariant-non-increasing')
|
||||
nonIncreasing.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 3, delayMs: 1, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 3, delayMs: 1, failure,
|
||||
})
|
||||
nonIncreasing.append('step/start', { turn: 1, step: 2 })
|
||||
nonIncreasing.append('step/end', { turn: 1, step: 2 })
|
||||
expect(() => {
|
||||
nonIncreasing.append('llm/retry', {
|
||||
turn: 1, step: 2, retry: 1, maxRetries: 3, delayMs: 1, failure,
|
||||
turn: 1, step: 2, ...normal, retry: 1, maxRetries: 3, delayMs: 1, failure,
|
||||
})
|
||||
}).toThrow(/must increase/)
|
||||
}).toThrow(/must equal provider policy retry 2/)
|
||||
})
|
||||
|
||||
it('validates existing histories on late registration', async () => {
|
||||
@@ -140,7 +287,7 @@ describe('llm-retry invariants', () => {
|
||||
const session = ctx.sessions.create(SessionId('retry-invariant-late'))
|
||||
session.append('step/end', { turn: 1, step: 1 })
|
||||
session.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
})
|
||||
await ctx.plugin(InvariantService)
|
||||
await expect(ctx.plugin(RetryInvariant)).rejects.toThrow(/inside an open turn/)
|
||||
|
||||
@@ -9,8 +9,8 @@ import Include from '@cordisjs/plugin-include'
|
||||
import AgentRegistry from '@deepseek-ai/dsh-agent'
|
||||
import type { Agent } from '@deepseek-ai/dsh-agent'
|
||||
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
|
||||
import LlmService, { LlmAdapter, LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import LlmService, { LlmAdapter, LlmError, resolveRetryPolicy } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, ResolvedRetryPolicy, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
|
||||
import ToolRegistry from '@deepseek-ai/dsh-tools'
|
||||
@@ -21,6 +21,16 @@ let context: Context | undefined
|
||||
|
||||
class TransientOnceAdapter extends LlmAdapter {
|
||||
requests = 0
|
||||
private readonly retryPolicy = resolveRetryPolicy({
|
||||
mode: 'normal',
|
||||
maxRetries: 1,
|
||||
retryableCodes: ['RATE_LIMIT', 'SERVER'],
|
||||
backoff: { initialDelayMs: 1, maxDelayMs: 1, jitterRatio: 0 },
|
||||
}, 'loader test provider retryPolicy')
|
||||
|
||||
override providerRetryPolicy(_provider: string): ResolvedRetryPolicy {
|
||||
return this.retryPolicy
|
||||
}
|
||||
|
||||
async * stream(_options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
this.requests += 1
|
||||
@@ -87,7 +97,7 @@ describe('real Loader composition', () => {
|
||||
// Real-Loader composition resolves workspace packages through tsx at test
|
||||
// time; first resolution after the host/client program split is slow enough
|
||||
// to trip the default 5s budget on cold caches.
|
||||
it('loads the flat policy and records recovery through the shipping loop', { timeout: 60_000 }, async () => {
|
||||
it('loads provider-supplied policy and records recovery through the shipping loop', { timeout: 60_000 }, async () => {
|
||||
const loaded = await loadYaml([
|
||||
"- name: '@deepseek-ai/dsh-llm'",
|
||||
"- name: '@deepseek-ai/dsh-session'",
|
||||
@@ -95,12 +105,6 @@ describe('real Loader composition', () => {
|
||||
"- name: '@deepseek-ai/dsh-tools'",
|
||||
"- name: '@deepseek-ai/dsh-agent'",
|
||||
"- name: '@deepseek-ai/dsh-llm-retry'",
|
||||
' config:',
|
||||
' maxTransientRetries: 1',
|
||||
' initialDelayMs: 1',
|
||||
' maxDelayMs: 1',
|
||||
' jitterRatio: 0',
|
||||
' retryableCodes: [RATE_LIMIT, SERVER]',
|
||||
"- name: '@deepseek-ai/dsh-agent-loop'",
|
||||
])
|
||||
|
||||
|
||||
@@ -34,12 +34,17 @@ describe.each(['jsonl', 'sqlite'] as const)('%s retry-event persistence', (kind)
|
||||
const session = ctx.sessions.create(SessionId(`retry-${kind}`))
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
session.append('request/header', {
|
||||
header: { config: { provider: 'mock', model: 'mock' } },
|
||||
reason: 'initial',
|
||||
})
|
||||
session.append('step/end', { turn: 1, step: 1 })
|
||||
const event = session.append('llm/retry', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
provider: 'mock',
|
||||
mode: 'always',
|
||||
retry: 1,
|
||||
maxRetries: 2,
|
||||
delayMs: 750,
|
||||
failure: { message: 'provider busy', code: 'RATE_LIMIT', status: 429 },
|
||||
})
|
||||
|
||||
@@ -1,8 +1,16 @@
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import type { Fiber } from 'cordis'
|
||||
import LlmService, { CallId, LlmAdapter, LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import LlmService, { CallId, LlmAdapter, LlmError, resolveRetryPolicy } from '@deepseek-ai/dsh-llm'
|
||||
import type {
|
||||
AlwaysRetryPolicyConfig,
|
||||
BackoffConfig,
|
||||
GenerateOptions,
|
||||
NormalRetryPolicyConfig,
|
||||
ResolvedRetryPolicy,
|
||||
RetryPolicyConfig,
|
||||
StreamChunk,
|
||||
} from '@deepseek-ai/dsh-llm'
|
||||
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
|
||||
@@ -10,13 +18,13 @@ import ToolRegistry, { defineContentToolFixture } from '@deepseek-ai/dsh-tools'
|
||||
import AgentRegistry from '@deepseek-ai/dsh-agent'
|
||||
import type { Agent, RequestErrorDecision } from '@deepseek-ai/dsh-agent'
|
||||
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
import * as retry from '../src/index.ts'
|
||||
|
||||
type ScriptEntry = Error | Iterable<StreamChunk> | AsyncIterable<StreamChunk>
|
||||
|
||||
class ScriptedAdapter extends LlmAdapter {
|
||||
readonly requests: GenerateOptions[] = []
|
||||
private retryPolicies: Readonly<Record<string, ResolvedRetryPolicy | undefined>> = {}
|
||||
|
||||
constructor(private readonly entries: ScriptEntry[]) {
|
||||
super()
|
||||
@@ -29,6 +37,21 @@ class ScriptedAdapter extends LlmAdapter {
|
||||
if (entry instanceof Error) throw entry
|
||||
yield* entry
|
||||
}
|
||||
|
||||
configureRetryPolicies(
|
||||
policies: Readonly<Record<string, RetryPolicyConfig | undefined>>,
|
||||
): void {
|
||||
this.retryPolicies = Object.fromEntries(Object.entries(policies).map(([provider, policy]) => [
|
||||
provider,
|
||||
policy === undefined
|
||||
? undefined
|
||||
: resolveRetryPolicy(policy, `retry test provider "${provider}" retryPolicy`),
|
||||
]))
|
||||
}
|
||||
|
||||
override providerRetryPolicy(provider: string): ResolvedRetryPolicy | undefined {
|
||||
return this.retryPolicies[provider]
|
||||
}
|
||||
}
|
||||
|
||||
async function* partialToolFailure(error: Error): AsyncGenerator<StreamChunk> {
|
||||
@@ -52,8 +75,8 @@ function textResponse(text: string): StreamChunk[] {
|
||||
}
|
||||
|
||||
async function harness(
|
||||
adapter: LlmAdapter,
|
||||
config: retry.Config = {},
|
||||
adapter: ScriptedAdapter,
|
||||
policies: Readonly<Record<string, RetryPolicyConfig | undefined>> = { mock: normalConfig() },
|
||||
beforeRetry?: (ctx: Context) => void,
|
||||
internals: retry.RetryInternals = {},
|
||||
): Promise<{ ctx: Context; retryFiber: Fiber }> {
|
||||
@@ -64,20 +87,44 @@ async function harness(
|
||||
await ctx.plugin(ToolRegistry)
|
||||
await ctx.plugin(AgentRegistry)
|
||||
beforeRetry?.(ctx)
|
||||
const resolvedConfig = Object.assign({
|
||||
maxTransientRetries: 2,
|
||||
initialDelayMs: 500,
|
||||
maxDelayMs: 10_000,
|
||||
jitterRatio: 0,
|
||||
}, config)
|
||||
adapter.configureRetryPolicies(policies)
|
||||
const retryFiber = await ctx.plugin(Object.assign((inner: Context) => {
|
||||
retry.apply(inner, resolvedConfig, internals)
|
||||
retry.apply(inner, {}, internals)
|
||||
}, { inject: retry.inject }))
|
||||
await ctx.plugin(AgentLoop, { agents: [] })
|
||||
ctx.llm.registerAdapter(['mock'], adapter)
|
||||
ctx.llm.registerAdapter(['mock', 'other'], adapter)
|
||||
return { ctx, retryFiber }
|
||||
}
|
||||
|
||||
function normalConfig(
|
||||
overrides: Partial<Omit<NormalRetryPolicyConfig, 'mode'>> = {},
|
||||
): NormalRetryPolicyConfig {
|
||||
const { backoff, ...policy } = overrides
|
||||
return {
|
||||
mode: 'normal',
|
||||
maxRetries: 2,
|
||||
...policy,
|
||||
backoff: {
|
||||
initialDelayMs: 500,
|
||||
maxDelayMs: 10_000,
|
||||
jitterRatio: 0,
|
||||
...backoff,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function alwaysConfig(backoff: BackoffConfig = {}): AlwaysRetryPolicyConfig {
|
||||
return {
|
||||
mode: 'always',
|
||||
backoff: {
|
||||
initialDelayMs: 500,
|
||||
maxDelayMs: 10_000,
|
||||
jitterRatio: 0,
|
||||
...backoff,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function waitForIdle(ctx: Context, agent: Agent): Promise<void> {
|
||||
return new Promise((resolve) => {
|
||||
const dispose = ctx.on('agent/status', (subject, status) => {
|
||||
@@ -108,14 +155,14 @@ afterEach(async () => {
|
||||
context = undefined
|
||||
})
|
||||
|
||||
describe('bounded transient retry policy', () => {
|
||||
describe('provider-routed retry policy', () => {
|
||||
it('records the scheduled delay before opening a fresh request attempt', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('busy', 'RATE_LIMIT', { status: 429 }),
|
||||
textResponse('done'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter))
|
||||
;({ ctx: context } = await harness(adapter, {}, undefined, { random: () => 0.5 }))
|
||||
const agent = context.agentLoop.create(SessionId('retry-success'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
@@ -135,6 +182,8 @@ describe('bounded transient retry policy', () => {
|
||||
expect(event.data).toEqual({
|
||||
turn: 1,
|
||||
step: 1,
|
||||
provider: 'mock',
|
||||
mode: 'normal',
|
||||
retry: 1,
|
||||
maxRetries: 2,
|
||||
delayMs: 500,
|
||||
@@ -207,7 +256,9 @@ describe('bounded transient retry policy', () => {
|
||||
new LlmError('busy two', 'SERVER'),
|
||||
new LlmError('busy three', 'SERVER'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, { jitterRatio: 0.1 }, undefined, {
|
||||
;({ ctx: context } = await harness(adapter, { mock: normalConfig({
|
||||
backoff: { jitterRatio: 0.1 },
|
||||
}) }, undefined, {
|
||||
random: () => samples.shift() ?? 0.5,
|
||||
}))
|
||||
const agent = context.agentLoop.create(SessionId('retry-exhausted'), { provider: 'mock', model: 'mock' })
|
||||
@@ -238,11 +289,9 @@ describe('bounded transient retry policy', () => {
|
||||
new LlmError('busy', 'SERVER'),
|
||||
textResponse('done'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, {
|
||||
initialDelayMs: 1,
|
||||
maxDelayMs: 1,
|
||||
jitterRatio: 1,
|
||||
}, undefined, { random: () => 0 }))
|
||||
;({ ctx: context } = await harness(adapter, { mock: normalConfig({
|
||||
backoff: { initialDelayMs: 1, maxDelayMs: 1, jitterRatio: 1 },
|
||||
}) }, undefined, { random: () => 0 }))
|
||||
const agent = context.agentLoop.create(SessionId('retry-zero-delay'), { provider: 'mock', model: 'mock' })
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
|
||||
@@ -261,7 +310,9 @@ describe('bounded transient retry policy', () => {
|
||||
new LlmError('wait', 'RATE_LIMIT', { providerRetryAfterMs: 2_000 }),
|
||||
textResponse('done'),
|
||||
])
|
||||
;({ ctx: context } = await harness(accepted, { jitterRatio: 1 }))
|
||||
;({ ctx: context } = await harness(accepted, { mock: normalConfig({
|
||||
backoff: { jitterRatio: 1 },
|
||||
}) }))
|
||||
const acceptedAgent = context.agentLoop.create(SessionId('retry-after-accepted'), { provider: 'mock', model: 'mock' })
|
||||
const scheduled = waitForRetry(context, acceptedAgent, 1)
|
||||
acceptedAgent.send([{ type: 'text', text: 'go' }])
|
||||
@@ -284,6 +335,32 @@ describe('bounded transient retry policy', () => {
|
||||
expect(rejectedAgent.session.events.some(event => event.type === 'llm/retry')).toBe(false)
|
||||
})
|
||||
|
||||
it('uses local jittered backoff when always mode receives an over-cap Retry-After', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('wait too long', 'AUTH', { providerRetryAfterMs: 10 }),
|
||||
textResponse('done'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, { mock: alwaysConfig({
|
||||
initialDelayMs: 2,
|
||||
maxDelayMs: 4,
|
||||
jitterRatio: 0.5,
|
||||
}) }, undefined, { random: () => 1 }))
|
||||
const agent = context.agentLoop.create(SessionId('retry-always-over-cap'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
expect((await scheduled).data.delayMs).toBe(3)
|
||||
const idle = waitForIdle(context, agent)
|
||||
await vi.advanceTimersByTimeAsync(3)
|
||||
await idle
|
||||
|
||||
expect(adapter.requests).toHaveLength(2)
|
||||
})
|
||||
|
||||
it('delegates non-transient failures without scheduling a timer', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([new LlmError('bad key', 'AUTH')])
|
||||
@@ -297,13 +374,203 @@ describe('bounded transient retry policy', () => {
|
||||
expect(vi.getTimerCount()).toBe(0)
|
||||
})
|
||||
|
||||
it('selects policy by the failed request provider', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('mock auth failed', 'AUTH'),
|
||||
new LlmError('other auth failed', 'AUTH'),
|
||||
textResponse('other recovered'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, {
|
||||
other: alwaysConfig({ initialDelayMs: 1, maxDelayMs: 1, jitterRatio: 0 }),
|
||||
}))
|
||||
|
||||
const normalAgent = context.agentLoop.create(SessionId('retry-provider-normal'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const normalIdle = waitForIdle(context, normalAgent)
|
||||
normalAgent.send([{ type: 'text', text: 'normal' }])
|
||||
await normalIdle
|
||||
expect(normalAgent.session.events.some(event => event.type === 'llm/retry')).toBe(false)
|
||||
|
||||
const alwaysAgent = context.agentLoop.create(SessionId('retry-provider-always'), {
|
||||
provider: 'other',
|
||||
model: 'mock',
|
||||
})
|
||||
const scheduled = waitForRetry(context, alwaysAgent, 1)
|
||||
alwaysAgent.send([{ type: 'text', text: 'always' }])
|
||||
expect((await scheduled).data).toMatchObject({
|
||||
provider: 'other',
|
||||
mode: 'always',
|
||||
retry: 1,
|
||||
delayMs: 1,
|
||||
})
|
||||
const alwaysIdle = waitForIdle(context, alwaysAgent)
|
||||
await vi.advanceTimersByTimeAsync(1)
|
||||
await alwaysIdle
|
||||
|
||||
expect(adapter.requests.map(request => request.provider)).toEqual(['mock', 'other', 'other'])
|
||||
})
|
||||
|
||||
it('selects an always policy from the provider chosen by agent/request', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('rerouted auth failed', 'AUTH'),
|
||||
textResponse('rerouted recovery'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, {
|
||||
other: alwaysConfig({ initialDelayMs: 1, maxDelayMs: 1, jitterRatio: 0 }),
|
||||
}, (ctx) => {
|
||||
ctx.on('agent/request', async (_agent, _turn, _step, config) => ({
|
||||
...config,
|
||||
provider: 'other',
|
||||
}))
|
||||
}))
|
||||
const agent = context.agentLoop.create(SessionId('retry-provider-rerouted'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
|
||||
agent.send([{ type: 'text', text: 'reroute' }])
|
||||
expect((await scheduled).data).toMatchObject({ provider: 'other', mode: 'always' })
|
||||
const idle = waitForIdle(context, agent)
|
||||
await vi.advanceTimersByTimeAsync(1)
|
||||
await idle
|
||||
|
||||
expect(adapter.requests.map(request => request.provider)).toEqual(['other', 'other'])
|
||||
})
|
||||
|
||||
it('keeps always mode unbounded while preserving cancellable jittered backoff', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('auth one', 'AUTH'),
|
||||
new LlmError('auth two', 'AUTH'),
|
||||
new LlmError('auth three', 'AUTH'),
|
||||
new LlmError('auth four', 'AUTH'),
|
||||
textResponse('eventually recovered'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, { mock: alwaysConfig({
|
||||
initialDelayMs: 1,
|
||||
maxDelayMs: 4,
|
||||
jitterRatio: 0.1,
|
||||
}) }, undefined, { random: () => 1 }))
|
||||
const agent = context.agentLoop.create(SessionId('retry-always-unbounded'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const idle = waitForIdle(context, agent)
|
||||
|
||||
agent.send([{ type: 'text', text: 'keep trying' }])
|
||||
await vi.runAllTimersAsync()
|
||||
await idle
|
||||
|
||||
const events = agent.session.events.filter(event => event.type === 'llm/retry')
|
||||
expect(adapter.requests).toHaveLength(5)
|
||||
expect(events.map(event => ({
|
||||
provider: event.data.provider,
|
||||
mode: event.data.mode,
|
||||
retry: event.data.retry,
|
||||
delayMs: event.data.delayMs,
|
||||
hasMax: 'maxRetries' in event.data,
|
||||
}))).toEqual([
|
||||
{ provider: 'mock', mode: 'always', retry: 1, delayMs: 1.1, hasMax: false },
|
||||
{ provider: 'mock', mode: 'always', retry: 2, delayMs: 2.2, hasMax: false },
|
||||
{ provider: 'mock', mode: 'always', retry: 3, delayMs: 4, hasMax: false },
|
||||
{ provider: 'mock', mode: 'always', retry: 4, delayMs: 4, hasMax: false },
|
||||
])
|
||||
})
|
||||
|
||||
it('keeps failed error text and partial output out of every retried model context', async () => {
|
||||
vi.useFakeTimers()
|
||||
const diagnostic = 'private provider diagnostic must not enter context'
|
||||
const adapter = new ScriptedAdapter([
|
||||
partialToolFailure(new LlmError(diagnostic, 'AUTH')),
|
||||
textResponse('recovered without leaked context'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, { mock: alwaysConfig({
|
||||
initialDelayMs: 1,
|
||||
maxDelayMs: 1,
|
||||
}) }))
|
||||
const agent = context.agentLoop.create(SessionId('retry-always-context-isolation'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
|
||||
agent.send([{ type: 'text', text: 'safe input' }])
|
||||
await scheduled
|
||||
const idle = waitForIdle(context, agent)
|
||||
await vi.advanceTimersByTimeAsync(1)
|
||||
await idle
|
||||
|
||||
expect(adapter.requests).toHaveLength(2)
|
||||
expect(adapter.requests[1]?.messages).toEqual(adapter.requests[0]?.messages)
|
||||
const retriedContext = JSON.stringify(adapter.requests[1]?.messages)
|
||||
expect(retriedContext).not.toContain(diagnostic)
|
||||
expect(retriedContext).not.toContain('discarded partial output')
|
||||
expect(agent.session.events.some(event =>
|
||||
event.type === 'llm/retry' && event.data.failure.message === diagnostic,
|
||||
)).toBe(true)
|
||||
})
|
||||
|
||||
it('lets downstream specialized recovery run before always fallback', async () => {
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('requires specialized recovery', 'AUTH'),
|
||||
textResponse('specialized recovery won'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, { mock: alwaysConfig() }))
|
||||
context.on('agent/request-error', async () => ({ action: 'retry' }))
|
||||
const agent = context.agentLoop.create(SessionId('retry-always-composition'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const idle = waitForIdle(context, agent)
|
||||
|
||||
agent.send([{ type: 'text', text: 'recover' }])
|
||||
await idle
|
||||
|
||||
expect(adapter.requests).toHaveLength(2)
|
||||
expect(agent.session.events.some(event => event.type === 'llm/retry')).toBe(false)
|
||||
})
|
||||
|
||||
it.each([
|
||||
['synchronously', () => { throw new Error('downstream recovery failed') }],
|
||||
['asynchronously', async () => { throw new Error('downstream recovery failed') }],
|
||||
])('falls back to always retry when downstream recovery throws %s', async (_kind, failDownstream) => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('requires fallback', 'AUTH'),
|
||||
textResponse('always recovered'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, { mock: alwaysConfig({
|
||||
initialDelayMs: 1,
|
||||
maxDelayMs: 1,
|
||||
}) }))
|
||||
context.on('agent/request-error', failDownstream)
|
||||
const agent = context.agentLoop.create(SessionId('retry-always-downstream-error'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
|
||||
agent.send([{ type: 'text', text: 'recover' }])
|
||||
await scheduled
|
||||
const idle = waitForIdle(context, agent)
|
||||
await vi.advanceTimersByTimeAsync(1)
|
||||
await idle
|
||||
|
||||
expect(adapter.requests).toHaveLength(2)
|
||||
})
|
||||
|
||||
it('aborts and drains a captured backoff before plugin disposal completes', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('temporary', 'TRANSPORT'),
|
||||
textResponse('must not run'),
|
||||
])
|
||||
const mounted = await harness(adapter)
|
||||
const mounted = await harness(adapter, { mock: alwaysConfig() })
|
||||
context = mounted.ctx
|
||||
const agent = context.agentLoop.create(SessionId('retry-hmr'), { provider: 'mock', model: 'mock' })
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
@@ -322,7 +589,7 @@ describe('bounded transient retry policy', () => {
|
||||
|
||||
it('does not make plugin disposal wait for a delegated recovery policy', async () => {
|
||||
const adapter = new ScriptedAdapter([new LlmError('bad key', 'AUTH')])
|
||||
const mounted = await harness(adapter)
|
||||
const mounted = await harness(adapter, { mock: alwaysConfig() })
|
||||
context = mounted.ctx
|
||||
const downstream = Promise.withResolvers<RequestErrorDecision>()
|
||||
const entered = Promise.withResolvers<undefined>()
|
||||
@@ -353,6 +620,61 @@ describe('bounded transient retry policy', () => {
|
||||
expect(adapter.requests).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('lets turn cancellation interrupt a delegated recovery policy', async () => {
|
||||
const adapter = new ScriptedAdapter([new LlmError('bad key', 'AUTH')])
|
||||
const mounted = await harness(adapter, { mock: alwaysConfig() })
|
||||
context = mounted.ctx
|
||||
const downstream = Promise.withResolvers<RequestErrorDecision>()
|
||||
const entered = Promise.withResolvers<undefined>()
|
||||
context.on('agent/request-error', () => {
|
||||
entered.resolve(undefined)
|
||||
return downstream.promise
|
||||
})
|
||||
const agent = context.agentLoop.create(SessionId('retry-delegated-cancel'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const idle = waitForIdle(context, agent)
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
await entered.promise
|
||||
|
||||
agent.cancel({ kind: 'user' })
|
||||
await idle
|
||||
downstream.resolve({ action: 'fail' })
|
||||
|
||||
expect(adapter.requests).toHaveLength(1)
|
||||
expect(agent.session.events.at(-1)).toMatchObject({
|
||||
type: 'turn/end',
|
||||
data: { reason: { kind: 'aborted' } },
|
||||
})
|
||||
})
|
||||
|
||||
it('handles synchronous cancellation while entering delegated recovery', async () => {
|
||||
const adapter = new ScriptedAdapter([new LlmError('bad key', 'AUTH')])
|
||||
const mounted = await harness(adapter, { mock: alwaysConfig() })
|
||||
context = mounted.ctx
|
||||
const downstream = Promise.withResolvers<RequestErrorDecision>()
|
||||
context.on('agent/request-error', (agent) => {
|
||||
agent.cancel({ kind: 'user' })
|
||||
return downstream.promise
|
||||
})
|
||||
const agent = context.agentLoop.create(SessionId('retry-delegated-sync-cancel'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const idle = waitForIdle(context, agent)
|
||||
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
await idle
|
||||
downstream.resolve({ action: 'fail' })
|
||||
|
||||
expect(adapter.requests).toHaveLength(1)
|
||||
expect(agent.session.events.at(-1)).toMatchObject({
|
||||
type: 'turn/end',
|
||||
data: { reason: { kind: 'aborted' } },
|
||||
})
|
||||
})
|
||||
|
||||
it('fails a captured callback after disposal without entering downstream policy', async () => {
|
||||
const adapter = new ScriptedAdapter([new LlmError('bad key', 'AUTH')])
|
||||
const captured = Promise.withResolvers<undefined>()
|
||||
@@ -391,10 +713,10 @@ describe('bounded transient retry policy', () => {
|
||||
it('lets turn cancellation win during backoff without opening another step', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('temporary', 'TIMEOUT'),
|
||||
new LlmError('permanent', 'AUTH'),
|
||||
textResponse('must not run'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter))
|
||||
;({ ctx: context } = await harness(adapter, { mock: alwaysConfig() }))
|
||||
const agent = context.agentLoop.create(SessionId('retry-cancel'), { provider: 'mock', model: 'mock' })
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
@@ -411,13 +733,16 @@ describe('bounded transient retry policy', () => {
|
||||
expect(vi.getTimerCount()).toBe(0)
|
||||
})
|
||||
|
||||
it('lets an earlier recovery listener cancel before retry policy runs', async () => {
|
||||
it.each([
|
||||
['normal', normalConfig()],
|
||||
['always', alwaysConfig()],
|
||||
])('lets an earlier recovery listener cancel before %s retry policy runs', async (_mode, policy) => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('temporary', 'SERVER'),
|
||||
textResponse('must not run'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, {}, (ctx) => {
|
||||
;({ ctx: context } = await harness(adapter, { mock: policy }, (ctx) => {
|
||||
ctx.on('agent/request-error', async (agent, _turn, _step, _error, _failure, _history, _signal, next) => {
|
||||
agent.cancel({ kind: 'user' })
|
||||
return next()
|
||||
@@ -458,19 +783,15 @@ describe('bounded transient retry policy', () => {
|
||||
expect(vi.getTimerCount()).toBe(0)
|
||||
})
|
||||
|
||||
it.each([
|
||||
[{ maxTransientRetries: -1 }, /maxTransientRetries/],
|
||||
[{ maxTransientRetries: 1.5 }, /maxTransientRetries/],
|
||||
[{ initialDelayMs: 0 }, /initialDelayMs/],
|
||||
[{ maxDelayMs: Number.POSITIVE_INFINITY }, /maxDelayMs/],
|
||||
[{ initialDelayMs: MAX_TIMER_DELAY_MS + 1 }, /initialDelayMs/],
|
||||
[{ maxDelayMs: MAX_TIMER_DELAY_MS + 1 }, /maxDelayMs/],
|
||||
[{ initialDelayMs: 20, maxDelayMs: 10 }, /less than or equal/],
|
||||
[{ jitterRatio: 1.1 }, /jitterRatio/],
|
||||
[{ retryableCodes: [] }, /must not be empty/],
|
||||
[{ retryableCodes: ['SERVER', 'SERVER'] }, /duplicates/],
|
||||
[{ retryableCodes: [''] }, /non-empty strings/],
|
||||
] as const)('fails direct composition for invalid config %#', (config, message) => {
|
||||
expect(() => { retry.apply(new Context(), config as retry.Config) }).toThrow(message)
|
||||
it('rejects retry policy configured on the executor instead of a provider', () => {
|
||||
expect(() => {
|
||||
retry.apply(new Context(), { retryPolicy: { mode: 'always' } })
|
||||
}).toThrow(/retryPolicy belongs under each provider/)
|
||||
})
|
||||
|
||||
it('rejects unknown executor config', () => {
|
||||
expect(() => {
|
||||
retry.apply(new Context(), { retryPolciy: {} })
|
||||
}).toThrow(/unknown key "retryPolciy"/)
|
||||
})
|
||||
})
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
import { defineConfig } from 'tsdown'
|
||||
|
||||
/** Build the package root and invariant companion as independent bundles. */
|
||||
export default defineConfig([
|
||||
{
|
||||
entry: ['lib/types/index.js'],
|
||||
outDir: 'lib',
|
||||
format: ['esm'],
|
||||
platform: 'node',
|
||||
target: 'es2024',
|
||||
fixedExtension: false,
|
||||
dts: false,
|
||||
clean: false,
|
||||
},
|
||||
{
|
||||
entry: ['lib/types/invariant.js'],
|
||||
outDir: 'lib',
|
||||
format: ['esm'],
|
||||
platform: 'node',
|
||||
target: 'es2024',
|
||||
fixedExtension: false,
|
||||
dts: false,
|
||||
clean: false,
|
||||
},
|
||||
])
|
||||
Reference in New Issue
Block a user