Tianyi Cui
e2556c51bf
The reconnect supervisor used connection lost for every transition into backoff, including an initial startup attempt that never established a connection and later retry attempts that also failed. That wording implied a previously healthy generation and obscured whether any tools had ever been registered. Capture whether the generation had reached the established state before scheduling recovery. Established disconnects retain connection lost/reconnecting; startup and retry failures now report connection failed/retrying. The reconnect-disabled diagnostic uses the same distinction while preserving its concrete manual-recovery guidance. Unit assertions cover established loss, initial failure, retry failure, and both reconnect-disabled branches. Focused package coverage remains 100%, and the bilingual Agent Note records the observable state vocabulary.
8.4 KiB
Agent Note: MCP client auto-reconnect with bounded backoff
Status: implemented
English | 中文
Problem
The MCP client connected once at plugin load. When a stdio server crashed or was killed, its registered tools stayed visible but every call failed with Not connected until a human edited the config (HMR) or restarted the Host — v1 explicitly deferred reconnection. Long-running hosts (ACP automation, web) cannot be bounced because a child process died, and for stdio the harness composition is the only party that can respawn it. External feedback escalated this as a real operational gap (issue #1746).
Decision
packages/mcp/mcp-client/src/connection.ts owns a per-instance connection supervisor; apply() shrinks to config resolution plus two effects (the serverName reservation and the supervisor's lifecycle). The supervisor owns the client/transport generations, the live tool registrations, and the reconnect loop.
Trigger. The supervisor arms client.onclose per generation. The SDK fires it when the stdio child exits, so a crash is observed without polling. StreamableHTTPClientTransport fires onclose only for deliberate closes — it owns its internal SSE-stream recovery and surfaces request failures per call — so HTTP servers are effectively outside supervisor restarts; the package README records that limitation.
Generations without interleaving. Each attempt builds a fresh transport and Client (the SDK binds a Protocol to one transport for life). One per-supervisor queue serializes every syncTools call — initial syncs and list_changed re-syncs across all generations — and an isCurrent fence makes stale generations inert, so no two syncs can interleave the dispose-previous/register-next swap (which would double-dispose one generation and leak another). The queue also closes a pre-existing race where two rapid list_changed notifications re-synced concurrently. The activation attempt, rather than the first queue entrant, explicitly owns strict startup registration: an early list_changed notification uses contained re-sync semantics and cannot consume failOnStartupError. Failure signals are idempotent per generation: a connect rejection racing its own transport close schedules exactly one retry. A failed attempt cannot enter backoff until both Client.close() settles and the transport reports onclose, which for stdio proves the child exited; a missing close signal stops reconnection after the SDK's bounded termination window instead of allowing two server processes to overlap. Disposal uses the same bounded close-signal barrier and reports an incomplete shutdown without ever restarting.
Bounded backoff with an outage budget. Delays double from initialDelayMs up to maxDelayMs. One outage shares maxAttempts consecutive failed attempts; exhaustion unregisters the server's tools, logs at error level, and stops until disposal or reload. A connection that survives past the stability window — maxDelayMs, derived rather than a fifth tunable, as the longest configured backoff spacing — resets the budget, so an occasionally-crashing server recovers indefinitely while a crash loop whose connects briefly succeed cannot launder its budget into a restart storm.
Config and resolution. Both transports accept reconnect { enabled, initialDelayMs, maxDelayMs, maxAttempts } with schemastery defaults (on, 500ms, 30s, 10). resolveReconnectPolicy() is the explicit resolve step: it re-judges every bound and cross-field constraint because programmatic construction may bypass Schemastery, and misconfiguration fails the plugin instance at load.
Observable states. An initial or retry-attempt failure says connection failed; an established generation ending says connection lost. Retrying logs at warn with attempt count and delay, recovery at info, final failure and disabled recovery at error. During an outage the last good generation stays registered and calls against it fail — deterministic public names mean a recovered unchanged tool list reproduces identical definitions, keeping the model-visible schema prefix stable instead of flapping. With reconnect.enabled: false a lost connection keeps the v1 manual-recovery behavior.
Disposal. Dispose flips the fence, cancels any pending timer, closes the current client, then awaits the in-flight attempt and the sync queue before unregistering — quiescence, not just a request to stop. The reconnect timer is unref'd so a waiting backoff never holds a finishing process open.
Alternatives considered
Consecutive-failure counter that resets on every successful connect. Rejected: a crash-looping server whose connects briefly succeed would reset the budget each cycle and restart forever — exactly the restart storm the failure cap exists to prevent. The uptime-gated reset distinguishes a recovered server from a looping one without new configuration.
Reuse one SDK Client across reconnects. The Protocol clears its transport on close and can technically connect again, but the SDK's own guidance is one connection per Protocol instance, and reuse carries notification handlers and negotiated capability state across server incarnations. A fresh Client per generation plus the isCurrent fence is unambiguous.
Unregister tools immediately on disconnect, re-register on recovery. Rejected: a transient outage would flap the model-visible tool list (two schema-prefix invalidations per crash) for no information gain; failing calls already signal the outage, and the swap on recovery is atomic per generation. Tools are unregistered at final failure so a permanently dead server does not leak permanently broken tools.
Route Streamable HTTP request failures into the supervisor. Rejected for now: the HTTP transport already reconnects its SSE stream with its own backoff, per-request errors do not imply a dead server, and there is no child process the harness could respawn. Transport close stays the single trigger.
Restart through Loader/HMR machinery instead of an in-plugin supervisor. Rejected: the Loader owns config-driven recomposition, not runtime health. A plugin restarting itself through the Loader would conflate config generations with connection generations and lose the per-outage budget.
Testing
Unit (tests/reconnect.spec.ts, mocked SDK): recovery swaps generations without duplication or leaks and serves post-recovery calls, diagnostics distinguish initial or retry failure from established connection loss, strict startup registration survives a pre-connect list_changed notification, failed initialization waits for the old generation's close signal and fails closed when that signal never arrives, disposal waits for the same signal with a bounded incomplete-shutdown path, the failure cap unregisters tools and stops, dispose cancels a pending backoff and quiesces an in-flight sync, a close after dispose schedules nothing, disabled mode keeps the v1 behavior, the stability window resets the budget while a crash loop exhausts it, double failure signals schedule one retry, stale generations and handlers are inert, and resolveReconnectPolicy rejects each invalid bound. E2E (tests/mcp-client.e2e.ts, keyless): the fixture server gained a crash tool that replies then exits; real-process tests prove a stdio crash recovers end to end and that unloading the plugin mid-outage stops reconnection promptly. Snapshot: deliberately none, per the original note's rationale — reconnection adds no new presentation shape, and a snapshot composition spawning a crashing server would make replays timing-dependent.
Consequences
- A crashed stdio MCP server recovers without human intervention: bounded backoff, re-discovery, atomic generation swap. Default policy retries an outage for roughly 2.5 minutes before giving up.
- Connection state is genuinely more intricate than connect-once — the partial-availability window v1 avoided now exists (registered tools failing during an outage), concentrated in one module with the invariants named.
reconnectis new config surface on both transports, and the stability window is deliberately derived frommaxDelayMs; making it independently tunable is a compatible future change.- After final failure or with reconnect disabled, the plugin stays loaded with no (or failing) tools until reload — deliberate and logged, so a chronically broken server cannot restart forever.