Files

T

Pine f5f114dfb8 docs(vision): package READMEs, Model Experience, and the capability Agent Note

Document the four image-recognition packages and the vision group, register
the vision domain in the package index, add the capability decision as an
Agent Note, declare the invariants peer dep, and drop the unused agents
injection from the consumer.

Co-Authored-By: Claude <noreply@anthropic.com>

2026-08-14 11:15:59 +08:00

3.8 KiB

Raw Blame History

Agent Note: Image-recognition capability (opt-in bundle)

Status: implemented

English | 中文

Problem

The harness had no way to recognize image contents as a first-class step of a task. A multimodal model can read_image, but nothing detects an image task and routes it through a user-configured recognition provider first. The user wanted an image-capability plugin, modeled on the web-search provider: fill in a provider address + API key in the settings page, and have the model recognize an image before continuing.

Decision

Add a complete capability seam over four new packages under packages/vision/, mirroring the web-search three-role shape (Service Definition / Provider / Consumer), plus an opt-in bundle:

Service Definition @deepseek-ai/dsh-image-recognition: ctx.imageRecognition provider registry + provider-selecting recognize() (mirrors ctx.web): duplicate ids rejected, order-independent selection, ImageRecognitionError taxonomy.
Provider @deepseek-ai/dsh-image-recognition-http: a user-configurable OpenAI-compatible chat-completions vision provider. baseURL + model live in a live settings section; the key rides the credentials domain via a credential-ref. Logs the secret-free vision request as the image-recognition/llm-request session event.
Consumer @deepseek-ai/dsh-tool-image-recognition: registers a recognize_image tool and a bundled image-recognition skill, and hooks agent/pre-step to deterministically inject the skill body when a step input carries an image (an image content block or an image file path/URL), so the model recognizes the image before other actions.
Bundle @deepseek-ai/dsh-image-recognition-bundle: mounts the three above; opt-in — it is not in any default profile's bundles list.
Settings card: an image-recognition card in the Plugins settings page (endpoint + key), cloned from the web-search card.

The mechanism is deterministic-injection-plus-model-driven-execution: the pre-step injection guarantees the skill instructions are present, while the model still drives the actual recognize_image call (consistent with every existing plugin — the model is the final decider of tool use).

Enabling

A user opts in per profile by adding @deepseek-ai/dsh-image-recognition-bundle to the profile's dsh.profile.bundles (or dsh plugin --profile <name> add ...), then configures the endpoint and key in the Plugins settings page.

Verification

packages/vision/* unit tests: SD selection semantics, provider HTTP error/abort mapping, and the image-signal detection function.
Full host program typechecks.
Real use: opt in the bundle, attach an image, confirm recognize_image runs and the task continues from its result.

Alternatives considered

Built-in multimodal read_image as the mechanism. Rejected: the user wanted a configurable provider (address + key) surfaced in the settings page, modeled on web-search, not a fixed multimodal path.
Prompt-guidance only (register the skill, let the model self-route). Rejected: the user asked for a deterministic guarantee that the model recognizes an image first; the pre-step injection delivers that while the model still drives the actual tool call.

Consequences

Costs: a new capability seam of four packages plus a client card; the opt-in bundle needs a configured provider before it works; image-task path/URL detection is heuristic and can be disabled.
Buys: recognition becomes a first-class step with a provider the user controls, the harness follows the capability-seam and deterministic-injection-plus-model-driven-execution patterns, and the feature is opt-in per profile (no default-behavior change).

3.8 KiB Raw Blame History