docs(vision): package READMEs, Model Experience, and the capability Agent Note

Document the four image-recognition packages and the vision group, register the vision domain in the package index, add the capability decision as an Agent Note, declare the invariants peer dep, and drop the unused agents injection from the consumer. Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-14 11:15:59 +08:00
parent 19b39bc2e0
commit f5f114dfb8
12 changed files with 342 additions and 1 deletions
@@ -0,0 +1,6 @@
+# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
+# side as of the last confirmed-consistent state. Both languages carry equal authority;
+# after editing either side, bring the other along and re-record with:
+#   pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-08-14-image-recognition-capability.md
+2026-08-14-image-recognition-capability.md: bd8fe3406e200b23f9d25f54ca71fcf7c081e1e6
+2026-08-14-image-recognition-capability.zh.md: c958d6441f96f7c8b96cf3c6c3a4e8783a00fb58
@@ -0,0 +1,78 @@
+# Agent Note: Image-recognition capability (opt-in bundle)
+
+Status: implemented
+
+English | [中文](2026-08-14-image-recognition-capability.zh.md)
+
+## Problem
+
+The harness had no way to recognize image contents as a first-class step of a
+task. A multimodal model can `read_image`, but nothing detects an image task and
+routes it through a user-configured recognition provider first. The user wanted
+an image-capability plugin, modeled on the web-search provider: fill in a
+provider address + API key in the settings page, and have the model recognize an
+image before continuing.
+
+## Decision
+
+Add a complete capability seam over four new packages under `packages/vision/`,
+mirroring the web-search three-role shape (Service Definition / Provider /
+Consumer), plus an opt-in bundle:
+
+- **Service Definition** `@deepseek-ai/dsh-image-recognition`: `ctx.imageRecognition`
+  provider registry + provider-selecting `recognize()` (mirrors `ctx.web`):
+  duplicate ids rejected, order-independent selection,
+  `ImageRecognitionError` taxonomy.
+- **Provider** `@deepseek-ai/dsh-image-recognition-http`: a user-configurable
+  OpenAI-compatible chat-completions vision provider. `baseURL` + `model` live in
+  a live settings section; the key rides the credentials domain via a
+  credential-ref. Logs the secret-free vision request as the
+  `image-recognition/llm-request` session event.
+- **Consumer** `@deepseek-ai/dsh-tool-image-recognition`: registers a
+  `recognize_image` tool and a bundled `image-recognition` skill, and hooks
+  `agent/pre-step` to **deterministically inject** the skill body when a step
+  input carries an image (an image content block or an image file path/URL), so
+  the model recognizes the image before other actions.
+- **Bundle** `@deepseek-ai/dsh-image-recognition-bundle`: mounts the three above;
+  **opt-in** — it is not in any default profile's bundles list.
+- **Settings card**: an `image-recognition` card in the Plugins settings page
+  (endpoint + key), cloned from the web-search card.
+
+The mechanism is deterministic-injection-plus-model-driven-execution: the
+pre-step injection guarantees the skill instructions are present, while the model
+still drives the actual `recognize_image` call (consistent with every existing
+plugin — the model is the final decider of tool use).
+
+## Enabling
+
+A user opts in per profile by adding `@deepseek-ai/dsh-image-recognition-bundle`
+to the profile's `dsh.profile.bundles` (or `dsh plugin --profile <name> add
+...`), then configures the endpoint and key in the Plugins settings page.
+
+## Verification
+
+- `packages/vision/*` unit tests: SD selection semantics, provider HTTP
+  error/abort mapping, and the image-signal detection function.
+- Full host program typechecks.
+- Real use: opt in the bundle, attach an image, confirm `recognize_image` runs
+  and the task continues from its result.
+
+## Alternatives considered
+
+- **Built-in multimodal `read_image` as the mechanism.** Rejected: the user
+  wanted a configurable provider (address + key) surfaced in the settings page,
+  modeled on web-search, not a fixed multimodal path.
+- **Prompt-guidance only (register the skill, let the model self-route).**
+  Rejected: the user asked for a deterministic guarantee that the model
+  recognizes an image first; the pre-step injection delivers that while the model
+  still drives the actual tool call.
+
+## Consequences
+
+- **Costs:** a new capability seam of four packages plus a client card; the
+  opt-in bundle needs a configured provider before it works; image-task path/URL
+  detection is heuristic and can be disabled.
+- **Buys:** recognition becomes a first-class step with a provider the user
+  controls, the harness follows the capability-seam and
+  deterministic-injection-plus-model-driven-execution patterns, and the feature
+  is opt-in per profile (no default-behavior change).
@@ -0,0 +1,62 @@
+# Agent Note：图像识别能力（可选 bundle）
+
+Status: implemented
+
+English | [中文](2026-08-14-image-recognition-capability.md)
+
+## 问题
+
+harness 此前无法把"识别图像内容"作为任务的一等步骤。多模态模型可以 `read_image`，
+但没有东西检测图像任务、并通过用户配置的识别提供方先路由它。用户想要一个仿网页
+搜索的图像能力插件：在设置页填写供应商地址 + API Key，让模型先识别图像再继续任务。
+
+## 决策
+
+在 `packages/vision/` 下新增四个包，构成完整能力缝，镜像 web-search 的三角色形态
+（Service Definition / Provider / Consumer），外加一个可选 bundle：
+
+- **Service Definition** `@deepseek-ai/dsh-image-recognition`：`ctx.imageRecognition`
+  提供方注册表 + 按提供方选择的 `recognize()`（镜像 `ctx.web`）：重复 id 拒绝、
+  选择与注册顺序无关、`ImageRecognitionError` 分类。
+- **Provider** `@deepseek-ai/dsh-image-recognition-http`：可配置的
+  OpenAI 兼容 chat-completions 视觉提供方。`baseURL` + `model` 在实时 settings
+  区段；密钥经 credential-ref 走凭据域。把不含密钥的视觉请求记录为
+  `image-recognition/llm-request` 会话事件。
+- **Consumer** `@deepseek-ai/dsh-tool-image-recognition`：注册 `recognize_image`
+  工具和一个内置 `image-recognition` 技能，并挂接 `agent/pre-step`，当一步输入
+  携带图像（图像内容块或图像文件路径/URL）时**确定性注入**技能体，让模型先识别
+  再做其他动作。
+- **Bundle** `@deepseek-ai/dsh-image-recognition-bundle`：挂载以上三者；**可选**——
+  不在任何默认 profile 的 bundles 列表里。
+- **设置页卡片**：在插件设置页新增 `image-recognition` 卡片（接口地址 + 密钥），
+  仿 web-search 卡片。
+
+机制是"确定性注入 + 模型驱动执行"：pre-step 注入保证技能指令在场，而模型仍驱动
+实际的 `recognize_image` 调用（与所有既有插件一致——模型是工具使用的最终决策者）。
+
+## 启用
+
+用户按 profile 启用，把 `@deepseek-ai/dsh-image-recognition-bundle` 加入该 profile
+的 `dsh.profile.bundles`（或 `dsh plugin --profile <name> add ...`），然后在插件
+设置页配置接口地址和密钥。
+
+## 验证
+
+- `packages/vision/*` 单测：SD 选择语义、provider 的 HTTP 错误/中止映射、图像信号
+  检测函数。
+- 全量 host 程序类型检查通过。
+- 实机：启用 bundle，附加一张图，确认 `recognize_image` 运行且任务基于其结果继续。
+
+## 备选方案
+
+- **用内置多模态 `read_image` 作为机制。** 已否决：用户想要一个可配置的提供方
+  （地址 + 密钥），在设置页露出，仿 web-search，而不是固定的多模态路径。
+- **仅 prompt 引导（注册技能、让模型自主路由）。** 已否决：用户要求确定性保证——
+  模型先识别图像；pre-step 注入实现这一点，同时模型仍驱动实际工具调用。
+
+## 后果
+
+- **代价：** 新增四个包 + 一个客户端卡片组成的能力缝；可选 bundle 需要先配置提供方
+  才能工作；图像任务的路径/URL 检测是启发式的，可关闭。
+- **收益：** 识别成为一等步骤，且提供方由用户掌控；harness 遵循能力缝与
+  "确定性注入 + 模型驱动执行"模式；功能按 profile 可选，不改变默认行为。
@@ -34,6 +34,7 @@ Groups hold `packages/<group>/<pkg>/`; names stay `@deepseek-ai/dsh-<pkg>`. **Gr
 | [`workflow/`](workflow/README.md) | Workflow seam, worker-thread engine, and model-facing `workflow`/`ralph` tools | Product — stable API |
 | [`web/`](web/README.md) | Web capability family: seam, search/fetch provider impls, and the model-facing web tools | Product — stable API |
 | [`attachment/`](attachment/README.md) | Durable attachment identity, validation, local content-addressed storage | Product — stable API |
+| [`vision/`](vision/README.md) | Image-recognition capability family: seam, configurable HTTP provider, and model-facing recognition tool + skill | Product — stable API |
 | [`spill/`](spill/README.md) | Spill capability family: storage seam, local impl, tool-result spill policy | Product — stable API |
 | [`todo/`](todo/README.md) | The model-facing `todo_write` tool | Product — stable API |
 | [`plan/`](plan/README.md) | Plan collaboration state with a direct entry command and reviewed exit | Product — stable API |
@@ -0,0 +1,17 @@
+# @deepseek-ai/dsh-image-recognition-bundle
+
+Opt-in image-recognition bundle for a profile. Its `cordis.patch.yml` mounts the
+capability seam (`@deepseek-ai/dsh-image-recognition`), the configurable HTTP
+provider (`@deepseek-ai/dsh-image-recognition-http`), and the model-facing
+consumer (`@deepseek-ai/dsh-tool-image-recognition`).
+
+The bundle is **not** in any default profile's bundles list. Enable it per
+profile by adding this package to the profile's `dsh.profile.bundles` (or
+`dsh plugin --profile <name> add @deepseek-ai/dsh-image-recognition-bundle`),
+then configure the endpoint and key in the Plugins settings page
+(`image-recognition` card).
+
+## Known Limitations and Deferred Work
+
+- Requires the user to supply a vision provider endpoint and key; recognition is
+  unavailable until both are configured.
@@ -0,0 +1,17 @@
+# vision/
+
+Image-recognition capability family over the DeepSeek Harness: a Service
+Definition seam, a user-configurable HTTP vision provider, and the model-facing
+recognition consumer (tool + skill + deterministic image-task injection). The
+opt-in bundle composes them for a profile.
+
+| Package | Role |
+|---|---|
+| [`image-recognition/`](image-recognition/README.md) | Service Definition: `ctx.imageRecognition` provider registry + selection + `ImageRecognitionError` |
+| [`image-recognition-http/`](image-recognition-http/README.md) | Provider: user-configured OpenAI-compatible chat-completions vision endpoint |
+| [`tool-image-recognition/`](tool-image-recognition/README.md) | Consumer: `recognize_image` tool, `image-recognition` skill, deterministic image-task injection |
+| [`../bundle/image-recognition`](../bundle/image-recognition/README.md) | Opt-in bundle mounting the seam, provider, and consumer |
+
+A capability seam keeps Service Definition / Provider / Consumer roles separate
+([capability seams](../../.agents/notes/implemented/architecture/2026-06-13-capability-seams.md));
+the opt-in bundle is how a deployment turns image recognition on per profile.
@@ -0,0 +1,29 @@
+# @deepseek-ai/dsh-image-recognition-http
+
+A user-configurable HTTP vision provider for `ctx.imageRecognition`. It calls an
+OpenAI-compatible `/chat/completions` endpoint with the image encoded as a data
+URL and returns the recognized text.
+
+## Config
+
+| Key | Type | Meaning |
+|---|---|---|
+| `baseURL` | string | Endpoint base; `/chat/completions` is appended. Blank inherits `$DSH_IMAGE_RECOGNITION_BASE_URL`, else unavailable. |
+| `apiKey` | string (secret) | Literal key; prefer `apiKeyEnv`. |
+| `apiKeyEnv` | string (credential-ref) | Credential reference resolved per recognition; defaults to `DEEPSEEK_API_KEY`. |
+| `model` | string | Vision model name; defaults to `deepseek-v4-flash`. |
+| `maxTokens` | number | Generated-token bound; defaults to 2048. |
+
+The endpoint and key are editable live through the `image-recognition-http`
+settings section. The key never rides a response: it is resolved per recognition
+from the credentials domain, falling back to the launch environment.
+
+A recognition under an initiating Agent appends the log-only
+`image-recognition/llm-request` session event carrying the secret-free request
+body before dispatch.
+
+## Known Limitations and Deferred Work
+
+- The provider assumes an OpenAI-compatible vision endpoint; non-standard
+  endpoints must be adapted via `baseURL`/`model` (or a new provider).
+- `file-path` inputs require an inferable media type from the extension.
@@ -31,6 +31,9 @@
    "lib/types/**/*.d.ts"
  ],
  "license": "MIT",
+  "peerDependencies": {
+    "@deepseek-ai/dsh-invariants": "workspace:^"
+  },
  "dependencies": {
    "@deepseek-ai/schemastery": "workspace:^"
  },
@@ -0,0 +1,25 @@
+# @deepseek-ai/dsh-image-recognition
+
+Service Definition for the image-recognition capability seam (`ctx.imageRecognition`):
+a provider registry and provider-selecting execution, mirrored on the web seam.
+
+- A provider registers via `ctx.imageRecognition.registerProvider(provider)`; a
+  duplicate `id` throws `IMAGE_RECOGNITION_DUPLICATE_PROVIDER`.
+- `recognize(request, signal)` resolves the provider at call time. Selection is
+  order-independent: a configured id must be registered and `available()`;
+  without a configured id, exactly one usable provider auto-selects. See the
+  `ImageRecognitionError` codes for missing/unavailable/ambiguous providers.
+- `ImageInput` is a closed union of `file-path`, `base64`, and `url`; a provider
+  encodes the kind it serves.
+
+## Config
+
+`provider` pins which provider wins; `$DSH_IMAGE_RECOGNITION_PROVIDER` is the
+environment equivalent (not a hidden priority chain).
+
+## Known Limitations and Deferred Work
+
+- The seam is provider-neutral by design; image decoding/encoding lives in the
+  provider, not here.
+- `ImageRecognitionError` codes are open-string and provider-specific codes are
+  tolerated by consumers.
@@ -0,0 +1,100 @@
+# @deepseek-ai/dsh-tool-image-recognition
+
+Model-facing image-recognition consumer. Registers a bundled `image-recognition`
+skill and a `recognize_image` tool, and hooks `agent/pre-step` to deterministically
+inject the skill body when a step input carries an image, so the model recognizes
+the image before continuing the task.
+
+## Config
+
+| Key | Type | Meaning |
+|---|---|---|
+| `detectImageBlocks` | boolean | Inject the skill body when a step input carries an image content block (default true). |
+| `detectImagePaths` | boolean | Inject the skill body when a step input text names an image file path or URL (default true). |
+
+Detection is a cheap signal (image content block or path/URL in text), never an
+image-byte read. The injection guarantees the skill instructions are present; the
+model still drives the actual `recognize_image` call.
+
+## Model Experience
+
+### Tool schema
+
+#### What the model sees
+
+The model sees the generated [`recognize_image` schema](../../../docs/tool-catalog.md#deepseek-aidsh-tool-image-recognition). The tool takes an `image` (file path, https URL, or data URL) and an optional `prompt` naming what to extract, and returns the recognized text from the configured provider.
+
+#### Token effect
+
+Fixed schema cost per request while the tool is visible.
+
+#### KV Cache effect
+
+Prefix-stable while the tool definition and visibility are unchanged.
+
+### Skill body
+
+#### What the model sees
+
+The `image-recognition` skill appears in the session skill catalog and loads through the `skill` tool; its body instructs the model to recognize an image with `recognize_image` first and act on the recognized text as ground truth. The rendered body is:
+
+##### Skill body template
+
+```markdown
+<skill_content name="image-recognition">
+<skill_instructions>
+When the task involves an image (an attached image, an image file path, or an image URL), FIRST recognize its content before continuing the task:
+1. Determine which image the task refers to from the conversation (attachment, path, or URL).
+2. Call `recognize_image` with that image and, when useful, a `prompt` naming what to extract.
+3. Use the recognized text as ground truth to complete the original task.
+Do not guess at image contents from a filename or description — run `recognize_image` and act on its result.
+</skill_instructions>
+</skill_content>
+```
+
+#### Token effect
+
+The catalog entry costs one line; loading the skill body costs its full text once per load.
+
+#### KV Cache effect
+
+Prefix-stable until the skill body loads; the loaded body appends as a new suffix.
+
+### Image-task injection
+
+#### What the model sees
+
+When a step input carries an image (an image content block, or a text block naming an image file path or URL — see Config), the plugin injects the `image-recognition` skill body as a plugin-sourced user message at the front of the step.
+
+#### Token effect
+
+One injected user message per image-bearing step while detection is on.
+
+#### KV Cache effect
+
+Injected messages append after the reusable prefix, forming a per-step suffix that resets when no image is present.
+
+### System prompt section
+
+#### What the model sees
+
+A `tool:image-recognition` system-prompt section adds the guidance below:
+
+##### System prompt guidance
+
+```markdown
+When a task involves an image, recognize it first with `recognize_image` before continuing; do not infer image contents from a filename or description.
+```
+
+#### Token effect
+
+One fixed system-prompt sentence per request.
+
+#### KV Cache effect
+
+Prefix-stable while the guidance text is unchanged.
+
+## Known Limitations and Deferred Work
+
+- Path/URL detection is heuristic and can be disabled; misdetections are possible.
+- Requires a configured, available provider — see `dsh-image-recognition-http`.
@@ -31,6 +31,9 @@
    "lib/types/**/*.d.ts"
  ],
  "license": "MIT",
+  "peerDependencies": {
+    "@deepseek-ai/dsh-invariants": "workspace:^"
+  },
  "dependencies": {
    "@deepseek-ai/schemastery": "workspace:^"
  },
@@ -17,7 +17,7 @@ import { isModelInvocable, renderSkillContent } from '@deepseek-ai/dsh-skill'
 import type { ImageInput, ImageRecognitionResult } from '@deepseek-ai/dsh-image-recognition'

 export const name = 'tool-image-recognition'
-export const inject = ['agents', 'imageRecognition', 'skills', 'systemPrompt', 'tools']
+export const inject = ['imageRecognition', 'skills', 'systemPrompt', 'tools']

 const SKILL_NAME = 'image-recognition'