diff --git a/.agents/notes/implemented/feature/2026-08-14-image-recognition-capability.i18n.yaml b/.agents/notes/implemented/feature/2026-08-14-image-recognition-capability.i18n.yaml new file mode 100644 index 0000000000..cdee42496d --- /dev/null +++ b/.agents/notes/implemented/feature/2026-08-14-image-recognition-capability.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-08-14-image-recognition-capability.md +2026-08-14-image-recognition-capability.md: bd8fe3406e200b23f9d25f54ca71fcf7c081e1e6 +2026-08-14-image-recognition-capability.zh.md: c958d6441f96f7c8b96cf3c6c3a4e8783a00fb58 diff --git a/.agents/notes/implemented/feature/2026-08-14-image-recognition-capability.md b/.agents/notes/implemented/feature/2026-08-14-image-recognition-capability.md new file mode 100644 index 0000000000..bd8fe3406e --- /dev/null +++ b/.agents/notes/implemented/feature/2026-08-14-image-recognition-capability.md @@ -0,0 +1,78 @@ +# Agent Note: Image-recognition capability (opt-in bundle) + +Status: implemented + +English | [中文](2026-08-14-image-recognition-capability.zh.md) + +## Problem + +The harness had no way to recognize image contents as a first-class step of a +task. A multimodal model can `read_image`, but nothing detects an image task and +routes it through a user-configured recognition provider first. The user wanted +an image-capability plugin, modeled on the web-search provider: fill in a +provider address + API key in the settings page, and have the model recognize an +image before continuing. + +## Decision + +Add a complete capability seam over four new packages under `packages/vision/`, +mirroring the web-search three-role shape (Service Definition / Provider / +Consumer), plus an opt-in bundle: + +- **Service Definition** `@deepseek-ai/dsh-image-recognition`: `ctx.imageRecognition` + provider registry + provider-selecting `recognize()` (mirrors `ctx.web`): + duplicate ids rejected, order-independent selection, + `ImageRecognitionError` taxonomy. +- **Provider** `@deepseek-ai/dsh-image-recognition-http`: a user-configurable + OpenAI-compatible chat-completions vision provider. `baseURL` + `model` live in + a live settings section; the key rides the credentials domain via a + credential-ref. Logs the secret-free vision request as the + `image-recognition/llm-request` session event. +- **Consumer** `@deepseek-ai/dsh-tool-image-recognition`: registers a + `recognize_image` tool and a bundled `image-recognition` skill, and hooks + `agent/pre-step` to **deterministically inject** the skill body when a step + input carries an image (an image content block or an image file path/URL), so + the model recognizes the image before other actions. +- **Bundle** `@deepseek-ai/dsh-image-recognition-bundle`: mounts the three above; + **opt-in** — it is not in any default profile's bundles list. +- **Settings card**: an `image-recognition` card in the Plugins settings page + (endpoint + key), cloned from the web-search card. + +The mechanism is deterministic-injection-plus-model-driven-execution: the +pre-step injection guarantees the skill instructions are present, while the model +still drives the actual `recognize_image` call (consistent with every existing +plugin — the model is the final decider of tool use). + +## Enabling + +A user opts in per profile by adding `@deepseek-ai/dsh-image-recognition-bundle` +to the profile's `dsh.profile.bundles` (or `dsh plugin --profile add +...`), then configures the endpoint and key in the Plugins settings page. + +## Verification + +- `packages/vision/*` unit tests: SD selection semantics, provider HTTP + error/abort mapping, and the image-signal detection function. +- Full host program typechecks. +- Real use: opt in the bundle, attach an image, confirm `recognize_image` runs + and the task continues from its result. + +## Alternatives considered + +- **Built-in multimodal `read_image` as the mechanism.** Rejected: the user + wanted a configurable provider (address + key) surfaced in the settings page, + modeled on web-search, not a fixed multimodal path. +- **Prompt-guidance only (register the skill, let the model self-route).** + Rejected: the user asked for a deterministic guarantee that the model + recognizes an image first; the pre-step injection delivers that while the model + still drives the actual tool call. + +## Consequences + +- **Costs:** a new capability seam of four packages plus a client card; the + opt-in bundle needs a configured provider before it works; image-task path/URL + detection is heuristic and can be disabled. +- **Buys:** recognition becomes a first-class step with a provider the user + controls, the harness follows the capability-seam and + deterministic-injection-plus-model-driven-execution patterns, and the feature + is opt-in per profile (no default-behavior change). diff --git a/.agents/notes/implemented/feature/2026-08-14-image-recognition-capability.zh.md b/.agents/notes/implemented/feature/2026-08-14-image-recognition-capability.zh.md new file mode 100644 index 0000000000..c958d6441f --- /dev/null +++ b/.agents/notes/implemented/feature/2026-08-14-image-recognition-capability.zh.md @@ -0,0 +1,62 @@ +# Agent Note:图像识别能力(可选 bundle) + +Status: implemented + +English | [中文](2026-08-14-image-recognition-capability.md) + +## 问题 + +harness 此前无法把"识别图像内容"作为任务的一等步骤。多模态模型可以 `read_image`, +但没有东西检测图像任务、并通过用户配置的识别提供方先路由它。用户想要一个仿网页 +搜索的图像能力插件:在设置页填写供应商地址 + API Key,让模型先识别图像再继续任务。 + +## 决策 + +在 `packages/vision/` 下新增四个包,构成完整能力缝,镜像 web-search 的三角色形态 +(Service Definition / Provider / Consumer),外加一个可选 bundle: + +- **Service Definition** `@deepseek-ai/dsh-image-recognition`:`ctx.imageRecognition` + 提供方注册表 + 按提供方选择的 `recognize()`(镜像 `ctx.web`):重复 id 拒绝、 + 选择与注册顺序无关、`ImageRecognitionError` 分类。 +- **Provider** `@deepseek-ai/dsh-image-recognition-http`:可配置的 + OpenAI 兼容 chat-completions 视觉提供方。`baseURL` + `model` 在实时 settings + 区段;密钥经 credential-ref 走凭据域。把不含密钥的视觉请求记录为 + `image-recognition/llm-request` 会话事件。 +- **Consumer** `@deepseek-ai/dsh-tool-image-recognition`:注册 `recognize_image` + 工具和一个内置 `image-recognition` 技能,并挂接 `agent/pre-step`,当一步输入 + 携带图像(图像内容块或图像文件路径/URL)时**确定性注入**技能体,让模型先识别 + 再做其他动作。 +- **Bundle** `@deepseek-ai/dsh-image-recognition-bundle`:挂载以上三者;**可选**—— + 不在任何默认 profile 的 bundles 列表里。 +- **设置页卡片**:在插件设置页新增 `image-recognition` 卡片(接口地址 + 密钥), + 仿 web-search 卡片。 + +机制是"确定性注入 + 模型驱动执行":pre-step 注入保证技能指令在场,而模型仍驱动 +实际的 `recognize_image` 调用(与所有既有插件一致——模型是工具使用的最终决策者)。 + +## 启用 + +用户按 profile 启用,把 `@deepseek-ai/dsh-image-recognition-bundle` 加入该 profile +的 `dsh.profile.bundles`(或 `dsh plugin --profile add ...`),然后在插件 +设置页配置接口地址和密钥。 + +## 验证 + +- `packages/vision/*` 单测:SD 选择语义、provider 的 HTTP 错误/中止映射、图像信号 + 检测函数。 +- 全量 host 程序类型检查通过。 +- 实机:启用 bundle,附加一张图,确认 `recognize_image` 运行且任务基于其结果继续。 + +## 备选方案 + +- **用内置多模态 `read_image` 作为机制。** 已否决:用户想要一个可配置的提供方 + (地址 + 密钥),在设置页露出,仿 web-search,而不是固定的多模态路径。 +- **仅 prompt 引导(注册技能、让模型自主路由)。** 已否决:用户要求确定性保证—— + 模型先识别图像;pre-step 注入实现这一点,同时模型仍驱动实际工具调用。 + +## 后果 + +- **代价:** 新增四个包 + 一个客户端卡片组成的能力缝;可选 bundle 需要先配置提供方 + 才能工作;图像任务的路径/URL 检测是启发式的,可关闭。 +- **收益:** 识别成为一等步骤,且提供方由用户掌控;harness 遵循能力缝与 + "确定性注入 + 模型驱动执行"模式;功能按 profile 可选,不改变默认行为。 diff --git a/packages/README.md b/packages/README.md index 7904ec1fea..0f57a9de40 100644 --- a/packages/README.md +++ b/packages/README.md @@ -34,6 +34,7 @@ Groups hold `packages///`; names stay `@deepseek-ai/dsh-`. **Gr | [`workflow/`](workflow/README.md) | Workflow seam, worker-thread engine, and model-facing `workflow`/`ralph` tools | Product — stable API | | [`web/`](web/README.md) | Web capability family: seam, search/fetch provider impls, and the model-facing web tools | Product — stable API | | [`attachment/`](attachment/README.md) | Durable attachment identity, validation, local content-addressed storage | Product — stable API | +| [`vision/`](vision/README.md) | Image-recognition capability family: seam, configurable HTTP provider, and model-facing recognition tool + skill | Product — stable API | | [`spill/`](spill/README.md) | Spill capability family: storage seam, local impl, tool-result spill policy | Product — stable API | | [`todo/`](todo/README.md) | The model-facing `todo_write` tool | Product — stable API | | [`plan/`](plan/README.md) | Plan collaboration state with a direct entry command and reviewed exit | Product — stable API | diff --git a/packages/bundle/image-recognition/README.md b/packages/bundle/image-recognition/README.md new file mode 100644 index 0000000000..67634c9ff0 --- /dev/null +++ b/packages/bundle/image-recognition/README.md @@ -0,0 +1,17 @@ +# @deepseek-ai/dsh-image-recognition-bundle + +Opt-in image-recognition bundle for a profile. Its `cordis.patch.yml` mounts the +capability seam (`@deepseek-ai/dsh-image-recognition`), the configurable HTTP +provider (`@deepseek-ai/dsh-image-recognition-http`), and the model-facing +consumer (`@deepseek-ai/dsh-tool-image-recognition`). + +The bundle is **not** in any default profile's bundles list. Enable it per +profile by adding this package to the profile's `dsh.profile.bundles` (or +`dsh plugin --profile add @deepseek-ai/dsh-image-recognition-bundle`), +then configure the endpoint and key in the Plugins settings page +(`image-recognition` card). + +## Known Limitations and Deferred Work + +- Requires the user to supply a vision provider endpoint and key; recognition is + unavailable until both are configured. diff --git a/packages/vision/README.md b/packages/vision/README.md new file mode 100644 index 0000000000..d8221a5733 --- /dev/null +++ b/packages/vision/README.md @@ -0,0 +1,17 @@ +# vision/ + +Image-recognition capability family over the DeepSeek Harness: a Service +Definition seam, a user-configurable HTTP vision provider, and the model-facing +recognition consumer (tool + skill + deterministic image-task injection). The +opt-in bundle composes them for a profile. + +| Package | Role | +|---|---| +| [`image-recognition/`](image-recognition/README.md) | Service Definition: `ctx.imageRecognition` provider registry + selection + `ImageRecognitionError` | +| [`image-recognition-http/`](image-recognition-http/README.md) | Provider: user-configured OpenAI-compatible chat-completions vision endpoint | +| [`tool-image-recognition/`](tool-image-recognition/README.md) | Consumer: `recognize_image` tool, `image-recognition` skill, deterministic image-task injection | +| [`../bundle/image-recognition`](../bundle/image-recognition/README.md) | Opt-in bundle mounting the seam, provider, and consumer | + +A capability seam keeps Service Definition / Provider / Consumer roles separate +([capability seams](../../.agents/notes/implemented/architecture/2026-06-13-capability-seams.md)); +the opt-in bundle is how a deployment turns image recognition on per profile. diff --git a/packages/vision/image-recognition-http/README.md b/packages/vision/image-recognition-http/README.md new file mode 100644 index 0000000000..c1dd7866d3 --- /dev/null +++ b/packages/vision/image-recognition-http/README.md @@ -0,0 +1,29 @@ +# @deepseek-ai/dsh-image-recognition-http + +A user-configurable HTTP vision provider for `ctx.imageRecognition`. It calls an +OpenAI-compatible `/chat/completions` endpoint with the image encoded as a data +URL and returns the recognized text. + +## Config + +| Key | Type | Meaning | +|---|---|---| +| `baseURL` | string | Endpoint base; `/chat/completions` is appended. Blank inherits `$DSH_IMAGE_RECOGNITION_BASE_URL`, else unavailable. | +| `apiKey` | string (secret) | Literal key; prefer `apiKeyEnv`. | +| `apiKeyEnv` | string (credential-ref) | Credential reference resolved per recognition; defaults to `DEEPSEEK_API_KEY`. | +| `model` | string | Vision model name; defaults to `deepseek-v4-flash`. | +| `maxTokens` | number | Generated-token bound; defaults to 2048. | + +The endpoint and key are editable live through the `image-recognition-http` +settings section. The key never rides a response: it is resolved per recognition +from the credentials domain, falling back to the launch environment. + +A recognition under an initiating Agent appends the log-only +`image-recognition/llm-request` session event carrying the secret-free request +body before dispatch. + +## Known Limitations and Deferred Work + +- The provider assumes an OpenAI-compatible vision endpoint; non-standard + endpoints must be adapted via `baseURL`/`model` (or a new provider). +- `file-path` inputs require an inferable media type from the extension. diff --git a/packages/vision/image-recognition-http/package.json b/packages/vision/image-recognition-http/package.json index 777a6a39a4..2e0154ea2b 100644 --- a/packages/vision/image-recognition-http/package.json +++ b/packages/vision/image-recognition-http/package.json @@ -31,6 +31,9 @@ "lib/types/**/*.d.ts" ], "license": "MIT", + "peerDependencies": { + "@deepseek-ai/dsh-invariants": "workspace:^" + }, "dependencies": { "@deepseek-ai/schemastery": "workspace:^" }, diff --git a/packages/vision/image-recognition/README.md b/packages/vision/image-recognition/README.md new file mode 100644 index 0000000000..026554ccc3 --- /dev/null +++ b/packages/vision/image-recognition/README.md @@ -0,0 +1,25 @@ +# @deepseek-ai/dsh-image-recognition + +Service Definition for the image-recognition capability seam (`ctx.imageRecognition`): +a provider registry and provider-selecting execution, mirrored on the web seam. + +- A provider registers via `ctx.imageRecognition.registerProvider(provider)`; a + duplicate `id` throws `IMAGE_RECOGNITION_DUPLICATE_PROVIDER`. +- `recognize(request, signal)` resolves the provider at call time. Selection is + order-independent: a configured id must be registered and `available()`; + without a configured id, exactly one usable provider auto-selects. See the + `ImageRecognitionError` codes for missing/unavailable/ambiguous providers. +- `ImageInput` is a closed union of `file-path`, `base64`, and `url`; a provider + encodes the kind it serves. + +## Config + +`provider` pins which provider wins; `$DSH_IMAGE_RECOGNITION_PROVIDER` is the +environment equivalent (not a hidden priority chain). + +## Known Limitations and Deferred Work + +- The seam is provider-neutral by design; image decoding/encoding lives in the + provider, not here. +- `ImageRecognitionError` codes are open-string and provider-specific codes are + tolerated by consumers. diff --git a/packages/vision/tool-image-recognition/README.md b/packages/vision/tool-image-recognition/README.md new file mode 100644 index 0000000000..6c8c1addec --- /dev/null +++ b/packages/vision/tool-image-recognition/README.md @@ -0,0 +1,100 @@ +# @deepseek-ai/dsh-tool-image-recognition + +Model-facing image-recognition consumer. Registers a bundled `image-recognition` +skill and a `recognize_image` tool, and hooks `agent/pre-step` to deterministically +inject the skill body when a step input carries an image, so the model recognizes +the image before continuing the task. + +## Config + +| Key | Type | Meaning | +|---|---|---| +| `detectImageBlocks` | boolean | Inject the skill body when a step input carries an image content block (default true). | +| `detectImagePaths` | boolean | Inject the skill body when a step input text names an image file path or URL (default true). | + +Detection is a cheap signal (image content block or path/URL in text), never an +image-byte read. The injection guarantees the skill instructions are present; the +model still drives the actual `recognize_image` call. + +## Model Experience + +### Tool schema + +#### What the model sees + +The model sees the generated [`recognize_image` schema](../../../docs/tool-catalog.md#deepseek-aidsh-tool-image-recognition). The tool takes an `image` (file path, https URL, or data URL) and an optional `prompt` naming what to extract, and returns the recognized text from the configured provider. + +#### Token effect + +Fixed schema cost per request while the tool is visible. + +#### KV Cache effect + +Prefix-stable while the tool definition and visibility are unchanged. + +### Skill body + +#### What the model sees + +The `image-recognition` skill appears in the session skill catalog and loads through the `skill` tool; its body instructs the model to recognize an image with `recognize_image` first and act on the recognized text as ground truth. The rendered body is: + +##### Skill body template + +```markdown + + +When the task involves an image (an attached image, an image file path, or an image URL), FIRST recognize its content before continuing the task: +1. Determine which image the task refers to from the conversation (attachment, path, or URL). +2. Call `recognize_image` with that image and, when useful, a `prompt` naming what to extract. +3. Use the recognized text as ground truth to complete the original task. +Do not guess at image contents from a filename or description — run `recognize_image` and act on its result. + + +``` + +#### Token effect + +The catalog entry costs one line; loading the skill body costs its full text once per load. + +#### KV Cache effect + +Prefix-stable until the skill body loads; the loaded body appends as a new suffix. + +### Image-task injection + +#### What the model sees + +When a step input carries an image (an image content block, or a text block naming an image file path or URL — see Config), the plugin injects the `image-recognition` skill body as a plugin-sourced user message at the front of the step. + +#### Token effect + +One injected user message per image-bearing step while detection is on. + +#### KV Cache effect + +Injected messages append after the reusable prefix, forming a per-step suffix that resets when no image is present. + +### System prompt section + +#### What the model sees + +A `tool:image-recognition` system-prompt section adds the guidance below: + +##### System prompt guidance + +```markdown +When a task involves an image, recognize it first with `recognize_image` before continuing; do not infer image contents from a filename or description. +``` + +#### Token effect + +One fixed system-prompt sentence per request. + +#### KV Cache effect + +Prefix-stable while the guidance text is unchanged. + +## Known Limitations and Deferred Work + +- Path/URL detection is heuristic and can be disabled; misdetections are possible. +- Requires a configured, available provider — see `dsh-image-recognition-http`. diff --git a/packages/vision/tool-image-recognition/package.json b/packages/vision/tool-image-recognition/package.json index 6fff441e33..fbbc61fe25 100644 --- a/packages/vision/tool-image-recognition/package.json +++ b/packages/vision/tool-image-recognition/package.json @@ -31,6 +31,9 @@ "lib/types/**/*.d.ts" ], "license": "MIT", + "peerDependencies": { + "@deepseek-ai/dsh-invariants": "workspace:^" + }, "dependencies": { "@deepseek-ai/schemastery": "workspace:^" }, diff --git a/packages/vision/tool-image-recognition/src/index.ts b/packages/vision/tool-image-recognition/src/index.ts index 557f1bf776..6e6c5eef46 100644 --- a/packages/vision/tool-image-recognition/src/index.ts +++ b/packages/vision/tool-image-recognition/src/index.ts @@ -17,7 +17,7 @@ import { isModelInvocable, renderSkillContent } from '@deepseek-ai/dsh-skill' import type { ImageInput, ImageRecognitionResult } from '@deepseek-ai/dsh-image-recognition' export const name = 'tool-image-recognition' -export const inject = ['agents', 'imageRecognition', 'skills', 'systemPrompt', 'tools'] +export const inject = ['imageRecognition', 'skills', 'systemPrompt', 'tools'] const SKILL_NAME = 'image-recognition'