refactor: narrow web image input v1
This commit is contained in:
Tianyi Cui
+2
-2
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.md
|
||||
2026-07-22-web-multimodal-image-input-and-durable-attachments.md: 7c14a8bdbb6d8164b2a6eb4432d196936271e4d1
|
||||
2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md: e9d7f8ebe78aa76285367e83374a6ace3a9eef21
|
||||
2026-07-22-web-multimodal-image-input-and-durable-attachments.md: ab1bd449f065346bc327984eacc85f71d4fd5a28
|
||||
2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md: d51f9135352ba09b2cd2a083ade6b62f4a276216
|
||||
|
||||
+9
-9
@@ -112,21 +112,21 @@ type PromptInputPart =
|
||||
}
|
||||
```
|
||||
|
||||
Base64 crosses JSON-RPC once and is discarded after persistence. The host validates canonical base64, image count, aggregate bytes, individual bytes, magic-byte MIME, intrinsic dimensions, and decoded-pixel count — the complete batch, through the seam's storage-free `validateImage`, before persisting any member, so one malformed image cannot strand the batch's valid members as unreferenced objects. Only after every image succeeds does it call the agent with normalized text and durable image blocks. A failure appends no user event and exposes no attachment path or raw bytes.
|
||||
Base64 crosses JSON-RPC once and is discarded after persistence. Version one accepts at most one image per prompt. The host validates canonical base64, individual bytes, magic-byte MIME, intrinsic dimensions, and intrinsic pixel count while durably saving that image. Only after the save succeeds does it call the agent with normalized text and a durable image block. A failure appends no user event and exposes no attachment path or raw bytes.
|
||||
|
||||
`session.attachment` is a read-only, session-scoped endpoint. The host serves bytes only when a durable event in that session references the requested attachment identifier. The client deduplicates loads by session and attachment identifier while that session is rendered, revokes resolved URLs on rendered-session disposal, and invalidates late loads so an unmounted session cannot repopulate the cache.
|
||||
|
||||
### Model capabilities and provider behavior
|
||||
|
||||
Model catalog entries gain optional merge-extensible input and output modality declarations. A missing declaration means unknown; a present list without `image` is an explicit negative capability.
|
||||
Model catalog entries gain optional merge-extensible input modality declarations. A missing declaration means unknown; a present list without `image` is an explicit negative capability.
|
||||
|
||||
The host is the authoritative preflight boundary. It resolves the session's latest routed provider/model, falling back through agent options to host defaults; if that model explicitly excludes image input, it rejects the prompt before writing any attachment or event, and the client restores the draft. Unknown capability proceeds to the adapter guard so uncatalogued model identifiers remain usable. `host.describe` projects the host-default active model and the image limits into `SessionsService`; the composer applies only the deployment limits before allocating object URLs or base64. Model capability is deliberately not gated client-side: the handshake snapshot cannot represent a session's current target after `session.selectModel`, so the host preflight is the sole capability authority and its rejection renders through the composer error strip. Decoded-pixel validation and the session's actual route remain authoritative on the host.
|
||||
The host is the authoritative preflight boundary. It resolves the session's latest routed provider/model, falling back through agent options to host defaults; if that model explicitly excludes image input, it rejects the prompt before writing any attachment or event, and the client restores the draft. Unknown capability proceeds to the adapter guard so uncatalogued model identifiers remain usable. `host.describe` projects the image limits into `SessionsService`; the composer applies only those deployment limits before allocating object URLs or base64. Model capability is deliberately not gated client-side: the handshake snapshot cannot represent a session's current target after `session.selectModel`, so the host preflight is the sole capability authority and its rejection renders through the composer error strip. Intrinsic-pixel validation and the session's actual route remain authoritative on the host.
|
||||
|
||||
The Pi-AI adapter is the first visual-input route: it resolves `ctx.attachments` at request time, then resolves each durable reference and emits native image content only for models that declare image input. The shipped Web assembly reaches it through `dsh web --provider <name> --model <id>`: the yml pi-ai row already routes openai/anthropic with ambient credentials, and only a catalog provider absent from that row is mounted dynamically; the DeepSeek-only default remains text-only. Request-time service resolution keeps Cordis load order from freezing optional attachment availability. The hand-written DeepSeek adapter throws typed `UNSUPPORTED_CONTENT` for an image anywhere in the request, including nested tool results. No adapter may flatten or skip an image.
|
||||
The Pi-AI adapter is the first visual-input route: it resolves `ctx.attachments` at request time, then resolves each durable reference and emits native image content only for models that declare image input. The shipped Web assembly reaches the statically configured openai/anthropic routes through `dsh web --provider <name> --model <id>` with provider-native ambient credentials; other routes must be registered by the composition, and the default DeepSeek route remains text-only. Request-time service resolution keeps Cordis load order from freezing optional attachment availability. The hand-written DeepSeek adapter throws typed `UNSUPPORTED_CONTENT` for an image anywhere in the request, including nested tool results. No adapter may flatten or skip an image.
|
||||
|
||||
Core supports structured assistant image blocks, but no current production provider route is certified for image output. Any future output-capable adapter must retrieve provider bytes under bounded size and time policy, validate them through the same attachment service, persist them, and only then publish the atomic `ImageBlock`. A URL in assistant Markdown remains text and is never downloaded automatically.
|
||||
|
||||
Token estimation accounts for image dimensions without counting base64 or attachment locators as text. Provider-reported usage remains authoritative. ACP renders an explicit image marker until that protocol surface gains native image support rather than silently omitting the block.
|
||||
Provider-neutral token estimation does not guess visual pricing from image dimensions; provider-reported usage remains authoritative. ACP renders an explicit image marker until that protocol surface gains native image support rather than silently omitting the block.
|
||||
|
||||
Compaction replays the selected conversation prefix, including image references, into the configured summarization route. A visual-capable route resolves those references through its adapter; a text-only route fails explicitly instead of silently dropping the visual context. The synthesized checkpoint remains text-only, and `compact-basic` rejects image summary output with `UNSUPPORTED_CONTENT`.
|
||||
|
||||
@@ -138,7 +138,7 @@ Composer thumbnails and each `MessageImage` own ephemeral original-preview state
|
||||
|
||||
### Limits and trust boundaries
|
||||
|
||||
Version one accepts PNG, JPEG, WebP, and GIF only. SVG and remote URLs are excluded. Default limits are 5 MiB per image, 10 images and 20 MiB aggregate image bytes per message, and 40 million intrinsic pixels per image. These deployment-varying limits are validated backend configuration and are projected to the client for fast-path guidance; host validation remains authoritative. The client connection carrier independently caps buffered API request bodies, deriving the cap from the aggregate image limit plus base64 and envelope expansion; a body without a declared length is rejected the moment it crosses the cap rather than drained to its end.
|
||||
Version one accepts at most one PNG, JPEG, WebP, or GIF per prompt. SVG and remote URLs are excluded. Default deployment limits are 5 MiB and 40 million intrinsic pixels per image; they are validated backend configuration and projected to the client for fast-path guidance, while host validation remains authoritative. The client connection carrier independently caps buffered API request bodies from the single-image byte limit plus base64 and envelope expansion; a body without a declared length is rejected the moment it crosses the cap rather than drained to its end.
|
||||
|
||||
Malformed base64, unsupported or mismatched media, truncated headers, excess bytes, excess image count, excess pixels, missing objects, and integrity mismatches return stable structured failures. Original filenames are reduced to a display basename, control characters are removed, and no local path is logged or returned to the browser.
|
||||
|
||||
@@ -148,7 +148,7 @@ Malformed base64, unsupported or mismatched media, truncated headers, excess byt
|
||||
| --- | --- |
|
||||
| `packages/attachment/attachment` | Opaque attachment identifier, image reference, limits, failures, and `ctx.attachments` service. |
|
||||
| `packages/attachment/attachment-local` | Private content-addressed storage, image-header validation, integrity verification, and configuration. |
|
||||
| `packages/llm/llm` and `packages/llm/token-meter` | Role-neutral `ImageBlock`, modality metadata, and image cost estimation. |
|
||||
| `packages/llm/llm` | Role-neutral `ImageBlock` and input-modality metadata. |
|
||||
| `packages/llm/llm-pi-ai` | Resolve durable supported image input into native provider content. |
|
||||
| `packages/llm/llm-deepseek` | Reject image content explicitly. |
|
||||
| `packages/compact/compact-basic` | Preserve images in summary input and reject non-text checkpoint output explicitly. |
|
||||
@@ -161,7 +161,7 @@ The attachment packages form the interface/implementation side of one capability
|
||||
|
||||
### Implementation
|
||||
|
||||
The implemented slice includes the attachment seam, role-neutral image block, image-aware token estimation, Pi-AI input conversion, DeepSeek rejection, durable host ordering, Web upload/read protocol, host-capability projection, bounded Web request bodies, in-memory draft images, paste/drop rail, user and assistant history rendering, double-click preview, compaction handling, and keyless assembled Web coverage.
|
||||
The implemented slice includes the attachment seam, role-neutral image block, Pi-AI input conversion, DeepSeek rejection, durable host ordering, Web upload/read protocol, image-limit projection, bounded Web request bodies, in-memory draft images, paste/drop rail, user and assistant history rendering, double-click preview, compaction handling, and keyless assembled Web coverage.
|
||||
|
||||
No compatibility shim is required for the pre-release prompt wire; all call sites and fixtures change with the introducing slice.
|
||||
|
||||
@@ -198,7 +198,7 @@ UI state can be stale and does not protect direct SDK, ACP, replay, or uncatalog
|
||||
- Client unit tests cover paste and drop, mixed clipboard text, image-only send, draft restoration, ordering, and draft/session-scope/application object-URL cleanup; the keyless assembled built-client lane (`apps/web/tests/image-display.snapshot.ts`, `DSH_EXAMPLE_MODE=lib pnpm run test:snapshot`) covers the historical user and assistant galleries over the authorized attachment route, the original-size lightbox, and the composer paste rail.
|
||||
- Adapter and compaction tests cover native Pi-AI image conversion, late attachment-service composition, text-only rejection, nested tool-result images, preserved summary input, and explicit image-output rejection.
|
||||
- A credentialed real-API test sends a PNG through the Anthropic `claude-opus-4-8` route and requires the model to identify its QR code.
|
||||
- The current production adapter set declares text-only output; output-provider certification remains outside version one.
|
||||
- The current production adapter set has no certified image-output route; output-provider certification remains outside version one.
|
||||
|
||||
## Consequences
|
||||
|
||||
|
||||
+9
-9
@@ -112,21 +112,21 @@ type PromptInputPart =
|
||||
}
|
||||
```
|
||||
|
||||
Base64 只跨越一次 JSON-RPC，并在持久化后丢弃。宿主会校验规范 base64、图片数量、总字节数、单张图片字节数、与魔数字节一致的 MIME、固有尺寸和解码像素数——并在持久化任何成员之前，通过服务边界上不触碰存储的 `validateImage` 校验完整批次，因此一张畸形图片不会把批次中的有效成员留成无引用的孤儿对象。只有每张图片都成功后，宿主才会用规范化文本和持久图片块调用 agent。失败时不追加用户事件，也不公开任何附件路径或原始字节。
|
||||
Base64 只跨越一次 JSON-RPC，并在持久化后丢弃。第一版每条提示词最多接受一张图片。宿主在持久保存该图片时，会校验规范 base64、单张图片字节数、与魔数字节一致的 MIME、固有尺寸和固有像素数。只有保存成功后，宿主才会用规范化文本和一个持久图片块调用 agent。失败时不追加用户事件，也不公开任何附件路径或原始字节。
|
||||
|
||||
`session.attachment` 是只读且限定于会话作用域的端点。只有该会话中的持久事件引用了所请求的附件标识符，宿主才提供字节。会话处于渲染状态时，客户端会按会话和附件标识符对加载操作去重；已渲染会话释放时会撤销已解析的 URL，并使延迟完成的加载失效，以免已卸载的会话重新写入缓存。
|
||||
|
||||
### 模型能力与提供方行为
|
||||
|
||||
模型目录项增加可选且可合并扩展的输入与输出模态声明。缺少声明表示未知；声明存在但不含 `image`，则明确表示不支持图片。
|
||||
模型目录项增加可选且可合并扩展的输入模态声明。缺少声明表示未知；声明存在但不含 `image`，则明确表示不支持图片。
|
||||
|
||||
宿主是权威的前置检查边界。它会解析会话最新路由到的提供方和模型，并在缺失时依次回退到 agent 选项和宿主默认值；如果该模型明确排除图片输入，宿主会在写入任何附件或事件前拒绝提示词，客户端则恢复草稿。能力未知时继续进入适配器强制检查，使未收录的模型标识符仍然可用。`host.describe` 把宿主默认的当前模型和图片限制投影到 `SessionsService`；composer 在分配对象 URL 或 base64 前只应用部署级限制。模型能力刻意不在客户端把关：握手快照无法表达 `session.selectModel` 之后会话的当前目标，因此宿主前置检查是唯一的能力权威，其拒绝通过 composer 错误条呈现。解码像素校验与会话的实际路由仍由宿主作出权威判定。
|
||||
宿主是权威的前置检查边界。它会解析会话最新路由到的提供方和模型，并在缺失时依次回退到 agent 选项和宿主默认值；如果该模型明确排除图片输入，宿主会在写入任何附件或事件前拒绝提示词，客户端则恢复草稿。能力未知时继续进入适配器强制检查，使未收录的模型标识符仍然可用。`host.describe` 把图片限制投影到 `SessionsService`；composer 在分配对象 URL 或 base64 前只应用这些部署级限制。模型能力刻意不在客户端把关：握手快照无法表达 `session.selectModel` 之后会话的当前目标，因此宿主前置检查是唯一的能力权威，其拒绝通过 composer 错误条呈现。固有像素校验与会话的实际路由仍由宿主作出权威判定。
|
||||
|
||||
Pi-AI 适配器是首条视觉输入路径：它在请求时解析 `ctx.attachments`，再解析每个持久引用，并且仅为声明支持图片输入的模型生成提供方原生图片内容。交付的 Web 组装通过 `dsh web --provider <name> --model <id>` 到达这条路径：yml 的 pi-ai row 已用环境凭据路由 openai/anthropic，只有该 row 之外的目录 provider 才会动态挂载；仅含 DeepSeek 的默认组装仍是纯文本。在请求时解析服务，可避免 Cordis 加载顺序将可选附件服务的可用性固化。手写 DeepSeek 适配器遇到请求中任何位置的图片时都会抛出类型化的 `UNSUPPORTED_CONTENT` 错误，包括嵌套工具结果中的图片。任何适配器都不得将图片展平或跳过。
|
||||
Pi-AI 适配器是首条视觉输入路径：它在请求时解析 `ctx.attachments`，再解析每个持久引用，并且仅为声明支持图片输入的模型生成提供方原生图片内容。交付的 Web 组装通过 `dsh web --provider <name> --model <id>`，使用提供方原生环境凭据访问静态配置的 openai/anthropic 路由；其他路由必须由组合注册，而默认 DeepSeek 路由仍仅支持文本。在请求时解析服务，可避免 Cordis 加载顺序将可选附件服务的可用性固化。手写 DeepSeek 适配器遇到请求中任何位置的图片时都会抛出类型化的 `UNSUPPORTED_CONTENT` 错误，包括嵌套工具结果中的图片。任何适配器都不得将图片展平或跳过。
|
||||
|
||||
核心层支持结构化助手图片块，但当前没有任何生产提供方路径通过图片输出认证。未来任何支持输出的适配器都必须在有界的大小和时间策略下获取提供方字节，通过同一个附件服务校验并持久化字节，之后才能以原子方式发布 `ImageBlock`。助手 Markdown 中的 URL 仍是文本，绝不自动下载。
|
||||
|
||||
token 估算根据图片尺寸计量，但不把 base64 或附件定位符作为文本计数。提供方返回的用量仍是权威值。在 ACP（Agent Client Protocol）接口原生支持图片前，ACP 会渲染明确的图片标记，而不是静默省略该块。
|
||||
提供方无关的 token 估算不会根据图片尺寸猜测视觉定价；提供方返回的用量仍是权威值。在 ACP（Agent Client Protocol）接口原生支持图片前，ACP 会渲染明确的图片标记，而不是静默省略该块。
|
||||
|
||||
压缩（compaction）会把选定的会话前缀（包含图片引用）回放到已配置的摘要生成路径中。支持视觉的路径会通过适配器解析这些引用；仅文本路径会明确失败，而不是静默丢弃视觉上下文。合成的检查点仍仅包含文本，`compact-basic` 会以 `UNSUPPORTED_CONTENT` 拒绝包含图片的摘要输出。
|
||||
|
||||
@@ -138,7 +138,7 @@ token 估算根据图片尺寸计量，但不把 base64 或附件定位符作为
|
||||
|
||||
### 限制与信任边界
|
||||
|
||||
第一版仅接受 PNG、JPEG、WebP 和 GIF。不接受 SVG 和远程 URL。默认限制为每张图片 5 MiB、每条消息 10 张图片和 20 MiB 图片总字节数，以及每张图片 4,000 万个固有像素。这些随部署变化的限制属于经过校验的后端配置，并会投影给客户端以提供快速路径指引；宿主校验仍是权威结果。客户端连接载体会独立限制 API 请求体的缓冲大小，并根据图片总量限制加上 base64 和请求封装的膨胀量推导上限；未声明长度的请求体在越过上限的瞬间即被拒绝，而不是先读完再拒。
|
||||
第一版每条提示词最多接受一张 PNG、JPEG、WebP 或 GIF 图片。不接受 SVG 和远程 URL。默认部署限制为每张图片 5 MiB 和 4,000 万个固有像素；这些限制属于经过校验的后端配置，并会投影给客户端以提供快速路径指引，而宿主校验仍是权威结果。客户端连接载体根据单张图片字节上限加上 base64 和请求封装的膨胀量，独立限制 API 请求体的缓冲大小；未声明长度的请求体在越过上限的瞬间即被拒绝，而不是先读完再拒。
|
||||
|
||||
格式错误的 base64、不支持或不匹配的媒体、截断的文件头、超出字节限制、超出图片数量、超出像素限制、对象缺失和完整性不匹配都会返回稳定的结构化错误。原始文件名只保留用于显示的末段，控制字符会被移除，并且任何本地路径都不会写入日志或返回浏览器。
|
||||
|
||||
@@ -148,7 +148,7 @@ token 估算根据图片尺寸计量，但不把 base64 或附件定位符作为
|
||||
| --- | --- |
|
||||
| `packages/attachment/attachment` | 不透明附件标识符、图片引用、限制、错误和 `ctx.attachments` 服务。 |
|
||||
| `packages/attachment/attachment-local` | 私有内容寻址存储、图片头校验、完整性校验和配置。 |
|
||||
| `packages/llm/llm` 和 `packages/llm/token-meter` | 角色无关的 `ImageBlock`、模态元数据和图片成本估算。 |
|
||||
| `packages/llm/llm` | 角色无关的 `ImageBlock` 和输入模态元数据。 |
|
||||
| `packages/llm/llm-pi-ai` | 将持久且受支持的图片输入解析为提供方原生内容。 |
|
||||
| `packages/llm/llm-deepseek` | 明确拒绝图片内容。 |
|
||||
| `packages/compact/compact-basic` | 在摘要输入中保留图片，并明确拒绝非文本检查点输出。 |
|
||||
@@ -161,7 +161,7 @@ token 估算根据图片尺寸计量，但不把 base64 或附件定位符作为
|
||||
|
||||
### 实现
|
||||
|
||||
已实现的范围包括附件服务边界、角色无关的图片块、图片感知的 token 估算、Pi-AI 输入转换、DeepSeek 拒绝、宿主持久化顺序、Web 上传与读取协议、宿主能力投影、大小受限的 Web 请求体、内存草稿图片、粘贴与拖放附件栏、用户与助手历史图片渲染、双击预览、压缩处理，以及组装后无需密钥的 Web 覆盖。
|
||||
已实现的范围包括附件服务边界、角色无关的图片块、Pi-AI 输入转换、DeepSeek 拒绝、宿主持久化顺序、Web 上传与读取协议、图片限制投影、大小受限的 Web 请求体、内存草稿图片、粘贴与拖放附件栏、用户与助手历史图片渲染、双击预览、压缩处理，以及组装后无需密钥的 Web 覆盖。
|
||||
|
||||
预发布提示词协议不需要兼容包装层；引入相应切片时会同时修改所有调用点和 fixture。
|
||||
|
||||
@@ -198,7 +198,7 @@ UI 状态可能陈旧，也无法保护直接 SDK、ACP、回放或未收录模
|
||||
- 客户端单元测试覆盖粘贴与拖放、混合剪贴板文本、仅图片发送、草稿恢复、顺序，以及草稿、会话作用域和应用层级的对象 URL 清理；keyless 的组装后构建产物通道（`apps/web/tests/image-display.snapshot.ts`，`DSH_EXAMPLE_MODE=lib pnpm run test:snapshot`）覆盖经授权附件路由渲染的历史用户与助手图片画廊、原图 lightbox，以及 composer 粘贴缩略图条。
|
||||
- 适配器与压缩测试覆盖 Pi-AI 原生图片转换、后置附件服务组合、仅文本拒绝、嵌套工具结果中的图片、保留摘要输入，以及明确拒绝图片输出。
|
||||
- 需要凭据的实际 API 测试会通过 Anthropic `claude-opus-4-8` 路径发送一张 PNG，并要求模型识别其中的二维码。
|
||||
- 当前生产适配器集合声明仅支持文本输出；输出提供方认证不在第一版范围内。
|
||||
- 当前生产适配器集合没有经过认证的图片输出路由；输出提供方认证仍不在第一版范围内。
|
||||
|
||||
## 后果
|
||||
|
||||
|
||||
Reference in New Issue
Block a user