Merge remote-tracking branch 'origin/master' into xtr/agent-loop-message-machine

This commit is contained in:
_Kerman
2026-07-27 17:18:12 +08:00
144 changed files with 3233 additions and 827 deletions
@@ -0,0 +1,6 @@
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write
2026-07-26-consolidate-gate-scripts-on-existing-deps.md: 6370c8f92eff7296327e941e698ec4f733100bb2
2026-07-26-consolidate-gate-scripts-on-existing-deps.zh.md: 3587c92e0e9655d8b38d24d184c1c68c44b131d4
@@ -0,0 +1,33 @@
# Agent Note: Consolidate gate scripts on already-present deps and builtins
Status: implemented
English | [中文](2026-07-26-consolidate-gate-scripts-on-existing-deps.zh.md)
## Problem
The `scripts/` gates mostly used the right tools (`node:fs` `globSync` in 15+ gates, mdast/micromark in the markdown gates), but a handful of stragglers hand-rolled what a sibling gate already did with an existing dependency or builtin:
- **Duplicated fence scanners.** `scripts/md-fences.ts` (~55 lines, consumed by `doc-typecheck.ts`) and `extractEquivBlocks` in `scripts/verify-type-equiv.ts` (~39 lines) were two copies of the same regex line-scanner for fenced code blocks, while `scripts/verify-mermaid.ts` already extracted fences by visiting mdast `code` nodes — and `markdownProseLines` in `scripts/markdown.ts` itself parsed to mdast but then hand-tracked fence state with a second regex. The regex scanners only recognized backtick fences at column 0, so they silently disagreed with the mdast-based gates on tilde and indented fences.
- **Hand-rolled argv parsing.** `parseOptions` in `scripts/publint-all.ts` and its near-identical copy in `scripts/verify-built-package-invariants.mjs` (~26 lines) stepped argv indexes manually, while sibling scripts (`verify-runtime-closure.ts`, `build-exe-for-python-sdk.ts`, `packages/sdk/scripts/src/args.ts`) already used the `node:util` `parseArgs` builtin.
- **Hand-rolled directory walks.** Five sites re-derived nested `readdirSync` walks that `globSync` covers: `verify-runtime-closure.ts` (packages + vendor manifests), `dev-web.ts` `discoverPluginDirs`, `verify-package-paths.ts` `realPackageNames`, `verify-client-domain-graph.ts` `listSources`, and `publint-all.ts` `addPath` (~5565 lines total). `scripts/package-invariants.ts` shows the one-line `globSync` template.
No new dependency was needed anywhere; every replacement is an existing devDep or a Node builtin.
## Decision
- A shared mdast fence helper, `markdownFences` in `scripts/markdown.ts`, visits `code` nodes for the language, full info string, body, and 1-based opening-fence line; `doc-typecheck.ts` and `verify-type-equiv.ts` extract fences through it. `md-fences.ts` and the duplicated `extractEquivBlocks` scanner are deleted, and `markdownProseLines` derives fenced lines from the parsed `code` nodes' positions instead of a second regex.
- Both CLIs parse argv via `parseArgs`; unknown options and missing values still fail loud, with `parseArgs`'s own error text instead of the bespoke usage strings.
- The five straggler walks use `globSync`. The walks in `check-workspace-constraints.ts` and `clean.ts` stay: they need dirent-level detail to diagnose malformed trees, which glob-by-pattern cannot report.
## Alternatives considered
- **A new glob/walking dependency (`tinyglobby`, `fdir`).** Rejected: the builtin already won repo-wide; these were stragglers, not a gap.
- **`p-map` for `publint-all.ts`'s ~19-line ordered worker pool.** Deliberately left out: one new devDep for one small deletion is at the edge of the [dependency policy](../process/2026-07-26-dependencies-over-hand-rolling.md) bar, and the pool's requirements (bounded workers, deterministic order, env override) are documented in the [parallel-gates note](../process/2026-07-06-parallel-pre-push-gates.md). Fold it in only if `p-map` earns a second consumer.
- **Leaving the fence scanners.** Rejected: two drifting copies of a parser beside a third correct implementation is exactly the duplication the shared `markdown.ts` helper exists to prevent, and the column-0-backtick-only limitation was a latent inconsistency between sibling gates.
## Consequences
- One fence parser: every markdown gate now classifies fences through mdast, so tilde, indented, and 4-backtick container fences behave identically everywhere. The docs tree contained no fence shape the regex scanners mishandled, so gate results are unchanged on the tree that landed the swap: `pnpm run doc-sync` and each rewritten gate ran before and after with byte-identical output (`doc-typecheck` block/opt-out counts, `verify-type-equiv` match counts, `publint`, `verify-built-package-invariants`, `verify-runtime-closure`, `verify-package-paths`, `verify-client-domain-graph`, and both package-README prose gates).
- `verify-type-equiv` still rejects an unterminated type-equivalence fence: mdast silently closes an unterminated block at end-of-file (its comparisons could then pass), so the shared helper reports whether a closing delimiter exists and the gate errors on an unclosed block, preserving the removed scanner's rejection. The `doc-typecheck` scanner never had that error path.
- `parseArgs` keeps the last value of a duplicated option instead of erroring — a dev-tool edge case the tests don't pin, accepted in exchange for deleting the two bespoke parsers. (Strict mode still rejects a `--`-prefixed token where a value is expected, matching the replaced parsers.)
@@ -0,0 +1,33 @@
# Agent Note: 把门禁脚本统一到已有依赖与内置模块上
Status: implemented
[English](2026-07-26-consolidate-gate-scripts-on-existing-deps.md) | 中文
## 问题
`scripts/` 下的门禁大多本已在用正确的工具(15 个以上的门禁使用 `node:fs``globSync`markdown 门禁使用 mdast/micromark),但少数几个掉队的脚本曾手写同类门禁早已用既有依赖或内置模块完成的事情:
- **重复的围栏扫描器。**`scripts/md-fences.ts`(约 55 行,由 `doc-typecheck.ts` 消费)和 `scripts/verify-type-equiv.ts` 中的 `extractEquivBlocks`(约 39 行)曾是同一个围栏代码块正则行扫描器的两份拷贝,而 `scripts/verify-mermaid.ts` 早已通过访问 mdast `code` 节点来提取代码围栏;`scripts/markdown.ts` 自己的 `markdownProseLines` 也曾先解析成 mdast,再用第二个正则手工跟踪围栏状态。这两个正则扫描器只识别第 0 列的反引号围栏,因此在波浪线围栏和缩进围栏上与基于 mdast 的门禁悄悄不一致。
- **手写的 argv 解析。**`scripts/publint-all.ts` 中的 `parseOptions``scripts/verify-built-package-invariants.mjs` 中与之几乎相同的拷贝(约 26 行)曾手工推进 argv 下标,而同类脚本(`verify-runtime-closure.ts``build-exe-for-python-sdk.ts``packages/sdk/scripts/src/args.ts`)早已在使用 `node:util` 的内置 `parseArgs`
- **手写的目录遍历。**五处代码曾各自重写 `globSync` 已覆盖的嵌套 `readdirSync` 遍历:`verify-runtime-closure.ts` 对 packages 与 vendor manifest(元数据清单)的扫描、`dev-web.ts``discoverPluginDirs``verify-package-paths.ts``realPackageNames``verify-client-domain-graph.ts``listSources`,以及 `publint-all.ts``addPath`(合计约 5565 行)。`scripts/package-invariants.ts` 展示了一行式的 `globSync` 模板。
所有替换都不需要引入新依赖;每一处替换用的都是既有的 devDependency 或 Node 内置模块。
## 决策
- `scripts/markdown.ts` 中的共享 mdast 围栏辅助函数 `markdownFences` 访问 `code` 节点,读取语言、完整 info string、块体以及以 1 起始的开围栏行号;`doc-typecheck.ts``verify-type-equiv.ts` 通过它提取代码围栏。`md-fences.ts` 和重复的 `extractEquivBlocks` 扫描器已删除,`markdownProseLines` 也改为从解析出的 `code` 节点位置推导围栏内的行,而不再用第二个正则。
- 两个 CLI 都改用 `parseArgs` 解析 argv;未知选项和缺失取值仍然大声失败,只是错误文案换成了 `parseArgs` 自带的文本,而非原先手写的用法字符串。
- 那五处掉队的目录遍历改用 `globSync``check-workspace-constraints.ts``clean.ts` 中的遍历保留:它们需要 dirent 级别的细节来诊断结构异常的目录树,按模式匹配的 glob 报告不了这些信息。
## 曾考虑的替代方案
- **新的 glob/目录遍历依赖(`tinyglobby``fdir`)。**不予采纳:内置模块已在全仓库范围内胜出;这几处只是掉队者,不是能力缺口。
- **用 `p-map` 替换 `publint-all.ts` 中约 19 行的有序 worker 池。**刻意未纳入:为一次小删除引入一个新 devDependency,正处在[依赖策略](../process/2026-07-26-dependencies-over-hand-rolling.md)门槛的边缘,而且该池的需求(worker 数量有界、确定性顺序、环境变量覆盖)已记录在[并行 pre-push 门禁决策记录](../process/2026-07-06-parallel-pre-push-gates.md)中。仅当 `p-map` 赢得第二个消费方时再顺带纳入。
- **保留这两个围栏扫描器。**不予采纳:在第三个正确实现旁边放着两份逐渐漂移的解析器拷贝,正是共享的 `markdown.ts` 辅助函数要防止的那种重复;「只认第 0 列反引号」的限制也是同类门禁之间的潜在不一致。
## 后果
- 只剩一个围栏解析器:所有 markdown 门禁现在都经由 mdast 归类代码围栏,因此波浪线围栏、缩进围栏和四反引号容器围栏在各处的行为完全一致。文档树中不存在正则扫描器处理有误的围栏形态,所以在落地这次替换的代码树上门禁结果不变:`pnpm run doc-sync` 及每个被改写的门禁在改动前后各跑一遍,输出逐字节相同(`doc-typecheck` 的块数/opt-out 计数、`verify-type-equiv` 的匹配计数、`publint``verify-built-package-invariants``verify-runtime-closure``verify-package-paths``verify-client-domain-graph`,以及两个包 README 散文门禁)。
- `verify-type-equiv` 仍然拒绝未闭合的类型等价围栏:mdast 会在文件末尾静默闭合未闭合的代码块(其比较随后可能通过),因此共享辅助函数会报告闭合定界符是否存在,门禁在块未闭合时报错,保留了被删扫描器的这条拒绝路径。`doc-typecheck` 的扫描器本来就没有这条错误路径。
- `parseArgs` 对重复出现的选项保留最后一个值而不报错——一个测试未固定的开发工具边缘用例,作为删除两份手写解析器的交换被接受。(严格模式下,需要取值处遇到以 `--` 开头的 token 仍会拒绝,与被替换的解析器行为一致。)
@@ -0,0 +1,6 @@
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/simplification/2026-07-26-turndown-for-tool-web-html-markdown.md
2026-07-26-turndown-for-tool-web-html-markdown.md: 0e387021e3d3be3011cc0d64d37864b30aec4fdf
2026-07-26-turndown-for-tool-web-html-markdown.zh.md: 44e7d08c1db40a1203e8cda955b521774335e774
@@ -0,0 +1,37 @@
# Agent Note: Replace tool-web's regex HTML-to-markdown converter with turndown
Status: implemented
English | [中文](2026-07-26-turndown-for-tool-web-html-markdown.zh.md)
## Problem
`dsh-tool-web`'s `src/html.ts` (~86 lines, ~40 lines of dedicated tests; deleted by this change) converted fetched HTML to markdown with regexes: strip script/style/noscript/comments, convert `<a>`/`<h1-6>`/`<li>`, decode numeric entities plus a 12-entry named-entity table, collapse whitespace. The module's own JSDoc said "A richer converter can replace it without changing the seam or tool schema", and the README's Known Limitations documented it as "a minimal regex converter, not an HTML parser — tables, images, and nested formatting are lost." The [web capability seam note](../architecture/2026-06-24-web-capability-seam.md) assigns HTML→markdown to this package as presentation, so the swap point was exactly here. The converter's output is model-visible on every fetched HTML page; no keyless snapshot exercised `web_fetch`, so no expected outputs pinned it.
## Decision
`packages/web/tool-web/src/fetch.ts` owns a module-level [`turndown`](https://github.com/mixmark-io/turndown) instance (`headingStyle: 'atx'`, `codeBlockStyle: 'fenced'`, `bulletListMarker: '-'` — fixed model-facing presentation, not deployment tunables) with `@joplin/turndown-plugin-gfm`'s composite `gfm` plugin for tables/strikethrough and `remove(['script', 'style', 'noscript'])` replacing the old wholesale drops. `formatFetchOutput` limits both the source prefix converted synchronously and the complete rendered output with `fetchMaxOutputChars` (default 200,000), so a custom provider cannot make conversion work unbounded before the output cap applies. The HTML arm then guards conversion twice: a conservative linear lexical pass treats comment contents conservatively, skips raw-text bodies, honors quoted tag text, and passes a body through raw when its stack crosses 512 levels; a try/catch also falls back to raw HTML when turndown rejects markup the guard cannot model. The GFM cell rule is overridden to ignore `colspan`, which Markdown cannot represent, rather than letting an untrusted numeric attribute synthesize arbitrary empty cells. `html.ts` and its conversion tests are deleted; the source/output bounds, fallback, and status-header/truncation-footer formatting are tested in `tests/tool-web.spec.ts`, and the README's Known Limitations trades the regex-converter caveat for the bounded degradation cases. The gfm plugin ships no types; `src/turndown-plugin-gfm.d.ts` declares the one imported export over `@types/turndown` (a devDependency).
The dependency-weight question the proposal flagged resolves in favor of the swap: `@deepseek-ai/dsh-tool-web` is in the single-file-executable closure ([single-exe note](../architecture/2026-07-10-single-file-executable-sdk-runtime-distribution.md)), and the exe's asset globs would pack ~7.9 MB of the three packages as published — but ~6 MB of that is `@mixmark-io/domino`'s test corpus (`test/**`), with runtime `lib/` at ~550 KB against a ~174 MB artifact, under 0.5% either way.
## Snapshot coverage
The previously-missing keyless `web_fetch` snapshot ships with the change as the acp-agent scenario `web-fetch`: `examples/acp-agent/web.cordis.yml` composes the web seam, the real `dsh-web-fetch-local` provider, `tool-web` with `search: false`, and `web-fetch-fixture-server.mjs` — a loopback HTTP fixture on a fixed port (the fetched URL is part of the recorded transcript) serving deterministic HTML with named entities, a GFM table, and nested formatting. Recording and keyless replay both drive the real HTTP fetch and conversion; the pinned tool result is the turndown output, and the scenario pins the `web` header class (the `web_fetch` schema and guidance).
## Alternatives considered
- **`@mozilla/readability` + a DOM.** Solves a different problem (content extraction, not conversion) and drags a heavier DOM dependency; the seam only asks for markdown rendering of whatever the fetch returned.
- **Keep the regex converter.** It was an explicit v1 placeholder per its own JSDoc; keeping it meant model-visible quality (tables, images, nested formatting) stayed lost for the cost of maintaining bespoke entity tables.
- **The minimal `entities`-only variant.** The proposal's fallback position: replace only the entity-decoding third of `html.ts` with the zero-dependency `entities` package, deleting less but avoiding the dependency-weight question. Not taken because the closure math above made the weight immaterial while the full swap deletes the whole hand-rolled converter and its documented quality gaps.
- **`turndown-plugin-gfm` (the original) instead of `@joplin/turndown-plugin-gfm`.** The original is unmaintained (last publish 2018); the Joplin fork is current against turndown 7 and actively released.
## Consequences
- **Bought**: standards-based model-visible markdown — ordinary tables, images, strikethrough, nested emphasis, fenced code blocks, and the complete named-entity set — plus the deletion of the bespoke converter and its entity tables.
- **Paid**: two runtime dependencies (`turndown``@mixmark-io/domino`) enter tool-web and therefore the exe closure (~550 KB of runtime code as measured above); overlong input is converted only through a bounded prefix, pathological nesting falls back to raw HTML, and spanning table cells are flattened because GFM has no corresponding syntax.
- Model-visible output changed on every fetched HTML page; nothing pinned the old output, and the new snapshot pins the new one.
## Testing
- `packages/web/tool-web/tests/tool-web.spec.ts` covers the turndown conversion surface (entities, links, tables, nesting, script/style/noscript removal), ignored table spans, source-prefix and complete-output bounds, fast raw-HTML passthrough for deep or deceptively closed nesting, linear handling of malformed tags, the residual converter-throw fallback, and exact and tiny output budgets; per-file coverage on the package src is 100%.
- The `web-fetch` acp-agent snapshot pins the assembled behavior keylessly end to end (real Loader composition, real HTTP fetch, real conversion).
@@ -0,0 +1,37 @@
# Agent Note: 用 turndown 替换 tool-web 的正则 HTML 转 markdown 转换器
Status: implemented
[English](2026-07-26-turndown-for-tool-web-html-markdown.md) | 中文
## 问题
`dsh-tool-web``src/html.ts`(约 86 行,另有约 40 行专属测试;已由本变更删除)曾用正则表达式把抓取到的 HTML 转成 markdown:剥离 script、style、noscript 标签与注释,转换 `<a>`/`<h1-6>`/`<li>`,解码数字实体外加一张 12 项的命名实体表,并折叠空白。该模块自身的 JSDoc 写明「A richer converter can replace it without changing the seam or tool schema」,README 的 Known Limitations 章节也把它记载为「a minimal regex converter, not an HTML parser — tables, images, and nested formatting are lost」。[web 能力 seam 决策记录](../architecture/2026-06-24-web-capability-seam.md)把 HTML 转 markdown 作为呈现职责划归本包(package),因此替换点恰好就在这里。每个抓取到的 HTML 页面上,该转换器的输出都对模型可见;此前没有任何无密钥快照执行到 `web_fetch`,因此没有预期输出固定它的行为。
## 决策
`packages/web/tool-web/src/fetch.ts` 持有一个模块级 [`turndown`](https://github.com/mixmark-io/turndown) 实例(`headingStyle: 'atx'``codeBlockStyle: 'fenced'``bulletListMarker: '-'`——固定的面向模型呈现方式,不是部署可调项),配合 `@joplin/turndown-plugin-gfm` 的组合 `gfm` 插件提供表格/删除线支持,并用 `remove(['script', 'style', 'noscript'])` 替代旧实现的整体剥离。`formatFetchOutput` 通过 `fetchMaxOutputChars`(默认 200,000)同时限制同步转换的源前缀和完整渲染输出,因此自定义提供方无法在输出上限生效前造成无界的转换工作。随后,HTML 分支对转换做双重防护:保守的线性词法扫描会保守处理注释内容,跳过原始文本元素的内容,正确处理标签内的引号文本,并在栈深超过 512 层时将主体作为原始 HTML 直接透传;当 turndown 拒绝守卫无法建模的标记时,try/catch 同样回退为原始 HTML。GFM 单元格规则被覆写为忽略 `colspan`;Markdown 无法表示它,这也避免了不受信任的数值属性凭空合成任意数量的空单元格。`html.ts` 及其转换测试已删除;源/输出上限、回退以及状态头/截断页脚格式化均在 `tests/tool-web.spec.ts` 中有测试覆盖,README 的 Known Limitations 用有界降级情形替换了正则转换器警示。gfm 插件不带类型声明;`src/turndown-plugin-gfm.d.ts` 基于 `@types/turndown`devDependency)声明了唯一被导入的导出。
提案标记的依赖体积问题的裁决结果支持替换:`@deepseek-ai/dsh-tool-web` 在单文件可执行文件闭包内([single-exe 决策记录](../architecture/2026-07-10-single-file-executable-sdk-runtime-distribution.md)),可执行文件的资产 glob 会把这三个包按发布原样打入约 7.9 MB——但其中约 6 MB 是 `@mixmark-io/domino` 的测试语料(`test/**`),运行时 `lib/` 仅约 550 KB,相对约 174 MB 的产物,两种口径都不到 0.5%。
## 快照覆盖
此前缺失的无密钥 `web_fetch` 快照随本变更以 acp-agent 场景 `web-fetch` 落地:`examples/acp-agent/web.cordis.yml` 组合了 web seam、真实的 `dsh-web-fetch-local` 提供方、`search: false``tool-web`,以及 `web-fetch-fixture-server.mjs`——一个固定端口(抓取的 URL 是录制 transcript(文本记录)的一部分)上的回环 HTTP fixture,提供包含命名实体、GFM 表格与嵌套格式的确定性 HTML。录制与无密钥回放都驱动真实的 HTTP 抓取与转换;固定住的工具结果就是 turndown 的输出,该场景同时固定 `web` header 类(`web_fetch` 的 schema 与指引)。
## 曾考虑的替代方案
- **`@mozilla/readability` 加一个 DOM。** 它解决的是另一个问题(内容提取,而非格式转换),还会拖入更重的 DOM 依赖;这个 seam 只要求把抓取返回的内容渲染成 markdown。
- **保留正则转换器。** 按其自身 JSDoc 的说法,它本来就是明确的 v1 占位实现;保留它意味着模型可见的质量(表格、图片、嵌套格式)继续缺失,代价还是维护一套自制实体表。
- **仅引入 `entities` 的最小变体。** 提案中的退守方案:只用零依赖的 `entities` 包替换 `html.ts` 中的实体解码部分,删得更少但完全避开依赖体积问题。未采纳:上述闭包测算表明体积无关紧要,而完整替换能删掉整个手写转换器及其记录在案的质量缺口。
- **用原版 `turndown-plugin-gfm` 而非 `@joplin/turndown-plugin-gfm`。** 原版已无人维护(最后发布于 2018 年);Joplin 分叉与 turndown 7 保持同步并持续发布。
## 后果
- **收益**:基于标准的模型可见 markdown——普通表格、图片、删除线、嵌套强调、围栏代码块以及完整的命名实体集——并删除了自制转换器及其实体表。
- **代价**:两个运行时依赖(`turndown``@mixmark-io/domino`)进入 tool-web 进而进入可执行文件闭包(如上实测约 550 KB 运行时代码);超长输入只转换有界前缀,病态嵌套回退为原始 HTML,跨列表格单元格会被展平,因为 GFM 没有对应语法。
- 每个抓取到的 HTML 页面上模型可见的输出都已变化;旧输出本无任何固定,新快照固定了新输出。
## 测试
- `packages/web/tool-web/tests/tool-web.spec.ts` 覆盖 turndown 转换面(实体、链接、表格、嵌套、script/style/noscript 移除)、被忽略的表格跨列、源前缀与完整输出上限、深层或带欺骗性闭合嵌套的快速原始 HTML 透传、畸形标签的线性处理、残余的转换器抛错回退,以及恰好达到上限和极小的输出预算;该包 src 的逐文件覆盖率为 100%。
- acp-agent 的 `web-fetch` 快照无密钥地端到端固定组装后的行为(真实 Loader 组合、真实 HTTP 抓取、真实转换)。