fix(tools): normalize the two class-name joins camelCase's own call misses

camelCase normalized `joined` and then prefixed, so the seam the `Tool`
prefix creates was never covered: `Tool` ends in `l`, a combining-mark
head composes with it, and a name headed by U+0301 was emitted as
`Tool` + U+0301 while CPython compiles `Too` + U+013A. childClassName
has the same shape -- both sides separately NFKC-stable, their join not:
a base ending in a Hangul L jamo or LV syllable composes with a V or T
jamo head. Beyond the declared-name/compiled-symbol mismatch, two
byte-distinct names can fold onto one, and usedClassNames dedupes by raw
bytes, so the collision counter never sees it. Normalize after the
prefix decision and at the join, before the cap. The remaining joins
need nothing: `Args`/`Output` and the digit suffix cannot compose
backwards.

Also record the Unicode-table skew. The predicate reads the engine's
tables (Node 22.23.1: 17.0) and the interpreter reads its own (CPython
3.9.6: 13.0.0), so an interpreter older than the engine takes a bare
name its tokenizer refuses -- U+1C89, U+10570, U+1E290 and U+1E4D0 are
accepted here and rejected there. The other direction only degrades a
legal name to subscript. Closing it needs the CPython floor, which the
backend PR owns; state the asymmetry in the docstring and make the
decision an explicit obligation in the note.
This commit is contained in:
Chinesezjc
2026-08-05 20:39:11 +08:00
parent 2cb0dddb40
commit 8c001d9928
5 changed files with 117 additions and 11 deletions
+48 -7
View File
@@ -17,7 +17,11 @@ import { assertSupportedJsonSchema } from './json-schema.ts'
import type { JsonSchemaNode, JsonSchemaScalar } from './json-schema.ts'
import type { ToolSdkSchema } from './ts-types.ts'
/** The reference grammar's `xid_start xid_continue*`, the same set `str.isidentifier()` accepts. */
/**
* The reference grammar's `xid_start xid_continue*` — the set
* `str.isidentifier()` accepts on a CPython whose Unicode tables match the
* engine's. See {@link isBareIdentifier} for what a version skew does.
*/
const IDENTIFIER = /^[\p{XID_Start}_]\p{XID_Continue}*$/u
/**
@@ -36,6 +40,24 @@ const IDENTIFIER = /^[\p{XID_Start}_]\p{XID_Continue}*$/u
* that normalize together would collapse into one declaration. Those names
* take the subscript path, which carries their exact bytes.
*
* Both conditions are evaluated against the ENGINE's Unicode tables, and the
* two sides are versioned independently — `\p{XID_Start}` follows the running
* engine (Node 22.23.1 reports Unicode 17.0) while CPython follows its own
* (3.9.6 reports 13.0.0). The skew is not symmetric. A CPython older than the
* engine is the dangerous direction: a character added to `XID_Start` since its
* tables (U+1C89, U+10570, U+1E290, U+1E4D0 are all NFKC-stable and accepted
* here, and all rejected by that 3.9.6) is emitted bare and its tokenizer
* refuses the character, taking the whole SDK block down — the same
* parseability invariant {@link UNPRINTABLE}, {@link LONE_SURROGATE} and
* {@link MAX_LIST_NESTING} exist for. A CPython newer than the engine only
* routes a legal name to the subscript path: less readable, still correct. The
* NFKC condition reduces to the same skew, since normalization stability
* guarantees an assigned character's normalization never changes afterwards.
*
* Closing the exposure needs the target interpreter's version, which the
* backend reporting `language: 'python'` owns and which is unpublished on this
* base; the note records it as that PR's decision.
*
* The `ts-types` sibling keeps its own ASCII rule rather than sharing this
* one: ECMAScript identifiers are a different set (`$`, ZWJ/ZWNJ) and are
* never normalized, so one predicate cannot be correct for both.
@@ -186,10 +208,19 @@ function docLines(description: unknown, indent: number): string[] {
* split words, `_` splits too (it is `XID_Continue`, so the split set names it
* explicitly), and a head that cannot start an identifier takes a `Tool`
* prefix. Unicode survives, so a `路径` field yields `路径`-based class names
* instead of collapsing to the bare prefix. The result is NFKC-normalized:
* these names are generated, never matched against a JSON key, so normalizing
* is free here and keeps what CPython compiles identical to what is emitted —
* unlike {@link isBareIdentifier}, which must reject unstable names outright.
* instead of collapsing to the bare prefix. A character that is not
* `XID_Continue` splits even when it is a letter, so a name whose NFKC folding
* would leave the identifier set is not carried through — the split set is the
* grammar's, not an ASCII approximation of it.
*
* The result is NFKC-normalized: these names are generated, never matched
* against a JSON key, so normalizing is free here and keeps what CPython
* compiles identical to what is emitted — unlike {@link isBareIdentifier},
* which must reject unstable names outright. Normalizing AFTER the prefix
* decision is what makes that hold at the seam the prefix creates: `Tool` +
* a combining-mark head composes there (`U+0301` gives `Tooĺ`, U+013A), so
* normalizing only the un-prefixed part would emit a name CPython compiles to
* a different symbol. The second call is idempotent on the un-prefixed arm.
* @param raw - the schema field or tool name to derive from.
* @returns a class-name segment safe to emit.
*/
@@ -200,7 +231,7 @@ function camelCase(raw: string): string {
.map(part => `${part.charAt(0).toUpperCase()}${part.slice(1)}`)
.join('')
.normalize('NFKC')
return /^\p{XID_Start}/u.test(joined) ? joined : `Tool${joined}`
return (/^\p{XID_Start}/u.test(joined) ? joined : `Tool${joined}`).normalize('NFKC')
}
/** Class-name base cap keeping each emitted name — and total text — linear in schema depth. */
@@ -291,9 +322,19 @@ function allocateClassName(base: string, state: RenderState): string {
* object-chain would otherwise carry an ever-growing ConsString down the tree
* and re-materialize it (via `.length`/`.slice`) at every level — Θ(depth²).
* The bounded base plus the collision counter still yields unique names.
*
* The join is NFKC-normalized because both sides are separately normalized yet
* their concatenation need not be: a base ending in a Hangul L jamo or LV
* syllable composes with a following V or T jamo head (`가` + `ᆨ` gives `각`),
* so the emitted class name would differ from the symbol CPython compiles, and
* two byte-distinct names could fold onto one — `usedClassNames` dedupes by the
* raw bytes, so the collision counter would not see it. Normalizing costs
* O(cap + segment) per level, the same order as the `slice` it feeds. The other
* two join points need no counterpart: `Args`/`Output` start with `A`/`O` and
* {@link allocateClassName}'s suffix is digits, none of which compose backwards.
*/
function childClassName(base: string, segment: string): string {
return capClassNameBase(`${base}${segment}`)
return capClassNameBase(`${base}${segment}`.normalize('NFKC'))
}
/**
+63 -2
View File
@@ -494,8 +494,69 @@ describe('renderToolsSdkPy', () => {
// whole characters; one ASCII character of padding puts the boundary inside
// the 60th pair, and that half is dropped rather than emitted.
expect(className('')).toBe(AHSA.repeat(60))
expect(className('x')).toBe(`X${AHSA.repeat(59)}`)
expect(className('x')).toHaveLength(119)
const padded = className('x')
expect(padded).toBe(`X${AHSA.repeat(59)}`)
expect(padded).toHaveLength(119)
})
it('normalizes the seam the Tool prefix creates, which the prefixed part alone does not cover', () => {
// U+0301 COMBINING ACUTE ACCENT is XID_Continue but not XID_Start, so a name
// headed by it takes the `Tool` prefix — and `Tool` ends in `l`, which
// composes with it. Normalizing only the part being prefixed would emit
// `Tool` + U+0301, which CPython compiles as `Too` + U+013A: the class
// the SDK declares would not be the class the interpreter defines. Every
// code point below is an escape — the two forms render identically.
const text = renderToolsSdkPy([
{
name: '\u0301abc',
description: 'Combining-mark head.',
parameters: { type: 'object', additionalProperties: false, properties: { q: { type: 'string' } }, required: ['q'] },
output: { type: 'string' },
},
])
expect(text).toContain('class Too\u013AabcArgs(TypedDict):')
expect(text).toContain('# tools["\u0301abc"](args: Too\u013AabcArgs) -> str')
expect(text).not.toContain('Tool\u0301')
})
it('normalizes a class-name join where two separately stable segments compose', () => {
// Hangul jamo compose ACROSS the join `childClassName` makes: the parent
// base ends in U+1100 (L jamo) and the child segment starts with U+1161 (V
// jamo), each NFKC-stable alone, together U+AC00. Unnormalized, the declared
// name differs from the compiled symbol, and two byte-distinct names can
// fold onto one — `usedClassNames` dedupes by raw bytes, so the collision
// counter never sees it and the later declaration shadows the earlier one
// under CPython. Escapes again, for the same reason as above.
const text = renderToolsSdkPy([
{
name: 'x',
description: 'Jamo field names.',
parameters: {
type: 'object',
additionalProperties: false,
required: ['\uAC00\u1100'],
properties: {
'\uAC00\u1100': {
type: 'object',
additionalProperties: false,
required: ['\u1161x'],
properties: {
'\u1161x': { type: 'object', additionalProperties: false, properties: { q: { type: 'string' } } },
},
},
},
},
output: { type: 'string' },
},
])
// The join is `XArgs` + U+AC00 U+1100 followed by U+1161 `x`, whose
// trailing L+V pair composes into a second U+AC00.
expect(text).toContain('class XArgs\uAC00\uAC00x(TypedDict):')
expect(text).toContain(' \u1161x: XArgs\uAC00\uAC00x')
expect(text).not.toContain('\u1100\u1161')
// The level above it is a join that composes nothing (LV + L), so it stays
// byte-identical — normalizing is not silently rewriting every name.
expect(text).toContain('class XArgs\uAC00\u1100(TypedDict):')
})
it('declares a closed empty object with omitted properties as an empty TypedDict, not dict[str, Any]', () => {