docs(tools): record LS/PS as tokenizer non-terminators, with a test

Review read `JSON.stringify`'s raw pass-through of U+0085/U+2028/U+2029 as a
parse hazard: an LS in a `Literal[...]` value or in a `# tools["..."]` comment
would end the physical line and take the SDK block down. Measured on CPython
3.9.6 (Unicode 13.0) and 3.12.13 (15.0): all three are accepted in both a
string literal and a `#` comment, value round-tripping, and only LF and CR
terminate either. The set is the tokenizer's, not `str.splitlines()`'.

Both existing claims were accurate, so nothing changes behaviorally. Name the
distinction where it was assumed: `UNPRINTABLE`'s terminator sentence now says
which set it means, and `pyScalar`'s raw-pass-through list, previously "DEL and
the C1 controls", now also names LS/PS, which are neither. A test pins the raw
form for `const` and `enum` so escaping them later cannot land as a silent
divergence from the TypeScript flavor.
This commit is contained in:
Chinesezjc
2026-08-05 22:25:45 +08:00
parent ab0c275494
commit c5b09c108f
2 changed files with 25 additions and 4 deletions
@@ -67,6 +67,19 @@ describe('jsonSchemaToPy', () => {
expect(jsonSchemaToPy({ type: 'string', const: 'ends\\' })).toBe(String.raw`Literal["ends\\"]`)
})
it('passes the paragraph separators through raw, which CPython does not treat as line terminators', () => {
// `JSON.stringify` escapes LF and CR but not LS/PS (U+2028/U+2029), which
// is safe here and not by accident: they are `str.splitlines()` boundaries,
// not tokenizer line terminators, so they end neither a string literal nor
// a `#` comment — measured on CPython 3.9.6 and 3.12.13. Pinning the raw
// form keeps a later "escape them for symmetry with LF" change from
// landing as a silent both-flavors divergence from `ts-types`.
// Escapes below — the two forms denote the same bytes, and neither
// character has a visible width.
expect(jsonSchemaToPy({ type: 'string', const: 'a\u2028b' })).toBe('Literal["a\u2028b"]')
expect(jsonSchemaToPy({ type: 'string', enum: ['a\u2029b'] })).toBe('Literal["a\u2029b"]')
})
it('emits exact digits for a beyond-safe-range integer literal', () => {
// Python integers are arbitrary-precision, so the emitted digits ARE the
// value the model programs against. `String(2 ** 60)` prints the rounded