docs: document package KV cache effects

2026-07-19 17:39:50 +08:00
parent b3c4b732d8
commit bcc920369c
93 changed files with 366 additions and 34 deletions
@@ -67,12 +67,16 @@ Unit tests use pi-ai catalog models redirected to local mock servers and cover p

 **Token effect**: Provider tokenization governs exact input. Conversion adds no model-visible text; replay metadata may let a native API reuse provider-side state.

+**KV Cache effect**: Conversion preserves logical request order without adding text, while the selected provider's serialization and replay state determine reuse. Changing adapter instance, provider, model, or any upstream request token may prevent reuse from the first difference.
+
 ### Provider response

 **What the model sees**: pi-ai events become harness reasoning, text, tool-call, usage, and finish chunks. Parsed tool arguments cross the harness boundary as raw JSON strings.

 **Token effect**: Generated content affects later inputs only after the loop records it. pi-ai folds reasoning tokens into output usage when the provider does not report them separately.

+**KV Cache effect**: Recorded response content appends to the next request and does not invalidate its earlier reusable prefix. Unrecorded transport metadata and usage accounting do not affect cache identity.
+
 ## Known Limitations and Deferred Work

 - **Catalog membership is required** — custom model ids that are absent from the installed pi-ai catalog fail with `UNKNOWN_MODEL`, even when a provider profile supplies a custom endpoint.