From 207677e9948dc5699073a4d6620d21139033c72d Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 22 Nov 2026 19:08:29 +0000 Subject: [PATCH] Prepare the v0.25.0 release documentation --- README.md | 6 + docs/releases/v0.25.0.md | 88 ++++ docs/roadmap/implementation.md | 448 ---------------- docs/roadmap/invocation-scoped-token-usage.md | 497 ------------------ 4 files changed, 94 insertions(+), 945 deletions(-) create mode 100644 docs/releases/v0.25.0.md delete mode 100644 docs/roadmap/implementation.md delete mode 100644 docs/roadmap/invocation-scoped-token-usage.md diff --git a/README.md b/README.md index da3d8ef..10ae644 100644 --- a/README.md +++ b/README.md @@ -45,6 +45,12 @@ boundary and constraints that framework work must preserve. ## Release Guidance +The [v0.25.0 release notes](docs/releases/v0.25.0.md) cover invocation-scoped +observation, provider conventions, reasoning and cache usage, and optional USD +cost reporting. Upgrading consumers should review provider selection, +authentication diagnostic defaults, and the corrected OpenRouter cache-write +mapping. + The [v0.24.0 release notes](docs/releases/v0.24.0.md) cover application-owned configuration loading, prompt-owned schemas, default output repair, and the prepared message-content hash. Upgrading consumers should review the breaking diff --git a/docs/releases/v0.25.0.md b/docs/releases/v0.25.0.md new file mode 100644 index 0000000..854abc1 --- /dev/null +++ b/docs/releases/v0.25.0.md @@ -0,0 +1,88 @@ +# Promptkit v0.25.0 + +This minor release adds invocation-scoped execution observation, provider-specific +reasoning and cache usage, and optional USD cost reporting. Backend definitions +can now select API conventions independently of their names and endpoints. +Consumers upgrading from v0.24.x should review the configuration and accounting +changes below. + +## Added And Changed + +- `WithExecutionObserver` selects observation through a caller-owned context + across nested, concurrent, coordinated, and multi-engine calls. Request + observers run before contextual observers. Prepared handles retain their + explicit request observer but select contextual observers when executed, + not when prepared. Each execution retains independent accounting; groups + count each run's first terminal once. See the + [observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run) + and [offline example](../../examples/go-library/observe/main.go). +- Backend `provider` selects `openai-compatible`, `openrouter`, or `vllm` + conventions. Omitted, null, or blank selects the generic OpenAI-compatible + behavior. The bundled OpenRouter backend explicitly selects `openrouter`. + `InspectBackend` exposes declared provider and `zero_cost` settings without + manufacturing runtime defaults. See the + [backend reference](../formats.md#backend-definitions). +- Usage values, presence indicators, observations, and group summaries include + `ReasoningTokens`. OpenRouter and vLLM decoding includes their respective + reasoning, cache-read, and cache-write fields. Reasoning is a subset of + completion tokens; cache activity does not add to reported token totals. + `ObservedTokenUsage.CacheHitRatio` derives a ratio from complete, consistent + input and cache-read totals. See the + [response contract](../integrations/openai-compatible-chat.md#response-handling). +- `RunResult.CostUSD`, execution events, and group summaries report a USD + subtotal with availability, completeness, and source metadata. OpenRouter + supplies optional `usage.cost`; injected clients can supply + `GenerateResponse.CostUSD`. Explicit backend `zero_cost: true` supplies a + zero fallback when no usable amount is reported. A reported amount, including + zero, takes precedence. See the [cost GoDoc](../../cost.go). +- Authentication diagnostics can use provider defaults: `openrouter_key` for + OpenRouter and the new `vllm_models` adapter for vLLM. The latter probes + base-relative `/models` and reports recognized response facts without + asserting authentication enforcement or inference authorization. See the + [diagnostic integration contract](../integrations/openai-compatible-chat.md#explicit-smoke-diagnostics). + +## Compatibility And Upgrade + +- For application-defined OpenRouter backends, set `provider: openrouter` to + enable OpenRouter-specific usage and cost decoding and its authentication + default. Backend IDs and endpoint hostnames do not select conventions. + Generic backends retain their existing request and token conventions and + do not decode monetary cost. Unsupported nonblank provider names are + configuration errors; supplied names are trimmed and lowercased. +- OpenRouter cache-write reporting now reads + `usage.prompt_tokens_details.cache_write_tokens`, not the generic top-level + `usage.cache_write_tokens` field. vLLM uses + `usage.prompt_tokens_details.created_cache_tokens`. Missing or unusable new + details remain unknown rather than invalidating otherwise usable content. + Availability depends on the provider and server settings; consult the + [response contract](../integrations/openai-compatible-chat.md#response-handling). +- A supplied authentication object no longer requires `kind`: an omitted or + null kind uses the provider default. Clearing the authentication block also + restores that default. Set `kind: none` to declare authentication testing + inapplicable. The generic provider has no authentication default; explicit + adapters remain supported independently of provider selection. Probes still + run only when explicitly requested. See the + [backend reference](../formats.md#backend-definitions). +- Cost includes all dispatched attempts, including transient retries and + structural repairs, and is available on successful results without an + observer. On operational failure, observations can retain cost even though + no partial result is returned. Existing `RunResult.Usage` continues to sum + successful generation responses; presence-aware observations cover all + dispatched attempts. Do not add terminal totals to attempt totals. +- Unknown cost is distinct from reported or configured zero. No backend is + automatically declared zero-cost, and no token-rate estimation is performed. + Group cost completeness describes included terminals, independently of + terminal coverage. Consumers own presentation and interpretation; complete + reporting does not verify billing. +- Result JSON adds `cost_usd`, and token JSON adds `reasoning_tokens`. + Update strict decoders if necessary. Older result JSON without cost decodes + to unknown cost. Consumers using positional literals for expanded public + structs may need to update them. See the [public types](../../types.go). + +The Go consumer minimum remains 1.27.0; dependencies are unchanged from v0.24.0. + +```sh +go get go.maximumdirect.net/promptkit@v0.25.0 +``` + +[COPYRIGHT](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.25.0/COPYRIGHT) | [LICENSE](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.25.0/LICENSE) | [ATTRIBUTION](https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.25.0/docs/oss/attribution.md) diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 52f3f5e..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,448 +0,0 @@ -# Invocation Observation And Usage Implementation Plan - -## Objective And Working Rules - -Implement the accepted [target-state roadmap](invocation-scoped-token-usage.md). -It owns scope, consumer intent, and reporting policy. This plan owns concrete -implementation decisions and the ordered work sequence. Baseline: `a47df4e`. - -Each numbered stage is one Luna-sized implementation prompt. Execute them in -order within one end-to-end sprint. Intermediate stages may leave unresolved -references or failing compilation; do not add compatibility scaffolding merely -to keep those intermediate states green. There are no intermediate human-review, -external-publication, or live-provider gates. Run focused checks when their -dependencies are ready; the completed sprint must pass the final checks. - -Read [development guidance](../development.md) and the applicable agent, -architecture, testing, and documentation policies. Follow -[ADR 0001](../adr/0001-configuration-boundary.md): interpretation and validation -use final ConfigKit values, not raw sources or historical inspection. Follow the -shared-settings facade exception without introducing public aliases for runtime -implementation types. Preserve unrelated work and follow the agent commit policy. - -Planning and implementing are separate tasks. This document does not authorize -a release, dependency upgrade, CLI implementation, or changes to published notes. - -## Selected Design - -- Store contextual observers as internal `domain.ExecutionObserver` callbacks - under a context key separate from the recorder, claim sink, and cost state. - The public helper wraps public event conversion at registration. Internal - composition preserves explicit-first and inherited-to-new contextual ordering. -- Add `Provider *string` and `ZeroCost *bool` to shared backend settings and - `BackendConfig`, with ordinary `provider`/`zero_cost` field names. Resource - inspection returns declared normalized values; configuration inspection owns - history. Neither inspection API needs a new provenance model. -- Capture runtime `Provider string` and `ZeroCost bool` on `domain.Backend` and - carry them on internal `domain.ExecutionTarget` as local metadata excluded - from JSON/YAML. Empty provider resolves to `openai-compatible` at runtime. - Keep these fields out of the root execution-target API and override API: - backend inspection is the requested public configuration surface. Endpoint-only - profiles use the generic provider and disabled zero fallback. Existing - model/endpoint overrides retain the selected backend's metadata. -- Keep provider ID recognition, response interpretation, and authentication - adapter resolution in `internal/llm`, using simple functions/dispatch. Config - validation may call that package's supported-name checks as it already does - for reserved request fields. No public provider registration mechanism or new - providers package is needed. -- Add `ReasoningTokens int` to token values, `ReasoningTokens bool` to presence, - and `ReasoningTokens ObservedTokenCount` to observed usage. Public token JSON - uses `reasoning_tokens`. Add the roadmap's two helper methods/functions exactly - as specified; do not add aliases for cache reads or a stored cache-hit ratio. -- Define public `CostSource string` constants `CostSourceNone = ""`, - `CostSourceProvider = "provider"`, `CostSourceConfiguredZero = "configured_zero"`, - and `CostSourceMixed = "mixed"`. Define `ObservedCost` with `Value float64`, - `Known bool`, `Complete bool`, and `Source CostSource`, JSON-tagged `value`, - `known`, `complete`, and `source`, without omitting zero fields. -- Add `GenerateResponse.CostUSD *float64` (`json:"cost_usd,omitempty"`) and - `RunResult.CostUSD ObservedCost` (`json:"cost_usd"`). Add `CostUSD ObservedCost` - to `ExecutionEvent` and `ExecutionGroupSummary`; the latter uses `cost_usd`. - Define equivalent internal domain values and explicit facade conversions. - Older result JSON lacking cost decodes to unknown cost, not reported zero. -- Own pure cost resolution/merge and run-local cost state in - `internal/observation`. Cost is collected once per dispatched attempt regardless - of observer selection. The recorder references that same run accumulator for - terminal cost; it must not independently add attempts a second time. Groups - reuse the merge rules but remain terminal-only collectors. -- New optional reasoning/cache-write details use local tolerant integer decoding; - cost uses local tolerant finite-float decoding. A typed `*int`/`*float64` field - in the main envelope would fail the whole decode for malformed optional data. - Preserve existing common token decoding, framing, content validation, and - error identities. Do not add a general numeric coercion framework. - -## Stage 1 — Context Helper And Ordinary Run Selection - -**Completed.** - -Touch `observations.go`, the root `Engine.Run` in `engine.go`, and context support -in `internal/observation/recorder.go` or a focused adjacent file. - -- Implement `WithExecutionObserver(context.Context, ExecutionObserver) - context.Context` using a private internal observer-selection key. Nil is a - no-op. Compose inherited callbacks before the new callback using the existing - composition facility; keep registration state immutable. -- In ordinary `Run`, compose a converted explicit request observer with the - contextual selection before deciding whether to create a recorder. Preserve - current start timing, run IDs, public error mapping, timeout finalization, - permit release, and terminal delivery. With neither observer, preserve the - existing no-observation path. -- Do not mask the contextual key in `WithoutExecutionState`; continue masking - inherited recorder and claim state. Each nested public call creates its own - recorder. Do not modify `Coordinate` to append contextual selection: its - explicit request/coordinator composition already leads to ordinary `Run`. -- Add focused public tests for no observer, contextual-only selection, explicit - plus nested context ordering, nil registration, unrelated contexts, and two - independent run IDs from a nested injected-client call. Group duplication - uses existing terminal deduplication, not callback equality checks. - -Validation when buildable: root observation/boundary contract tests and focused -internal observer tests. No provider requests are needed. - -## Stage 2 — Execution-Time Context For Prepared Handles - -**Completed.** - -Touch root `RunPrepared`, internal `usecase/prepared_execution.go`, and observer -conversion only where required. - -- Retain only the explicitly supplied request observer in the prepared payload. - Do not snapshot contextual observers during `PrepareExecution` or capture - preparation contexts in the handle. -- After a successful claim, compose that observer with the contextual selection - from the execution context before recorder creation. Preserve the private - claim sink so root terminal delivery still follows public error mapping and - finalized timeout attribution. -- Preserve silence on preparation, discard, nil/wrong-engine/already-used handle - rejection, and existing payload clearing. Do not expose private callbacks in - `Details` or create a second lifecycle around the claimed execution. -- Test a handle prepared in scope A and executed in B: retained explicit - observer and B receive events; A does not. Include a context-only execution, - rejected claims, and a continuation selected by `Coordinate` to protect its - explicit-before-context order. - -Validation: prepared execution and coordinate observation contract tests. - -## Stage 3 — Backend Configuration And Provider Selection - -**Completed.** - -Touch `internal/settings/settings.go`, `backends.go`, configuration ownership, -resolution/validation code, and the provider-selection functions in `internal/llm`. - -- Add the two pointer fields and presence-preserving source-helper encoding. - Nil omits a field; false and blank pointers remain supplied values. Raw - ConfigKit mappings remain the way to supply explicit null. -- Normalize supplied provider strings with trimming/lowercasing in the existing - final resolution callback and record normalizations using existing ConfigKit - APIs. Do not materialize the runtime default into the settings object. -- Recognize exactly `openai-compatible`, `openrouter`, and `vllm`; empty means - generic. Reject other nonempty values through existing final configuration - validation/error wrapping. Do not validate overwritten contributions. -- Validate `zero_cost` as the typed Boolean setting; absent/null/false disables - fallback. Do not infer it or make `LocalBackend` set it automatically. -- Relax authentication-object validation to allow absent/null kind, retain - existing kind normalization and supported explicit kinds, and add - `vllm_models`. A missing kind with no provider default remains an unsupported - diagnostic selection, not a new load-time requirement for a provider. -- Cover layered final-value validation, inheritance, null clearing, normalization, - unsupported names, helper presence, and absence of invented defaults. - -Validation: focused configuration/settings/source-helper tests. - -## Stage 4 — Runtime Capture, Freezing, And Backend Inspection - -**Completed.** - -Touch `internal/configuration/targets.go`, `ownership.go`, `resources.go`, domain -backend/target values and clones, and `bundled/openrouter.yaml` under that package. - -- Copy declared settings into owned records and inspection results. Preserve - explicit false while allowing absent/null collapse in resource inspection. - Leave full null/omission history to `InspectConfiguration`. -- Capture effective provider and zero policy on the internal backend and the - backend-derived execution target. Ensure scalar copying through profile - capture, runtime overrides, prepared details/payloads, and repair requests. - Retain backend metadata when a profile/request overrides only its endpoint. -- Keep root public execution targets and provider HTTP bodies unchanged. Audit - serialization/copy helpers so excluded internal fields are not accidentally - lost by a JSON-based copy. Do not recapture policy from configuration during - `RunPrepared` or traverse provenance during execution. -- Set the bundled OpenRouter backend's provider explicitly; do not add zero-cost - defaults or new backend instances. Keep API keys and default profiles unchanged. -- Test renamed/namespaced/inherited OpenRouter backends, generic endpoint-only - profiles, frozen prepared policy, and inspection without manufactured values. - -Validation: configuration capture, backend inspection, bundle, and preparation tests. - -## Stage 5 — Generic Authentication Dispatch And OpenRouter Defaults - -**Completed.** - -Current provider branches occur in `internal/usecase/smoke_authentication.go` and -`smoke_selection.go`, including `ProbeOpenRouterKey` and endpoint derivation. - -- Replace the OpenRouter-specific collaborator operation with a generic - authentication-probe operation owned by `internal/llm`. Carry a resolved - adapter selection to it; leave usecase coordination provider-neutral. -- Resolve adapter kind and destination with pure provider-boundary functions. - Explicit kind wins, absent/null kind uses the provider default, and `none` - is inapplicable. Explicit endpoint wins; otherwise use the adapter-relative - fallback. Missing selection remains unsupported. Do not rewrite loaded - settings, captured declared smoke metadata, or inspection provenance. -- OpenRouter defaults to `openrouter_key`; generic has no default. Preserve - explicit OpenRouter adapters on any backend and existing HTTP evidence, - bounded recognition, credential, cancellation, and overflow handling. -- Clear an authentication block to restore the provider default; an endpoint-only - block may use that default. Preserve reachability and inference selection. -- Update affected interfaces, engine assembly, and test doubles. Test default, - explicit/none, endpoint-only, cleared-block, and unsupported combinations at - their owning boundaries rather than duplicating every permutation publicly. - -Validation: smoke selection/authentication tests and existing OpenRouter contract tests. - -## Stage 6 — vLLM Models Authentication Adapter - -**Completed.** - -- Implement `vllm_models` beside the OpenRouter adapter, sharing generic HTTP - execution and bounded recognition helpers rather than duplicating transport. - Select it by default for `vllm` and allow explicit use on other providers. -- Resolve `models` beneath the configured API base, preserving base prefixes; - honor an explicit full diagnostic endpoint. Do not hard-code a host or probe - `/health`. Use existing optional credential rules without forcing a key. -- Recognize exactly the roadmap's HTTP 200 object/list/data-array shape, - including empty arrays. HTTP 401 is rejection; other/unrecognized responses - follow existing unknown/failure reporting. Do not inspect list members for - model existence, inference permissions, or authentication enforcement. -- Test accepted empty/nonempty lists, wrong shapes/framing/overflow, 401, other - statuses, optional credentials, overridden destinations, and `kind: none`. - Ensure unsupported injected HTTP diagnostics still do not fall back to a - built-in transport. - -Validation: focused adapter and smoke coordinator tests with controlled HTTP. - -## Stage 7 — Reasoning Values And Provider-Specific Token Decoding - -**Completed.** - -Touch public/domain token declarations, facade conversions, and -`internal/llm/openai_compatible_client.go` plus focused provider files. - -- Add `ReasoningTokens` to `TokenUsage`, `TokenUsagePresence`, and observed usage - on both sides of the facade; include JSON field/conversions and mutable-response - ownership where applicable. Do not rename `CachedTokens`. -- Keep common counts and cached reads on their existing decode path. For - OpenRouter, read reasoning from `completion_tokens_details.reasoning_tokens` - and writes from `prompt_tokens_details.cache_write_tokens`; for vLLM, read - the same reasoning path and `prompt_tokens_details.created_cache_tokens`. - Generic keeps its existing top-level write mapping and does not interpret the - new reasoning field. Ignore other providers' alternative write locations; - do not add/fallback between the nested and top-level values. -- Decode newly interpreted integer fields independently with local raw JSON - helpers. Absent/null, wrong-type, fractional, or int-overflow values become - unknown. Do not coerce strings. Retain independently valid fields; adding a - typed optional field must not create a new whole-response failure. Existing - malformed common fields and framing/content failures retain their behavior. -- Preserve decoded usage on later missing-content/choice failure as today. - Do not infer missing reasoning or writes, enforce new nonnegative restrictions, - change generation controls, or collect reasoning text. -- Test provider-specific raw fixtures with conflicting write locations, - partial/malformed new fields, known zero, and disabled-detail vLLM responses. - Keep fixtures synthetic; reviewed upstream paths are in the feature roadmap. - -Validation: model-client parsing tests and public token JSON round trips. - -## Stage 8 — Token Aggregation And Cache-Hit Helper - -**Completed.** - -Touch observation usage accumulators/recorder/group, runner `addTokenUsage`, -public event/group conversions, and related fixtures. - -- Include reasoning counts in existing successful-result summation, attempt and - terminal usage, group merging, and explicit/legacy injected presence behavior. - Do not replace legacy result token sums with all-attempt observation sums. -- Implement `ObservedTokenUsage.CacheHitRatio() (float64, bool)` using the - roadmap's known/complete/positive-input/valid-read bounds. Unknown or invalid - data returns `(0, false)`; known zero reads with positive input returns `(0, true)`. - Divide totals as floats, not integer division. Do not persist or average ratios. -- Cover per-field incomplete reasoning, injected known zero, retry/repair sums, - duplicate group enrollment, weighted aggregate ratios, zero denominator, - inconsistent counts without generation failure, and independence from writes. -- Preserve reported total tokens: reasoning and cache details are subsets, not - extra values to add into input/output/total counts. - -Validation: focused token accumulator/group tests and public observation contracts. - -## Stage 9 — Public Cost Values And Shared Cost Algebra - -**Completed.** - -Create public cost declarations in a focused root file and matching domain -values; add the selected fields to responses, results, events, and summaries. - -- Implement the selected cost/source declarations and conversions. Copy optional - amount pointers at the injected-client conversion boundary so caller mutation - cannot change retained data. Include cost in custom result JSON marshal/unmarshal - paths as needed, preserving existing duration and omission behavior elsewhere. -- Implement pure internal cost resolution: finite reported amount first, then - configured zero, else unknown. A reported zero remains provider-sourced. - Negative finite reported amounts are not newly prohibited. Unknown cost is - zero-valued with false Known/Complete and empty source. -- Implement one small `CostAccumulator` in `internal/observation` that merges - known values, ORs Known, ANDs Complete, and unions sources. Its empty snapshot - is known/complete zero; its first unknown contribution must reset that empty - special case rather than accidentally making unknown cost known. Include - known zero contributions in source union. Unknown contributions add no source. -- Allow merging aggregate costs for groups with the same helper; leave the - existing no-terminal group special case to group coverage logic. -- Unit-test precedence, source union including zeros, all-unknown/empty cases, - completeness, and approximate floating-point sums. Add JSON tests for old - result JSON without cost, current round trips, and optional client zero. - -Validation: pure accounting and public serialization tests; no pricing library. - -## Stage 10 — Optional Cost At The Model-Client Boundary - -**Completed.** - -- Decode OpenRouter `usage.cost` independently using local tolerant finite-float - handling. Missing/null/string/object/overflow values are unknown; invalid - framing or common envelope fields retain existing failures. Generic/vLLM do - not interpret the field. Do not consume `cost_details` or send a new usage flag. -- Carry raw optional cost on built-in `AttemptOutcome` and `GenerateResponse` - even when a later decoded-content check fails. Non-2xx/invalid framing remain - without provider usage. Do not apply zero-cost fallback inside provider code. -- Preserve optional cost supplied through an injected client. The adapter already - converts non-nil responses alongside errors: retain cost for accounting even - then, without changing existing token treatment or public error/result rules. - Normalize nonfinite injected cost to unknown before applying fallback. -- Test cost independent of token fields, usable zero, malformed optional cost - without generation failure, ignored cost on generic/vLLM, and reported cost - retained alongside a classified injected error. - -Validation: provider decoder and injected-adapter tests. - -## Stage 11 — Per-Attempt Cost Collection And Observation - -**Completed.** - -Touch generation `recordAttempt` and built-in/injected execution paths, -observation attempt data/events, recorder terminal cost, and group aggregation. - -- At every attempt completion, if dispatched, resolve raw cost against the - frozen target's ZeroCost and add it exactly once to an invocation accumulator. - Do this for transport errors, eligible retry statuses, cancellation after - dispatch, malformed bodies, and missing content as well as success. A failed - acquisition/pre-dispatch cancellation contributes nothing. -- Keep cost collection independent of the optional recorder and diagnostic - evidence capture flag. Preserve permit-release-before-callback behavior. - Do not count retry waits, terminal events, or generation responses again. -- Carry resolved cost on attempt events. Recorder terminal cost reads the same - accumulator; groups merge only first terminal costs via shared algebra. - Extend public event/group projection without changing existing token totals. -- Introduce private cost-state context access. Extend `WithoutExecutionState` to - mask inherited cost state, but not contextual observer selection. Smoke - diagnostics must continue masking private execution accounting and emit no - ordinary lifecycle or enclosing-run cost contributions. -- Exercise controlled retry/error sequences and zero-cost fallback without an - observer. Test duplicate terminal delivery and independence of group terminal - coverage from cost completeness. Do not retain an attempt history for cost. - -Validation: generation, observation recorder/group, and diagnostic-boundary tests. - -## Stage 12 — Ordinary Run Cost Lifetime And Result Accounting - -**Completed.** - -- Create/install a new private accumulator at the public ordinary Run boundary, - before optional recorder creation. Pass that same instance to the recorder. - Ensure direct internal runner execution creates a local accumulator if one - was not supplied, without replacing one supplied by the owning facade. -- Retain it through initial generation and all repair calls. Set successful - `domain.RunResult.CostUSD` from its snapshot, not by summing responses. Keep - the legacy `Usage` success-response sum and error handling unchanged. -- Ensure errors still return no partial public `RunResult`; available cost is - reported in terminal observations. Runs rejected before dispatch have the - established empty aggregate, not a configured-zero contribution source. -- Test a retry with unreported cost followed by a reported charge, mixed sources - within repairs, reported-cost precedence over zero policy, and identical - successful cost metadata with observation enabled/disabled. - -Validation: assembled ordinary run/reliable execution contract tests. - -## Stage 13 — Prepared Cost Lifetime And Nested Isolation - -**Completed.** - -- In internal RunPrepared, after successful claim create/install its own cost - accumulator before recorder creation and generation. Use the same result and - terminal accounting path as ordinary Run. Do not store an accumulator or - contextual observer in the reusable preparation details/private ready payload. -- The claimed handle uses its frozen backend Provider/ZeroCost, even when run - from another invocation scope. Preserve admission, deadline attribution, - single-claim semantics, discard behavior, and cleared callbacks/credentials. -- Verify ordinary/prepared child runs launched from injected clients or - prepared callbacks do not inherit the parent's accumulator, even when both - inherit one contextual group observer. Each terminal contributes once to the - invocation total; parent cost includes only its own dispatched generations. -- Cover prepared zero-cost failure and repair, rejected claims without events, - independent contexts on a reused engine, and diagnostic calls nested in a run - without cost/event leakage. - -Validation: prepared and nested-boundary contracts, including race-enabled focused tests. - -## Stage 14 — End-To-End Consumer Regression Coverage - -**Completed.** - -- Add one representative offline invocation workflow using two engines and - concurrent runs under one group. Use controlled transport/injected clients - for OpenRouter reported cost, vLLM details/configured zero, a retry, and a - failed operation after available usage. Join application-owned goroutines - before taking the final snapshot. -- Assert independently predictable terminal membership, sums, sources, token - presence/completeness, and weighted ratio (or its unavailability when an - attempt leaves input/cache-read reporting incomplete). Compare floating-point - sums with tolerance; do not assert map order, run-ID bytes, or exact timing. -- Cover a snapshot with completed fully costed members and one started member: - cost completeness concerns the included terminals while terminal coverage is - false. Cover empty and members-with-no-terminals snapshots separately. -- Reuse narrower tests from earlier stages rather than reproduce every case in - this workflow. Keep observations metadata-only; do not add bodies, credentials, - URLs, or error objects to the event contract. - -Validation: focused new integration tests plus `go test -race ./...` when the -implementation has reached its final buildable state. - -## Stage 15 — Canonical Documentation And Final Verification - -**Completed.** - -- Finalize GoDoc for helpers, backend fields, reasoning counts, cost values and - sources, client ownership, and observer/prepared lifetimes. Update stale - authentication Kind requirements in shared settings and public docs. -- Update the consumer guide and maintained offline observation example for - invocation scoping, cache/reasoning subsets, known/complete cost, configured - zeros, sources, and failure-path reporting. Maintain semantic example tests. -- Update `docs/formats.md`, the OpenAI-compatible integration contract, and - internal overview/observation/generation/model-client/runner documentation for - actual ownership and flow. Explain provider defaults versus declared inspection, - vLLM reporting settings, and the exact diagnostic response recognition. -- Document migration from required authentication kind and block-clearing - semantics. Record the change in the next unpublished release note when such a - note exists; otherwise leave release-note creation to release preparation. - Do not edit published v0.24.0 notes or tag/publish anything in this sprint. -- Use the newest already installed Go toolchain per the development guide, - `GOTOOLCHAIN=local`, and `GOWORK=off`. Format changed Go files, run - `go test -race ./...`, `go vet ./...`, and `git diff --check`. Reuse successful - results for unchanged content; do not add redundant builds or live-provider - checks. Verify local Markdown links and maintained examples through their - normal offline tests. Review the entire diff for unrelated changes and secrets. -- Audit completion against every target-state section, especially no-observer - cost, failed dispatched attempts, optional-field tolerance, prepared policy - freezing, and nested/diagnostic separation. Resolve remaining implementation - defects before reporting sprint completion. Roadmap retirement is later - housekeeping, not part of this implementation plan. - -There are no unresolved product decisions required to begin these stages. diff --git a/docs/roadmap/invocation-scoped-token-usage.md b/docs/roadmap/invocation-scoped-token-usage.md deleted file mode 100644 index 1582153..0000000 --- a/docs/roadmap/invocation-scoped-token-usage.md +++ /dev/null @@ -1,497 +0,0 @@ -# Invocation-Scoped Execution Observation - -Status: accepted scope; implementation complete. Current behavior is documented -in the linked canonical contracts; roadmap retirement is separate housekeeping. - -## Purpose And Scope - -Let consumers select observation once for an application-owned operation and -obtain usage across its nested, concurrent, coordinated, and multi-engine -PromptKit calls. Extend that reporting with reasoning tokens, cache activity, a -token cache-hit ratio, and optional USD cost with its source and completeness. - -CommandKit's optional `ckprompt` integration motivates this work, but the API -remains application-neutral. PromptKit owns observation, provider response -interpretation, and aggregation. Consumers own context propagation, waiting for -their work, invocation lifecycle, presentation, and interpretation. - -This document defines the target state and the changes required to reach it. -A separate implementation plan will own sequencing and prompt-sized stages. - -## Current Baseline - -PromptKit v0.24.0 already provides -[execution observers and ordered composition](../../observations.go), -a concurrency-safe [execution group](../../execution_group.go), and -[request-level observation](../consumers/pkg-promptkit.md#observe-an-ordinary-run). -Groups aggregate the first terminal event for each observed run. Existing -observation distinguishes known, unknown, and incomplete prompt, completion, -total, cached, and cache-write token counts across retries and repairs. - -Today, consumers explicitly enroll requests. Prepared handles retain the request -observer; [coordinator observers](../../coordinate.go) select continuation runs. -There is no contextual selection, reasoning-token count, cache-hit helper, or -monetary cost reporting. OpenRouter cache writes also need their documented -nested mapping: the current client reads `usage.cache_write_tokens` instead. - -## Contextual Observer Selection - -Add the public helper: - -```go -func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context -``` - -The returned context composes the supplied observer with any inherited contextual -observers. A nil observer leaves that selection unchanged. Executions using the -context, or derived contexts retaining its values, participate automatically; -unrelated contexts do not. No engine option or process-wide default is needed. - -A consumer can create an invocation-owned group, associate `group.Observe` with -the context, run its pipeline, and snapshot the group after its owned work -finishes, including when the pipeline returns an error. Group coverage describes -observed members, not future work or overall application progress. - -### Delivery And Execution Boundaries - -- `Run` selects contextual observers from its invocation context. -- `RunPrepared` selects them from its execution context, not the context used - to prepare the handle. The explicitly supplied request observer remains - retained by the handle and composes with those execution-time observers. -- Preparation emits no execution events. Discarded handles and rejected - execution claims retain their existing observation behavior. -- Coordinated continuations acquire contextual observation through their - ordinary `Run` boundary. `CoordinateRequest.Observer` still selects - continuation runs only; additional calls made by application callbacks - participate when they use the scoped context. -- Every execution, including a nested one using a callback or injected-client - context, has its own identity, recorder, and totals. Clearing an enclosing - execution's private state at a public boundary preserves contextual observer - selection, not the enclosing recorder. - -Deliver events to explicit request observers first, then any coordinator observer -already composed with them, then contextual observers from inherited to newly -registered. Incorporate contextual selection once per execution, not again in -forwarding or coordination layers. - -Preserve synchronous callback delivery and panic behavior. Concurrent runs may -invoke a shared observer concurrently; consumers synchronize their callback state. -The existing group remains concurrency-safe. - -Registering the same callback through multiple paths may deliver duplicate -events; arbitrary Go functions are not deduplicated. Retain one lifecycle per -execution and reuse the group's terminal-event deduplication. Terminal totals -are aggregates, not additional contributions to add to attempt totals. - -## Backend Provider And Accounting Metadata - -Add two optional backend fields: - -| Field | Meaning | -| --- | --- | -| `provider` | Selects the supported API conventions, independently of backend identity, namespace, endpoint hostname, and model vendor. | -| `zero_cost` | Enables a configured-zero cost fallback when no usable reported cost is available. | - -For example: - -```yaml -promptkit: - backends: - - id: research-router - provider: openrouter - endpoint: https://openrouter.ai/api/v1 - api_key_env: OPENROUTER_API_KEY - - id: local-inference - provider: vllm - endpoint: http://localhost:8000/v1 - zero_cost: true -``` - -Trim and lowercase the final effective provider value. Omitted, null, empty, and -whitespace-only values select `openai-compatible`. Reject unsupported nonempty -normalized values as ordinary configuration errors. The supported selections are -`openai-compatible`, `openrouter`, and `vllm`. - -`zero_cost` is Boolean. Omitted, null, or false disables its fallback; true -enables it. It is independent of provider selection and does not suppress a -usable reported cost. - -Resolve both fields through ordinary final ConfigKit composition and inheritance. -Carry their effective selections through capture and prepared execution without -reloading configuration. Declare `provider: openrouter` in the bundled -OpenRouter backend. Do not automatically declare any backend zero-cost. - -These fields are local metadata, not outbound request parameters or entries -added to `extra_params`. Provider selection does not choose an endpoint or model, -and endpoint overrides retain the selected provider and accounting policy. - -### Inspection Boundary - -`InspectBackend` shows the normalized/resolved declared provider and zero-cost -setting, preserving explicit false for `zero_cost`. Omitted/null declarations -may remain absent, and blank provider values need no extra history representation. -Do not manufacture an explicit `openai-compatible` declaration or false -`zero_cost` value from runtime defaults. - -`InspectConfiguration` owns ConfigKit source history and explicit/null/omitted -distinctions. Resource inspection does not acquire source-history fields or -null-tracking machinery. Provider diagnostic defaults likewise remain runtime -fallbacks, not values written into loaded configuration or recorded as explicit -source declarations. - -## Provider Conventions - -All selections reuse the existing -[OpenAI-compatible text Chat Completions foundation](../integrations/openai-compatible-chat.md): -base-relative `/chat/completions` JSON POST requests, existing credential rules, -text messages, selected generation controls and structured output, extra -parameters, first-choice content extraction, common token fields, and existing -response validation, errors, cancellation, timeouts, and transient retries. - -| Provider | Additional response interpretation | Default authentication adapter | -| --- | --- | --- | -| `openai-compatible` | Existing token mappings; no provider-reported monetary cost decoding. | None. | -| `openrouter` | OpenRouter reasoning/cache details and `usage.cost` in USD. | `openrouter_key`. | -| `vllm` | vLLM reasoning/cache details; no provider-reported monetary cost decoding. | `vllm_models`. | - -The default provider names PromptKit's current supported conventions, not the -entire OpenAI API. "Lowest common denominator" does not strip existing optional -features or promise every backend supports them. Continue sending selected -supported settings and return backend errors normally. - -Provider selection supplies no default endpoint, model, credential environment -variable, or sampling settings, and does not create backend definitions. Keep -vLLM reasoning, chat-template, and other model-specific controls caller-selected; -do not automatically enable thinking or translate those controls. No new -generation protocol, capability negotiation, or version discovery is needed. -vLLM documents its -[OpenAI-compatible interface](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/). - -### Authentication Diagnostics - -Provider defaults apply only to explicitly requested authentication diagnostics. -Loading configuration, constructing an engine, and ordinary generation do not -initiate probes. Existing reachability and inference-default behavior remains. - -Explicit `smoke_tests.authentication` metadata overrides provider defaults: - -- A supplied `kind` selects the adapter; `none` declares authentication testing - inapplicable. -- Omitted/null `kind` uses the provider default, allowing endpoint-only overrides. -- A supplied endpoint overrides the selected adapter's destination; omitted/null - uses that adapter's existing fallback. -- An absent/null authentication block removes its overrides and leaves provider - defaults applicable. `kind: none`, not clearing the block, suppresses them. - -Without a provider default or explicit adapter, preserve the existing unsupported -diagnostic outcome rather than assuming authentication is inapplicable. Explicit -adapters remain selectable without a matching provider declaration. - -Retain `openrouter_key` behavior. The new `vllm_models` adapter makes one GET to -`/models` relative to the configured API base, supplying an available credential -under existing rules and allowing dispatch without one. A base ending in `/v1` -therefore produces `/v1/models`; explicit diagnostic endpoint overrides apply. - -For `vllm_models`, HTTP 200 with a valid JSON object containing `object: "list"` -and array-valued `data` means acceptance of the probe request. An empty array is -accepted; recognition does not inspect model capabilities or require a particular -model. HTTP 401 means rejection; unrecognized responses leave acceptance unknown. -Reuse existing bounded diagnostic decoding and framing. Report observed -credential presence and response facts without claiming authentication is -enforced or models-list acceptance establishes inference authorization. - -vLLM documents `/v1/models` as protected when API-key authentication is configured; -`/health` is not an authentication check. See its -[authentication documentation](https://docs.vllm.ai/en/stable/usage/security/#api-key-authentication-limitations). -No health-monitoring facility is added here. - -## Token And Cache Reporting - -### Provider Response Mapping - -Use common `usage.prompt_tokens`, `usage.completion_tokens`, and -`usage.total_tokens` for `PromptTokens`, `CompletionTokens`, and `TotalTokens`. -Normalize these additional provider details, relative to `usage`: - -| Public field | OpenRouter | vLLM | -| --- | --- | --- | -| New `ReasoningTokens` | `completion_tokens_details.reasoning_tokens` | `completion_tokens_details.reasoning_tokens` | -| Existing `CachedTokens` | `prompt_tokens_details.cached_tokens` | `prompt_tokens_details.cached_tokens` | -| Existing `CacheWriteTokens` | `prompt_tokens_details.cache_write_tokens` | `prompt_tokens_details.created_cache_tokens` | - -OpenRouter documents these in its -[response schema](https://openrouter.ai/docs/api_reference/overview), -[usage-accounting guide](https://openrouter.ai/docs/cookbook/administration/usage-accounting), -and [caching guide](https://openrouter.ai/docs/guides/best-practices/prompt-caching). -vLLM's [response construction](https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/chat_completion/serving/) -uses `created_cache_tokens`; its -[output definitions](https://docs.vllm.ai/en/stable/api/vllm/outputs/) -describe local prefix-cache writes, not a billable write charge. - -Preserve `CachedTokens` as the public name and clarify its meaning as cache reads. -Use each provider's documented cache-write field, not a sum or guessed alias; -a missing nested value is unknown even if an alternative field is present. -Retain the generic provider's existing mappings, including its top-level -`usage.cache_write_tokens` handling. - -Add `ReasoningTokens` to public/domain token values, presence indicators, observed -usage, conversions, and result serialization, and carry it through attempt events, -terminal events, and group summaries. Injected clients may supply canonical -counts and presence through the existing model-client boundary, retaining its -legacy `UsagePresence` fallback; explicit presence distinguishes measured zero. - -### Availability And Optional Metadata - -Omitted/null details are unknown, not zero. Explicit integer zero is known zero. -For the newly interpreted provider-specific reasoning and cache-write details, -wrong-type, nonintegral, or unrepresentable integer values are also unknown. -An unusable new detail does not invalidate otherwise usable content or erase -independently available usage. Preserve existing common-field decoding and -response framing/content validation; this is a targeted optional-metadata rule, -not a general relaxation of the response contract. Keep this interpretation local -to the model-client boundary without a general numeric-conversion framework. - -vLLM detail availability depends on version and configuration: prompt details -currently require `--enable-prompt-tokens-details`, and reasoning details depend -on a configured reasoning parser. Document those reporting settings without -configuring or discovering them automatically. Older or differently configured -servers may omit details and still produce usable responses. Do not estimate -reasoning counts from text or cache writes from input minus cache reads. - -### Count Relationships And Aggregation - -Reasoning is included in completion tokens, not additional output to add to them. -OpenRouter explains this in its -[reasoning documentation](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens); -vLLM similarly includes reasoning detail within generated-token usage. Cache reads -and writes are input-token details, not extra input consumption. Preserve the -reported total without recomputation or reconciliation. - -Consumers can derive non-reasoning output from consistent, known completion and -reasoning counts; no additional stored count is needed. Capturing counts does -not capture reasoning text or change generation/caching settings. - -Reuse existing per-field token accounting: aggregates sum known contributions, -`Known` identifies available reporting, and `Complete` describes its coverage -across dispatched attempts. Preserve initial-generation, retry, and repair -observation accounting and terminal-only group deduplication. Existing integer -result usage remains distinct from presence-aware observations; a zero result -field alone does not establish that the provider reported it. - -### Token Cache-Hit Ratio - -Add a derived helper, not another stored total: - -```go -func (u ObservedTokenUsage) CacheHitRatio() (float64, bool) -``` - -Compute `CachedTokens / PromptTokens` as a fraction from zero to one. This is the -proportion of input tokens served from cache, not request hit frequency or cost -savings. For aggregates, divide summed reads by summed input; do not average -individual percentages or use `reads / (reads + writes)`. - -Return an available ratio only when input and cache-read counts are known and -complete, input is positive, and reads lie between zero and input inclusive. -Known zero reads with positive input yields an available zero. Missing, -incomplete, zero-denominator, or inconsistent data yields zero and false while -preserving raw counts; it does not fail generation or clamp reported values. -Cache-write availability is irrelevant. - -The same helper serves attempts, terminals, and group summaries. A group ratio -describes its included observations and does not supersede terminal coverage or -imply all application work has finished. - -## Cost Reporting - -### Reported Cost And Configured-Zero Precedence - -Initially, only the OpenRouter built-in provider interprets `usage.cost` as the -account charge in USD. See its -[usage-accounting reference](https://openrouter.ai/docs/cookbook/administration/usage-accounting) -and [USD explanation](https://openrouter.ai/blog/insights/cost-vs-quality-tradeoff-framework-for-agent-models/). -Do not add `cost_details` amounts to that charge or interpret arbitrary providers' -fields named `cost`. The generic and vLLM providers do not decode monetary cost. - -Use the cost included in the ordinary response; no extra request or usage lookup -is needed. At the model-client boundary, represent an optional USD amount as -`*float64`: nil means unknown; a pointer to zero means reported zero. Injected -clients may supply the same value regardless of provider selection. - -Resolve cost at completion of each dispatched attempt, outside provider-specific -decoding and whether or not decoding occurred or succeeded: - -- A usable provider/model-client-reported cost takes precedence, including zero. -- Otherwise, `zero_cost: true` on the effective backend supplies configured USD zero. -- Otherwise, cost remains unknown. - -This applies to built-in and injected clients, including failed or canceled -dispatched attempts and undecodable responses. Without a selected backend -declaration, no configured-zero fallback applies. Non-dispatched work contributes -no cost. Do not infer zero cost from local hosting, provider name, credentials, -or token counts, or install the declaration automatically on bundled backends. - -An unusable reported amount, including a nonnumeric value or one not representable -as a finite `float64`, is unknown before fallback; it does not make an otherwise -usable generation fail. Keep available cost from a decoded successful envelope -when a later choice/content check fails. Invalid framing and non-2xx bodies supply -no provider usage, although the configured-zero fallback still applies to a -dispatched attempt. Existing framing, content, and error handling remain intact. - -### Monetary Representation And Sources - -Expose cost separately from token counts and their presence indicators. Use -explicit USD field naming such as `CostUSD` and a small public cost value for -results, events, and group summaries with: - -- `Value float64`: the available USD amount or subtotal; -- `Known bool`: availability; -- `Complete bool`: coverage under the selected cost policy; and -- `Source`: the contribution-source discriminator below. - -| Source | Meaning | -| --- | --- | -| `provider` | Supplied by the provider or injected model-client response; identifies the reporting path, not independent billing verification. | -| `configured_zero` | Supplied by the explicit backend fallback. | -| `mixed` | An aggregate includes both reported and configured-zero contributions. | -| Empty | No known contribution has a source, including an unknown cost or an aggregate with no dispatched attempts. | - -Use binary floating point for operational telemetry, without rounding to cents -or another fixed scale during capture or aggregation. Consumers choose display -precision. Exact-decimal ledgers, decimal dependencies, fixed-scale currency -rounding, pricing inference, and invoice reconciliation are not part of this work. - -### Attempt, Run, And Group Accounting - -A dispatched attempt with usable reported cost or configured zero is known and -complete for cost, independently of token completeness. Sum known amounts across -all dispatched attempts, including retries and repairs. Aggregate `Known` means -at least one known contribution; `Complete` means every dispatched attempt has a -cost from either source. Unknown contributions add no amount and make coverage -incomplete. Completeness does not assert that every provider reported a charge. - -Aggregate sources by union of known contributions, including zero amounts. -Unknown contributions add no source: a provider-only subtotal can be incomplete -without becoming `mixed`. A reported zero plus a configured zero is `mixed`. - -Successful results expose cost even without an observer. Result and terminal -cost agree on amount, availability, completeness, and source, and enabling -observation does not change result cost. For example, an unreported failed -attempt followed by a reported USD 0.002 yields an incomplete provider-source -subtotal when no zero-cost fallback applies. - -Groups aggregate terminal costs only, counting each run's first terminal once. -Combine known amounts, OR `Known`, AND `Complete`, and union sources. Preserve -existing empty/no-terminal group semantics: a completed execution with no -dispatched attempts yields known, complete zero with empty source; an empty group -does likewise, while a group with members but no terminals has unknown, -incomplete usage. - -Cost completeness concerns included terminal observations, independently of -`TerminalCoverageComplete`. Two completed runs with full cost coverage can yield -a complete subtotal while a third observed run lacks a terminal. Conversely, -fully reported OpenRouter runs and explicitly zero-cost local runs can yield a -complete, mixed-source total. Neither case verifies a provider invoice or future -application progress. - -## Internal Responsibilities And Scope Limits - -Provider-specific metadata interpretation and authentication conventions belong -in `internal/llm`, sharing common transport, request construction, credentials, -framing, and response fields. Use one provider-selection mechanism there. -Generation/observation coordination consumes normalized facts or generic -diagnostic operations, replacing the OpenRouter-specific diagnostic collaborator -method with a generic operation while retaining explicit adapter selection. - -Provider-specific files and straightforward dispatch suffice; no public provider -registration API, new `internal/providers` package, or broad mandatory-capability -interface is needed. This is configuration-driven selection among supported -implementations, not downstream registration of implementations. - -Reuse a small shared cost-accounting component for result and observation paths -so dispatched attempts, fallback, and source union follow the same rules even -when observation is disabled. Provider code interprets wire metadata; common -accounting owns fallback and aggregation. No ConfigKit provenance traversal -belongs in ordinary execution. - -Smoke diagnostics retain their separate report/evidence model and are not -enrolled in ordinary execution observation. CLI integration, output delivery, -persistent history, logging infrastructure, reasoning-text capture, changes to -reasoning/caching request behavior, server administration/metrics, additional -built-in cost decoders, multiple currencies, configurable token rates, fixed -nonzero request prices, and cost estimation remain outside scope. - -## Work Needed And Acceptance - -The following areas describe required changes, not an execution order. - -- Extend the public context API and ordinary/prepared execution wiring while - preserving independent nested lifecycles, explicit observers, coordinator - selection, and existing preparation/discard/rejected-claim behavior. -- Extend backend settings, typed helpers, final-value normalization/validation, - capture, prepared metadata, and inspection for `provider` and `zero_cost`. - Update the bundled OpenRouter declaration. Preserve configuration provenance - and resource-inspection simplicity, and keep metadata out of request bodies. -- Consolidate provider behavior in `internal/llm`, adding vLLM usage normalization - and the models-list diagnostic alongside OpenRouter interpretation and the - shared default path. Preserve common decoding while tolerating unusable new - optional details and cost locally to the model-client boundary. -- Extend public/domain token types, presence, serialization, conversions, events, - and groups with reasoning counts and correct provider-specific cache writes. - Add the presence-aware cache-hit helper without changing existing token - meanings or introducing a separate ratio accumulator. -- Carry optional reported cost and configured-zero policy through attempt - completion and shared accounting into results, events, and groups, including - source metadata and observer-independent result accounting. - -Select focused offline checks under the [testing policy](../policy/testing.md). -Use real internal collaborators and controlled HTTP/injected-client boundaries, -with proportionate public contract coverage. Important behaviors include: - -- Context inheritance, registration order, nil/duplicate enrollment, concurrent - calls, multiple engines, independent scopes, nested calls, and execution-time - selection for a handle prepared elsewhere. -- Provider selection independent of backend identity; normalization and unsupported - value handling; omitted/null/blank defaults; inheritance, endpoint overrides, - explicit false, and inspection without manufactured values or source history. -- Provider-default and explicit diagnostic selection, endpoint-only overrides, - clearing metadata versus `kind: none`, vLLM acceptance including an empty list, - rejection, unrecognized/framing-limited responses, and optional credentials. -- Provider token mappings, absent/null/zero details, malformed new optional counts - without generation failure, unchanged common-field handling, and responses - representative of vLLM servers with detail reporting disabled. -- Cache-hit availability, inconsistent counts without generation failure, - token-weighted aggregates, independence from cache writes, and no double-counting - of reasoning or cache details. -- Reported cost precedence, configured-zero fallback on dispatched failure and - cancellation, built-in/injected clients, retries/repairs, mixed/unknown sources, - zero-valued contributions, and independently incomplete token reporting. -- Result/terminal cost agreement with observation enabled or disabled, duplicate - terminal deduplication, empty/no-terminal groups, and subtotal completeness - distinct from terminal coverage. - -Synthetic fixtures and offline examples suffice; live providers are not required. -Keep ownership localized and avoid repeating the same assertions across layers. - -## Documentation And Migration - -When implemented, exact exported contracts belong in GoDoc; the -[consumer guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run) should show -invocation-scoped aggregation, reasoning/cache breakdowns without double-counting, -and cost availability, source, and completeness. Document provider wire mappings, -optional-metadata handling, and diagnostic response recognition in the -[integration contract](../integrations/openai-compatible-chat.md#response-handling). -The [framework format reference](../formats.md#backend-definitions) owns the new -backend fields, default selection, inheritance, and diagnostic overrides. -Update internal documentation for changed ownership and flows. - -Explain vLLM's server reporting settings, unknown-versus-zero behavior, and cache -creation meaning without promising availability on every deployment. Consumers -own presentation. Models-list acceptance does not establish inference -authorization, and complete cost metadata does not independently verify billing. - -Release notes should call out that authentication objects no longer require -`kind` when a provider default supplies it, endpoint-only overrides are supported, -and clearing an authentication block restores provider defaults. `kind: none` -suppresses that default. Preserve existing explicit adapters without requiring a -provider declaration.