diff --git a/docs/roadmap/invocation-scoped-token-usage.md b/docs/roadmap/invocation-scoped-token-usage.md index e73fee1..7880781 100644 --- a/docs/roadmap/invocation-scoped-token-usage.md +++ b/docs/roadmap/invocation-scoped-token-usage.md @@ -8,9 +8,11 @@ Allow consumers to select execution observation once for an application-owned operation, rather than attach an observer to every PromptKit request. Nested, concurrent, and multi-engine calls using that operation's context should participate automatically. Include optional provider-reported cost alongside -token usage, with built-in cost support initially limited to OpenRouter. Extend -OpenRouter and vLLM token reporting with reasoning counts, documented cache reads -and writes, and a shared derived token cache-hit ratio. +token usage, with built-in cost decoding initially limited to OpenRouter and an +explicit backend zero-cost fallback. Distinguish reported and configured cost +sources in individual and aggregate metadata. Extend OpenRouter and vLLM token +reporting with reasoning counts, documented cache reads and writes, and a shared +derived token cache-hit ratio. The motivating consumer is CommandKit's optional `ckprompt` integration, which could use this facility to report token usage for an entire command or pipeline. @@ -31,8 +33,9 @@ PromptKit v0.24.0 already provides: execution, and [CoordinateRequest.Observer](../../coordinate.go) for coordinator-selected continuation runs. -The missing capabilities are contextual observer selection, provider-reported -cost accounting, reasoning-token counts, and a presence-aware cache-hit helper. +The missing capabilities are contextual observer selection, cost accounting and +source metadata, an explicit zero-cost declaration, reasoning-token counts, and +a presence-aware cache-hit helper. OpenRouter cache writes also need their documented nested response mapping; the current client reads a top-level `usage.cache_write_tokens` field instead. Applications currently have to enroll each request explicitly, and PromptKit @@ -186,11 +189,12 @@ conventions. Preserve the existing shared behavior documented in the transient retries. This selection supplies no default endpoint, model, API-key environment variable, -or sampling settings. It does not interpret monetary cost or provide a default -authentication-probe adapter. Without an explicit authentication adapter, report -the existing unsupported diagnostic outcome rather than assuming authentication -is inapplicable. Existing reachability and inference diagnostics and explicit -authentication adapters remain available. +or sampling settings. It does not decode provider-reported monetary cost or +provide a default authentication-probe adapter. The backend zero-cost declaration +below remains available independently of provider selection. Without an explicit +authentication adapter, report the existing unsupported diagnostic outcome rather +than assuming authentication is inapplicable. Existing reachability and inference +diagnostics and explicit authentication adapters remain available. "Lowest common denominator" identifies the shared protocol foundation; it does not remove existing optional features or promise every backend supports them. @@ -211,9 +215,11 @@ thinking, or translate model-specific controls based on the provider label. No separate generation protocol is needed; vLLM documents an [OpenAI-compatible Chat Completions API](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/). -The token mappings are specified below. Monetary cost remains unknown, not zero, -including when vLLM is locally hosted. This provider adds no price estimator, -server administration, metrics collection, or automatic capability/version probe. +The token mappings are specified below. vLLM supplies no provider-reported cost; +cost remains unknown unless the backend explicitly declares the zero-cost +fallback below. Local hosting alone does not select that fallback. This provider +adds no price estimator, server administration, metrics collection, or automatic +capability/version probe. ### Smoke-Test Defaults And Explicit Overrides @@ -419,7 +425,7 @@ For example, input 1,250 and cache reads 250 yield 0.2 (20%) when both counts are known and complete, even if cache writes are unknown. Output 840 with reasoning 600 still represents 840 output tokens, not 1,440. -## Optional Provider-Reported Cost +## Optional Cost Reporting ### Initial Provider Support @@ -433,14 +439,54 @@ top-level charge; those describe different facts and could double-count cost. For the built-in client, select this interpretation through `provider: openrouter`, regardless of backend identity or namespace. Do not infer USD semantics from an arbitrary backend's field named `cost`, or add endpoint discovery or verification. -Built-in cost interpretation is an extension unique to `openrouter` initially; -`openai-compatible` and `vllm` leave cost unknown even if a response contains -`usage.cost`. No separate cost-reporting selector is needed. Aggregation remains -provider-neutral after the model-client boundary supplies interpreted USD facts. +Built-in provider-reported cost interpretation is an extension unique to +`openrouter` initially; `openai-compatible` and `vllm` do not interpret +`usage.cost`. Without an applicable zero-cost declaration, their built-in cost +remains unknown. No separate provider cost-decoding selector is needed. +Aggregation remains provider-neutral after the model-client boundary supplies +interpreted USD facts. Use the cost included in the normal completion response. No additional usage lookup, pricing request, or opt-in provider request parameter is needed. +### Explicit Backend Zero-Cost Declaration + +Add optional Boolean backend metadata `zero_cost`, independent of `provider`: + +```yaml +promptkit: + backends: + - id: local-inference + provider: vllm + endpoint: http://localhost:8000/v1 + zero_cost: true +``` + +For each dispatched attempt, use a usable provider/model-client-reported cost +first, including an explicitly reported zero. If no usable reported cost is +available and the effective backend has `zero_cost: true`, supply USD zero with +a configured-zero source. Omitted, null, or false leaves the fallback disabled; +it does not suppress a reported cost. Apply this policy to built-in and injected +client execution using the selected backend, regardless of its provider type. + +The fallback also applies to dispatched attempts that fail, are canceled, or +produce no decodable response. It is an operator declaration, not a provider +observation. Non-dispatched work contributes no cost. Do not infer this setting +from a local address, provider name, token counts, or missing credentials, and +do not install it automatically on any backend. Without a selected backend +declaration, no configured-zero fallback applies. + +Resolve the field through ordinary final ConfigKit composition and inheritance; +include it in typed backend helpers, configuration capture, and backend +inspection. Retain the effective policy through prepared execution without +reloading configuration. It is local accounting metadata, not an outbound +request parameter. Preserve explicit/null/omitted source provenance in inspection +rather than manufacturing an explicit false value for the disabled fallback. + +This supports an explicit zero declaration, not configurable input/output or +cache-read/write token rates, fixed nonzero request charges, pricing tables, or +cost estimation. It does not alter credentials, generation, or token reporting. + ### Monetary Representation Use `float64` amounts in USD. Make the unit explicit in public field names such @@ -453,9 +499,10 @@ OpenRouter backend; no provider-specific interface is needed. Expose cost separately from `TokenUsage` and `TokenUsagePresence`, whose fields remain token counts. For run results, execution events, and group summaries, use a small presence-aware cost value with `Value float64`, `Known bool`, and -`Complete bool`, following the existing observed-count conventions. Successful -run results should expose cost even when no observer was registered; observers -also provide per-attempt and terminal cost, including available facts on failure. +`Complete bool`, following the existing observed-count conventions, plus the +source metadata below. Successful run results should expose cost even when no +observer was registered; observers also provide per-attempt and terminal cost, +including available facts on failure. Binary floating point is sufficient for operational telemetry and keeps decoding, addition, and the public API lightweight. Do not round to cents or another fixed @@ -465,42 +512,75 @@ exact financial ledgers, while fixed-scale integers would require a selected precision and rounding policy. Neither is warranted for this reporting feature. The API does not promise exact decimal arithmetic or invoice reconciliation. +### Cost-Source Metadata + +Give the presence-aware cost value a `Source` discriminator with these meanings: + +- `provider`: a cost supplied by the provider or injected model-client response. + This records the reporting path, not independent verification of a charge. +- `configured_zero`: zero supplied by the explicit backend fallback. +- `mixed`: an aggregate includes both provider and configured-zero contributions. +- Empty: no known contribution has a source. This covers unknown cost and the + known, complete zero aggregate when no attempts were dispatched. + +Derive aggregate source from the union of known contribution sources, including +zero-valued contributions. Unknown contributions supply no source but affect +completeness. A provider-only subtotal can therefore be incomplete without being +`mixed`. Carry this metadata consistently through attempt events, terminal +events, successful results, and group summaries. Reuse existing aggregation and +terminal deduplication rather than adding a parallel collector or cost history. + +Consumers can distinguish reported charges, configured zeros, and mixed totals +without treating configured values as provider reports. Source metadata is +separate from configuration inspection and does not require provenance traversal +during execution. + ### Presence, Aggregation, And Failures -- An absent or null cost is unknown. Explicit numeric zero is known zero. - A backend that reports no cost remains unknown even when a consumer knows it - has no external charge. -- On a dispatched attempt, a usable reported cost is known and complete for - that attempt. Cost completeness is independent of token-field completeness. +- An absent or null provider cost is unknown unless the backend's explicit + zero-cost fallback supplies it. A reported numeric zero remains known zero + with a provider source, even when the fallback is enabled. +- On a dispatched attempt, a usable reported cost or configured-zero fallback + is known and complete for that attempt. Cost completeness is independent of + token-field completeness and does not assert provider reporting. - Sum known costs across dispatched attempts, including initial generations, - retries, and repairs. `Known` means at least one attempt reported cost; - `Complete` means every dispatched attempt did. Unknown values contribute no - amount, but make the total incomplete. With no dispatched attempts, follow - the existing known, complete zero convention. + retries, and repairs. `Known` means at least one attempt has an available cost + from either source; `Complete` means every dispatched attempt does under this + policy. Unknown values contribute no amount, but make the total incomplete. + With no dispatched attempts, follow the existing known, complete zero + convention, with an empty source. - Group summaries reuse existing terminal-only aggregation and deduplication: sum known terminal values, OR their `Known` flags, and AND their `Complete` flags. Retain existing empty-group and terminal-coverage semantics. A subtotal - from OpenRouter mixed with unreported local-backend cost is incomplete. + from OpenRouter mixed with unreported local-backend cost is incomplete unless + those local attempts have an explicit configured-zero fallback. Combine + cost-source metadata as described above without changing token accounting. - Keep valid cost observed from a decoded successful envelope when a later choice/content check fails, as with token usage. Invalid framing and non-2xx - error bodies continue to supply no usage. Cancellation or failure may leave - incomplete reporting even when billable upstream work occurred. + error bodies continue to supply no provider usage, but the configured-zero + fallback still applies to dispatched attempts. Without that fallback, + cancellation or failure may leave incomplete cost reporting even when billable + upstream work occurred. - Treat an unusable optional cost value, including a nonnumeric value or one not representable as a finite `float64`, as unknown rather than making an - otherwise valid generation fail. This does not relax existing response framing - or content validation. An injected client likewise cannot supply a nonfinite - usable cost. + otherwise valid generation fail; the zero-cost fallback can then apply. This + does not relax existing response framing or content validation. An injected + client likewise cannot supply a nonfinite usable cost. -Completeness describes coverage of reported values, not verification of the -provider's billing. Preserve reported usable amounts without pricing inference -or reconciliation. +Completeness describes coverage by available reported or explicitly configured +cost, not verification of the provider's billing or proof that every provider +reported a charge. Preserve usable reported amounts without pricing inference or +reconciliation. For a successful run, result cost and terminal-event cost describe the same -dispatched attempts and must agree on value, presence, and completeness. Result -cost must also be the same with observation enabled or disabled. For example, -an unreported failed attempt followed by a successful attempt reporting USD -0.002 yields `Value: 0.002`, `Known: true`, and `Complete: false`. Apply the same -accounting through repairs and ordinary or prepared execution. +dispatched attempts and must agree on value, presence, completeness, and source. +Result cost must also be the same with observation enabled or disabled. For +example, an unreported failed attempt followed by a successful attempt reporting USD +0.002 yields `Value: 0.002`, `Known: true`, `Complete: false`, and a provider +source when no zero-cost fallback applies. Apply the same accounting through +repairs and ordinary or prepared execution. An invocation with fully reported +OpenRouter charges and configured-zero local calls can have a complete subtotal +with a mixed source, without claiming all providers reported costs. Group cost completeness concerns the terminal observations included in its subtotal. If two completed runs reported all their costs but a third observed @@ -523,7 +603,8 @@ application-specific concepts to PromptKit. Smoke diagnostics retain their separate report/evidence model and are not enrolled in ordinary execution observation by this change. Reasoning-text capture, changes to reasoning or caching request behavior, additional built-in -cost providers, multiple currencies, currency conversion, pricing estimates, +cost providers, configurable token-rate estimation, fixed nonzero request +pricing, multiple currencies, currency conversion, pricing estimates, billing reconciliation, persistent history, and logging infrastructure are outside this scope. @@ -537,6 +618,11 @@ outside this scope. - Add `provider` to the backend configuration model, typed source helpers, capture, inspection, and execution plumbing. Update the bundled OpenRouter backend to select it explicitly. Retain the selection through prepared runs. +- Add `zero_cost` to backend settings, typed helpers, capture, inspection, and + prepared-execution plumbing. Apply the fallback outside provider-specific + decoding so it works with built-in and injected clients. Verify composition + and inheritance, omitted/null/false behavior, explicit overrides, provider + independence, retained inspection provenance, and absence from provider bodies. - Consolidate provider-specific response and authentication-probe behavior in `internal/llm`; keep coordination and accounting provider-neutral. Cover provider-default probes, explicit kind and endpoint overrides, `kind: none`, @@ -561,7 +647,8 @@ outside this scope. recognized acceptance, rejection, unrecognized responses, provider defaults, and explicit kind/endpoint overrides, including `none` and adapter selection without a provider declaration. Verify that ordinary generation performs no - diagnostic requests and leaves monetary cost unknown. + diagnostic requests and leaves monetary cost unknown unless the backend + explicitly declares the zero-cost fallback. - Cover vLLM fixtures with reported, zero, omitted, and null reasoning/cache details, including responses representative of servers without detail reporting enabled. Verify `created_cache_tokens` mapping independently of @@ -575,6 +662,16 @@ outside this scope. conversions, successful run results, attempt and terminal events, and existing group summaries. Keep result cost available without registering an observer. Preserve existing token fields and accounting. +- Carry cost-source metadata through results, events, conversions, and group + snapshots. Cover provider-only, configured-zero-only, mixed, and unknown costs, + including zero-valued reported contributions and empty/non-dispatched cases. + Verify source union and terminal deduplication independently of cost value. +- Verify that usable reported cost wins over `zero_cost: true`, and missing, + null, or unusable reported cost permits fallback. Cover dispatched failures, + cancellation, undecodable responses, retries, repairs, ordinary/prepared + execution, and observer-enabled/disabled results. Verify that mixed OpenRouter + and declared zero-cost local calls can produce complete cost totals while + token completeness remains independent. - Verify that successful-result cost agrees with terminal-event cost and is unchanged when observation is disabled, including the unreported-retry example above. Verify group snapshots with fully reported completed runs alongside a @@ -595,15 +692,18 @@ outside this scope. turn telemetry into an exact-decimal guarantee. - Document the exact helper contract in GoDoc and add a focused invocation-level aggregation example to the consumer observation guide, including cost - availability, units, and completeness. Document the reasoning/cache fields and - cache-hit helper in their owning GoDoc and illustrate input/output breakdowns - without double-counting. Document OpenRouter/vLLM token decoding and OpenRouter - cost decoding in the + availability, units, completeness, and source. Explain that configured zeros + can make cost complete without provider reporting. Document the reasoning/cache + fields and cache-hit helper in their owning GoDoc and illustrate input/output + breakdowns without double-counting. Document OpenRouter/vLLM token decoding and + OpenRouter cost decoding in the [integration contract](../integrations/openai-compatible-chat.md#response-handling), and update internal observation and model-client documentation for changed flows. Document `provider`, its smoke-test defaults and override semantics in the framework format reference and relevant public GoDoc; update inspection and - diagnostic documentation with the implemented behavior. Include the diagnostic + diagnostic documentation with the implemented behavior. Document `zero_cost` + and its fallback precedence, inspection behavior, and prepared lifetime in the + relevant format, GoDoc, and consumer owners. Include the diagnostic migration in release notes. Include vLLM usage availability, server reporting settings, the meaning of cache creation, and the limits of its authentication diagnostic in the relevant consumer and integration documentation. @@ -616,12 +716,13 @@ Organize the eventual implementation plan around two workstreams: and nested execution boundaries and the existing group collector. 2. Backend provider selection and consolidation within `internal/llm`, including smoke-test defaults and explicit overrides, followed by OpenRouter/vLLM token - extensions and provider-reported cost through decoding, accounting, public - results, events, and group summaries, plus the derived cache-hit helper. - Include the corresponding configuration and API documentation. + extensions, provider-reported cost, and configured-zero fallback through + decoding, accounting, public results, events, and group summaries, plus the + derived cache-hit helper. Include the corresponding configuration and API + documentation. The second area is more extensive than the context helper. Keep backend provider -configuration, diagnostic consolidation, token/cache reporting, and cost +configuration, diagnostic consolidation, token/cache reporting, and cost/source accounting in separately reviewable implementation units rather than one large implementation prompt. Avoid coupling delivery of contextual observation to provider cost support. @@ -637,8 +738,9 @@ contract. Keep tolerant optional-cost decoding local to the model-client boundary. Adding a plain `*float64` JSON field to the typed envelope would make a nonnumeric or overflowing value fail decoding of the entire response. Decode the optional -field separately so an unusable cost becomes unknown without changing existing -framing or content validation. This needs no general numeric-conversion layer. +field separately so an unusable provider cost becomes unknown before applying +any configured-zero fallback, without changing existing framing or content +validation. This needs no general numeric-conversion layer. Use the completion criteria above to select focused offline checks for each unit. This roadmap owns target behavior; a separate implementation plan owns