Files
promptkit/docs/roadmap/invocation-scoped-token-usage.md

497 lines
27 KiB
Markdown

# Invocation-Scoped Execution Observation
Status: accepted scope; implementation pending.
## Purpose And Scope
Let consumers select observation once for an application-owned operation and
obtain usage across its nested, concurrent, coordinated, and multi-engine
PromptKit calls. Extend that reporting with reasoning tokens, cache activity, a
token cache-hit ratio, and optional USD cost with its source and completeness.
CommandKit's optional `ckprompt` integration motivates this work, but the API
remains application-neutral. PromptKit owns observation, provider response
interpretation, and aggregation. Consumers own context propagation, waiting for
their work, invocation lifecycle, presentation, and interpretation.
This document defines the target state and the changes required to reach it.
A separate implementation plan will own sequencing and prompt-sized stages.
## Current Baseline
PromptKit v0.24.0 already provides
[execution observers and ordered composition](../../observations.go),
a concurrency-safe [execution group](../../execution_group.go), and
[request-level observation](../consumers/pkg-promptkit.md#observe-an-ordinary-run).
Groups aggregate the first terminal event for each observed run. Existing
observation distinguishes known, unknown, and incomplete prompt, completion,
total, cached, and cache-write token counts across retries and repairs.
Today, consumers explicitly enroll requests. Prepared handles retain the request
observer; [coordinator observers](../../coordinate.go) select continuation runs.
There is no contextual selection, reasoning-token count, cache-hit helper, or
monetary cost reporting. OpenRouter cache writes also need their documented
nested mapping: the current client reads `usage.cache_write_tokens` instead.
## Contextual Observer Selection
Add the public helper:
```go
func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context
```
The returned context composes the supplied observer with any inherited contextual
observers. A nil observer leaves that selection unchanged. Executions using the
context, or derived contexts retaining its values, participate automatically;
unrelated contexts do not. No engine option or process-wide default is needed.
A consumer can create an invocation-owned group, associate `group.Observe` with
the context, run its pipeline, and snapshot the group after its owned work
finishes, including when the pipeline returns an error. Group coverage describes
observed members, not future work or overall application progress.
### Delivery And Execution Boundaries
- `Run` selects contextual observers from its invocation context.
- `RunPrepared` selects them from its execution context, not the context used
to prepare the handle. The explicitly supplied request observer remains
retained by the handle and composes with those execution-time observers.
- Preparation emits no execution events. Discarded handles and rejected
execution claims retain their existing observation behavior.
- Coordinated continuations acquire contextual observation through their
ordinary `Run` boundary. `CoordinateRequest.Observer` still selects
continuation runs only; additional calls made by application callbacks
participate when they use the scoped context.
- Every execution, including a nested one using a callback or injected-client
context, has its own identity, recorder, and totals. Clearing an enclosing
execution's private state at a public boundary preserves contextual observer
selection, not the enclosing recorder.
Deliver events to explicit request observers first, then any coordinator observer
already composed with them, then contextual observers from inherited to newly
registered. Incorporate contextual selection once per execution, not again in
forwarding or coordination layers.
Preserve synchronous callback delivery and panic behavior. Concurrent runs may
invoke a shared observer concurrently; consumers synchronize their callback state.
The existing group remains concurrency-safe.
Registering the same callback through multiple paths may deliver duplicate
events; arbitrary Go functions are not deduplicated. Retain one lifecycle per
execution and reuse the group's terminal-event deduplication. Terminal totals
are aggregates, not additional contributions to add to attempt totals.
## Backend Provider And Accounting Metadata
Add two optional backend fields:
| Field | Meaning |
| --- | --- |
| `provider` | Selects the supported API conventions, independently of backend identity, namespace, endpoint hostname, and model vendor. |
| `zero_cost` | Enables a configured-zero cost fallback when no usable reported cost is available. |
For example:
```yaml
promptkit:
backends:
- id: research-router
provider: openrouter
endpoint: https://openrouter.ai/api/v1
api_key_env: OPENROUTER_API_KEY
- id: local-inference
provider: vllm
endpoint: http://localhost:8000/v1
zero_cost: true
```
Trim and lowercase the final effective provider value. Omitted, null, empty, and
whitespace-only values select `openai-compatible`. Reject unsupported nonempty
normalized values as ordinary configuration errors. The supported selections are
`openai-compatible`, `openrouter`, and `vllm`.
`zero_cost` is Boolean. Omitted, null, or false disables its fallback; true
enables it. It is independent of provider selection and does not suppress a
usable reported cost.
Resolve both fields through ordinary final ConfigKit composition and inheritance.
Carry their effective selections through capture and prepared execution without
reloading configuration. Declare `provider: openrouter` in the bundled
OpenRouter backend. Do not automatically declare any backend zero-cost.
These fields are local metadata, not outbound request parameters or entries
added to `extra_params`. Provider selection does not choose an endpoint or model,
and endpoint overrides retain the selected provider and accounting policy.
### Inspection Boundary
`InspectBackend` shows the normalized/resolved declared provider and zero-cost
setting, preserving explicit false for `zero_cost`. Omitted/null declarations
may remain absent, and blank provider values need no extra history representation.
Do not manufacture an explicit `openai-compatible` declaration or false
`zero_cost` value from runtime defaults.
`InspectConfiguration` owns ConfigKit source history and explicit/null/omitted
distinctions. Resource inspection does not acquire source-history fields or
null-tracking machinery. Provider diagnostic defaults likewise remain runtime
fallbacks, not values written into loaded configuration or recorded as explicit
source declarations.
## Provider Conventions
All selections reuse the existing
[OpenAI-compatible text Chat Completions foundation](../integrations/openai-compatible-chat.md):
base-relative `/chat/completions` JSON POST requests, existing credential rules,
text messages, selected generation controls and structured output, extra
parameters, first-choice content extraction, common token fields, and existing
response validation, errors, cancellation, timeouts, and transient retries.
| Provider | Additional response interpretation | Default authentication adapter |
| --- | --- | --- |
| `openai-compatible` | Existing token mappings; no provider-reported monetary cost decoding. | None. |
| `openrouter` | OpenRouter reasoning/cache details and `usage.cost` in USD. | `openrouter_key`. |
| `vllm` | vLLM reasoning/cache details; no provider-reported monetary cost decoding. | `vllm_models`. |
The default provider names PromptKit's current supported conventions, not the
entire OpenAI API. "Lowest common denominator" does not strip existing optional
features or promise every backend supports them. Continue sending selected
supported settings and return backend errors normally.
Provider selection supplies no default endpoint, model, credential environment
variable, or sampling settings, and does not create backend definitions. Keep
vLLM reasoning, chat-template, and other model-specific controls caller-selected;
do not automatically enable thinking or translate those controls. No new
generation protocol, capability negotiation, or version discovery is needed.
vLLM documents its
[OpenAI-compatible interface](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/).
### Authentication Diagnostics
Provider defaults apply only to explicitly requested authentication diagnostics.
Loading configuration, constructing an engine, and ordinary generation do not
initiate probes. Existing reachability and inference-default behavior remains.
Explicit `smoke_tests.authentication` metadata overrides provider defaults:
- A supplied `kind` selects the adapter; `none` declares authentication testing
inapplicable.
- Omitted/null `kind` uses the provider default, allowing endpoint-only overrides.
- A supplied endpoint overrides the selected adapter's destination; omitted/null
uses that adapter's existing fallback.
- An absent/null authentication block removes its overrides and leaves provider
defaults applicable. `kind: none`, not clearing the block, suppresses them.
Without a provider default or explicit adapter, preserve the existing unsupported
diagnostic outcome rather than assuming authentication is inapplicable. Explicit
adapters remain selectable without a matching provider declaration.
Retain `openrouter_key` behavior. The new `vllm_models` adapter makes one GET to
`/models` relative to the configured API base, supplying an available credential
under existing rules and allowing dispatch without one. A base ending in `/v1`
therefore produces `/v1/models`; explicit diagnostic endpoint overrides apply.
For `vllm_models`, HTTP 200 with a valid JSON object containing `object: "list"`
and array-valued `data` means acceptance of the probe request. An empty array is
accepted; recognition does not inspect model capabilities or require a particular
model. HTTP 401 means rejection; unrecognized responses leave acceptance unknown.
Reuse existing bounded diagnostic decoding and framing. Report observed
credential presence and response facts without claiming authentication is
enforced or models-list acceptance establishes inference authorization.
vLLM documents `/v1/models` as protected when API-key authentication is configured;
`/health` is not an authentication check. See its
[authentication documentation](https://docs.vllm.ai/en/stable/usage/security/#api-key-authentication-limitations).
No health-monitoring facility is added here.
## Token And Cache Reporting
### Provider Response Mapping
Use common `usage.prompt_tokens`, `usage.completion_tokens`, and
`usage.total_tokens` for `PromptTokens`, `CompletionTokens`, and `TotalTokens`.
Normalize these additional provider details, relative to `usage`:
| Public field | OpenRouter | vLLM |
| --- | --- | --- |
| New `ReasoningTokens` | `completion_tokens_details.reasoning_tokens` | `completion_tokens_details.reasoning_tokens` |
| Existing `CachedTokens` | `prompt_tokens_details.cached_tokens` | `prompt_tokens_details.cached_tokens` |
| Existing `CacheWriteTokens` | `prompt_tokens_details.cache_write_tokens` | `prompt_tokens_details.created_cache_tokens` |
OpenRouter documents these in its
[response schema](https://openrouter.ai/docs/api_reference/overview),
[usage-accounting guide](https://openrouter.ai/docs/cookbook/administration/usage-accounting),
and [caching guide](https://openrouter.ai/docs/guides/best-practices/prompt-caching).
vLLM's [response construction](https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/chat_completion/serving/)
uses `created_cache_tokens`; its
[output definitions](https://docs.vllm.ai/en/stable/api/vllm/outputs/)
describe local prefix-cache writes, not a billable write charge.
Preserve `CachedTokens` as the public name and clarify its meaning as cache reads.
Use each provider's documented cache-write field, not a sum or guessed alias;
a missing nested value is unknown even if an alternative field is present.
Retain the generic provider's existing mappings, including its top-level
`usage.cache_write_tokens` handling.
Add `ReasoningTokens` to public/domain token values, presence indicators, observed
usage, conversions, and result serialization, and carry it through attempt events,
terminal events, and group summaries. Injected clients may supply canonical
counts and presence through the existing model-client boundary, retaining its
legacy `UsagePresence` fallback; explicit presence distinguishes measured zero.
### Availability And Optional Metadata
Omitted/null details are unknown, not zero. Explicit integer zero is known zero.
For the newly interpreted provider-specific reasoning and cache-write details,
wrong-type, nonintegral, or unrepresentable integer values are also unknown.
An unusable new detail does not invalidate otherwise usable content or erase
independently available usage. Preserve existing common-field decoding and
response framing/content validation; this is a targeted optional-metadata rule,
not a general relaxation of the response contract. Keep this interpretation local
to the model-client boundary without a general numeric-conversion framework.
vLLM detail availability depends on version and configuration: prompt details
currently require `--enable-prompt-tokens-details`, and reasoning details depend
on a configured reasoning parser. Document those reporting settings without
configuring or discovering them automatically. Older or differently configured
servers may omit details and still produce usable responses. Do not estimate
reasoning counts from text or cache writes from input minus cache reads.
### Count Relationships And Aggregation
Reasoning is included in completion tokens, not additional output to add to them.
OpenRouter explains this in its
[reasoning documentation](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens);
vLLM similarly includes reasoning detail within generated-token usage. Cache reads
and writes are input-token details, not extra input consumption. Preserve the
reported total without recomputation or reconciliation.
Consumers can derive non-reasoning output from consistent, known completion and
reasoning counts; no additional stored count is needed. Capturing counts does
not capture reasoning text or change generation/caching settings.
Reuse existing per-field token accounting: aggregates sum known contributions,
`Known` identifies available reporting, and `Complete` describes its coverage
across dispatched attempts. Preserve initial-generation, retry, and repair
observation accounting and terminal-only group deduplication. Existing integer
result usage remains distinct from presence-aware observations; a zero result
field alone does not establish that the provider reported it.
### Token Cache-Hit Ratio
Add a derived helper, not another stored total:
```go
func (u ObservedTokenUsage) CacheHitRatio() (float64, bool)
```
Compute `CachedTokens / PromptTokens` as a fraction from zero to one. This is the
proportion of input tokens served from cache, not request hit frequency or cost
savings. For aggregates, divide summed reads by summed input; do not average
individual percentages or use `reads / (reads + writes)`.
Return an available ratio only when input and cache-read counts are known and
complete, input is positive, and reads lie between zero and input inclusive.
Known zero reads with positive input yields an available zero. Missing,
incomplete, zero-denominator, or inconsistent data yields zero and false while
preserving raw counts; it does not fail generation or clamp reported values.
Cache-write availability is irrelevant.
The same helper serves attempts, terminals, and group summaries. A group ratio
describes its included observations and does not supersede terminal coverage or
imply all application work has finished.
## Cost Reporting
### Reported Cost And Configured-Zero Precedence
Initially, only the OpenRouter built-in provider interprets `usage.cost` as the
account charge in USD. See its
[usage-accounting reference](https://openrouter.ai/docs/cookbook/administration/usage-accounting)
and [USD explanation](https://openrouter.ai/blog/insights/cost-vs-quality-tradeoff-framework-for-agent-models/).
Do not add `cost_details` amounts to that charge or interpret arbitrary providers'
fields named `cost`. The generic and vLLM providers do not decode monetary cost.
Use the cost included in the ordinary response; no extra request or usage lookup
is needed. At the model-client boundary, represent an optional USD amount as
`*float64`: nil means unknown; a pointer to zero means reported zero. Injected
clients may supply the same value regardless of provider selection.
Resolve cost at completion of each dispatched attempt, outside provider-specific
decoding and whether or not decoding occurred or succeeded:
- A usable provider/model-client-reported cost takes precedence, including zero.
- Otherwise, `zero_cost: true` on the effective backend supplies configured USD zero.
- Otherwise, cost remains unknown.
This applies to built-in and injected clients, including failed or canceled
dispatched attempts and undecodable responses. Without a selected backend
declaration, no configured-zero fallback applies. Non-dispatched work contributes
no cost. Do not infer zero cost from local hosting, provider name, credentials,
or token counts, or install the declaration automatically on bundled backends.
An unusable reported amount, including a nonnumeric value or one not representable
as a finite `float64`, is unknown before fallback; it does not make an otherwise
usable generation fail. Keep available cost from a decoded successful envelope
when a later choice/content check fails. Invalid framing and non-2xx bodies supply
no provider usage, although the configured-zero fallback still applies to a
dispatched attempt. Existing framing, content, and error handling remain intact.
### Monetary Representation And Sources
Expose cost separately from token counts and their presence indicators. Use
explicit USD field naming such as `CostUSD` and a small public cost value for
results, events, and group summaries with:
- `Value float64`: the available USD amount or subtotal;
- `Known bool`: availability;
- `Complete bool`: coverage under the selected cost policy; and
- `Source`: the contribution-source discriminator below.
| Source | Meaning |
| --- | --- |
| `provider` | Supplied by the provider or injected model-client response; identifies the reporting path, not independent billing verification. |
| `configured_zero` | Supplied by the explicit backend fallback. |
| `mixed` | An aggregate includes both reported and configured-zero contributions. |
| Empty | No known contribution has a source, including an unknown cost or an aggregate with no dispatched attempts. |
Use binary floating point for operational telemetry, without rounding to cents
or another fixed scale during capture or aggregation. Consumers choose display
precision. Exact-decimal ledgers, decimal dependencies, fixed-scale currency
rounding, pricing inference, and invoice reconciliation are not part of this work.
### Attempt, Run, And Group Accounting
A dispatched attempt with usable reported cost or configured zero is known and
complete for cost, independently of token completeness. Sum known amounts across
all dispatched attempts, including retries and repairs. Aggregate `Known` means
at least one known contribution; `Complete` means every dispatched attempt has a
cost from either source. Unknown contributions add no amount and make coverage
incomplete. Completeness does not assert that every provider reported a charge.
Aggregate sources by union of known contributions, including zero amounts.
Unknown contributions add no source: a provider-only subtotal can be incomplete
without becoming `mixed`. A reported zero plus a configured zero is `mixed`.
Successful results expose cost even without an observer. Result and terminal
cost agree on amount, availability, completeness, and source, and enabling
observation does not change result cost. For example, an unreported failed
attempt followed by a reported USD 0.002 yields an incomplete provider-source
subtotal when no zero-cost fallback applies.
Groups aggregate terminal costs only, counting each run's first terminal once.
Combine known amounts, OR `Known`, AND `Complete`, and union sources. Preserve
existing empty/no-terminal group semantics: a completed execution with no
dispatched attempts yields known, complete zero with empty source; an empty group
does likewise, while a group with members but no terminals has unknown,
incomplete usage.
Cost completeness concerns included terminal observations, independently of
`TerminalCoverageComplete`. Two completed runs with full cost coverage can yield
a complete subtotal while a third observed run lacks a terminal. Conversely,
fully reported OpenRouter runs and explicitly zero-cost local runs can yield a
complete, mixed-source total. Neither case verifies a provider invoice or future
application progress.
## Internal Responsibilities And Scope Limits
Provider-specific metadata interpretation and authentication conventions belong
in `internal/llm`, sharing common transport, request construction, credentials,
framing, and response fields. Use one provider-selection mechanism there.
Generation/observation coordination consumes normalized facts or generic
diagnostic operations, replacing the OpenRouter-specific diagnostic collaborator
method with a generic operation while retaining explicit adapter selection.
Provider-specific files and straightforward dispatch suffice; no public provider
registration API, new `internal/providers` package, or broad mandatory-capability
interface is needed. This is configuration-driven selection among supported
implementations, not downstream registration of implementations.
Reuse a small shared cost-accounting component for result and observation paths
so dispatched attempts, fallback, and source union follow the same rules even
when observation is disabled. Provider code interprets wire metadata; common
accounting owns fallback and aggregation. No ConfigKit provenance traversal
belongs in ordinary execution.
Smoke diagnostics retain their separate report/evidence model and are not
enrolled in ordinary execution observation. CLI integration, output delivery,
persistent history, logging infrastructure, reasoning-text capture, changes to
reasoning/caching request behavior, server administration/metrics, additional
built-in cost decoders, multiple currencies, configurable token rates, fixed
nonzero request prices, and cost estimation remain outside scope.
## Work Needed And Acceptance
The following areas describe required changes, not an execution order.
- Extend the public context API and ordinary/prepared execution wiring while
preserving independent nested lifecycles, explicit observers, coordinator
selection, and existing preparation/discard/rejected-claim behavior.
- Extend backend settings, typed helpers, final-value normalization/validation,
capture, prepared metadata, and inspection for `provider` and `zero_cost`.
Update the bundled OpenRouter declaration. Preserve configuration provenance
and resource-inspection simplicity, and keep metadata out of request bodies.
- Consolidate provider behavior in `internal/llm`, adding vLLM usage normalization
and the models-list diagnostic alongside OpenRouter interpretation and the
shared default path. Preserve common decoding while tolerating unusable new
optional details and cost locally to the model-client boundary.
- Extend public/domain token types, presence, serialization, conversions, events,
and groups with reasoning counts and correct provider-specific cache writes.
Add the presence-aware cache-hit helper without changing existing token
meanings or introducing a separate ratio accumulator.
- Carry optional reported cost and configured-zero policy through attempt
completion and shared accounting into results, events, and groups, including
source metadata and observer-independent result accounting.
Select focused offline checks under the [testing policy](../policy/testing.md).
Use real internal collaborators and controlled HTTP/injected-client boundaries,
with proportionate public contract coverage. Important behaviors include:
- Context inheritance, registration order, nil/duplicate enrollment, concurrent
calls, multiple engines, independent scopes, nested calls, and execution-time
selection for a handle prepared elsewhere.
- Provider selection independent of backend identity; normalization and unsupported
value handling; omitted/null/blank defaults; inheritance, endpoint overrides,
explicit false, and inspection without manufactured values or source history.
- Provider-default and explicit diagnostic selection, endpoint-only overrides,
clearing metadata versus `kind: none`, vLLM acceptance including an empty list,
rejection, unrecognized/framing-limited responses, and optional credentials.
- Provider token mappings, absent/null/zero details, malformed new optional counts
without generation failure, unchanged common-field handling, and responses
representative of vLLM servers with detail reporting disabled.
- Cache-hit availability, inconsistent counts without generation failure,
token-weighted aggregates, independence from cache writes, and no double-counting
of reasoning or cache details.
- Reported cost precedence, configured-zero fallback on dispatched failure and
cancellation, built-in/injected clients, retries/repairs, mixed/unknown sources,
zero-valued contributions, and independently incomplete token reporting.
- Result/terminal cost agreement with observation enabled or disabled, duplicate
terminal deduplication, empty/no-terminal groups, and subtotal completeness
distinct from terminal coverage.
Synthetic fixtures and offline examples suffice; live providers are not required.
Keep ownership localized and avoid repeating the same assertions across layers.
## Documentation And Migration
When implemented, exact exported contracts belong in GoDoc; the
[consumer guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run) should show
invocation-scoped aggregation, reasoning/cache breakdowns without double-counting,
and cost availability, source, and completeness. Document provider wire mappings,
optional-metadata handling, and diagnostic response recognition in the
[integration contract](../integrations/openai-compatible-chat.md#response-handling).
The [framework format reference](../formats.md#backend-definitions) owns the new
backend fields, default selection, inheritance, and diagnostic overrides.
Update internal documentation for changed ownership and flows.
Explain vLLM's server reporting settings, unknown-versus-zero behavior, and cache
creation meaning without promising availability on every deployment. Consumers
own presentation. Models-list acceptance does not establish inference
authorization, and complete cost metadata does not independently verify billing.
Release notes should call out that authentication objects no longer require
`kind` when a provider default supplies it, endpoint-only overrides are supported,
and clearing an authentication block restores provider defaults. `kind: none`
suppresses that default. Preserve existing explicit adapters without requiring a
provider declaration.