Add reasoning and cache usage reporting to the roadmap
This commit is contained in:
@@ -8,7 +8,9 @@ Allow consumers to select execution observation once for an application-owned
|
||||
operation, rather than attach an observer to every PromptKit request. Nested,
|
||||
concurrent, and multi-engine calls using that operation's context should
|
||||
participate automatically. Include optional provider-reported cost alongside
|
||||
token usage, with built-in cost support initially limited to OpenRouter.
|
||||
token usage, with built-in cost support initially limited to OpenRouter. Extend
|
||||
OpenRouter token reporting with reasoning counts, documented cache reads and
|
||||
writes, and a derived token cache-hit ratio.
|
||||
|
||||
The motivating consumer is CommandKit's optional `ckprompt` integration, which
|
||||
could use this facility to report token usage for an entire command or pipeline.
|
||||
@@ -29,15 +31,18 @@ PromptKit v0.24.0 already provides:
|
||||
execution, and [CoordinateRequest.Observer](../../coordinate.go) for
|
||||
coordinator-selected continuation runs.
|
||||
|
||||
The missing capabilities are contextual observer selection and provider-reported
|
||||
cost accounting. Applications currently have to enroll each request explicitly,
|
||||
and PromptKit does not expose provider charges. The existing
|
||||
The missing capabilities are contextual observer selection, provider-reported
|
||||
cost accounting, reasoning-token counts, and a presence-aware cache-hit helper.
|
||||
OpenRouter cache writes also need their documented nested response mapping;
|
||||
the current client reads a top-level `usage.cache_write_tokens` field instead.
|
||||
Applications currently have to enroll each request explicitly, and PromptKit
|
||||
does not expose provider charges. The existing
|
||||
[consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run)
|
||||
describes that workflow.
|
||||
|
||||
## Target Public API
|
||||
|
||||
Add one public helper:
|
||||
Add a public context helper:
|
||||
|
||||
```go
|
||||
func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context
|
||||
@@ -49,8 +54,8 @@ associated with the context; it does not replace the inherited observer. A nil
|
||||
observer leaves the inherited selection unchanged.
|
||||
|
||||
No engine option, process-wide default, or replacement group type is needed.
|
||||
Extend the existing result, event, and group surfaces with cost metadata as
|
||||
described below.
|
||||
Extend the existing token types and result, event, and group surfaces with the
|
||||
token and cost metadata described below, plus a derived cache-hit helper.
|
||||
|
||||
The intended consumer workflow is:
|
||||
|
||||
@@ -258,6 +263,91 @@ requiring every provider to implement every capability is needed.
|
||||
This is configurable selection among PromptKit-supported provider implementations,
|
||||
not a public API for downstream registration of additional implementations.
|
||||
|
||||
## OpenRouter Token And Cache Reporting
|
||||
|
||||
### Response Mapping
|
||||
|
||||
Use OpenRouter's documented Chat Completions usage fields. Its
|
||||
[response schema](https://openrouter.ai/docs/api_reference/overview),
|
||||
[usage-accounting guide](https://openrouter.ai/docs/cookbook/administration/usage-accounting),
|
||||
and [caching guide](https://openrouter.ai/docs/guides/best-practices/prompt-caching)
|
||||
define the following mapping, relative to `usage`:
|
||||
|
||||
| Quantity | Response field | PromptKit field |
|
||||
| --- | --- | --- |
|
||||
| Input tokens | `prompt_tokens` | Existing `PromptTokens` |
|
||||
| Output tokens, including reasoning | `completion_tokens` | Existing `CompletionTokens` |
|
||||
| Reasoning tokens | `completion_tokens_details.reasoning_tokens` | New `ReasoningTokens` |
|
||||
| Cache reads | `prompt_tokens_details.cached_tokens` | Existing `CachedTokens` |
|
||||
| Cache writes | `prompt_tokens_details.cache_write_tokens` | Existing `CacheWriteTokens` |
|
||||
| Total tokens | `total_tokens` | Existing `TotalTokens` |
|
||||
|
||||
Preserve `CachedTokens` as the public field name and clarify its meaning as
|
||||
cache-read tokens. Correct the OpenRouter cache-write mapping to the nested
|
||||
field above; retain the generic provider's existing top-level handling. Do not
|
||||
add the nested and top-level values together. OpenRouter uses its documented
|
||||
nested value; an omitted nested value remains unknown.
|
||||
|
||||
Add `ReasoningTokens` to the public/domain token values, presence indicators,
|
||||
and observed token usage, including conversions, result serialization, attempt
|
||||
and terminal events, and group summaries. Initially, OpenRouter supplies this
|
||||
additional built-in response interpretation; the generic provider preserves its
|
||||
existing behavior. Injected clients may supply the canonical count and its
|
||||
presence through the existing model-client boundary.
|
||||
|
||||
### Count Relationships And Availability
|
||||
|
||||
Reasoning tokens are included in completion tokens, as explained in OpenRouter's
|
||||
[reasoning documentation](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens).
|
||||
Cache reads and writes describe input-token breakdowns. Present these as subsets
|
||||
or details rather than additional consumption to add to input/output totals.
|
||||
Preserve the provider's reported `TotalTokens`; do not recompute or reconcile it.
|
||||
|
||||
Consumers can derive non-reasoning output as completion tokens minus reasoning
|
||||
tokens when both counts are available and consistent. No additional stored
|
||||
non-reasoning count is needed in this scope. Capturing reasoning-token counts
|
||||
does not capture reasoning text or change reasoning request settings.
|
||||
|
||||
Reuse existing per-field presence and aggregation rules: omitted/null means
|
||||
unknown, explicit zero means known zero, and aggregate `Known` and `Complete`
|
||||
flags describe the reported contributions across dispatched attempts. Include
|
||||
initial generations, retries, and repairs through the existing observation
|
||||
accounting, retaining terminal-only group aggregation and deduplication. Preserve
|
||||
the existing distinction between integer-only result usage and presence-aware
|
||||
observations; zero in `TokenUsage` alone does not establish provider reporting.
|
||||
|
||||
### Token Cache-Hit Ratio
|
||||
|
||||
Add this derived helper on the existing presence-aware usage type:
|
||||
|
||||
```go
|
||||
func (u ObservedTokenUsage) CacheHitRatio() (float64, bool)
|
||||
```
|
||||
|
||||
The ratio is `CachedTokens / PromptTokens`, represented as a fraction from zero
|
||||
to one. It measures the proportion of input tokens served from cache, not the
|
||||
proportion of requests with a cache hit or monetary savings. The Boolean reports
|
||||
whether a ratio is available; an unavailable result returns zero and false.
|
||||
|
||||
Return an available ratio only when both counts are known and complete, input
|
||||
tokens are positive, and cache reads are between zero and input tokens inclusive.
|
||||
Zero cache reads with known positive input yields an available zero ratio. Missing
|
||||
or incomplete counts, zero input, or inconsistent counts leave it unavailable;
|
||||
preserve the raw reported counts rather than clamping them or failing generation.
|
||||
Cache-write availability does not affect this calculation.
|
||||
|
||||
For a run or group, divide the aggregated cache-read count by the aggregated
|
||||
input count; do not average per-attempt or per-run percentages. Do not use
|
||||
`reads / (reads + writes)`, because cache writes need not account for all
|
||||
uncached input. The same helper works on attempts, terminal events, and group
|
||||
summaries without storing another independently maintained field.
|
||||
|
||||
The ratio describes the included observations. A group ratio does not supersede
|
||||
`TerminalCoverageComplete` or imply that all application work has finished.
|
||||
For example, input 1,250 and cache reads 250 yield 0.2 (20%) when both counts
|
||||
are known and complete, even if cache writes are unknown. Output 840 with
|
||||
reasoning 600 still represents 840 output tokens, not 1,440.
|
||||
|
||||
## Optional Provider-Reported Cost
|
||||
|
||||
### Initial Provider Support
|
||||
@@ -360,10 +450,11 @@ delivery, and interpretation. This feature adds no CommandKit dependency or
|
||||
application-specific concepts to PromptKit.
|
||||
|
||||
Smoke diagnostics retain their separate report/evidence model and are not
|
||||
enrolled in ordinary execution observation by this change. Reasoning-token
|
||||
capture, additional built-in cost providers, multiple currencies, currency
|
||||
conversion, pricing estimates, billing reconciliation, persistent history, and
|
||||
logging infrastructure are outside this scope.
|
||||
enrolled in ordinary execution observation by this change. Reasoning-text
|
||||
capture, changes to reasoning or caching request behavior, additional built-in
|
||||
cost providers, multiple currencies, currency conversion, pricing estimates,
|
||||
billing reconciliation, persistent history, and logging infrastructure are
|
||||
outside this scope.
|
||||
|
||||
## Work Needed And Completion Criteria
|
||||
|
||||
@@ -389,6 +480,15 @@ logging infrastructure are outside this scope.
|
||||
- Verify that provider diagnostic defaults do not become explicit values in the
|
||||
loaded configuration or replace source provenance in inspection. Cover the
|
||||
changed clearing behavior and endpoint-only authentication overrides.
|
||||
- Carry OpenRouter reasoning-token counts through token values, presence,
|
||||
results, events, and group summaries. Verify nested OpenRouter cache-write
|
||||
decoding, including omitted/null/zero values and the nested value taking
|
||||
precedence when a top-level value is also present. Preserve generic-provider
|
||||
token mappings and existing field meanings.
|
||||
- Add `ObservedTokenUsage.CacheHitRatio`. Cover known zero reads, unknown and
|
||||
incomplete counts, zero input, inconsistent counts, independence from
|
||||
cache-write availability, and token-weighted aggregate ratios. Verify subset
|
||||
accounting without adding reasoning or cached tokens to the reported totals.
|
||||
- Carry optional OpenRouter cost through response decoding, internal/public
|
||||
conversions, successful run results, attempt and terminal events, and existing
|
||||
group summaries. Keep result cost available without registering an observer.
|
||||
@@ -413,7 +513,9 @@ logging infrastructure are outside this scope.
|
||||
turn telemetry into an exact-decimal guarantee.
|
||||
- Document the exact helper contract in GoDoc and add a focused invocation-level
|
||||
aggregation example to the consumer observation guide, including cost
|
||||
availability, units, and completeness. Document OpenRouter cost decoding in
|
||||
availability, units, and completeness. Document the reasoning/cache fields and
|
||||
cache-hit helper in their owning GoDoc and illustrate input/output breakdowns
|
||||
without double-counting. Document OpenRouter token and cost decoding in
|
||||
the [integration contract](../integrations/openai-compatible-chat.md#response-handling),
|
||||
and update internal observation and model-client documentation for changed flows.
|
||||
Document `provider`, its smoke-test defaults and override semantics in the
|
||||
@@ -428,13 +530,15 @@ Organize the eventual implementation plan around two workstreams:
|
||||
1. Contextual observer selection, including ordinary, prepared, coordinated,
|
||||
and nested execution boundaries and the existing group collector.
|
||||
2. Backend provider selection and consolidation within `internal/llm`, including
|
||||
smoke-test defaults and explicit overrides, followed by provider-reported cost
|
||||
through decoding, accounting, public results, events, and group summaries.
|
||||
smoke-test defaults and explicit overrides, followed by OpenRouter token
|
||||
extensions and provider-reported cost through decoding, accounting, public
|
||||
results, events, and group summaries, plus the derived cache-hit helper.
|
||||
Include the corresponding configuration and API documentation.
|
||||
|
||||
The second area is more extensive than the context helper. Keep backend provider
|
||||
configuration, diagnostic consolidation, and cost accounting in separately
|
||||
reviewable implementation units rather than one large implementation prompt.
|
||||
configuration, diagnostic consolidation, token/cache reporting, and cost
|
||||
accounting in separately reviewable implementation units rather than one large
|
||||
implementation prompt.
|
||||
Complete the remaining provider discussion before preparing that plan; avoid
|
||||
coupling delivery of contextual observation to provider cost support.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user