Clarify observation accounting and implementation planning
This commit is contained in:
@@ -144,6 +144,12 @@ override on that backend continues to select its OpenRouter response contract.
|
||||
Other built-in backend selections report unknown cost initially. Additional
|
||||
provider adapters or cost-capability configuration are not needed for this scope.
|
||||
|
||||
This baseline selects by exact backend identity: a separately named or
|
||||
collection-qualified backend pointing to OpenRouter would report unknown cost.
|
||||
Backend configuration as an alternative to identity-based selection is now an
|
||||
open discussion; settle that selection before implementing provider cost support.
|
||||
The contextual-observation stage does not depend on that decision.
|
||||
|
||||
Use the cost included in the normal completion response. No additional usage
|
||||
lookup, pricing request, or opt-in provider request parameter is needed.
|
||||
|
||||
@@ -201,6 +207,19 @@ Completeness describes coverage of reported values, not verification of the
|
||||
provider's billing. Preserve reported usable amounts without pricing inference
|
||||
or reconciliation.
|
||||
|
||||
For a successful run, result cost and terminal-event cost describe the same
|
||||
dispatched attempts and must agree on value, presence, and completeness. Result
|
||||
cost must also be the same with observation enabled or disabled. For example,
|
||||
an unreported failed attempt followed by a successful attempt reporting USD
|
||||
0.002 yields `Value: 0.002`, `Known: true`, and `Complete: false`. Apply the same
|
||||
accounting through repairs and ordinary or prepared execution.
|
||||
|
||||
Group cost completeness concerns the terminal observations included in its
|
||||
subtotal. If two completed runs reported all their costs but a third observed
|
||||
run has no terminal event, the subtotal may have `Complete: true` while
|
||||
`TerminalCoverageComplete` is false. Preserve this distinction using the existing
|
||||
coverage field; no additional completion flag or group lifecycle is needed.
|
||||
|
||||
## Accounting And Responsibility Boundaries
|
||||
|
||||
Reuse existing events, token presence/completeness, and group summaries, extending
|
||||
@@ -230,6 +249,10 @@ logging infrastructure are outside this scope.
|
||||
conversions, successful run results, attempt and terminal events, and existing
|
||||
group summaries. Keep result cost available without registering an observer.
|
||||
Preserve existing token fields and accounting.
|
||||
- Verify that successful-result cost agrees with terminal-event cost and is
|
||||
unchanged when observation is disabled, including the unreported-retry example
|
||||
above. Verify group snapshots with fully reported completed runs alongside a
|
||||
member without a terminal observation.
|
||||
- Cover ordinary and prepared execution, coordinated continuations, nested calls
|
||||
using derived contexts, concurrent calls, multiple engines sharing a scope,
|
||||
and independent scopes on a reused engine.
|
||||
@@ -250,5 +273,33 @@ logging infrastructure are outside this scope.
|
||||
the [integration contract](../integrations/openai-compatible-chat.md#response-handling),
|
||||
and update internal observation and model-client documentation for changed flows.
|
||||
|
||||
This roadmap defines the target behavior. A separate implementation plan can
|
||||
divide the work into appropriately sized stages.
|
||||
## Implementation Planning Guidance
|
||||
|
||||
Plan two separately reviewable stages:
|
||||
|
||||
1. Contextual observer selection, including ordinary, prepared, coordinated,
|
||||
and nested execution boundaries and the existing group collector.
|
||||
2. Provider-reported cost, carried through decoding, accounting, public results,
|
||||
events, and group summaries, with the corresponding documentation.
|
||||
|
||||
The second stage is more extensive than the context helper. Resolve the backend
|
||||
selection discussion before finalizing that stage; avoid coupling delivery of
|
||||
contextual observation to provider cost support.
|
||||
|
||||
Today, successful results accumulate generation responses in the runner, while
|
||||
optional observation recorders account for dispatched attempts. Adding cost to
|
||||
response objects alone does not preserve completeness across unreported retries.
|
||||
Reuse a small accounting component for result and observation paths so they
|
||||
apply the same rules, including when no observer is selected. Preserve existing
|
||||
token accounting rather than expanding this work into a redesign of its public
|
||||
contract.
|
||||
|
||||
Keep tolerant optional-cost decoding local to the model-client boundary. Adding
|
||||
a plain `*float64` JSON field to the typed envelope would make a nonnumeric or
|
||||
overflowing value fail decoding of the entire response. Decode the optional
|
||||
field separately so an unusable cost becomes unknown without changing existing
|
||||
framing or content validation. This needs no general numeric-conversion layer.
|
||||
|
||||
Use the completion criteria above to select focused offline checks for each
|
||||
stage. This roadmap owns target behavior; a separate implementation plan owns
|
||||
the detailed work sequence.
|
||||
|
||||
Reference in New Issue
Block a user