Document invocation observation and cost reporting
This commit is contained in:
@@ -728,6 +728,40 @@ counts, but its zero counts remain unknown to observations; supply explicit
|
||||
presence when measured zeros matter. Existing `RunResult.Usage` sums returned
|
||||
values independently of observation presence.
|
||||
|
||||
For an application-owned invocation spanning several calls or engines, attach
|
||||
an observer once to a context and propagate it to those calls:
|
||||
|
||||
```go
|
||||
var group promptkit.ExecutionGroup
|
||||
ctx = promptkit.WithExecutionObserver(ctx, group.Observe)
|
||||
result, err := engine.Run(ctx, request)
|
||||
// After joining all application-owned work, inspect group.Snapshot(), even on error.
|
||||
```
|
||||
|
||||
Explicit request observers run first, followed by contextual observers in
|
||||
registration order. Coordinated continuations pick up the context through
|
||||
`Run`; prepared handles retain the request observer but use the execution
|
||||
context's observers, not the preparation context's. Each nested call has its
|
||||
own run ID and accounting. Duplicate registration can deliver duplicate events;
|
||||
groups count each run's first terminal once.
|
||||
|
||||
Reasoning tokens are a subset of completion tokens; cache reads and writes are
|
||||
input details, not extra tokens. `ObservedTokenUsage.CacheHitRatio()` derives
|
||||
the cache-read fraction only when input and read totals are known, complete,
|
||||
positive-denominator, and consistent. It divides aggregate counts rather than
|
||||
averaging per-run ratios and is independent of cache writes.
|
||||
|
||||
`RunResult.CostUSD` is available even without an observer; attempt and terminal
|
||||
events and group snapshots also report cost. `Known` denotes an available
|
||||
subtotal, `Complete` covers dispatched attempts, and `Source` distinguishes
|
||||
provider reports, an explicitly configured zero fallback, and mixed sources.
|
||||
Reported cost, including zero, takes precedence over fallback. Unknown failed
|
||||
attempts make a subtotal incomplete; configured zero may cover them. Terminal
|
||||
events can report cost when a failed operation returns no result. Neither
|
||||
complete reporting nor provider attribution verifies an invoice; consumers
|
||||
choose presentation precision. See [backend definitions](../formats.md#backend-definitions)
|
||||
and the [integration contract](../integrations/openai-compatible-chat.md#response-handling).
|
||||
|
||||
Callbacks run synchronously, in order for one operation, and can overlap for
|
||||
different operations. Synchronize shared callback state and avoid blocking on
|
||||
work that needs the same run's admission slot. Callback panics propagate;
|
||||
@@ -879,15 +913,15 @@ outcome, err := engine.Coordinate(ctx, promptkit.CoordinateRequest{
|
||||
})
|
||||
```
|
||||
|
||||
`assess` is application code and explicitly selects its observer on any nested
|
||||
run. A completed turn releases its admission lease before the callback, so that
|
||||
`assess` is application code and can select its observer explicitly or inherit
|
||||
one from the context. A completed turn releases its admission lease before the callback, so that
|
||||
assessment may use the same backend. `stopped` and `exhausted` describe control
|
||||
flow, not content acceptance. Cancellation before callback delivery can return one
|
||||
completed response in `Undelivered`; errors retain their ordinary identities.
|
||||
|
||||
### Collect Selected Execution Facts
|
||||
|
||||
Groups are independent of continuation and enrollment is explicit:
|
||||
Groups are independent of continuation and can be enrolled explicitly:
|
||||
|
||||
```go
|
||||
var group, validatorGroup promptkit.ExecutionGroup
|
||||
@@ -904,7 +938,7 @@ _, _ = summary, validatorSummary
|
||||
```
|
||||
|
||||
Only the first terminal event per run contributes counts and usage; coverage of
|
||||
currently observed members is separate from token-usage completeness. Empty
|
||||
currently observed members is separate from token and cost completeness. Empty
|
||||
groups report complete, known zero totals, while members without terminals do
|
||||
not invent totals. Overlapping groups are not generally additive. See the
|
||||
[offline example](../../examples/go-library/observe/main.go) for complete setup.
|
||||
|
||||
@@ -223,7 +223,7 @@ Backend definitions are supplied through caller-selected configuration
|
||||
sources. The conventional backend IDs are names only; PromptKit does not
|
||||
install definitions for them.
|
||||
|
||||
A backend definition may have `endpoint`, `api_key_env`,
|
||||
A backend definition may have `provider`, `zero_cost`, `endpoint`, `api_key_env`,
|
||||
`concurrency_limit`, `queue_capacity`, `extra_params`, and `smoke_tests`. The
|
||||
final endpoint is required and must be an absolute HTTP(S) URL with a host, no
|
||||
user information, query, or fragment. An omitted or null API-key environment
|
||||
@@ -237,6 +237,17 @@ unrecognized backend field is application data and is not automatically redacted
|
||||
by Promptkit. Applications own sensitivity declarations and disclosure policy
|
||||
for their extension fields.
|
||||
|
||||
`provider` selects response and diagnostic conventions independently of the
|
||||
backend ID, endpoint, or model. Its final value is trimmed and lowercased;
|
||||
omitted, null, or blank selects `openai-compatible` at runtime. Supported
|
||||
nonblank values are `openai-compatible`, `openrouter`, and `vllm`. `zero_cost`
|
||||
is an optional Boolean: true supplies a zero-cost fallback when a dispatched
|
||||
attempt has no usable reported cost; omitted, null, and false disable it.
|
||||
Neither setting is sent to the provider. They follow normal layering and
|
||||
inheritance, and endpoint overrides retain the selected backend's policy.
|
||||
Resource inspection shows declared normalized values without inventing runtime
|
||||
defaults; configuration inspection retains source history.
|
||||
|
||||
`smoke_tests` is optional backend metadata for explicit diagnostic operations.
|
||||
It may contain `default_profile`, `reachability`, and `authentication`:
|
||||
|
||||
@@ -274,6 +285,10 @@ Without a selected adapter, authentication diagnostics report an unsupported
|
||||
selection. An endpoint-only authentication
|
||||
block overrides the destination of a provider default; `kind: none` suppresses
|
||||
that default.
|
||||
Previously, a supplied authentication object required `kind`. Endpoint-only
|
||||
objects now use a provider default where available; clearing the object
|
||||
restores that default rather than suppressing it. Use `kind: none` to suppress
|
||||
authentication testing.
|
||||
These values are configuration only: engine construction does not contact a
|
||||
backend.
|
||||
|
||||
@@ -364,7 +379,7 @@ does not disable the caller context or HTTP transport timeout.
|
||||
|
||||
### Bundled OpenRouter Configuration
|
||||
|
||||
The default bundle installs root backend `openrouter`, endpoint
|
||||
The default bundle installs root backend `openrouter`, provider `openrouter`, endpoint
|
||||
`https://openrouter.ai/api/v1`, optional credential environment name
|
||||
`OPENROUTER_API_KEY`, and smoke metadata selecting `gpt-luna-latest` with the
|
||||
`openrouter_key` authentication adapter. Existing diagnostic endpoint fallback
|
||||
|
||||
@@ -173,6 +173,10 @@ otherwise usable content. Existing common count decoding remains strict.
|
||||
OpenRouter's `usage.cost` is an optional USD amount; malformed or non-finite
|
||||
amounts are ignored without invalidating content. The generic and vLLM
|
||||
providers do not interpret that field.
|
||||
vLLM prompt-token details depend on server configuration, including
|
||||
`--enable-prompt-tokens-details`; reasoning details depend on a configured
|
||||
reasoning parser. Servers without these details remain usable. vLLM cache
|
||||
writes describe local prefix-cache creation, not a billable write charge.
|
||||
|
||||
Each common token field retains whether the provider reported a
|
||||
non-null integer, including zero. Omitted and `null` fields are unknown for
|
||||
|
||||
@@ -79,7 +79,7 @@ other backend IDs.
|
||||
The coordinator does not retry injected clients; it normalizes a nil successful
|
||||
response as malformed generation. The built-in HTTP client remains responsible
|
||||
for provider transport behavior. The runner, rather than the
|
||||
capacity layer, sums all five usage fields from the initial response and every
|
||||
capacity layer, sums usage fields from the initial response and every
|
||||
completed repair response into the successful run result.
|
||||
|
||||
An invocation-local observer receives attempt events after active permit
|
||||
|
||||
@@ -35,6 +35,12 @@ unknown usage. Transport, timeout, malformed, and noneligible status failures
|
||||
stop the sequence without retry exhaustion. Context is checked before each
|
||||
wait, acquisition, and dispatch.
|
||||
|
||||
At every dispatched completion the coordinator resolves any reported cost
|
||||
against the frozen backend zero-cost policy and adds it once to the run's
|
||||
[cost accumulator](observation.md), independently of observer selection.
|
||||
Injected responses may retain reported cost alongside an error. Undispatched
|
||||
acquisitions and retry waits contribute no cost.
|
||||
|
||||
The public client adapter translates explicitly supplied public transport and
|
||||
malformed-response identities into internal categories while retaining the
|
||||
original error chain. This keeps injected attempt classifications consistent
|
||||
|
||||
@@ -142,6 +142,14 @@ known. Strictly decoded usage can still be observed if later choice/content
|
||||
checks fail; invalid framing or a non-success error body contributes none.
|
||||
Existing successful-result accounting remains value-based.
|
||||
|
||||
The captured provider selects additional response details and authentication
|
||||
diagnostic defaults. Optional new reasoning, cache-write, and OpenRouter cost
|
||||
fields are decoded independently, without relaxing common-field or framing
|
||||
validation. Cost remains an optional raw amount at this boundary; shared
|
||||
accounting applies configured-zero fallback after dispatch. The integration
|
||||
contract specifies the [wire mappings](../integrations/openai-compatible-chat.md#response-handling)
|
||||
and [probe recognition](../integrations/openai-compatible-chat.md#explicit-smoke-diagnostics).
|
||||
|
||||
For a non-success response, `ProviderHTTPError` retains the HTTP status and
|
||||
only normalized detail from the bounded recognized envelope. It retains
|
||||
`ErrUnexpectedStatus` through unwrapping. The client owns response closure;
|
||||
|
||||
@@ -21,7 +21,7 @@ contributor workflow and validation.
|
||||
| `internal/capacity` | Owns engine-local bounded execution admission and FIFO model-generation permits for limited backend IDs, including cancellation-safe waiter removal and client wrapping. | [Internal capacity management](capacity.md) |
|
||||
| `internal/domain` | Defines internal framework values for requests, immutable conversation exchanges, artifacts, prompt definitions, requirement inspection, profiles, execution targets, rendering, generation, validation, and source-neutral smoke diagnostics, and owns source-neutral invariants for shared execution settings, OpenAI-compatible base endpoints, session identifiers, output contracts, and resource-identity syntax and qualification. Source parsing, required fields, other source-specific normalization, defaulting, and boundary-specific error classification remain with their callers. | [Domain declarations](../../internal/domain/domain.go), [exchange storage](../../internal/domain/exchange.go), [requirement values](../../internal/domain/requirements.go), [endpoint invariant](../../internal/domain/endpoint.go) |
|
||||
| `internal/engine` | Assembles an already captured configuration into shared capacity, rendering, validation, generation, repair, runner, and smoke-test collaborators. Immutable namespace views share those collaborators and qualify selections without rewriting captured references. Built-in engines provide the diagnostic HTTP collaborator; injected-client engines do not. It delegates resource inspection and listing to the capture. | [Engine assembly](../../internal/engine/engine.go), [internal sources](sources.md#typed-configuration-loading), [internal runner](runner.md) |
|
||||
| `internal/generation` | Coordinates frozen built-in HTTP attempts and one-shot injected calls through active backend permits, including bounded status retries, cancellable backoff, ordinary attempt observations, and invocation-local evidence for diagnostics. | [Internal generation coordination](generation.md) |
|
||||
| `internal/generation` | Coordinates frozen built-in HTTP attempts and one-shot injected calls through active backend permits, including bounded retries, per-attempt cost, ordinary observations, and separate diagnostic evidence. | [Internal generation coordination](generation.md) |
|
||||
| `internal/defaults` | Defines application-neutral framework constants and constructs the default execution target. It contains no CLI, server, or inbound HTTP limits. | [Framework defaults](../../internal/defaults/defaults.go) |
|
||||
| `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |
|
||||
| `internal/jsonvalue` | Validates and deeply copies bounded JSON-compatible provider values, and detaches previously accepted values without repeating validation. | [JSON values](../../internal/jsonvalue/jsonvalue.go) |
|
||||
@@ -30,8 +30,8 @@ contributor workflow and validation.
|
||||
| `internal/prompt` | Renders prompt messages from Go templates with artifact, variable, session, and cache-control data, and conservatively enumerates dependencies from captured template syntax. | [Go-template renderer](../../internal/prompt/go_renderer.go), [requirement discovery](../../internal/prompt/requirements.go) |
|
||||
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
|
||||
| `internal/validate` | Creates operation-local validation plans from inline schema documents or bound filesystem resources and validates basic, JSON, and JSON Schema output. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
|
||||
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including bounded response decoding, authentication, deadline handling, per-invocation HTTP evidence and built-in diagnostic probes, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
|
||||
| `internal/observation` | Owns invocation-local run IDs, presence-aware usage and phase accounting, synchronous metadata event delivery and composition, and explicit terminal-based group aggregation; smoke diagnostics keep their private evidence outside ordinary observer events. | [Internal execution observation](observation.md) |
|
||||
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible requests, provider-specific response interpretation and authentication probes, bounded decoding, deadlines, and diagnostic evidence. | [Internal model client](llm.md) |
|
||||
| `internal/observation` | Owns invocation-local run IDs, presence-aware token and cost accounting, contextual observer selection, synchronous event delivery, and terminal-based group aggregation; smoke diagnostics retain separate evidence. | [Internal execution observation](observation.md) |
|
||||
| `internal/usecase` | Selects captured prompts and profiles, enumerates requirements from captured prompt syntax, applies request overrides and completion checks, coordinates ordinary and single-claim prepared execution across rendering, artifact loading, operation-local validation plans, generation, capacity, and bounded repair, advances a single conversation branch, and runs explicit smoke diagnostics with private cases, shared generation/capacity, and optional built-in HTTP probes. | [Internal runner](runner.md), [requirement inspection](../../internal/usecase/requirements_inspection.go), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
|
||||
|
||||
The internal engine assembles these internal components without exposing their
|
||||
|
||||
@@ -216,10 +216,16 @@ ordinary `Run` reuses its run ID and carries it to the generation coordinator
|
||||
through context. Prompt and profile identities are enriched only after their
|
||||
respective resolutions succeed, before admission. The facade starts observation
|
||||
before request conversion and emits its terminal event after public error
|
||||
mapping and after the runner's deferred admission release. Preparation and
|
||||
prepared execution follows the same terminal mapping after a successful claim;
|
||||
mapping and after the runner's deferred admission release. Preparation emits no
|
||||
events; prepared execution follows the same terminal mapping after a successful claim;
|
||||
invalid or discarded handles do not acquire an observer.
|
||||
|
||||
Ordinary runs establish fresh private cost state before observation; a claimed
|
||||
prepared execution establishes its own state after claim. Contextual observers
|
||||
remain available at public boundaries, while enclosing recorder and cost state
|
||||
are masked. Prepared handles retain the request observer and frozen provider
|
||||
policy, not preparation-context selection or an accumulator.
|
||||
|
||||
The facade establishes an optional operation scope for ordinary runs before
|
||||
observation and preparation. A claimed prepared execution establishes its own
|
||||
scope from the retained duration; preparation and idle time do not consume it.
|
||||
@@ -273,11 +279,13 @@ run identifier, and UTC timing. The same effective session reaches initial
|
||||
generation and any repair attempt through the rendered prompt. The same
|
||||
effective target and presence metadata, including backend identity and direct
|
||||
credential during execution, reaches generation and every repair attempt.
|
||||
Result usage is the field-wise sum of all five usage values from the initial
|
||||
Result usage is the field-wise sum of usage values from the initial
|
||||
response and every completed repair response. Final raw output, artifact, and
|
||||
validation state still come from the last candidate. A repair error returns no
|
||||
partial run result or partial result usage, although the invocation-local
|
||||
observer retains completed-attempt usage and reports it in the terminal event.
|
||||
Result cost instead snapshots the shared accumulator across all dispatched
|
||||
attempts, including retries and repairs, whether or not observation is enabled.
|
||||
|
||||
## Failure Categories
|
||||
|
||||
|
||||
@@ -417,6 +417,8 @@ implementation has reached its final buildable state.
|
||||
|
||||
## Stage 15 — Canonical Documentation And Final Verification
|
||||
|
||||
**Completed.**
|
||||
|
||||
- Finalize GoDoc for helpers, backend fields, reasoning counts, cost values and
|
||||
sources, client ownership, and observer/prepared lifetimes. Update stale
|
||||
authentication Kind requirements in shared settings and public docs.
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# Invocation-Scoped Execution Observation
|
||||
|
||||
Status: accepted scope; implementation pending.
|
||||
Status: accepted scope; implementation complete. Current behavior is documented
|
||||
in the linked canonical contracts; roadmap retirement is separate housekeeping.
|
||||
|
||||
## Purpose And Scope
|
||||
|
||||
|
||||
Reference in New Issue
Block a user