Clarify provider selection and diagnostic defaults in the roadmap
This commit is contained in:
@@ -1,6 +1,7 @@
|
||||
# Invocation-Scoped Execution Observation
|
||||
|
||||
Status: accepted scope; implementation pending.
|
||||
Status: agreed direction; default-provider and vLLM scope under discussion;
|
||||
implementation pending.
|
||||
|
||||
## Purpose
|
||||
|
||||
@@ -143,11 +144,18 @@ promptkit:
|
||||
api_key_env: OPENROUTER_API_KEY
|
||||
```
|
||||
|
||||
An omitted or null provider retains ordinary OpenAI-compatible behavior without
|
||||
provider-specific defaults or metadata interpretation. Initially, `openrouter`
|
||||
selects the OpenRouter conventions described here. A vLLM-specific selection can
|
||||
follow when its relevant response contract and representative fixtures have been
|
||||
reviewed; this work does not claim vLLM-specific support.
|
||||
Normalize the final effective provider value by trimming whitespace and
|
||||
lowercasing. Omitted, null, empty, and whitespace-only values are equivalent.
|
||||
Reject an unsupported nonempty normalized value as an ordinary configuration
|
||||
error rather than silently using generic behavior. `openrouter` selects the
|
||||
OpenRouter conventions described here.
|
||||
|
||||
The current generic fallback retains ordinary OpenAI-compatible behavior without
|
||||
provider-specific defaults or metadata interpretation. Before finalizing this
|
||||
roadmap, discuss making this fallback an explicit lowest-common-denominator
|
||||
provider and adding a `vllm` selection. Their names, supported conventions, and
|
||||
final scope remain to be settled; this document does not yet select a vLLM
|
||||
contract.
|
||||
|
||||
Resolve the field through ordinary ConfigKit composition and inheritance. Carry
|
||||
the selected value through configuration capture and prepared execution, expose
|
||||
@@ -182,6 +190,21 @@ existing behavior.
|
||||
These are conventions for explicitly requested diagnostics. Loading a backend,
|
||||
constructing an engine, and ordinary generation do not initiate probes.
|
||||
|
||||
Resolve provider-supplied diagnostic defaults at the provider boundary without
|
||||
writing them into the loaded configuration or recording them as explicit source
|
||||
values. Configuration inspection preserves the distinction between configured
|
||||
overrides and provider-supplied fallbacks, including their existing source
|
||||
provenance.
|
||||
|
||||
### Diagnostic Migration
|
||||
|
||||
This changes the current authentication metadata contract: an authentication
|
||||
object currently requires `kind`, and clearing the object removes its configured
|
||||
adapter. Under this roadmap, endpoint-only overrides become valid, and clearing
|
||||
the object restores any selected provider's default. `kind: none` suppresses
|
||||
that default. Explain these changes in the framework format documentation and
|
||||
release notes, alongside the new provider field.
|
||||
|
||||
### Provider Behavior Ownership
|
||||
|
||||
Centralize provider-specific behavior within `internal/llm`, which already owns
|
||||
@@ -200,6 +223,8 @@ method to a generic operation as part of this consolidation.
|
||||
Provider-specific files and straightforward dispatch are sufficient initially.
|
||||
No new `internal/providers` package, registration framework, or broad interface
|
||||
requiring every provider to implement every capability is needed.
|
||||
This is configurable selection among PromptKit-supported provider implementations,
|
||||
not a public API for downstream registration of additional implementations.
|
||||
|
||||
## Optional Provider-Reported Cost
|
||||
|
||||
@@ -321,9 +346,14 @@ logging infrastructure are outside this scope.
|
||||
provider-default probes, explicit kind and endpoint overrides, `kind: none`,
|
||||
and existing explicit probes without a provider selection.
|
||||
- Verify provider selection independently of backend names and namespaces,
|
||||
including renamed and inherited backends, generic behavior when omitted/null,
|
||||
including renamed and inherited backends, equivalent omitted/null/blank values,
|
||||
trimming and case normalization, rejection of unsupported nonempty values,
|
||||
and retention through endpoint overrides. Confirm provider metadata is not
|
||||
serialized into the completion request body.
|
||||
serialized into the completion request body. Use the generic fallback selected
|
||||
when the remaining provider discussion is complete.
|
||||
- Verify that provider diagnostic defaults do not become explicit values in the
|
||||
loaded configuration or replace source provenance in inspection. Cover the
|
||||
changed clearing behavior and endpoint-only authentication overrides.
|
||||
- Carry optional OpenRouter cost through response decoding, internal/public
|
||||
conversions, successful run results, attempt and terminal events, and existing
|
||||
group summaries. Keep result cost available without registering an observer.
|
||||
@@ -353,11 +383,12 @@ logging infrastructure are outside this scope.
|
||||
and update internal observation and model-client documentation for changed flows.
|
||||
Document `provider`, its smoke-test defaults and override semantics in the
|
||||
framework format reference and relevant public GoDoc; update inspection and
|
||||
diagnostic documentation with the implemented behavior.
|
||||
diagnostic documentation with the implemented behavior. Include the diagnostic
|
||||
migration in release notes.
|
||||
|
||||
## Implementation Planning Guidance
|
||||
|
||||
Plan two separately reviewable stages:
|
||||
Organize the eventual implementation plan around two workstreams:
|
||||
|
||||
1. Contextual observer selection, including ordinary, prepared, coordinated,
|
||||
and nested execution boundaries and the existing group collector.
|
||||
@@ -366,9 +397,11 @@ Plan two separately reviewable stages:
|
||||
through decoding, accounting, public results, events, and group summaries.
|
||||
Include the corresponding configuration and API documentation.
|
||||
|
||||
The second stage is more extensive than the context helper and may be divided
|
||||
into focused implementation units. Backend provider selection is decided above;
|
||||
avoid coupling delivery of contextual observation to provider cost support.
|
||||
The second area is more extensive than the context helper. Keep backend provider
|
||||
configuration, diagnostic consolidation, and cost accounting in separately
|
||||
reviewable implementation units rather than one large implementation prompt.
|
||||
Complete the remaining provider discussion before preparing that plan; avoid
|
||||
coupling delivery of contextual observation to provider cost support.
|
||||
|
||||
Today, successful results accumulate generation responses in the runner, while
|
||||
optional observation recorders account for dispatched attempts. Adding cost to
|
||||
@@ -385,5 +418,19 @@ field separately so an unusable cost becomes unknown without changing existing
|
||||
framing or content validation. This needs no general numeric-conversion layer.
|
||||
|
||||
Use the completion criteria above to select focused offline checks for each
|
||||
stage. This roadmap owns target behavior; a separate implementation plan owns
|
||||
unit. This roadmap owns target behavior; a separate implementation plan owns
|
||||
the detailed work sequence.
|
||||
|
||||
## Topics For Further Discussion
|
||||
|
||||
Before finalizing the roadmap, settle:
|
||||
|
||||
1. A default, lowest-common-denominator OpenAI API-compatible provider used when
|
||||
the backend does not specify a provider, including whether it has an explicit
|
||||
selectable identifier and which common conventions it owns.
|
||||
2. A `vllm` provider with vLLM-specific conventions, identifying the response and
|
||||
diagnostic behavior to support from reviewed upstream contracts and fixtures.
|
||||
|
||||
The normalization, unsupported-value handling, override precedence, and
|
||||
configuration-inspection boundaries above remain agreed decisions. These topics
|
||||
do not yet add further provider behavior or an implementation commitment.
|
||||
|
||||
Reference in New Issue
Block a user