promptkit/docs/roadmap/inspect-backend.md

# Operator-Facing Resource Inspection And Configuration API Roadmap

## Purpose And Status

Help operators using CommandKit-based applications inspect the configured
prompt, profile, or backend selected by ID. CommandKit owns commands and report
presentation; Promptkit exposes the selected resource's resolved configuration.

The direction is accepted for planning; implementation has not begun. Selected
work includes fuller prompt inspection, profile inspection of the resolved
profile itself, a new `InspectBackend`, `BackendConfig.Extends`, typed prompt
construction through `PromptConfig`/`PromptSource`, generically renderable
inspection results, and simplification of Promptkit's post-handoff null handling.
Execution-policy reporting
and prepared-request enhancements have been removed from this roadmap. The
decisions below record the accepted scope and representation approach.

## Resource-Inspection Boundary

“Fully resolved” means that ConfigKit has applied source composition and
inheritance to the selected resource. A child profile should show its resulting
keys and values, not an unresolved `extends` directive. It does not mean that
inspection assembles everything affecting execution.

| API | Resource shown |
| --- | --- |
| `InspectPrompt` | Resolved prompt definition, including captured imported message content. |
| `InspectProfile` | Resolved profile definition, before backend-to-profile execution completion. |
| Proposed `InspectBackend` | Resolved backend definition, including configured diagnostic metadata. |

For example, given:

```yaml
profiles:
  - id: base
    backend: openrouter
    model: example-model
    temperature: 0.5
  - id: precise
    extends: {id: base}
    temperature: 0
```

The resource settings shown for `precise` are:

```yaml
id: precise
backend: openrouter
model: example-model
temperature: 0
```

ConfigKit inheritance contributes the parent's model and backend reference.
Promptkit's execution-time completion would obtain settings from that backend,
but those settings do not become fields of the inspected profile. The operator
can inspect the referenced backend separately. Likewise, prompt inspection
reports a default profile reference without incorporating that profile's values.

Use the captured, resolved configuration, including normalizations/defaults
already recorded in that configuration. Do not add framework execution defaults,
request overrides, credential values, or provider-derived defaults to fill
absent resource fields. For example, an absent profile timeout is not populated
from the framework generation timeout, while a queue default already recorded
in the resolved backend remains visible. See the
[format reference](../formats.md#source-headers-collections-and-resource-ids).

## Design Balance And Reporting

Provide useful resource information already captured or available with modest
projection. Reuse ConfigKit's resolved handoff and existing resource capture;
avoid another composition pass, execution resolver, parallel provenance system,
or duplicate retained graphs solely for inspection. These are proportionality
guidelines, not prohibitions on a small helper or type when it serves the API.

**Selected presentation:** Omit absent optional settings. Preserve explicit
values, including zero. Absence is not displayed as zero or `unspecified`.
Treat null like omission for nullable resource fields after ConfigKit resolution,
under the selected post-handoff semantics below. Literal nulls inside provider
data retain their meaning; they are not optional resource-field presence states.
Accepted decision 4 selects typed results without a third state for nullable
resource fields.

Inspection reports effective resource values rather than original YAML layout,
comments, or the history of each inherited field. `InspectConfiguration` remains
available for provenance. Existing opaque prompt identity/hash information can
remain alongside the resource view; it does not turn inspection into a source
round-trip API.

Retain existing captured-state, namespace, ownership, and credential-handling
contracts unless a change is deliberately selected. Mutable results should not
let a consumer change retained engine state; that does not require repeated
copies between private helpers or copying immutable strings. Ordinary resource
inspection needs no template rendering, provider calls, or credential
availability checks.
Templates and arbitrary provider settings remain application data whose
presentation and disclosure are controlled by the consuming application.

## Generic Rendering And Presence

**Requirement:** Consumers must be able to render `InspectPrompt`,
`InspectProfile`, and `InspectBackend` results generically, without maintaining
parallel resource schemas or field-by-field adapters. Keep typed Go results for
programmatic use and make them directly encodable as JSON and YAML using
ordinary encoders. A consumer calls inspection, handles the error, and passes
the result to its selected encoder; it need not interpret individual settings
or register each nested Promptkit type.

Promptkit owns serialization field names, nesting, and omission semantics for
these results. Use consistent names aligned with configuration fields where
applicable. CommandKit owns command selection, output format, presentation, and
delivery. Adding an inspection field should not require another CommandKit
field mapping after updating its Promptkit dependency. Consumers may still
choose custom layouts, but generic rendering must be sufficient for a complete
resource report.

Capture presence before converting configured values to execution values that
collapse omission into zero, false, or an empty string. This applies across all
three resource kinds, including prompt input `required`, output
`repair_attempts`, and backend capacity settings. Use small inspection-specific
values where existing types such as `PromptInputDefinition` or `OutputContract`
cannot preserve that distinction. Do not change shared execution types merely
to make them suitable for inspection. Existing configuration validation rules,
including rejection of null for non-nullable fields, remain intact.

Omit absent optional settings and nullable fields resolved to null. Retain
explicit zero, false, empty strings, and explicit empty collections where
represented in the resolved resource, along with literal nulls inside provider
data. Generic encoding must preserve these distinctions; a blanket `omitempty`
on scalar or collection fields is not sufficient. Prefer ordinary optional
fields and serialization tags, with focused encoding support only where needed
for the selected semantics. Consumers must not reconstruct presence themselves.

This is a modest serialization contract for inspection results, not a new
reporting framework, configuration round-trip API, or promise of byte-for-byte
output stability. Formatting choices such as indentation and mapping-key order
belong to the encoder or consumer; semantic sequence order remains meaningful.
Promptkit need not provide CLI layouts or a generic reflection framework.

## Expanded Prompt Inspection

Expose description, ordered message templates with roles and cache-control
metadata, and the session-ID template, alongside the existing identity, hash,
default profile reference, input declarations, and output contract. Include
schema references and repair settings as part of the prompt definition.

Captured prompt definitions already retain the effective content of inline and
imported text/Markdown messages. Return that text without rendering or reopening
files. Original source layout and parsed-template objects are unnecessary.
The schema remains a reference; inspecting the prompt does not load it.

Keep prompt inspection independent of request readiness and default-profile
lookup. A captured prompt remains inspectable without a runnable default profile.

## Resolved Profile Inspection

This is a deliberate change to existing `InspectProfile` behavior. It currently
returns backend-completed `EffectiveModelParams`; the new resource boundary
requires the resolved profile's own fields instead.

Report fields supplied by the profile or inherited through ConfigKit, including
backend reference, endpoint/model where present, generation controls, credential
metadata, and provider parameters. Do not merge in backend provider defaults or
resolve an environment variable. Profiles may be partial, and their references
remain references. Distinguish omitted settings from explicit values, including
zero, using the resolved resource's information rather than inferring presence
from an execution target. Null uses the same absent representation as omission.

Execution target presence and prepared metadata describe a later resolution
stage and are not substitutes for profile configuration presence. Use the
existing configuration capture path to expose the needed information without
introducing a second configuration interpreter. Follow accepted decision 4 for
typed results and migration from `EffectiveModelParams`.

This roadmap adds no preparation API surface. Execution continues to apply
backend completion, framework defaults, and request overrides through its
existing path, with the deliberate configuration behavior change below.

## Post-Handoff Null Semantics

**Selected behavior:** For Promptkit's interpretation of nullable resource
fields, null and omission mean the same thing. Apply this consistently to
resource inspection and execution, rather than hiding a runtime distinction
only in inspection results.

ConfigKit owns everything before the handoff of its resolved configuration,
including source composition, inheritance, and null's effects during those
operations. This work makes no upstream change or request, removes no nulls
from input sources, and does not reinterpret or repeat ConfigKit's composition.
The retained ConfigKit result and its provenance remain available through
`InspectConfiguration`; resource projection does not rewrite that tree.

Remove the existing special treatment of null in these profile fields:

| Field | Selected meaning of either omission or null after handoff |
| --- | --- |
| `endpoint` | Permit the selected backend's endpoint fallback. |
| `api_key_env` | Permit the ordinary backend credential-source fallback, subject to the existing `api_key_required` rule. |
| Whole-map `extra_params` | Permit the backend's configured provider parameters. |

Retain explicit values, including zero, false, and the existing empty-string
clearing behavior for `endpoint` and `api_key_env`. Retain the ordinary map
overlay behavior: `extra_params: {}` does not clear backend defaults, and
`extra_params: ""` is not a valid mapping. Profile-level suppression of the
whole backend provider-parameter map via null is deliberately removed; do not
replace it with another clearing option without a concrete requirement.
Operators remain free to remove or replace parameters in backend definitions
through ordinary ConfigKit composition. This does not prohibit those workflows
or change ConfigKit's null semantics.

Null entries within arbitrary provider maps and arrays remain literal provider
data. This is not recursive null removal. Required-field checks and fields that
currently reject null remain governed by their existing validation rules; this
decision does not broaden accepted configuration shapes.

For example, ConfigKit may resolve a child's null over a parent's endpoint.
Promptkit then treats the resulting unset profile endpoint like omission and
uses the selected backend's endpoint, if available. The parent's endpoint is
not restored or recomputed by Promptkit.

## Backend Inspection

Proposed entry point:

```go
func (e *Engine) InspectBackend(
    ctx context.Context,
    backendID string,
) (*BackendInspection, error)
```

Expose the full backend identity and its resolved fields: base endpoint,
credential environment-variable name, provider defaults, concurrency/queue
settings, and configured `smoke_tests` metadata. Reuse suitable public nested
values where their semantics match; exact names and nesting follow decision 4.

Diagnostic metadata includes the configured authentication kind, profile
reference, and endpoint overrides after ConfigKit composition/inheritance.
An inherited endpoint override is a resolved configured value. An endpoint
computed later by a smoke-test adapter is not. Do not synthesize `/key` URLs,
reachability fallbacks, or a `smoke_tests` block that is absent from the resolved
backend. No shared diagnostic URL derivation work is required for this feature.

Inspecting a backend does not resolve its smoke-test default profile or contact
an endpoint. Report the configured reference even if that profile cannot be
selected. Capacity values describe configured policy, not current occupancy.

Follow the existing namespace-view selection rules: qualify slashless caller
IDs with the backend scope, pass qualified IDs through, and return the selected
full ID. Bundled and caller-defined resources use the same lookup rules.
Follow existing inspection conventions for invalid engines, invalid selections,
and cancellation. Missing-backend classification follows accepted decision 1.

## Consumer Integration And Compatibility

CommandKit can offer navigation between prompt, profile, and backend references
without combining their settings into the inspected resource. Stored references
retain their exact meaning; caller namespace defaults do not rewrite them.
A particular connected layout is a consumer choice, not a new Promptkit API.

Coordinate CommandKit adoption of generically rendered resource results,
replacing parallel report structures and per-field conversions where they are
no longer needed. In particular, migrate consumers of `EffectiveModelParams`.
Also account for prompt inspection fields whose types need to change to retain
presence; compatibility work is not limited to `ProfileInspection`. Document
the Go and serialized-result migrations with the final types, and follow the
existing release policy. The profile semantic change must be documented as such,
not presented as merely adding more fields.

Document the post-handoff null change as a configuration compatibility change
as well. Existing null endpoint/credential-source clearing can use the existing
explicit empty-string form when that behavior is desired. There is no new
replacement for profile-level whole-map suppression of backend parameters.
Include a migration example showing that a profile's null now permits backend
endpoint, credential-source, or provider-parameter fallback that it previously
suppressed; this changes execution as well as inspection. Update
the [format reference](../formats.md#defaults-and-overrides) when implementation
lands; it describes the current behavior until then.

Existing preparation APIs remain available for separate concrete-request uses;
this roadmap adds neither preview commands nor execution-policy report fields.
Transport timeouts, engine retry settings, caller deadlines, and injected-client
policy are outside the inspected resource unless they are actual fields of its
resolved definition. This is the scope of these APIs, not a pending engine-policy
feature decision.

## Typed Backend Inheritance

Add `Extends *string` to `BackendConfig`, matching `ProfileConfig.Extends`.
`BackendSource` encodes a non-nil value as `extends: {id: value}` and omits it
when nil. ConfigKit owns sibling inheritance and its diagnostics. The source
helper does not look up parents or populate inherited fields itself.

Keep existing root/collection identity and ConfigKit composition semantics,
including its handling of null. Raw mappings remain available for generic
controls outside the convenience helper.
See [resource inheritance](../formats.md#resource-inheritance) and the existing
[profile helper](../../profiles.go).

## Typed Prompt Sources

Add `PromptConfig` and `PromptSource` to complete the typed source-helper set
alongside `BackendConfig`/`BackendSource` and `ProfileConfig`/`ProfileSource`.
The proposed helper follows their calling convention:

```go
func PromptSource(label string, prompts []PromptConfig) configkit.Source
```

Cover prompt identity, sibling inheritance through `Extends`, default profile,
description, session template, inputs, messages with inline or file content and
cache controls, and output settings. Choose nested input types for configuration
semantics; reuse existing types where suitable without forcing execution or
inspection values into the source-input role.

Keep the helper a straightforward mapping constructor. Encode supplied fields
under a `prompts` list, then delegate to `configkit.Mapping`. Optional nil fields
mean omission; explicit values, including zero, false, and empty strings where
supported, must remain representable. Preserve nil-versus-empty collections
where that distinction matters to composition. Raw ConfigKit mappings remain
the escape hatch for ConfigKit's explicit-null composition operations and other
advanced controls outside the typed convenience surface. The post-handoff
interpretation above does not change how source helpers supply values to ConfigKit.

The helper does not resolve inheritance, apply execution defaults, render
messages, read content files, or independently validate a prompt. Existing
ConfigKit composition and engine loading retain those responsibilities. Relative
content references use the ordinary input/source resource-context mechanism;
creating a typed source does not select a new filesystem root or file-loading
workflow. Reuse ConfigKit's source snapshot behavior rather than adding another
copier or recursive compatibility converter.

Keep all three source-helper families consistent in their scope and omission
conventions. They need not reproduce every generic configuration operation or
share a reflection-based encoding framework. Maintain the field mappings and
focused behavioral tests as the configuration model evolves.

## Related Work

Listing return-type consistency is a possible later cleanup, not required here.
Collections, schemas, and engine options do not need identical APIs merely for symmetry.
The [deferred fingerprint proposal](deferred.md#semantic-execution-target-fingerprints)
remains separate. Live capacity monitoring, provider health guarantees, and
source round-tripping are not selected deliverables.

## Decisions

All five decisions below are resolved, including generic rendering and the
post-handoff null simplification in decision 4. Numbering is retained for
continuity with the discussion; implementation is pending.

### 1. How Should An Unknown Backend Be Classified?

**Status:** Accepted; implementation pending.

**Decision:** Add public `ErrBackendNotFound` for an absent backend selected by
`InspectBackend`, following the existing prompt/profile sentinel pattern.
An ordinary missing-backend lookup should match `ErrBackendNotFound` through
`errors.Is` and should not also match `ErrInvalidRequest`. Reuse the registry's
existing internal missing-backend classification at the public boundary.

Invalid selections follow existing ID rules and match `ErrInvalidRequest`;
invalid engines match `ErrInvalidConfig`. Preserve context cancellation through
the existing inspection conventions. Invalid backend definitions remain an
engine-construction concern. Exact error wording is not a new contract.

Initially apply the new public classification to `InspectBackend`; do not
implicitly reclassify construction or smoke-test failures. CommandKit owns
operator wording and exit-status selection.

**Rationale:** This gives consumers a useful distinction with a small addition
consistent with prompt and profile inspection, without a new error hierarchy.

**Alternatives considered:** Using `ErrInvalidRequest` saves one public sentinel
but combines malformed selections and missing identities. A typed generic
resource-not-found error could carry resource kind and ID, but adds a broader
contract and integration questions for existing sentinels. Neither is selected;
the caller already knows the operation and requested ID.

### 2. Resource Inspection Does Not Include Execution-Policy Reporting

**Status:** Resolved; the former engine-policy proposal is removed from scope.

**Decision:** Inspect the selected resolved resource, not all settings that may
affect its execution. ConfigKit composition and inheritance are included;
backend-to-profile execution completion, framework execution defaults, request
overrides, and engine transport/retry reporting are not.

Remove the proposed engine-settings API, prepared-result policy fields, and
prepared-result presence enhancements from this roadmap. No decision about
where to place those additions remains necessary for resource inspection.

### 3. Should Typed Prompt Construction Be Added In This Round?

**Status:** Accepted; implementation pending.

**Decision:** Add `PromptConfig` and `PromptSource`, including inheritance,
following the thin mapping-constructor pattern described above. Retain the
backend/profile helpers and raw ConfigKit mapping support.

**Rationale:** API symmetry and convenient programmatic construction have value
as Promptkit approaches 1.0. The existing helpers are small explicit encoders;
a prompt helper adds nested input values and mappings, not a new authoring,
composition, or validation subsystem. That bounded maintenance cost is justified
by consistent discoverability and compiler assistance across all three resources.

**Alternatives considered:** Deferring prompt helpers would leave asymmetric Go
conveniences. Removing the existing typed helpers would simplify Promptkit's
surface but move string-keyed mapping construction to consumers and require
migration. Neither is selected. Full typed coverage of all ConfigKit operations
is unnecessary; raw mappings continue to serve those cases.

### 4. How Should Resolved Resource Results And Presence Be Represented?

**Status:** Accepted; implementation pending.

**Decision (Option A):** Use resource-oriented typed inspection results, reusing
suitable public nested values. Change `ProfileInspection` deliberately to report
the resolved profile rather than retaining a second execution-settings view for
compatibility. Apply the same resource semantics to prompt/backend additions.
Require generic rendering under the contract above; typed access must not force
consumers to maintain parallel report types or interpret every setting.

Represent absent settings and explicit values, including zero, false, and empty
strings where valid. Treat nullable resource fields set to null as absent in
the result and adopt the post-handoff behavior described above. This supersedes
the earlier proposal to preserve three profile-specific null distinctions.
ConfigKit composition and literal nulls within provider data retain their
existing ownership and meaning. No third resource-field presence state or
generic resource framework is needed. Execution-target presence does not
represent the uncompleted profile and must not substitute for it.

**Rationale:** Typed results provide discoverable Go fields without making
consumers interpret a generic configuration mapping. Endpoint and credential
overrides already take precedence without first clearing a value, and explicit
empty strings already support clearing those settings. No concrete requirement
justifies retaining profile-level whole-map suppression of backend parameters.
Removing these exceptions simplifies interpretation and avoids carrying source
syntax and inheritance history into every field.

**Alternative considered:** A caller-owned resource mapping naturally represents
missing keys and null and can reduce per-field projection work, but sacrifices
typed discoverability and introduces a broader mapping-value contract. It is not
selected as the primary resource-inspection result; `InspectConfiguration`
remains available for explicit configuration/provenance inspection.

Document migration for existing profile and prompt inspection consumers with
the final Go types and their serialization semantics. Generic JSON/YAML encoding
is selected; bespoke consumer reports and custom serializers are not inherently
required. Do not leak internal package types or mutable captured state.

### 5. Backend Diagnostics Show Configured Values, Not Derived Destinations

**Status:** Resolved by the resource-inspection boundary.

**Decision:** Return diagnostic settings present in the resolved backend,
including inherited endpoint overrides. Leave omitted endpoint settings absent;
deriving a default destination belongs to diagnostic execution. The earlier
proposal to expose both declared and computed destinations is removed. This
avoids an additional representation and provider-specific derivation machinery
in resource inspection.

## Acceptance Criteria

The companion [implementation plan](implementation.md) owns the staged work
and verification sequence. The target is complete when:

- Prompt, profile, and backend inspection expose the selected resolved resource
  with the boundary and presence semantics above, using ordinary JSON/YAML
  encoding without per-field consumer adapters.
- Inherited values, explicit zero/false/empty settings, and provider-data nulls
  are represented accurately; nullable resource nulls and omission are absent.
- Profile inspection excludes backend-completed execution values, and backend
  inspection reports configured diagnostics without synthesizing destinations
  or requiring diagnostic readiness.
- Prompt inspection includes captured inline/imported message content and
  metadata without rendering or loading related resources.
- Execution applies the selected post-handoff null behavior while preserving
  explicit clearing, map overlay, and ConfigKit's retained result and provenance.
- Selection, errors, captured-state isolation, and credential handling follow
  the selected contracts. Requirement discovery retains its lightweight path.
- All three resource kinds have typed source helpers, including inheritance,
  using ordinary ConfigKit composition, context, and snapshot behavior.
- Canonical guidance and the next minor release note explain changed Go
  inspection types, generic rendering, source helpers, and null migration.

Promptkit owns resource serialization checks; CommandKit owns presentation and
delivery. No engine-policy or prepared-request additions, benchmark threshold,
or new release gate is part of this feature.