base-layer/commandkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document PromptKit catalogs and smoke testing

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-17 19:15:58 +00:00
parent 8d1f33a40e
commit e653b7d6ec
15 changed files with 386 additions and 177 deletions
Download Patch File Download Diff File Expand all files Collapse all files

394
docs/cli.md
View File

@@ -175,14 +175,36 @@ sources:
kind: file
underlay_index: 0
imported_from: null
destination: /server
placement: mapping
destination_unresolved: false
version_check:
path: /version
version:
present: true
null: false
text: "1"
location:
source_id: source-1
line: 2
column: 1
redacted: false
```
Each source includes `id`, `label`, `path`, `layer`, `kind`, `underlay_index`,
and `imported_from`. The import location is null when absent; otherwise it has
`source_id`, `line`, and `column`. Owning values and source order are preserved,
including empty fields, zero coordinates, and synthetic sources without paths.
An empty inventory is `sources: []`. Text output begins with the scope heading
and then prints the same source fields, quoting labels and paths.
and `imported_from`, plus the native destination pointer, placement, unresolved
destination flag, and nullable `version_check`. Placement is `mapping` or
`sequence_item`. The import location is null when absent; otherwise it has
`source_id`, `line`, and `column`. Version evidence preserves `path`, `present`,
`null`, `text`, optional source `location`, and `redacted` as returned by
ConfigKit. Owning values and source order are preserved, including empty fields,
zero coordinates, false and null observations, and synthetic sources without
paths. An absent version check is null. An empty inventory is `sources: []`.
Text output begins with the scope heading and prints the same source fields,
quoting labels and paths. These fields project the selected ConfigKit source
records; CommandKit does not reload files or rerun version checks. See the
[ConfigKit v0.8.0 inspection API](https://gitea.maximumdirect.net/eric/ConfigKit/src/commit/137769f3f18a370c68444242fb5d88e620e307e5/inspection.go)
for native evidence definitions.
### Explain a configuration path
@@ -306,38 +328,35 @@ existing incomplete-report delivery outcome and can leave partial output.
## PromptKit definition inspection
Applications can install prompt and profile inspection independently with
`ckprompt.Command(suite)`, passing the resulting declaration to core. The
`ckprompt.FromEngine(loadEngine)` adapter supplies callbacks that acquire an
engine only after input checks pass and inspection is reached. The loader
receives invocation context, selection, input, and state; its implementation
controls whether an engine is constructed or reused. Direct suite callbacks
retain their own implementation and lifecycle.
`ckprompt.Command(suite)`. The `ckprompt.FromEngine(loadEngine)` adapter
supplies these callbacks alongside native requirements discovery and the five
catalogs. Its loader receives invocation context, configuration selection,
typed input, and shared state. Direct suite callbacks retain their own
implementation and lifecycle.
CommandKit's inspection path does not execute prompts, contact model providers,
or resolve provider credential values. Application acquisition and direct
callback effects remain application-owned. Prompt definitions and profile
resolution remain owned by
[PromptKit's inspection API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/4b8169da286917fde7875425bb32b799b12c6af3/engine.go).
Prompt inspection uses an ID only. PromptKit returns any descriptive version
metadata as an optional report field; there is no `--version` flag for selecting
a prompt version. The prompt format and native selection behavior remain owned by
[PromptKit v0.16.0](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/docs/consumers/pkg-promptkit.md).
```text
example promptkit show prompt customer/summary --version 2 --format yaml
example promptkit show prompt customer/summary --format yaml
example promptkit show prompt customer/summary
example promptkit show profile customer-support --format yaml
```
Both commands require one ID operand and accept `--config`, `--overlay`, and
`--format text|yaml` (default `text`). Prompt inspection also accepts singleton
`--version`, defaulting to the empty string. PromptKit owns version selection:
an omitted version works for a single-version ID, while an ambiguous ID is an
operational error. These commands have no built-in `--scope` selection; their
namespace is `promptkit`.
`--format text|yaml` (default `text`). These commands have no built-in
`--scope` selection; their namespace is `promptkit`.
Prompt YAML has these fields. Input order is preserved and an empty input list is
`[]`. The prompt hash is an opaque owning-library equality value.
Prompt YAML preserves native input order and uses an optional `version` field.
The key is omitted when PromptKit returns no version and is retained when the
version is present, including an empty string. The prompt hash is an opaque
owning-library equality value.
```yaml
prompt_id: customer/summary
prompt_version: "2"
version: "2"
prompt_hash: opaque-value
default_profile_id: customer-support
inputs:
@@ -376,47 +395,41 @@ api_key_required: true
```
Text uses labeled fields and includes nested `extra_params` values. A missing ID
is a usage error. Blank, unknown, and ambiguous definition selections reach the
owning inspection API and produce operational results with supported public
error identity. Cancellation is also operational: standard handlers check the
caller's context at inspection boundaries, and PromptKit can return cancellation
from its own inspection methods. Supported cancellation identity is preserved.
is a usage error. Definition selection reaches the owning inspection API and
preserves supported error identity and cancellation.
## PromptKit requirements
When `Suite.InspectRequirements` is present,
`ckprompt.Command(suite)` installs `promptkit requirements show`. It accepts one
required prompt ID and uses PromptKit's native selection and discovery rules:
When `Suite.InspectRequirements` is present, `ckprompt.Command(suite)` installs
`promptkit requirements show`. It accepts one prompt ID and uses PromptKit's
native selection and discovery rules:
```text
example promptkit requirements show briefing --campaign spring
example promptkit requirements show briefing --version 2 --profile editorial --var mode=full
example promptkit requirements show briefing --profile editorial --var mode=full
example promptkit requirements show briefing --session-id session-42 --var note= --format yaml
```
The command accepts these flags:
The command accepts:
| Flag | Values | Default | Meaning |
| --- | --- | --- | --- |
| `--config` | path string | omitted | Select the configuration input; explicit presence is retained. |
| `--overlay` | path string | omitted | Select one overlay; explicit presence is retained. Repeated occurrences use the last value. |
| Suite selectors | application-defined | per selector | The flags captured from `Suite.Selectors`. |
| `--version` | string | empty | Pass the prompt version to PromptKit; empty uses its native version-selection rule. |
| `--profile` | string | empty | Pass the profile ID to PromptKit; empty uses its native profile-selection rule. |
| `--profile` | string | empty | Pass the profile ID to PromptKit; empty uses its native selection rule. |
| `--session-id` | string | empty | Pass session context to PromptKit. |
| `--var` | repeatable `KEY=VALUE` | none | Supply caller-variable context for native discovery. |
| `--format` | `yaml` | `yaml` | Select the requirements report format. Other formats are usage errors for this operation. |
The `promptkit` namespace identifies the PromptKit operation, so there is no
built-in `--scope` flag. The version, profile, and session fields use native
empty-string semantics: an explicit empty string has the same meaning as
omission. PromptKit owns their interpretation, including default version and
profile selection, session context, and selection errors. Requirements
inspection does not require a profile to be execution-ready. See the pinned
[PromptKit requirements API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/143c6e9b3a675872117b0471e2ff8bcb25055a54/requirements.go)
for the owning discovery contract. The built-in flags other than `--var` are
singleton bindings; when repeated, the last value is used. Suite selectors retain
their declared binding behavior.
The `promptkit` namespace identifies the operation, so there is no built-in
`--scope` flag. An explicitly empty profile or session value retains PromptKit's
native empty-string meaning. Requirements inspection does not require a profile
to be execution-ready. See the
[PromptKit v0.16.0 consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/docs/consumers/pkg-promptkit.md)
for native configuration, selection, and requirements behavior. The built-in
flags other than `--var` are singleton bindings; when repeated, the last value
is used. Suite selectors retain their declared binding behavior.
Each `--var` occurrence must contain `=`. CommandKit splits at the first
separator and preserves the exact key and remaining value. Empty keys and
@@ -424,20 +437,21 @@ values, whitespace, commas, and additional `=` characters are retained. All
occurrences are parsed; duplicate keys use the last value, while any malformed
occurrence is a usage error even if a later assignment is valid.
The successful report is YAML with selected identities, native completeness,
and the input, variable, and issue inventories. False values and empty
inventories are retained. Input metadata preserves null versus an empty string.
Variable entries contain `name`, `kind`, `supplied`, and `explanation`.
`supplied` reports native exact-key presence, including an explicitly empty
value, independently of `kind` and `complete`. Explanations and issue descriptions
are upstream-owned human-readable summaries, not machine-readable details.
The YAML report contains selected identities, native completeness, and input,
variable, and issue inventories. False values and empty inventories are
retained. Input metadata preserves null versus an empty string. Variable entries
contain `name`, `kind`, `supplied`, and `explanation`. `supplied` reports native
exact-key presence, including an explicitly empty value, independently of
`kind` and `complete`. Explanations and issue descriptions are upstream-owned
human-readable summaries, not machine-readable details. The optional `version`
key is omitted when absent and retained when supplied by PromptKit.
For a prompt with the template `{{if .mode}}{{input "notes"}}{{end}}` and no
input declarations, omitting `mode` can produce this partial report:
input declarations, omitting `mode` can produce a partial report:
```yaml
prompt_id: briefing
prompt_version: "2"
version: "2"
profile_id: editorial
complete: false
inputs:
@@ -459,28 +473,24 @@ issues:
description: A missing variable prevents selecting a dependency-bearing branch.
```
PromptKit owns the inventory, classifications, explanations, completeness,
and structured issues. Completeness means discovery covered the relevant
dependencies; it does not mean inputs or variables are supplied, artifacts are
valid, credentials are available, or execution is ready. A complete report can
contain unsupplied required variables, and an incomplete report can still be
useful. Successfully acquired and delivered partial reports return status `0`;
structured issues remain in YAML and do not independently fail the command.
Provider errors, native selection errors, cancellation, and report delivery
failures use the operational path (status `1`) with supported cause identity.
Syntax, operand, variable-assignment, and format errors use usage status `2`.
PromptKit owns the inventory, classifications, explanations, completeness, and
structured issues. Successfully acquired and delivered partial reports return
status `0`; structured issues do not independently fail the command. Provider
errors, native selection errors, cancellation, and report delivery failures use
the operational path (status `1`) with supported cause identity. Syntax,
operand, variable-assignment, and format errors use usage status `2`.
### Concrete request validation
When `Suite.ValidateRequest` is present, `ckprompt.Command(suite)` also installs
`promptkit requirements validate`. It can be installed without discovery, and
discovery can remain installed without validation. The command accepts one
required prompt ID and passes a concrete request to PromptKit's native
prompt ID and passes a concrete request to PromptKit's native
`Engine.ValidateRequest` preparation path:
```text
example promptkit requirements validate briefing --campaign spring --input topic=topic.txt
example promptkit requirements validate briefing --version 2 --profile editorial --session-id session-42 --var mode=full --input notes=notes.txt
example promptkit requirements validate briefing --profile editorial --session-id session-42 --var mode=full --input notes=notes.txt
```
The command accepts:
@@ -490,7 +500,6 @@ The command accepts:
| `--config` | path string | omitted | Select the configuration input; explicit presence is retained. |
| `--overlay` | path string | omitted | Select one overlay; explicit presence is retained. Repeated occurrences use the last value. |
| Suite selectors | application-defined | per selector | The flags captured from `Suite.Selectors`. |
| `--version` | string | empty | Pass the prompt version to PromptKit. |
| `--profile` | string | empty | Pass the profile ID to PromptKit. |
| `--session-id` | string | empty | Pass session context to PromptKit. |
| `--var` | repeatable `KEY=VALUE` | none | Supply caller variables for the concrete request. |
@@ -498,24 +507,17 @@ The command accepts:
| `--format` | `yaml` | `yaml` | Select the success confirmation format. Other formats are usage errors. |
The built-in flags other than repeatable `--var` and `--input` are singleton
bindings; when repeated, the last value is used. Suite selectors retain their
declared binding behavior. Explicit empty `--config` or `--overlay` values keep
their supplied-presence information.
bindings; when repeated, the last value is used. Explicit empty `--config` or
`--overlay` values keep their supplied-presence information. The profile and
session fields use native empty-string semantics. PromptKit owns their
interpretation and selection errors.
The version, profile, and session fields use native empty-string semantics:
an explicit empty string has the same meaning as omission. PromptKit owns their
interpretation and selection errors. The `promptkit` namespace identifies the
operation, so there is no built-in `--scope` flag.
Each `--var` and `--input` occurrence must contain `=`. CommandKit splits at
the first separator and preserves the exact key/name and remaining value/path.
Empty variable keys and values are passed to PromptKit. Empty input names and
paths are also passed to native validation. File paths are translated to
`promptkit.File(PATH)` references without reading, normalizing, expanding, or
rebasing them during argument binding. Duplicate keys or names use the last
value, but all occurrences are parsed, so any malformed occurrence is a usage
error even if a later assignment is valid. Syntax and path values after shell
processing are otherwise preserved.
Each `--var` and `--input` occurrence must contain `=`. CommandKit splits at the
first separator and preserves the exact key/name and remaining value/path.
Empty variable keys and values and empty input names and paths are passed to
native validation. File paths become `promptkit.File(PATH)` references without
being read, normalized, expanded, or rebased during argument binding. Duplicate
keys or names use the last value, but every occurrence is parsed.
On success, the YAML confirmation contains only the requested prompt ID and
`valid: true`:
@@ -525,46 +527,64 @@ prompt_id: briefing
valid: true
```
Validation delegates concrete request preparation to PromptKit. Depending on
the request and configuration, native preparation may resolve prompt/profile
resources, load artifact and schema resources, render messages and session
context, and perform applicable configuration and credential checks. Native
errors are returned as encountered rather than as an exhaustive inventory.
Validation does not make a model call or retain a prepared execution snapshot.
A successful report means preparation passed for this invocation; it does not
certify provider acceptance or guarantee that later execution will succeed.
See PromptKit's [v0.15.0 `Engine.ValidateRequest` API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/635674c066d0b432187f96e72b9c03d085443ac5/engine.go)
for the native contract.
Validation delegates concrete request preparation to PromptKit. Native
preparation may resolve prompt/profile resources, load artifact and schema
resources, render messages and session context, and perform applicable
configuration and credential checks. It does not make a model call or retain a
prepared execution snapshot. A successful report describes this preparation
invocation; it does not certify provider acceptance or guarantee later
execution. See PromptKit's
[`Engine.ValidateRequest` API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/engine.go).
Builder, engine-loading, and native validation errors use the operational path
(status `1`) with supported error identity and acquired diagnostics retained.
Malformed syntax, missing operands, and unsupported formats use usage status
`2`. Input-check failures keep their existing shared handling. Successful
diagnostics and report-delivery failures follow the common delivery contract.
`2`. Successful diagnostics and report-delivery failures follow the common
delivery contract.
### PromptKit catalogs
An installed `ckprompt.Command(suite)` exposes the catalog commands supplied by
the suite:
An installed `ckprompt.Command(suite)` exposes each catalog whose callback is
present:
```text
example promptkit list prompts
example promptkit list profiles --config settings.yml --campaign spring
example promptkit list backends --namespace team
example promptkit list namespaces --kind prompt --kind profile
example promptkit list collections
```
Each command accepts `--config`, `--overlay`, and applicable selectors from
`Suite.Selectors`. It takes no ID or version operand and has no `--version` or
`--scope` selector. `--format` defaults to `yaml`; `--format yaml` is accepted,
and other formats are usage errors.
All catalogs accept `--config`, `--overlay`, applicable flags from
`Suite.Selectors`, and `--format yaml` (the default). Other formats are usage
errors. Prompt, profile, and backend listings accept namespace selection:
Each invocation writes one bare YAML sequence. Prompt entries have explicit
string `id` and `version` fields; profile entries are strings. Provider order and
exact identity values are preserved. For example:
| Binding | Effect |
| --- | --- |
| omitted `--namespace` and `--all-namespaces` | Forward PromptKit's native default namespace filter. |
| `--namespace team` | Forward the named namespace filter. |
| `--namespace ""` | Select the unqualified namespace. |
| `--all-namespaces` | Forward the all-namespaces filter. |
| `--all-namespaces=false` | Do not select an override; a supplied namespace still applies, otherwise native default is used. |
The namespace binding is a singleton and its last occurrence is used.
Supplying a namespace while `--all-namespaces=true` is active is a usage error
before provider work. Namespace inventory accepts repeatable `--kind KIND`;
omitting `--kind` forwards PromptKit's native default kind selection. Repeated
values are forwarded in order. Collections have no namespace or kind filter.
Each operation writes one bare YAML sequence, retaining provider order and
native identities:
- Prompt entries are `{id, version?}`. The optional `version` key is omitted
when absent and retained for a present empty or nonempty value.
- Profiles, backends, and namespaces are strings.
- Collection entries are `{namespace, version?}`, where `version` projects
PromptKit's optional release version.
- Nil and allocated-empty inventories both render as `[]`.
```yaml
- id: briefing
version: "1"
- id: briefing
- id: team/briefing
version: "2"
```
@@ -573,11 +593,155 @@ exact identity values are preserved. For example:
- spring
```
Successful nil and allocated-empty inventories both render as `[]`. Catalog
membership and ordering follow the selected provider, and a listed definition
is not a readiness check or execution result. PromptKit owns the meaning of its
native listing APIs; custom providers own their results. Applications retain
their own disclosure choices for catalog data.
A catalog result is an inventory, not a readiness check or execution result.
The owning provider controls membership and order. Namespace filters are passed
to PromptKit without local membership interpretation. See the
[PromptKit v0.16.0 consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/docs/consumers/pkg-promptkit.md)
for native resource and selection definitions.
### PromptKit smoke testing
When `Suite.SmokeTest` is present, `ckprompt.Command(suite)` installs
`promptkit test KIND`. The special kind `all` runs the four concrete operations
sequentially in this order: `reachability`, `authentication`, `text_inference`,
and `structured_output`. Both forms use the same single-operation callback;
each aggregate call changes the seed kind before invoking it. The command
accepts:
```text
example promptkit test text_inference --campaign spring --prompt briefing --profile spring
example promptkit test all --campaign spring --prompt briefing --profile spring
example promptkit test authentication --backend local
```
| Flag | Values | Default | Meaning |
| --- | --- | --- | --- |
| `--config`, `--overlay` | path strings | omitted | Select PromptKit configuration; supplied presence is retained. |
| Suite selectors | application-defined | per selector | Captured flags from `Suite.Selectors`. |
| `--backend` | string | empty | Native backend ID. |
| `--prompt` | string | empty | Native prompt ID. |
| `--profile` | string | empty | Native profile ID. |
| `--endpoint` | string | omitted | Override the execution endpoint. |
| `--model` | string | omitted | Override the execution model. |
| `--temperature` | float | omitted | Override execution temperature. |
| `--max-tokens` | integer | omitted | Override execution maximum tokens. |
| `--top-p` | float | omitted | Override execution top-p. |
| `--timeout-seconds` | integer | omitted | Override the execution timeout. |
| `--service-tier` | string | omitted | Override the execution service tier. |
| `--reasoning-effort` | string | omitted | Override execution reasoning effort. |
| `--api-key-env` | string | omitted | Supply an API-key environment-variable name to native configuration. |
| `--transient-retries` | integer | omitted | Set the native inference transient retry count. |
| `--format` | `yaml` | `yaml` | Select the report format. Other formats are usage errors. |
The execution flags populate PromptKit's native execution override. The
`--api-key-env` value is a variable name, not a credential value. Application
selectors, loader policy, credentials, and client wiring remain application
owned. The `WithSmokeTesting` helper accepts an optional typed builder. A nil
builder forwards the seed; a supplied builder's complete native request is
authoritative. The helper retains a native report when PromptKit returns both a
report and an error.
A single report projects the native smoke report into YAML. It preserves native
false, unknown/null, and incomplete values and does not add a readiness score.
The report includes `kind`, `supported`, `applicable`, `completed`, `reason`,
`target`, `facts`, `timing`, `attempts`, `usage`, and `failure`. Its target
contains selected IDs, optional `version`, selection source, endpoint, effective
settings, and settings-presence flags. Attempt entries retain their own facts,
timing, failure, and token-usage evidence. Durations use Go duration strings;
token counts preserve `value`, `known`, and `complete`.
Here is a native projection as rendered by the command:
```yaml
kind: reachability
supported: false
applicable: null
completed: true
reason: http_diagnostics_unavailable
target:
backend_id: ""
profile_id: spring
prompt_id: briefing
version: "1"
default_source_backend_id: ""
selection_source: profile
endpoint: https://example.invalid/v1
effective_settings:
endpoint: https://example.invalid/v1
model: spring-model
temperature: 0
max_tokens: 0
top_p: 0
timeout_seconds: 600
service_tier: ""
reasoning_effort: ""
api_key_env: ""
extra_params:
campaign: spring
settings_presence:
temperature: false
max_tokens: false
top_p: false
timeout_seconds: false
facts:
credentials_supplied: null
endpoint_reachable: null
credentials_accepted: null
response_received: null
generation_completed: null
text_returned: null
expected_text_matched: null
structured_output_requested: null
valid_json: null
schema_matched: null
timing:
started_at: 2026-10-17T19:05:36Z
duration: 35.685µs
capacity_wait: null
request_duration: null
retry_delay: null
evaluation_duration: null
attempts: []
usage:
prompt_tokens: {value: 0, known: true, complete: true}
completion_tokens: {value: 0, known: true, complete: true}
total_tokens: {value: 0, known: true, complete: true}
cached_tokens: {value: 0, known: true, complete: true}
cache_write_tokens: {value: 0, known: true, complete: true}
failure: ""
```
Aggregate output is one YAML mapping with ordered `tests` entries. Each entry
has selected `kind`, nullable `report`, and optional `error` text. The inner
report keeps its native `kind`. This cancellation example shows two attempted
operations with no reports; later kinds are absent because cancellation stops
the sequence:
```yaml
tests:
- kind: reachability
report: null
error: "reachability operation failed"
- kind: authentication
report: null
error: "context canceled"
```
A successful delivered single operation returns status `0`. For `all`, all
four callbacks must return without error for status `0`. Native false, unknown,
unsupported, or incomplete observations do not independently change the
outcome. Operation errors use operational status `1`; aggregate runs continue
after individual errors, retain their reports and ordered error text, and join
causes for supported `errors.Is`/`errors.As` access. Cancellation stops later
operations and preserves results already acquired. Report encoding or delivery
failures use the common incomplete-report delivery outcome and retain operation
causes where applicable. Syntax, operand, unsupported format, and conflicting
namespace selection are usage status `2`.
Native configuration, selection, and diagnostic meanings remain owned by
PromptKit. See the [v0.16.0 consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/docs/consumers/pkg-promptkit.md),
[smoke API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/smoke_diagnostics.go),
and [engine API](https://gitea.maximumdirect.net/eric/promptkit/src/commit/9178f3eb91c2b2138caa78e22be93c048b4bc992/engine.go).
## Outcome status