Files

Eric Rakestraw e653b7d6ec Document PromptKit catalogs and smoke testing

2026-10-17 19:15:58 +00:00

32 KiB

Raw Blame History

Command line reference

CommandKit owns command routing and syntax for commands registered with its App. The caller supplies argv without the executable name, a context, and the output and diagnostic writers. App.Execute returns an Outcome; it does not exit the process.

Help

The root help command lists registered commands and their summaries. Help is available through a bare invocation, --help, -h, and help:

example
example --help
example -h
example help
example help config show
example config --help

help [COMMAND...] resolves an exact registered path. A command group without an execution handler displays its help when invoked by itself. A runnable command with children runs its handler when invoked without help; add --help to inspect that command and its children.

Help uses registered descriptions, command summaries, local flags, defaults, required and repeatable flag metadata, and operand names and counts. It does not construct the selected command's input or invoke its handler. Supplied flag and operand values are still parsed so malformed values and unknown flags remain usage errors. Missing execution-required values and input-combination checks are skipped for help.

Version

version reports the application identity by default:

example version
example version --scope commandkit
example version --format yaml

The supported flags are:

Flag	Values	Default	Meaning
`--scope`	`app`, `commandkit`	`app`	Select the caller's application identity or the embedded CommandKit module identity.
`--format`	`text`, `yaml`	`text`	Select the output format.

Text output is NAME VERSION, followed by (REVISION) when the selected identity has a revision. An empty application version is displayed as dev; the revision remains empty when omitted.

CommandKit identity uses the version of its module in build information. A versioned module replacement supplies the effective version. A local or versionless replacement, an untagged module, or missing build information displays dev. The CommandKit identity name is CommandKit; dependency build metadata does not supply a revision.

YAML output uses these explicit fields, including an empty revision:

name: ExampleApp
version: dev
revision: ""

Registered command syntax

Options are parsed after the complete command path and may be interspersed with declared operands. Flags are local to the selected command. Use -- to end option parsing; all later tokens are operands.

Value flags accept --name value and --name=value. Boolean flags accept presence as true and explicit values such as --enabled=false. A singleton flag accepts repeated occurrences; the last successfully parsed occurrence supplies its value. Every supplied occurrence is converted, so an earlier malformed value remains an error. Repeatable flags preserve occurrence order, with each occurrence treated as one value, including strings containing commas.

Typed inputs keep supplied state separate from defaults. An explicit empty, false, or zero value is supplied. Help and version are available without constructing application command inputs.

Configuration scopes

An application installs the declaration returned by ckconfig.Command(ckconfig.Suite[Input, State]{...}) through commandkit.New or App.Register. For the common single application scope, ckconfig.AppCommand returns a convenience declaration. The installed config scopes command lists scope names and the configuration operations currently available for those scopes:

example config scopes
example config scopes --format yaml

Text output uses one scope: operation, operation line per scope. YAML uses a scopes sequence with explicit name and operations fields. Scope order is app, then promptkit, then other names alphabetically. A metadata-only scope can be listed without an acquisition callback. Discovery does not construct a runtime context or acquire configuration. It reports the intersection between declared scope capabilities and installed handlers, so an operation is listed only when the corresponding command is available.

Applications can add ckprompt.ConfigScope(loadEngine) to the ConfigKit suite to expose PromptKit engine configuration through these handlers. The scope declares validate, show, sources, explain, and native diff. It does not install PromptKit prompt or profile definition commands.

Validate configuration

config validate loads and validates the selected scope. The default scope is app; --scope all validates every registered scope and is available when each scope declares validation. The command checks all selected capabilities before calling scope providers. Its default format is text; --format yaml emits a scopes sequence with name and valid: true for each scope that completed successfully.

example config validate
example config validate --scope promptkit --format yaml
example config validate --scope all

Each successful scope is reported as SCOPE: valid. Aggregate validation stops at the first load or output failure; records from earlier successful scopes may already have been written. Provider warnings go to the diagnostic stream and do not alone make validation fail. --config and --overlay preserve whether an empty value was explicitly supplied, so a scope callback can map those inputs to its own ConfigKit load policy. Help and invalid syntax do not call scope providers.

Show effective configuration

config show writes the provider's native effective YAML for one scope. Its default scope is app; --format yaml is accepted and YAML is the only initial format. Single-scope output is passed through unchanged, without a CommandKit header or document marker.

example config show
example config show --scope promptkit
example config show --scope all

--scope all acquires every selected scope and checks each effective writer before writing output. It then emits one YAML document per scope in discovery order, with a # scope: NAME comment at each document boundary. For ordinary block documents the comment follows the marker. When a native opening marker contains inline content, the comment precedes that marker so it does not interrupt the native value. Existing native markers and leading comments are retained; documents without a marker receive one. Scope labels containing line control characters are quoted to keep them on one comment line. A provider failure leaves the output stream empty. A rendering or output failure may leave an aggregate prefix, and the outcome reports incomplete report delivery.

List configuration sources

config sources reports the source inventory captured during the selected scope's acquisition. It reads the source facet once and does not reload configuration. The default scope is app; --scope all is unsupported. Text is the default format, and --format yaml returns a structured report:

scope: app
sources:
  - id: source-1
    label: "application config"
    path: /etc/example/config.yml
    layer: base
    kind: file
    underlay_index: 0
    imported_from: null
    destination: /server
    placement: mapping
    destination_unresolved: false
    version_check:
      path: /version
      version:
        present: true
        null: false
        text: "1"
      location:
        source_id: source-1
        line: 2
        column: 1
      redacted: false

Each source includes id, label, path, layer, kind, underlay_index, and imported_from, plus the native destination pointer, placement, unresolved destination flag, and nullable version_check. Placement is mapping or sequence_item. The import location is null when absent; otherwise it has source_id, line, and column. Version evidence preserves path, present, null, text, optional source location, and redacted as returned by ConfigKit. Owning values and source order are preserved, including empty fields, zero coordinates, false and null observations, and synthetic sources without paths. An absent version check is null. An empty inventory is sources: []. Text output begins with the scope heading and prints the same source fields, quoting labels and paths. These fields project the selected ConfigKit source records; CommandKit does not reload files or rerun version checks. See the ConfigKit v0.8.0 inspection API for native evidence definitions.

Explain a configuration path

config explain --path POINTER explains the effective and historical value at a JSON Pointer path. --path is required; an explicitly empty value selects the root. The default scope is app; --format yaml selects structured output, and text is the default. --scope all is unsupported.

example config explain --path /server/addr
example config explain --path /profiles/team~1summary --format yaml
example config explain --path= --format yaml

Pointer tokens use JSON Pointer escaping: ~1 represents / and ~0 represents ~. A malformed pointer is a usage error with status 2 and is rejected before acquisition. A well-formed path that is unknown in the selected configuration is an operational error with status 1. Text output comes from the scope's native explanation writer.

YAML output uses these root fields:

scope: app
path: /server/addr
effective:
  present: true
  redacted: false
  yaml: "localhost:8080\n"
origin:
  layer: base
  locations:
    - source_id: source-1
      line: 4
      column: 9
  operation: ""
  inheritance: []
history: []
secret_reference: null

Snapshot yaml values remain strings, including null and empty fragments. Origin and history retain source locations, operation names, inheritance paths, ordered reasons and dependencies. Secret references contain only the owning environment-variable name; effective and historical secret fragments remain redacted. Empty location, inheritance, history, reason, and dependency lists are serialized as [].

Compare configuration selections

config diff prepares two selections for one scope that declares comparison and invokes its comparison provider after input checks pass. The default scope is app; --scope all is unsupported. When an application installs ckprompt.ConfigScope(loadEngine), the PromptKit scope declares comparison and delegates it to PromptKit's captured configuration inspection API. PromptKit owns the meaning of that native comparison; CommandKit uses the existing config diff preparation and report path. --config and --overlay select the left side. --against-config and --against-overlay independently override the right side. When a right-side field is omitted, it inherits the corresponding left value and supplied state; an explicit empty string overrides it and remains supplied. Equal selections are allowed; their equality alone does not cause CommandKit to skip the provider.

YAML is the only format and the default. Explicit --format yaml is accepted; text and other formats are usage errors. For example:

example config diff --config before.yml --against-config after.yml
example config diff --config base.yml --against-overlay candidate
example config diff --config base.yml --overlay current --against-overlay ""
example config diff --scope promptkit --config before.yml --against-config after.yml

For ConfigKit-backed scopes, ConfigKit compares frozen effective typed values after defaults and resolution; source or provenance differences alone do not create a change. PromptKit delegates its native comparison to ConfigKit. The report is one YAML document with additions, changes, and deletions in that order. Additions and changes preserve semantic paths and contain right-side values provided by the native comparison. Deletions contain canonical JSON Pointer paths. Explicit null is a value; it is not a deletion. Changed sequences and native structural replacement regions stay atomic.

ConfigKit pairs sensitivity across both loads. A concealed change can therefore appear even when its two safe snapshots are equal. For those ConfigKit-backed comparisons, the report omits separate redaction and hidden-change metadata: a reported masked region changed, but that does not mean every member inside it changed. Marker-looking public strings stay ordinary YAML values. An empty comparison succeeds with this shape:

additions: {}
changes: {}
deletions: []

The reduced report is a partial safe view, not a reloadable configuration or an automatically applicable patch. Exceptional whole-root empty/null transitions can be ambiguous because the report has no root envelope; ordinary field-level null values and deletions retain their distinction.

Successful comparisons return status 0, including empty results. Unsupported scopes and format errors are usage errors; input checks retain the existing usage/operational classification. Load and native comparison failures are operational and preserve supported error identity.

The bundled FromConfigKit and AppCommand adapters invoke the supplied loader for each reached side, left before right, then call native comparison. They label native diagnostics with their left/right side and source legend, retain left diagnostics if right loading fails, and gather both loads' diagnostics before native comparison. Custom pair providers own their loading and diagnostics; application loaders and caches determine whether a loader call reads sources again.

The complete report is prepared before writing. Report preparation failures and cancellation observed before report delivery starts leave CommandKit's report output empty. Once writing starts, CommandKit does not interrupt the supplied writer or roll back delivered bytes. A write error or short write uses the existing incomplete-report delivery outcome and can leave partial output.

PromptKit definition inspection

Applications can install prompt and profile inspection independently with ckprompt.Command(suite). The ckprompt.FromEngine(loadEngine) adapter supplies these callbacks alongside native requirements discovery and the five catalogs. Its loader receives invocation context, configuration selection, typed input, and shared state. Direct suite callbacks retain their own implementation and lifecycle.

Prompt inspection uses an ID only. PromptKit returns any descriptive version metadata as an optional report field; there is no --version flag for selecting a prompt version. The prompt format and native selection behavior remain owned by PromptKit v0.16.0.

example promptkit show prompt customer/summary --format yaml
example promptkit show prompt customer/summary
example promptkit show profile customer-support --format yaml

Both commands require one ID operand and accept --config, --overlay, and --format text|yaml (default text). These commands have no built-in --scope selection; their namespace is promptkit.

Prompt YAML preserves native input order and uses an optional version field. The key is omitted when PromptKit returns no version and is retained when the version is present, including an empty string. The prompt hash is an opaque owning-library equality value.

prompt_id: customer/summary
version: "2"
prompt_hash: opaque-value
default_profile_id: customer-support
inputs:
  - name: topic
    required: true
    content_type: text/plain
    description: Topic to summarize
output_contract:
  format: json
  validation_mode: basic
  schema_path: ""
  repair_attempts: 2

Profile YAML preserves the effective values reported by PromptKit, including zero values and nested provider parameters. api_key_env is only an environment-variable name. api_key_required reports a direct credential requirement; neither field reports credential availability. A nil provider parameter map is displayed as {}.

profile_id: customer-support
effective_model_params:
  backend_id: local
  endpoint: https://example.invalid/v1
  model: support-model
  temperature: 0
  max_tokens: 0
  top_p: 0
  timeout_seconds: 600
  service_tier: ""
  reasoning_effort: ""
  api_key_env: ""
  extra_params: {}
api_key_required: true

Text uses labeled fields and includes nested extra_params values. A missing ID is a usage error. Definition selection reaches the owning inspection API and preserves supported error identity and cancellation.

PromptKit requirements

When Suite.InspectRequirements is present, ckprompt.Command(suite) installs promptkit requirements show. It accepts one prompt ID and uses PromptKit's native selection and discovery rules:

example promptkit requirements show briefing --campaign spring
example promptkit requirements show briefing --profile editorial --var mode=full
example promptkit requirements show briefing --session-id session-42 --var note= --format yaml

The command accepts:

Flag	Values	Default	Meaning
`--config`	path string	omitted	Select the configuration input; explicit presence is retained.
`--overlay`	path string	omitted	Select one overlay; explicit presence is retained. Repeated occurrences use the last value.
Suite selectors	application-defined	per selector	The flags captured from `Suite.Selectors`.
`--profile`	string	empty	Pass the profile ID to PromptKit; empty uses its native selection rule.
`--session-id`	string	empty	Pass session context to PromptKit.
`--var`	repeatable `KEY=VALUE`	none	Supply caller-variable context for native discovery.
`--format`	`yaml`	`yaml`	Select the requirements report format. Other formats are usage errors for this operation.

The promptkit namespace identifies the operation, so there is no built-in --scope flag. An explicitly empty profile or session value retains PromptKit's native empty-string meaning. Requirements inspection does not require a profile to be execution-ready. See the PromptKit v0.16.0 consumer guide for native configuration, selection, and requirements behavior. The built-in flags other than --var are singleton bindings; when repeated, the last value is used. Suite selectors retain their declared binding behavior.

Each --var occurrence must contain =. CommandKit splits at the first separator and preserves the exact key and remaining value. Empty keys and values, whitespace, commas, and additional = characters are retained. All occurrences are parsed; duplicate keys use the last value, while any malformed occurrence is a usage error even if a later assignment is valid.

The YAML report contains selected identities, native completeness, and input, variable, and issue inventories. False values and empty inventories are retained. Input metadata preserves null versus an empty string. Variable entries contain name, kind, supplied, and explanation. supplied reports native exact-key presence, including an explicitly empty value, independently of kind and complete. Explanations and issue descriptions are upstream-owned human-readable summaries, not machine-readable details. The optional version key is omitted when absent and retained when supplied by PromptKit.

For a prompt with the template {{if .mode}}{{input "notes"}}{{end}} and no input declarations, omitting mode can produce a partial report:

prompt_id: briefing
version: "2"
profile_id: editorial
complete: false
inputs:
  - name: notes
    kind: conditional
    declared: false
    content_type: null
    description: null
    explanation: This reference is not known to be reached on every execution path.
variables:
  - name: mode
    kind: required
    supplied: false
    explanation: ""
issues:
  - code: missing_variable
    input_name: ""
    variable_name: mode
    description: A missing variable prevents selecting a dependency-bearing branch.

PromptKit owns the inventory, classifications, explanations, completeness, and structured issues. Successfully acquired and delivered partial reports return status 0; structured issues do not independently fail the command. Provider errors, native selection errors, cancellation, and report delivery failures use the operational path (status 1) with supported cause identity. Syntax, operand, variable-assignment, and format errors use usage status 2.

Concrete request validation

When Suite.ValidateRequest is present, ckprompt.Command(suite) also installs promptkit requirements validate. It can be installed without discovery, and discovery can remain installed without validation. The command accepts one prompt ID and passes a concrete request to PromptKit's native Engine.ValidateRequest preparation path:

example promptkit requirements validate briefing --campaign spring --input topic=topic.txt
example promptkit requirements validate briefing --profile editorial --session-id session-42 --var mode=full --input notes=notes.txt

The command accepts:

Flag	Values	Default	Meaning
`--config`	path string	omitted	Select the configuration input; explicit presence is retained.
`--overlay`	path string	omitted	Select one overlay; explicit presence is retained. Repeated occurrences use the last value.
Suite selectors	application-defined	per selector	The flags captured from `Suite.Selectors`.
`--profile`	string	empty	Pass the profile ID to PromptKit.
`--session-id`	string	empty	Pass session context to PromptKit.
`--var`	repeatable `KEY=VALUE`	none	Supply caller variables for the concrete request.
`--input`	repeatable `NAME=PATH`	none	Bind a file-backed PromptKit artifact.
`--format`	`yaml`	`yaml`	Select the success confirmation format. Other formats are usage errors.

The built-in flags other than repeatable --var and --input are singleton bindings; when repeated, the last value is used. Explicit empty --config or --overlay values keep their supplied-presence information. The profile and session fields use native empty-string semantics. PromptKit owns their interpretation and selection errors.

Each --var and --input occurrence must contain =. CommandKit splits at the first separator and preserves the exact key/name and remaining value/path. Empty variable keys and values and empty input names and paths are passed to native validation. File paths become promptkit.File(PATH) references without being read, normalized, expanded, or rebased during argument binding. Duplicate keys or names use the last value, but every occurrence is parsed.

On success, the YAML confirmation contains only the requested prompt ID and valid: true:

prompt_id: briefing
valid: true

Validation delegates concrete request preparation to PromptKit. Native preparation may resolve prompt/profile resources, load artifact and schema resources, render messages and session context, and perform applicable configuration and credential checks. It does not make a model call or retain a prepared execution snapshot. A successful report describes this preparation invocation; it does not certify provider acceptance or guarantee later execution. See PromptKit's Engine.ValidateRequest API.

Builder, engine-loading, and native validation errors use the operational path (status 1) with supported error identity and acquired diagnostics retained. Malformed syntax, missing operands, and unsupported formats use usage status 2. Successful diagnostics and report-delivery failures follow the common delivery contract.

PromptKit catalogs

An installed ckprompt.Command(suite) exposes each catalog whose callback is present:

example promptkit list prompts
example promptkit list profiles --config settings.yml --campaign spring
example promptkit list backends --namespace team
example promptkit list namespaces --kind prompt --kind profile
example promptkit list collections

All catalogs accept --config, --overlay, applicable flags from Suite.Selectors, and --format yaml (the default). Other formats are usage errors. Prompt, profile, and backend listings accept namespace selection:

Binding	Effect
omitted `--namespace` and `--all-namespaces`	Forward PromptKit's native default namespace filter.
`--namespace team`	Forward the named namespace filter.
`--namespace ""`	Select the unqualified namespace.
`--all-namespaces`	Forward the all-namespaces filter.
`--all-namespaces=false`	Do not select an override; a supplied namespace still applies, otherwise native default is used.

The namespace binding is a singleton and its last occurrence is used. Supplying a namespace while --all-namespaces=true is active is a usage error before provider work. Namespace inventory accepts repeatable --kind KIND; omitting --kind forwards PromptKit's native default kind selection. Repeated values are forwarded in order. Collections have no namespace or kind filter.

Each operation writes one bare YAML sequence, retaining provider order and native identities:

Prompt entries are {id, version?}. The optional version key is omitted when absent and retained for a present empty or nonempty value.
Profiles, backends, and namespaces are strings.
Collection entries are {namespace, version?}, where version projects PromptKit's optional release version.
Nil and allocated-empty inventories both render as [].

- id: team/briefing
  version: "2"

- autumn
- spring

A catalog result is an inventory, not a readiness check or execution result. The owning provider controls membership and order. Namespace filters are passed to PromptKit without local membership interpretation. See the PromptKit v0.16.0 consumer guide for native resource and selection definitions.

PromptKit smoke testing

When Suite.SmokeTest is present, ckprompt.Command(suite) installs promptkit test KIND. The special kind all runs the four concrete operations sequentially in this order: reachability, authentication, text_inference, and structured_output. Both forms use the same single-operation callback; each aggregate call changes the seed kind before invoking it. The command accepts:

example promptkit test text_inference --campaign spring --prompt briefing --profile spring
example promptkit test all --campaign spring --prompt briefing --profile spring
example promptkit test authentication --backend local

Flag	Values	Default	Meaning
`--config`, `--overlay`	path strings	omitted	Select PromptKit configuration; supplied presence is retained.
Suite selectors	application-defined	per selector	Captured flags from `Suite.Selectors`.
`--backend`	string	empty	Native backend ID.
`--prompt`	string	empty	Native prompt ID.
`--profile`	string	empty	Native profile ID.
`--endpoint`	string	omitted	Override the execution endpoint.
`--model`	string	omitted	Override the execution model.
`--temperature`	float	omitted	Override execution temperature.
`--max-tokens`	integer	omitted	Override execution maximum tokens.
`--top-p`	float	omitted	Override execution top-p.
`--timeout-seconds`	integer	omitted	Override the execution timeout.
`--service-tier`	string	omitted	Override the execution service tier.
`--reasoning-effort`	string	omitted	Override execution reasoning effort.
`--api-key-env`	string	omitted	Supply an API-key environment-variable name to native configuration.
`--transient-retries`	integer	omitted	Set the native inference transient retry count.
`--format`	`yaml`	`yaml`	Select the report format. Other formats are usage errors.

The execution flags populate PromptKit's native execution override. The --api-key-env value is a variable name, not a credential value. Application selectors, loader policy, credentials, and client wiring remain application owned. The WithSmokeTesting helper accepts an optional typed builder. A nil builder forwards the seed; a supplied builder's complete native request is authoritative. The helper retains a native report when PromptKit returns both a report and an error.

A single report projects the native smoke report into YAML. It preserves native false, unknown/null, and incomplete values and does not add a readiness score. The report includes kind, supported, applicable, completed, reason, target, facts, timing, attempts, usage, and failure. Its target contains selected IDs, optional version, selection source, endpoint, effective settings, and settings-presence flags. Attempt entries retain their own facts, timing, failure, and token-usage evidence. Durations use Go duration strings; token counts preserve value, known, and complete.

Here is a native projection as rendered by the command:

kind: reachability
supported: false
applicable: null
completed: true
reason: http_diagnostics_unavailable
target:
  backend_id: ""
  profile_id: spring
  prompt_id: briefing
  version: "1"
  default_source_backend_id: ""
  selection_source: profile
  endpoint: https://example.invalid/v1
  effective_settings:
    endpoint: https://example.invalid/v1
    model: spring-model
    temperature: 0
    max_tokens: 0
    top_p: 0
    timeout_seconds: 600
    service_tier: ""
    reasoning_effort: ""
    api_key_env: ""
    extra_params:
      campaign: spring
  settings_presence:
    temperature: false
    max_tokens: false
    top_p: false
    timeout_seconds: false
facts:
  credentials_supplied: null
  endpoint_reachable: null
  credentials_accepted: null
  response_received: null
  generation_completed: null
  text_returned: null
  expected_text_matched: null
  structured_output_requested: null
  valid_json: null
  schema_matched: null
timing:
  started_at: 2026-10-17T19:05:36Z
  duration: 35.685µs
  capacity_wait: null
  request_duration: null
  retry_delay: null
  evaluation_duration: null
attempts: []
usage:
  prompt_tokens: {value: 0, known: true, complete: true}
  completion_tokens: {value: 0, known: true, complete: true}
  total_tokens: {value: 0, known: true, complete: true}
  cached_tokens: {value: 0, known: true, complete: true}
  cache_write_tokens: {value: 0, known: true, complete: true}
failure: ""

Aggregate output is one YAML mapping with ordered tests entries. Each entry has selected kind, nullable report, and optional error text. The inner report keeps its native kind. This cancellation example shows two attempted operations with no reports; later kinds are absent because cancellation stops the sequence:

tests:
  - kind: reachability
    report: null
    error: "reachability operation failed"
  - kind: authentication
    report: null
    error: "context canceled"

A successful delivered single operation returns status 0. For all, all four callbacks must return without error for status 0. Native false, unknown, unsupported, or incomplete observations do not independently change the outcome. Operation errors use operational status 1; aggregate runs continue after individual errors, retain their reports and ordered error text, and join causes for supported errors.Is/errors.As access. Cancellation stops later operations and preserves results already acquired. Report encoding or delivery failures use the common incomplete-report delivery outcome and retain operation causes where applicable. Syntax, operand, unsupported format, and conflicting namespace selection are usage status 2.

Native configuration, selection, and diagnostic meanings remain owned by PromptKit. See the v0.16.0 consumer guide, smoke API, and engine API.

Outcome status

CommandKit usage failures use status 2; operational failures use status 1. Successful operations use status 0. Application handlers may return a domain classification and status of their choice, which Execute preserves. The caller chooses how to map an outcome to process exit.

32 KiB Raw Blame History