base-layer/commandkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Plan PromptKit request validation adoption

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-12 21:37:43 +00:00
parent 700f10d78e
commit 87391903f4
1 changed files with 322 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

322
docs/roadmap/promptkit-validation.md Normal file
View File

@@ -0,0 +1,322 @@
# PromptKit concrete request validation
Status: proposed adoption and feature roadmap. The command placement is agreed;
the recommendations under Open Decisions await review. This document defines
the target and required work, rather than a staged implementation plan.
## Purpose
Adopt PromptKit v0.15.0 and expose its concrete request validation capability as
`app promptkit requirements validate PROMPT_ID`, alongside the delivered
`app promptkit requirements show PROMPT_ID` command.
The organizing principle is **CommandKit provides useful, consistent access to
capabilities owned elsewhere**. CommandKit owns vocabulary, registration,
selection, coordination, presentation, and invocation outcomes. PromptKit owns
what concrete request validation means. Applications supply their domain
mapping, defaults, artifacts, and runtime dependencies.
The command answers whether the constructed request passes native preparation
at the time of the invocation. It complements discovery without making
discovery a prerequisite for validation or validation a prerequisite for
application execution.
## Current baseline
The baseline is CommandKit commit `700f10d78ed0e410b0cc1d754fdd98415cd5817a`.
- The optional [ckprompt module](../../integrations/ckprompt/go.mod) requires
PromptKit v0.14.0, ConfigKit v0.7.1, and Go 1.26.0.
- Its [suite and engine adapters](../../integrations/ckprompt/adapters.go)
support independent prompt/profile catalogs, named definition inspection,
and requirements discovery. `FromEngine` supplies those callbacks using an
application-owned engine loader. There is no concrete validation callback.
- The [requirements command](../cli.md#promptkit-requirements) accepts a prompt
ID, configuration selection, typed suite selectors, version/profile/session
context, and repeatable variable assignments. It returns the native discovery
evidence as YAML. A partial inventory or missing variable is not itself an
invocation failure.
- `ConfigScope` separately exposes engine configuration inspection and
comparison through ckconfig. It does not install the PromptKit command family.
- The [composed consumer](../../integrations/ckprompt/examples/composed/main.go)
demonstrates discovery and application-owned configuration/engine loading.
It does not yet demonstrate construction or validation of a concrete request.
PromptKit v0.15.0 adds
[`Engine.ValidateRequest`](https://gitea.maximumdirect.net/eric/promptkit/src/commit/635674c066d0b432187f96e72b9c03d085443ac5/engine.go)
for an ordinary public `RunRequest`. Its
[consumer guide](https://gitea.maximumdirect.net/eric/promptkit/src/commit/635674c066d0b432187f96e72b9c03d085443ac5/docs/consumers/pkg-promptkit.md#validate-a-concrete-request)
and [release note](https://gitea.maximumdirect.net/eric/promptkit/src/commit/635674c066d0b432187f96e72b9c03d085443ac5/docs/releases/v0.15.0.md)
describe the addition. The reviewed release leaves the upstream minimum Go
version, module dependencies, and licensing material unchanged from v0.14.0.
## Targeted end state
### Command placement and meaning
The standard PromptKit family can expose both operations independently:
| Command | Operator purpose | Owning native capability |
| --- | --- | --- |
| `app promptkit requirements show PROMPT_ID` | Discover input and variable requirements for a prompt context. | `Engine.InspectRequirements` |
| `app promptkit requirements validate PROMPT_ID` | Check the actual constructed request, including supplied values and artifact bindings. | `Engine.ValidateRequest` |
The new handler delegates to concrete native validation. It does not infer
validity from the discovery report's `Complete`, inventory, or issues, and does
not independently interpret templates or decide which inputs are required.
Native validation shares `Prepare`'s request conversion and preparation path.
It may resolve prompt/profile resources, load artifacts and schema resources,
render messages and session context, and perform the applicable configuration
and credential checks. All supplied inputs follow native preparation behavior,
including inputs unused by the selected template branch. Errors are returned
as encountered rather than as an exhaustive inventory.
There is no model call or retained prepared snapshot. Success neither certifies
provider acceptance nor guarantees that a later execution will succeed. A later
run prepares independently. Applications needing execution tied to a checked
snapshot can use PromptKit's existing prepared-execution API in their own
workflow; that workflow is outside this feature.
### Operator inputs
Use the existing requirements command's common syntax for:
- One required prompt ID operand.
- `--config` and `--overlay`, with existing explicit-presence handling.
- Typed suite selectors.
- `--version`, `--profile`, and `--session-id`, forwarding their native
empty-string semantics.
- Repeatable `--var KEY=VALUE`, preserving the existing first-separator split,
exact keys and values, and last assignment for a duplicate key.
Keep singleton repetition and selector binding behavior consistent with the
current CLI contract. PromptKit owns semantic interpretation of the values.
The namespace identifies the library; configuration scope selection belongs
to the separate `config` family.
The artifact-binding syntax and success presentation are proposed in OD-2 and
OD-3. Application-specific settings continue to use typed application input
and callbacks. The command need not offer a built-in flag for every field in
`RunRequest` to permit applications to use those fields through their builder.
### Public integration and request construction
Add an independently optional concrete validation capability to `ckprompt.Suite`.
Installing validation alone must be useful; supplying discovery alone must
continue to install only discovery. Both operations belong to the same
`requirements` group when present. The existing installed-family ownership,
absent-suite behavior, and retained wiring errors apply.
Provide an engine-backed adapter that combines an application engine loader
with an application-owned concrete request builder. The builder receives the
parsed standard request context and bindings, selected application input, and
invocation state, and produces a native `promptkit.RunRequest`. It can supply
application artifacts, inline content, execution/output-contract overrides,
appended messages, credentials, or other supported native settings without
CommandKit defining parallel domain abstractions.
The adapter should make it straightforward to honor the selected prompt and
operator values while adding application defaults. Document how standard
values reach the builder; do not hide a second merging layer inside the adapter
or introduce independent native-field validation. Reusing a request builder
between validation and application execution is recommended consumer guidance,
not a requirement governing downstream application internals.
The public API shape and wiring recommendation are in OD-1. Exact declarations
will be finalized before preparing the staged implementation plan.
### Invocation lifecycle and outcomes
Reuse the integration's current typed input preparation, selector application,
input checking, invocation state, configuration diagnostic adaptation, and
outcome/delivery machinery. Construction, installation, help, and version must
leave acquisition and request-building callbacks uninvoked, under the existing
[architecture policy](../policy/architecture.md).
Complete applicable CLI preflight and application input checks before shared
state creation and acquisition. Pass the invocation context through request
construction, engine loading, and native validation. Preserve supported native
error identity and configuration diagnostics, including failures during engine
loading. Request building must be able to fail without a fabricated validation
success result.
Use the existing classification model: malformed CLI syntax and missing
operands are usage failures; engine loading and native request validation
failures are operational failures. Application input and builder errors retain
the supported classification behavior of CommandKit's error APIs. Preserve
native cancellation error categories; do not add a promise that every native
failure matches `ctx.Err()`.
Successful native validation produces the selected success presentation only
after it succeeds. Diagnostics and output failures follow the established
delivery contract. The presentation reports the validation outcome without
dumping the request, artifact content, credential values, or rendered messages.
### Package and dependency ownership
All new provider contracts, request adaptation, standard handlers, and
presentation belong in `integrations/ckprompt`. Core continues to own generic
declarations, bindings, dispatch, and outcomes. ckconfig remains responsible
for its configuration adaptation; ckprompt can reuse it where appropriate.
Adopt the released PromptKit v0.15.0 module. Preserve the current Go minimum
and dependency direction: ckprompt may depend on ckconfig; ckconfig and core
do not acquire a PromptKit dependency. Use public upstream APIs and released
module versions, without sibling-checkout requirements in committed manifests.
## Work needed from the baseline
### Dependency adoption
Update ckprompt's PromptKit requirement and checksums to v0.15.0. Review the
resulting selected module graph, and update the PromptKit attribution entry's
selected version and source/notice links to release commit
`635674c066d0b432187f96e72b9c03d085443ac5`. Update other attribution entries only
where the selected graph or incorporated material actually changes.
### Integration API and native adapter
Resolve OD-1, then add the optional validation callback and engine-backed
adapter. Reuse existing engine-loading diagnostics and supported error handling.
Support concrete request construction from typed application input and standard
CLI values, without copying PromptKit's preparation logic. Account for builder
failure and nil/invalid wiring using the established integration conventions.
Keep current discovery and configuration adapters available independently.
`FromEngine` alone cannot invent application artifact bindings or concrete
request defaults; enabling validation must be an explicit wiring choice.
### Command assembly, bindings, and presentation
Extend `ckprompt.Command` to assemble the requirements group from its supplied
show and validate capabilities. Add the validate handler and the selected
OD-2/OD-3 bindings and presentation. Share common binding/report mechanics where
that simplifies the code, preserving the delivered discovery contract.
### Maintained consumer and documentation
Extend the composed consumer with a synthetic concrete request builder and
offline validation examples for a successful request and a missing required
binding. Demonstrate the existing typed domain selectors alongside the new
standard bindings. The example should make the different questions answered
by show and validate clear without requiring a model service or real credential.
When implemented, update Go declarations/GoDoc, the CLI reference, consumer
guide, and affected orientation/internal descriptions under their canonical
owners. Describe native resource work and validation limits accurately, so
statements about metadata inspection do not accidentally claim that concrete
validation performs no rendering, artifact loading, or credential checks.
This capability fits the accepted optional-family and owning-library boundaries.
An API or presentation addition alone does not require rewriting accepted ADRs.
Reassess ADR/policy alignment if resolving an open decision changes an
architectural boundary rather than simply adding the capability within it.
### Completion evidence for implementation
Use proportionate public-boundary evidence for independent capability
availability, help without acquisition, standard/typed input mapping, fresh
invocation state, builder and loader failures, native validation error identity,
and diagnostic/output delivery. Use a real PromptKit engine for representative
successful validation and a missing binding, with a model-call double that
detects accidental execution. Protect continued successful discovery of a
partial inventory; do not turn it into a validation failure.
Check semantic output rather than YAML key ordering or native error wording.
Avoid duplicating PromptKit's complete preparation test suite. Validate the
owning integration and affected composed consumer; check isolated module
resolution and dependency boundaries for the upgrade, using the established
workspace and disposable-module practices in the
[developer guide](../development.md). Implementation evidence must not rely
solely on a workspace that silently substitutes a sibling PromptKit checkout.
## Open Decisions
### OD-1: Validation provider and engine-adapter API
**Recommendation:** Add a separate `Suite.ValidateRequest` callback receiving a
CommandKit request envelope containing configuration selection and a native
`RunRequest` populated from standard command inputs, plus `I` and `S` as with
existing callbacks. Return diagnostics and error through
`commandkit.Acquired[struct{}]` and `error`; validation has no native value to
invent. Provide a separate adapter factory taking the existing engine-loader
shape and a typed request-builder callback, whose result can be assigned to
`Suite.ValidateRequest`. The builder transforms the command's seed request
into the concrete request passed to `Engine.ValidateRequest`; the adapter does
not reapply or merge fields afterward.
This keeps capability opt-in explicit, reuses diagnostics machinery, and lets
applications share concrete request construction without requiring them to
write engine acquisition/error adaptation again. The builder's application
defaults must respect the selected prompt and native empty-value semantics;
document that responsibility rather than adding duplicate domain checks.
The factory's final name and declarations should be settled with a small
consumer mockup before implementation planning.
**Alternative:** Provide only the suite callback, and let applications load
the engine, construct the request, invoke native validation, and return
diagnostics themselves. This has the smallest public adapter surface and
supports unusual runtime arrangements. It leaves repeated engine-loading and
diagnostic adaptation in consumers, weakening the infrastructure-reduction
goal. Custom callbacks remain useful even if the recommended helper exists.
**Alternative:** Extend `FromEngine` with an explicit validation option carrying
the builder. This makes one constructor supply all engine-backed capabilities.
It adds a constructor option and makes validation availability conditional on
an extra argument, whereas a separate helper expresses the independent
capability directly. It is viable if the consumer mockup shows materially
simpler wiring.
### OD-2: Standard artifact-binding inputs
**Recommendation:** Add repeatable `--input NAME=PATH` for file-backed bindings,
translated to native `promptkit.File(PATH)` references without eagerly reading
the files. Reuse first-separator assignment parsing, preserve exact names and
paths, and use the last binding for a duplicate name. Let native validation
decide whether the resulting references are valid and loadable. Applications
can use their typed input and builder to supply inline or domain-derived
artifacts. No independent input-name whitelist or path restrictions are needed.
This supplies a useful common CLI workflow with simple semantics, while leaving
PromptKit responsible for artifact preparation. It avoids defining a new
serialization format for every native request field. Syntax missing `=` is a
usage error; semantic artifact failures belong to native validation.
**Alternative:** Also add `--input-inline NAME=CONTENT`. This gives all consumers
a direct inline workflow without an application flag. It adds another binding
source and requires an explicit duplicate-name precedence rule across the two
flags; it is justified if representative consumers need that common syntax.
Selecting it would require settling that precedence before implementation.
**Alternative:** Provide no standard artifact flag and accept bindings only
through application selectors and the builder. This fits applications whose
artifacts come entirely from domain state, but makes an ordinary file-validation
workflow less consistent and recreates avoidable CLI plumbing in consumers.
### OD-3: Successful validation presentation
**Recommendation:** Emit a small YAML report with `prompt_id` and `valid: true`,
using YAML as the default and only supported `--format` value for this operation.
The ID identifies the command's requested prompt; `valid` means native
preparation returned nil. Do not claim a resolved version/profile or return
discovery inventories: native validation supplies neither a report nor a
prepared snapshot. Failures use the existing diagnostic/error outcome path,
not a newly invented aggregate validation report.
This complements requirements show's YAML output and gives operators and
automation an explicit success result. Its limited meaning must be clear in
the CLI reference, especially when an application builder supplies additional
context. Output delivery becomes part of invocation success, as for other
CommandKit reports.
**Alternative:** Produce no success output and rely on exit status and
diagnostics. This closely follows the native error-only API, needs no output
schema or format flag, and is suitable for scripting. Interactive operators
receive less confirmation, and it differs from the existing configuration
validation reporting convention.
**Alternative:** Emit a short text confirmation, with optional YAML. This
matches `config validate`'s text-oriented default and can be easier to scan.
It introduces two presentations where the requirements command's current
YAML convention may already satisfy both interactive and automated use.