diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..2073c69 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,448 @@ +# PromptKit request validation implementation plan + +Status: ready for implementation; no implementation stages have been completed. + +## Authority, scope, and baseline + +Follow the stages below in order. Each stage is sized for one Luna implementation +prompt and identifies its changes, evidence, and completion condition. The +[accepted feature roadmap](promptkit-validation.md) owns the target state and +user choices. The [architecture policy](../policy/architecture.md), accepted +[ADRs](../adr/), and existing public contracts remain authoritative. + +The planning baseline is commit `1119bda6b720171c708f3e6d03c5c8690b65ea81`. +The implementation still requires PromptKit v0.14.0; the accepted roadmap adds +v0.15.0 adoption and `promptkit requirements validate PROMPT_ID`. + +The deliverable is the complete accepted feature, including the independently +optional callback, suite-returning helper, nil-builder path, file bindings, +YAML confirmation, maintained consumer, and canonical documentation. Existing +discovery, catalogs, definitions, configuration operations, and application +commands keep their current contracts. + +## Working instructions + +- Start with [development guidance](../development.md) and applicable policies. + Prefer the repository knowledge graph for code discovery; verify snippets + against current source when graph evidence is incomplete or stale. +- Implement in `integrations/ckprompt`, using public core, ckconfig, ConfigKit, + and PromptKit APIs. Core and ckconfig need no new public API or PromptKit + dependency for this feature. +- Use `Engine.ValidateRequest` directly. Discovery completeness is not a + validation gate. Avoid parallel native validation rules, identity enforcement, + request remerging, automatic execution, or prepared-run retention. +- Preserve caller ownership of callback objects, input, and state. Omitted + factories supply zero values, including nil pointers. No arbitrary deep + copies, automatic pointer repair, or concurrent-dispatch guarantee is added. +- Keep machinery proportionate. Private file and helper names below identify + a concrete implementation route; equivalent simple organization within the + owning package is acceptable. Public names and signatures are settled. +- Apply `gofmt` to changed Go files. Add focused tests alongside the stage that + owns each behavior; reuse existing helpers where useful without duplicating + PromptKit's suite. Current tests are evidence, not a source of new policy. +- Preserve the ignored local workspace and any existing replacements. Do not + commit workspace files or sibling-directory replacements. The selected + upstream PromptKit must be the released v0.15.0 module. +- Commit each completed nonempty stage under the + [agent commit policy](../policy/agents.md#git-commit-policy). Record concise + completion markers and actual validation evidence beneath the corresponding + stage here. Never mark a future stage complete merely because its tests pass + incidentally. Do not push, tag, publish, or rewrite historical release notes + as part of this sprint. + +The validation commands in this plan are for the implementation sprint. This +planning pass creates documentation only. + +## Stage 1: Adopt the released PromptKit dependency and notices + +**Primary files:** `integrations/ckprompt/go.mod`, its `go.sum`, and +`integrations/ckprompt/docs/oss/attribution.md`. + +1. Upgrade only the selected PromptKit requirement to v0.15.0. Keep Go 1.26.0, + ConfigKit v0.7.1, and the existing CommandKit prerequisite version selections. + Review any actual transitive graph changes rather than upgrading unrelated + dependencies. +2. Resolve module metadata with the real released PromptKit. If unpublished + CommandKit prerequisite versions prevent workspace-off operations, use + temporary copies of ckprompt's module and sum files as described under + Stage 9. Add replacements only for core and ckconfig in those copies, run + the targeted upgrade/tidy there, then remove those temporary replacements + before transferring the resulting metadata to the owning checked-in files. + Preserve unrelated existing checksum entries unless normal tidy removes them. +3. Update the **integration's** PromptKit attribution entry from v0.14.0 to + v0.15.0 and pin its source/notice links to + `635674c066d0b432187f96e72b9c03d085443ac5`. The root attribution document is + not the owner of this entry. Review released copyright/license material; + update other entries or texts only for actual changes in selected material. +4. Confirm the released dependency provides `Engine.ValidateRequest` and that + no committed manifest includes a sibling PromptKit replacement. + +**Validation:** Run ckprompt's existing suite with the upgraded dependency; +inspect its selected module version and metadata diff. Use the module practices +in Stage 9 if required. Prose and attribution checks follow +[OSS](../policy/oss.md) and [documentation](../policy/documentation.md) policy. + +**Complete when:** ckprompt selects released PromptKit v0.15.0, its current +suite passes, notices identify that release, and module boundaries are unchanged. + +## Stage 2: Add the public capability and bundled validation adapter + +**Primary files:** `integrations/ckprompt/adapters.go`, a new owning-package +`validation.go`, and focused adapter tests. + +1. Add these exact public contracts, with GoDoc: + + ```go + type ValidationRequest struct { + Selection ckconfig.Selection + Request promptkit.RunRequest + } + + // Additional field in Suite[I, S]: + ValidateRequest func( + context.Context, ValidationRequest, I, S, + ) (commandkit.Acquired[struct{}], error) + + func WithRequestValidation[I, S any]( + suite Suite[I, S], + load func(context.Context, ckconfig.Selection, I, S) (*promptkit.Engine, error), + build func(context.Context, ValidationRequest, I, S) (promptkit.RunRequest, error), + ) Suite[I, S] + ``` + +2. Return the supplied suite with its engine-backed `ValidateRequest` callback + attached. Preserve other callbacks, factories, selectors, checks, and + attached issues. With a nil loader, retain a wiring error on the returned + suite and return without constructing a usable new validation callback. + When adding an issue, use an independent issues slice so the operation does + not overwrite a source suite's retained metadata through slice capacity. + Do not deep-copy caller-owned callback objects or selector bindings here. +3. Leave `FromEngine`'s existing five capabilities unchanged. A nil builder is + valid and means forwarding the seed request unchanged. +4. The attached callback checks cancellation before building, uses the builder + if present, checks cancellation again after successful building, acquires + the engine through `loadPromptKitEngine`, and invokes + `engine.ValidateRequest(ctx, concreteRequest)`. Pass the same `I`, `S`, and + invocation context to builder and loader. Use the envelope's original + configuration selection for loading. +5. A builder failure returns its supported error identity and prevents engine + loading. A loader failure returns the acquired configuration diagnostics and + error. A native failure returns that error with the acquired diagnostics. + Check operation errors before substituting a later context error, preserving + PromptKit's native cancellation category. Reuse `promptKitContextError` and + the existing engine-loading diagnostic adaptation where applicable. +6. Treat `Acquired[struct{}]` as diagnostics plus an empty value, not a Boolean + result or nil-result protocol. Builder output is the authoritative concrete + request: forward it without reapplying seed fields, merging maps, or checking + its prompt identity independently. Successful builder calls have no separate + diagnostic-return field in this selected API. + +**Validation:** Add direct adapter tests for composition preserving existing +suite members, nil-builder success, builder failure before loading, and retained +nil-loader issues. Confirm installation reports retained issues even after all +capabilities are cleared. Run the focused tests and ckprompt suite. + +**Complete when:** the public API and adapter work without command dispatch, +with explicit capability opt-in and the selected wiring/error behavior. + +## Stage 3: Verify native delegation, request construction, and cancellation + +**Primary files:** a focused `integrations/ckprompt/validation_adapters_test.go` +or equivalent owning adapter tests. + +1. Use a real offline PromptKit engine with synthetic prompts/profiles and a + counting or fail-on-call model client. Demonstrate successful concrete + validation and a missing required binding, asserting zero model calls. +2. Compare representative adapter failures with native `ValidateRequest` using + the same concrete request, or assert the documented native category with + `errors.Is`. Cover an invalid selection and missing/unloadable artifact + without reproducing every native preparation rule. +3. Demonstrate that a builder receives the seed, context, selection, typed input, + and shared state, then can add inline bindings and request settings. Use an + observable native effect, such as supplying a needed variable/artifact or + invalid appended message, to detect discarded builder output. The adapter + must not silently restore seed fields afterward. +4. Cover a nil engine result and a loader's real `*configkit.LoadError`, including + its retained diagnostic records and error identity. Reuse the existing + fixture and diagnostic helpers rather than a new abstraction framework. +5. Cover cancellation before building, after successful building, and during + or after loading. Verify later work is skipped and already acquired + diagnostics survive. For cancellation returned during native artifact + loading, assert PromptKit's applicable operation category; do not require + a general `errors.Is(err, context.Canceled)` relationship for native errors. + +**Validation:** Run these focused adapter tests and the ckprompt suite. Prefer +behavioral assertions; exact ordering assertions belong only to the explicitly +selected builder-before-loader lifecycle. + +**Complete when:** the adapter's native authority, builder forwarding, failure +handling, and cancellation boundaries have meaningful public-boundary evidence. + +## Stage 4: Implement the validate command and requirements group composition + +**Primary files:** `integrations/ckprompt/command.go`, a new private +`validation_command.go` in the same package, and basic command tests. + +1. Include `Suite.ValidateRequest` in `Command`'s capability/absence check. + Assemble one `requirements` group from the independently supplied show and + validate declarations. Continue attaching retained suite issues and using + the existing root-only family declaration. Update help summaries to describe + inspection and validation without changing catalog/definition routing. +2. Introduce a private per-invocation validation input holding `ValidationRequest`, + the format, captured selector values, and prepared application input. Build + a normal typed `CommandSpec` named `validate`, with one required prompt ID. + Use `CaptureFlags` and `prepareSuiteInput` for application selectors/checks. +3. Bind the following fields; defaults and repetition follow the accepted + roadmap and current requirements bindings: + + | CLI input | Destination | + | --- | --- | + | Prompt ID operand | `Request.PromptID` | + | `--config`, `--overlay` | Presence-aware `Selection.Config`, `Selection.Overlay` | + | `--version` | `Request.PromptVersion` | + | `--profile` | `Request.ProfileID` | + | `--session-id` | `Request.SessionID` | + | Repeatable `--var KEY=VALUE` | `Request.Vars` | + | Repeatable `--input NAME=PATH` | `Request.Inputs[NAME] = promptkit.File(PATH)` | + | `--format` | Private format, default `yaml` | + | Suite selectors | Captured bindings applied to initialized `I` | + +4. Parse file assignments at the first `=` and preserve the name and path + exactly. Parse every occurrence, then let the last duplicate name win. + Missing separators are usage errors; empty names/paths go to native + validation. Do not read files, normalize paths, expand them, or rebase them + while binding. Reuse variable-assignment mechanics where this is simple, + keeping their existing behavior unchanged. +5. Reject unsupported formats before application input preparation. Use the + existing YAML-only format check where appropriate. Missing operands, + malformed bindings, and unknown flags retain existing usage handling. +6. In the handler, check cancellation, initialize state using the existing + zero-value convention, check cancellation again, and invoke the captured + validation callback. On error, return `definitionFailure` with diagnostics. + After successful validation, check cancellation before encoding/delivery. + A successful empty acquisition value is valid; there is no inspection value + to require or dereference. +7. Encode a private YAML DTO with explicit `prompt_id` and `valid` fields. Use + the command's requested ID and `valid: true`, with no resolved metadata or + request dump. Reuse `encodePromptYAMLBytes` and `writePromptReport`, returning + diagnostics for core delivery. Do not change established report/diagnostic + delivery order, short-write handling, or failure bookkeeping. + +**Validation:** Add a basic assembled command test with a capturing custom +callback and semantic YAML decoding. Verify the correct seed reaches the +callback, and both show and validate can be installed in one requirements group. +Run focused tests and the ckprompt suite. + +**Complete when:** the new command invokes the selected provider and produces +the agreed YAML confirmation using existing core mechanics. + +## Stage 5: Cover command grammar, capability availability, and input lifetimes + +**Primary files:** a focused `integrations/ckprompt/validation_command_test.go` +and existing requirements command tests where an extension is simpler. + +1. Exercise validation-only, discovery-only, both, and absent suites. Check + installed help/availability and the existing namespace collision/root-only + behavior for the new validation-only declaration, reusing core coverage + rather than retesting the registry in full. +2. Verify construction, installation, help, and version leave input/state + factories, builders, and loaders uninvoked. Capture a command, then mutate + source callbacks/selector metadata to confirm the established eager-capture + behavior includes the new callback. Callback-owned referenced objects stay + caller-owned. +3. Capture standard request fields and typed selectors in a custom provider. + Cover explicit empty config/overlay presence, empty scalar native semantics, + singleton repetition, and repeated variable/file assignments. Include paths + containing `=`, empty names/values, whitespace/commas, duplicate names, and + a malformed earlier occurrence followed by a valid one. Inspect native file + references without requiring nonexistent paths to pass actual validation. +4. Cover missing/extra operands, unsupported format, unknown flags, malformed + `--var`/`--input`, and required suite selectors. Confirm applicable syntax + failures stop before application preparation/acquisition. Input-check + failures stop state creation and provider work using current classifications. +5. Run multiple invocations on one App and verify parsed Vars/Inputs, selection, + and selector presence do not leak into later calls. Verify one reached input + preparation/check and state creation per selected validation invocation, + and omitted factories supply zero values including nil pointers. Do not + promise isolation for aliases created by application factories. + +**Validation:** Run the focused command tests and ckprompt suite. Decode each +YAML output into a fresh value; avoid output-prefix or key-order assertions. + +**Complete when:** the new capability obeys the selected standard grammar, +availability, capture, and invocation-lifetime contracts. + +## Stage 6: Cover native command outcomes and report delivery + +**Primary files:** validation command tests and small real-engine fixtures. + +1. Exercise `WithRequestValidation` through `Command` and `App.Execute`, with + real native success, missing required binding, and one selection failure. + Include a useful nil-builder request built entirely from standard flags. + Assert no model call and preserve native error identity; compare categories, + not exact human-readable error wording. +2. Cover custom-provider diagnostics on success and error. Builder, loader, + and native failures are operational outcomes/status 1. CLI usage failures + are status 2. Retain warning diagnostics without turning warning severity + alone into failure. Reuse existing input-check handling. +3. Verify cancellation before acquisition and after a successful callback + prevents later work/report output. Preserve callback errors and native error + categories rather than overwriting them with a general context error. +4. Reuse existing writer doubles to cover report errors, short writes, and + diagnostic writer errors. Preserve `ReportIncomplete`, `DeliveryError`, + diagnostic-delivery state, and supported cause identity as established by + the shared delivery contract. Do not add rollback or blocking-writer + interruption guarantees. +5. Decode the success report and assert only the agreed confirmation fields + are emitted, without values/artifact contents/rendered messages. Keep the + existing partial-discovery-success tests as regression coverage; do not + duplicate their whole matrix for validation. + +**Validation:** Run focused tests and the complete ckprompt suite, including +its maintained composed consumer tests. + +**Complete when:** assembled validation follows native semantics and existing +outcome/report delivery contracts under representative successes and failures. + +## Stage 7: Extend the maintained composed consumer + +**Primary files:** `integrations/ckprompt/examples/composed/main.go`, its tests, +README, and synthetic fixtures under its existing `fixtures/` directory. + +1. Compose `WithRequestValidation` into the existing `FromEngine` suite, keeping + its factories, campaign selector, catalogs, definitions, discovery, and + separately installed config/session commands. +2. Add a typed builder that starts from the seed and uses the existing + application configuration loader and invocation cache. Load the selected + campaign through that shared state, then default a missing `topic` variable + to a synthetic campaign-specific topic and missing `mode` to `""`. Use map + membership, preserving supplied empty values. Keep seed identity, scalar + context, and operator artifact bindings. The engine loader can then reuse + that application load through the same cache. This is example behavior, + not a CommandKit cache/defaults policy. +3. Keep the existing synthetic prompt template and required `topic` artifact. + Add secret-free `fixtures/topic.txt` and `fixtures/notes.txt`. Do not supply + the required topic artifact implicitly in the builder: its absence is the + useful validation failure. The existing profile uses an inert endpoint and + no real credentials; validation remains offline. +4. Add README commands, from the composed directory: + + ```sh + go run . promptkit requirements validate briefing --campaign spring --input topic=fixtures/topic.txt + go run . promptkit requirements validate briefing --campaign spring --input topic=fixtures/topic.txt --var mode=full --input notes=fixtures/notes.txt --format yaml + go run . promptkit requirements validate briefing --campaign spring + ``` + + The first two succeed; the last intentionally fails for the missing required + topic binding. Preserve the existing show examples, explaining their distinct + discovery result. Describe success as passing preparation at that invocation. +5. Add focused composed-consumer evidence for successful file-backed validation, + missing-topic failure, and the builder preserving explicitly empty variables + while supplying defaults only for absent keys. Test builder behavior directly + where needed to observe precedence without inventing a library enforcement + rule. Preserve current campaign and discovery tests. + +**Validation:** Run composed tests and the documented successful/failing +commands. Record the expected classifications and semantic reports rather than +making the intentionally failing example a failed completion check. + +**Complete when:** the maintained consumer demonstrates the accepted feature, +typed selection, shared state, and application-owned defaults without execution. + +## Stage 8: Update canonical documentation for implemented behavior + +**Primary files:** GoDoc, `docs/cli.md`, `docs/consumers/guide.md`, README, +`docs/development.md`, and `docs/internal/overview.md` as affected. + +1. Complete GoDoc for `ValidationRequest`, `Suite.ValidateRequest`, + `WithRequestValidation`, and command factory behavior. Update the existing + `InspectRequirements` comment so omission refers to `requirements show`, + rather than implying the entire requirements group is absent. State optionality, + nil-builder forwarding, retained wiring issues, builder-before-loader order, + callback ownership, and native error/diagnostic treatment. +2. Extend the CLI requirements section with separate show/validate descriptions, + exact flags and repetition rules, file-binding semantics, YAML confirmation, + and status meanings. Scope existing discovery-only statements appropriately: + concrete validation loads resources and renders through native preparation; + it returns encountered errors and no snapshot/model output. +3. Update consumer guidance from five independently optional callbacks to six, + distinguishing `FromEngine`'s five supplied callbacks from explicit validation + composition. Include minimal full-suite and validation-only/nil-builder + wiring, and link to the maintained consumer for the complete builder example. + Recommend shared validation/execution request construction without prescribing + downstream architecture. +4. Update module summaries and implemented internal inventory to include concrete + validation. Audit statements about no rendering/resource loading/credential + resolution so metadata-inspection descriptions do not accidentally cover the + new capability. Keep native semantics linked to adopted upstream contracts. +5. Leave accepted ADRs, policy boundaries, historical releases, and unrelated + deferred proposals intact. No new ADR or policy revision is required by this + accepted feature. Current API/CLI documentation becomes the canonical owner + when implementation lands. + +**Validation:** Review examples against implemented signatures and commands; +check local links/anchors and Markdown whitespace. Reuse Stage 7's unchanged +example evidence. Check prose for accidental stronger guarantees or consumer +mandates. + +**Complete when:** current documentation accurately describes the implemented +capability under its assigned owners, with no stale five-callback or +metadata-only claims covering request validation. + +## Stage 9: Validate the complete end state and finish + +1. Format changed Go files and inspect the final diff for scope, generated junk, + local replacements, and unrelated changes. Keep `deferred.md` intact unless + a statement actually becomes stale through this feature. +2. Run the complete workspace suites from all three module directories: + + ```sh + GOTOOLCHAIN=local go test -count=1 ./... + (cd integrations/ckconfig && GOTOOLCHAIN=local go test -count=1 ./...) + (cd integrations/ckprompt && GOTOOLCHAIN=local go test -count=1 ./...) + ``` + + Root `./...` does not traverse nested modules. These ordinary suites do not + authorize a release publication exercise or live model service tests. +3. Separately verify each module with `GOWORK=off`, `GOTOOLCHAIN=local`, and + `-mod=readonly`. For unpublished CommandKit prerequisites, create temporary + absolute-path copies of each affected module's `go.mod` and `go.sum`, keeping + paired names such as `ckprompt.mod` and `ckprompt.sum`. Run from that module's + real directory with `-modfile=/absolute/temp/path/ckprompt.mod`. + + - Core: use its actual manifest; no optional-library replacements. + - ckconfig: replace only its required core version with the current root. + - ckprompt: replace only its required core and ckconfig versions with their + current local directories. + - Use version-specific replacements for the versions in the manifests. + PromptKit and ConfigKit remain real released dependencies. Temporary + replacements are validation scaffolding, never committed metadata. + - Prepare any needed cache/checksums in the temporary copies, then record + their hashes before running readonly tests. Verify those files are unchanged + afterward. A workspace-only pass does not establish isolated resolution. + + In each module, use `go test -mod=readonly -count=1 ./...` with the applicable + `-modfile` and environment above. Inspect `go list -m all` and + `go list -deps -test ./...` with the same isolated manifest: core has no optional + integration dependency; ckconfig has no PromptKit/ckprompt dependency; + ckprompt selects released PromptKit v0.15.0. Do not change checked-in + prerequisite versions merely to make isolated checks easier. Remove temporary + module files after retaining the relevant validation evidence. +4. Review the Stage 7 commands and canonical docs together. Check that the + nil-builder path and typed builder are both covered, show still permits + partial discovery, errors preserve native categories, and only the agreed + YAML confirmation is emitted on validation success. +5. Run relevant prose whitespace/link checks, including staged documentation. + Confirm the integration attribution matches the selected released dependency. + No new coverage threshold, benchmark, platform matrix, release gate, or + threat scope is introduced by this plan. +6. Mark completed stages with actual evidence, update the feature roadmap's + implementation status, and record any genuine remaining limitation accurately. + Commit outstanding sprint changes using a concise plain-English message, + inspect status afterward, and report the final commit(s) and validation. + Retain planning documents until their cleanup is requested. + +**Complete when:** every accepted roadmap capability is implemented, all +applicable evidence above passes, public documentation and module metadata are +aligned, and the workspace is clean. No unresolved product or API decisions +were identified in preparing this plan. diff --git a/docs/roadmap/promptkit-validation.md b/docs/roadmap/promptkit-validation.md index 6188723..1cb174e 100644 --- a/docs/roadmap/promptkit-validation.md +++ b/docs/roadmap/promptkit-validation.md @@ -3,7 +3,7 @@ Status: accepted adoption and feature roadmap; implementation has not started. Command placement, adapter composition, request construction, artifact syntax, and initial presentation are settled. This document defines the target and -required work; a later implementation plan will sequence the work. +required work; the [implementation plan](implementation.md) sequences the work. ## Purpose