diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md index f96d1e44..0d9936ca 100644 --- a/docs/roadmap/remediation.md +++ b/docs/roadmap/remediation.md @@ -1,8 +1,8 @@ # Configuration Upgrade Review Remediation The earlier PromptKit findings below retain their original scope and status. -The ConfigKit 1.2.0 review and its two new findings are recorded in the final -section; they do not reopen or attribute earlier issues to that upgrade. +The ConfigKit 1.2.0 review and subsequent integration audit appear below; they +do not reopen or attribute earlier issues to that upgrade. ## Scope And Overall Assessment @@ -245,9 +245,10 @@ Required changes: and whole-record upsert; no new application inheritance machinery is needed. Acceptance: an operator can determine the correct syntax for both a pipeline -mapping and a step record from the canonical reference. Any added snippet loads -through the current configuration boundary. Reuse the existing behavioral test; -do not add a documentation snapshot or change runtime behavior to fit the prose. +mapping and an identified record from the canonical reference. Any example +presented as a runnable pipeline must pass pipeline resolution, not just loading; +see CK-3 for step-record limitations. Reuse behavioral coverage rather than adding +a documentation snapshot or changing runtime behavior to fit the prose. ### Finding CK-2 — Describe Dependency-Ready Step Assembly Order @@ -289,7 +290,7 @@ merging, or scheduler changes belong in this remediation. | 1 — Dependencies/toolchain | The selected graph builds with Go 1.27.0, both approved ConfigKit versions, PromptKit v0.12.0, and YAML rc.6; tidy reports no diff. | | 2 — API/handoff | Application imports use the new package, old types remain in the PromptKit boundary, and LoadSource/MainSourcePath retain frozen inspection and detached runtime values. | | 3 — Normalization/origins | Paths resolve before key movement; unchanged step identities retain leaf records, changed identities retain covering structural records. Existing tests and an additional temporary combined key/path-normalization probe pass. | -| 4 — Collections | The exact step-only identity registration delegates composition to ConfigKit. Real-file tests cover imports, upsert, replace, append, clearing, record inheritance, and source-relative references. | +| 4 — Collections | The exact step-only identity registration delegates composition to ConfigKit. Real-file tests cover imports, upsert, replace, append, clearing, and source-relative references. Identified-record inheritance is checked only at loading/configuration validation; CK-3 records the missing runtime-consumer check. | | 5 — Fixtures | Application fixtures use native controls and assignment timing; old PromptKit fixtures remain correctly scoped. The full suite passes. | | 6 — CLI discovery | The shared loader describes File/SearchConfig sources once and returns the recorded selected file. Manual probe helpers are removed. | | 7 — Discovery/failures | Controlled tests cover precedence, filename selection, required exhaustion, terminal failures, reports, symlinks, and debug provenance without relying on host system files. | @@ -317,3 +318,90 @@ merging, or scheduler changes belong in this remediation. Keep the earlier PromptKit findings separate from these two documentation fixes. The deferred PromptKit/ConfigKit v1.2.1 upgrade and release work remain out of scope. + +## ConfigKit Integration Audit + +Reviewed on 2026-11-17 against `6f0a75e`, with particular attention to how the +11 implementation stages interact with the existing pipeline and PromptKit +consumers. The implementation is unchanged from `c9744ad`; the intervening +commits changed roadmaps only. CK-1 and CK-2 remain open. + +The dependency/toolchain boundary, detached configuration conversion, path +normalization, CLI overrides, native discovery, selected-file reporting, and +old-format PromptKit source handoff are coherent. The maintained complete +example still reaches the existing multi-step runner and publishes its generated +registry/occurrence handoffs through an injected external client. Session, +checkpoint, warning, and output consumers continue to receive resolved domain +values, not ConfigKit source graphs. No new production correctness regression +was identified. One additional test/documentation integration gap remains. + +### Finding CK-3 — Verify Identified-Record Inheritance At The Runtime Boundary + +Priority: P3. Category: integration coverage and support-scope clarity. Status: open. +Affected stages: 4, 9, and 11. + +`TestIdentifiedStepRecordInheritance` in +`internal/core/config/collection_composition_test.go:182–208` supplies a parent +step containing only references and a child step containing an artifact lane. +It checks `Load` and `Config.Validate`, but never calls `Config.Resolve` with a +catalog. The existing runner's resolver requires every explicit step to have at +least one artifact lane (`internal/framework/pipeline/profile.go`, +`ResolvePipeline`). The parent is an ordinary executable step, not an excluded +template. The fixture therefore cannot be used as a resolved pipeline, despite +the positive test and the reference's unqualified mention of step-record +inheritance. + +Giving the parent a real artifact lane is not a simple repair to the fixture: +native inheritance recursively copies that named lane into the child, while +Notarius requires lane identifiers to be unique across steps. Adding a new +child-specific lane does not remove the inherited one. Pipeline-level inheritance +with step upsert is a different operation and remains supported and tested. + +Offline reproduction through the current binary: + +- A production-module equivalent of the test fixture loads for effective + inspection, but `config validate --pipeline session` fails because the parent + step must declare an artifact lane. +- A parent with `parent-spells` and a child declaring `child-spells` fails domain + validation because inherited `parent-spells` occurs in both steps. +- A native identity declaration on an extraction validator list, with + `generic/valid_json_schema` inheriting from `generic/valid_json` through + `extends: {module: generic/valid_json}`, passes production pipeline validation. + Generic identified-record inheritance is therefore usable; the gap is the + selected positive step fixture and its implied runtime applicability. + +Required changes: + +- Replace or reframe the positive step fixture. Cover a genuinely executable + identified-record use case in a compatible collection and verify it through + `Config.Resolve` with the appropriate catalog or the equivalent public CLI + validation path. Assert a meaningful inherited value, not only successful load. +- Make the step-specific constraints clear alongside CK-1's syntax correction. + Do not present the reference-only parent fixture as a runnable step template. + Preserve the distinction between pipeline inheritance/upsert and inheritance + between records of one final sequence. +- Preserve the existing runtime constraints. This finding does not authorize + template-only steps, implicit parent removal, lane renaming, or weaker lane-ID + uniqueness. A separate product decision would be needed for such features. + +Acceptance: positive inheritance coverage reaches the actual configuration +consumer, and operator guidance distinguishes native composition from executable +pipeline validity. The complete D&D example, generated-reference validation, and +ordinary pipeline inheritance continue to behave unchanged. No production fix +is indicated by the current evidence. + +### Integration Audit Validation + +- Reviewed the staged changes against configuration conversion, pipeline + resolution/reference materialization, PromptKit construction/inspection, + checkpoint identity, and CLI debug/output consumers. +- `GOTOOLCHAIN=go1.27.0 GOWORK=off go test ./internal/core/config ./internal/cli ./internal/framework/llm ./internal/framework/pipeline ./internal/modules/integration` + passed. This includes the maintained complete-example run, generated-reference + handoffs, command overrides, selected-path debug provenance, and profile reports. +- Temporary CLI fixtures outside the repository reproduced both step-inheritance + failures and verified the compatible validator-record alternative. An additional + author-identified validator list with whitespace-normalized module names also + passed pipeline validation. +- The full-suite, vet, race, tidy, and build results from the preceding review + remain applicable to the unchanged implementation. No live provider call or + production-quality evaluation was performed.