Record PromptKit migration review findings

This commit is contained in:
2026-10-04 15:24:35 +00:00
parent 6700ed2188
commit e690cae24f

View File

@@ -1,108 +1,193 @@
# ConfigKit Inheritance Review Remediation
# PromptKit Configuration Migration Review Remediation
## Review Scope And Overall Result
## Scope And Overall Assessment
Reviewed on 2026-10-04, comparing implementation commits `ec99d48` through
`6450994` with the baseline `9e74df0`, the ConfigKit feature roadmap, and its
eight-stage implementation plan.
Review baseline: Notarius `6700ed2`, compared with planning revision `f5b066b`,
the accepted [feature roadmap](promptkit.md), and
[implementation plan](implementation.md). PromptKit v0.12.0 is pinned at
`4b8169da286917fde7875425bb32b799b12c6af3`; ConfigKit remains v0.5.0, with both
PromptKit `/v3` catalogs at v3.20260930.0.
The architecture matches the intended design. ConfigKit v0.5.0 owns source
composition and native sibling inheritance before decoding. Notarius retains
its typed defaults, presence-sensitive validation, declaration-relative path
normalization, detached runtime configuration, catalog-dependent resolution,
and private redacted inspection boundary. The production configuration adapter
needed no changes; the work adds the dependency update, integration tests,
documentation, and the derived development pipeline example.
The migration is substantially implemented. Native sources, combined operator
documents and imports, recursive overlays, source-relative prompt content,
captured engine lifetime, immutable checkpoint identity, and credential-free
operator inspection are in place. The provider-neutral execution boundary,
prepared execution, retries, scheduling, and durable artifact contracts remain
intact. All 16 migrated prompt bodies match the previous definitions after
removing their old embedded identity fields; prompt fragments and schemas are
unchanged.
No production regression or inadvertently strengthened requirement was found.
In particular, the implementation adds no inheritance whitelist, new schema
version, custom merging, path-containment rule, template exemption, or release
gate. The newly reserved `extends` key is the expressly adopted upstream
compatibility change.
Four bounded follow-ups remain below. No production code is changed by this
review. No additional runtime security restriction, mandatory credential check,
filesystem permission policy, compatibility shim, or release gate was found.
Eager final-definition validation and the external keyed format are intentional
upstream changes, not accidental strengthening. The previously recorded
ConfigKit remediation was completed before this work set; it is not outstanding.
Two follow-ups were identified: a path-test blind spot and an incomplete
documented command. Completion is recorded under each finding. This document
records remediation work only; it does not authorize
unrelated runtime refactoring.
## Finding 1 — Compose Registered Application Assets In One Strict Group
## Finding 1 — Make The Path Fixture Distinguish Declaring Directories
Priority: P2. Category: architecture/correctness gap. Status: open.
Priority: P2. Category: verification gap.
Evidence:
Status: addressed on 2026-10-04. The fixture now uses distinct parent-, child-,
and overlay-local paths for the source-base assertions and retains a separate
`../` reference. `GOWORK=off go test ./internal/core/config` passed. No
production resolver change was needed.
- `internal/framework/llm/asset_registry.go:106–139` creates one underlay per
registered prompt or fallback-profile filesystem.
- `internal/framework/llm/asset_registry_sources_test.go`,
`TestPromptKitConfigurationSourcesKeepsNativeFilesystemGroups`, asserts that
separate registrations yield separate underlays, encoding this divergence.
- The plan's Sources And Engine Ownership section calls for one application
underlay whose inputs each retain their own filesystem context;
`docs/internal/llm.md:86–92` also describes that behavior.
Evidence: `internal/core/config/inheritance_test.go:338–436`,
`TestInheritedReferencePathsKeepDeclaringSource`.
Separate underlays have recursive overlay semantics. Conflicting application
definitions in different registrations therefore silently depend on module
registration order instead of receiving native strict-composition diagnostics.
An offline probe registered two fallback files with the same profile key and
different models. The current inspector accepted them and selected the second
model. The exact same filesystem inputs placed in one native underlay correctly
failed composition. Distinct filesystem contexts do not require separate layers.
The parent, child, and overlay files live in `root/parents`, `root/children`,
and `root/overlays`. Their resource paths all use `../fixtures/<name>`, and
the expected paths all point into `root/fixtures`. Resolving one of these
paths from any of the three directories yields the same result. Thus, choosing
the child's directory for an inherited step/nested reference, or choosing the
parent's directory for an overlay replacement, could still pass this test.
Required changes:
`TestInheritedInspectionRetainsOriginsRedactionAndOwnership` does distinguish
the parent directory for a simple inherited `./shared.txt` reference. It does
not close the blind spot for the nested, ordered-step, direct-child replacement,
and overlay cases in the larger path fixture.
- Gather the native `FilesystemInputs` from every application prompt and
fallback-profile registration into one application underlay. Preserve each
input's existing label, filesystem, root, and disabled-import setting. Keep
schemas on their separate runtime option and omit an empty application group.
- Keep the operator and explicit local-backend contributions as later groups.
Preserve their accepted recursive overrides and source precedence. Do not
add an application-specific duplicate-key detector or flatten filesystem
contexts into an artificial shared root.
- Replace the incidental group-count test with behavioral coverage: conflicting
application contributions fail through native strict composition; distinct
declarations retain source-relative content; an operator overlay over the
resulting application base still succeeds. Reuse existing origin coverage.
Required change:
Acceptance: application conflicts are not silently resolved by registration
order; source-relative imports/content and valid later operator overlays remain
unchanged. Runtime, preflight, and reports use the corrected shared construction.
- Revise the existing path fixture to use directory-distinguishing relative
paths for these cases, such as `./shared.txt`, `./step.txt`, and a child-local
replacement. Assert inherited paths against the parent's directory, direct
replacements against the child's directory, and overlay-supplied paths
against the overlay's directory.
- Retain a separate legitimate `../` example to protect the existing host-path
behavior. Avoid making every path converge on one shared sibling directory.
- Keep this as a focused real-file configuration test. Change production
resolution only if the improved assertions expose an actual defect.
Validation: `GOWORK=off go test ./internal/framework/llm ./internal/cli ./internal/modules/...`.
Acceptance: choosing a parent, child, or overlay directory incorrectly produces
a different absolute path and fails the relevant assertion. Nested inheritance,
inherited ordered-step references, child replacements, and overlay replacements
still pass with the actual resolver.
## Finding 2 — Validate Explanation Syntax Before Loading Configuration
Validation: `GOWORK=off go test ./internal/core/config`.
Priority: P3. Category: CLI correctness. Status: open.
## Finding 2 — Supply Input In The Development Pipeline Command
Evidence: `internal/cli/configkit_commands.go:77–106`. PromptKit scope loads
Notarius configuration and constructs the inspection engine before validating
the pointer; the default Notarius scope parses the pointer first.
Priority: P3. Category: documentation accuracy.
Reproduction, using the native temporary build and a nonexistent config path:
Status: addressed on 2026-10-04. The example now supplies the pipeline ID,
configuration path, and transcript input path, and links to the CLI run
reference. The flags were checked against that reference, and
`GOWORK=off go test ./internal/cli` passed without a live model request.
```sh
notarius config explain --scope promptkit --config /path/to/missing.yml '/profiles/~2'
```
Evidence: `docs/config.md:43–49`, in Maintained Examples.
The current PromptKit command returns exit 1 with a config-file error. The same
malformed pointer returns exit 2 with a valid config, and the Notarius scope
returns exit 2 even with the missing config. A bad PromptKit definition or
missing content file can likewise mask malformed command syntax. This differs
from the selected syntax/runtime distinction in the plan and CLI reference.
The new example says to select the development pipeline with
`notarius run dnd-session-development --config /path/to/dnd-complete.config.yml`.
The run command requires `--input`. Executing the equivalent command against
the maintained configuration returns `notarius: run requires --input` with CLI
exit status 2, before pipeline execution.
Required changes:
Required change: include a transcript input placeholder in the invocation,
for example `--input /path/to/transcript.json`, and retain the link to the CLI
reference for the full invocation contract. No CLI behavior change or new test
framework is needed.
- Validate canonical JSON Pointer syntax once, before either scope loads config
or constructs an engine. Reuse the existing parser; retain defensive parsing
inside the adapter for direct callers.
- Keep the empty root pointer, escaped literal keys, historical-only paths,
unknown-path exit 1, and writer-error behavior unchanged.
- Extend CLI coverage with a malformed pointer and unavailable configuration;
verify exit 2, no stdout, and no configuration/provider work. No broader
argument-parser refactor is needed.
Acceptance: the documented command supplies both required run inputs—the
pipeline ID and input path—and uses the implemented flag names. Verify against
the CLI reference and existing parser coverage without a live model request.
Acceptance: malformed pointers are syntax errors independent of configuration
availability for both scopes; valid selectors still load and inspect normally.
## Checks Completed During Review
Validation: `GOWORK=off go test ./internal/cli ./internal/framework/llm`.
## Finding 3 — Correct Lifecycle And Fallback Documentation
Priority: P3. Category: documentation accuracy. Status: open.
Evidence:
- `docs/config.md:321–323` says prompt **and schema** definitions are validated
at engine construction. Schemas remain preparation-time resources supplied
through `WithSchemaFS`, not captured configuration definitions.
- `docs/internal/llm.md:71–77` and `docs/internal/cli.md:55–61` describe preflight
as occurring without loading a prompt. Construction now loads and validates
effective prompt definitions and captures message files/static templates;
it does not prepare a request or generate output.
- The comment on `RegisterFallbackProfileFS` in
`internal/framework/llm/asset_registry.go:61–62` still says fallback assets
are used only when the operator source lacks the ID. Under recursive overlays,
their omitted fields remain active beneath a matching operator entry.
Required changes:
- Distinguish constructor-time definition/message capture from preparation-time
schema loading and request-variable/artifact resolution. State accurately
that preflight needs no transcript, credential value, or provider call; do
not promise it performs no prompt-content file reads.
- Describe fallback registration as a lower-precedence application contribution
rather than an all-or-nothing lookup fallback.
- While editing this text, clarify that an operator `backends.local` declaration
can supply the backend as well as Notarius's optional `local_backend` object.
The latter is a convenience contribution, not a required second declaration.
Link canonical source rules rather than duplicating them.
- Change documentation/comments only unless verification exposes a real runtime
defect. Do not move schema loading earlier to make the stale prose true.
Acceptance: public and internal guidance consistently explains actual lifecycle,
native layering, and operator backend configuration without inventing a stronger
preflight guarantee or an additional required config field.
Validation: inspect current constructors/prepared execution and upstream v0.12
contracts; review affected local links; run `git diff --check`. Reuse unchanged
behavioral checks rather than adding prose-snapshot tests.
## Finding 4 — Preserve ADR-0010's Historical Decision Body
Priority: P3. Category: documentation-policy compliance. Status: open.
Evidence: the migration diff rewrites ADR-0010's Decision and Consequences
paragraphs at `docs/adr/0010-workload-oriented-llm-profile-defaults.md:26–33`
and `:47–50`, as well as adding the appropriate status/supersession link.
`docs/policy/documentation.md` requires accepted decision content to remain
immutable; Stage 14 specified a concise status link rather than rewriting the
historical body.
The replacement correctly acknowledges the old behavior, but the historical
record should retain its original wording. ADR-0019 already owns the new native
overlay decision and accurately limits which part of ADR-0010 it supersedes.
Required change: restore ADR-0010's Decision and Consequences body from `f5b066b`,
keeping its new partial-supersession status and ADR-0019 link. Leave workload
ownership and ADR-0019 unchanged. Current runtime behavior remains documented
in the canonical current references, not retrofitted into the older decision.
Acceptance: the historical body matches the accepted original, the changed
clause's supersession remains discoverable, and no runtime behavior changes.
Validation: compare the body with `git show f5b066b:docs/adr/0010-workload-oriented-llm-profile-defaults.md`,
verify the ADR link, and run `git diff --check`.
## Review Evidence And Validation
- `GOWORK=off go test ./...` passed.
- `GOWORK=off go vet ./...` passed.
- `GOWORK=off go build` of `./cmd/notarius` to a temporary destination passed.
- `GOWORK=off go test -race ./internal/framework/llm ./internal/cli` passed.
- `GOWORK=off go mod tidy -diff` produced no changes.
- The selected ConfigKit module is v0.5.0; PromptKit remains v0.11.2.
- The implementation diff passed whitespace checks.
- The incomplete development invocation was reproduced without provider calls.
- Module inspection confirmed the selected ConfigKit, PromptKit, and catalog pins.
- A native build outside the repository succeeded. Credential-free profile and
explanation commands and complete/development pipeline validation succeeded.
- A YAML-aware comparison confirmed all 16 migrated prompt definition bodies
are unchanged after removing old embedded IDs/versions. Prompt fragments,
schemas, and the embedded D&D execution settings are unchanged.
- An offline Go overlay probe, stored entirely outside the repository, confirmed
the application-group conflict behavior in Finding 1. CLI invocations
reproduced Finding 2 without generation or provider calls.
- The implementation diff passed whitespace checks. No production inference
or production model-quality evaluation was performed.
These checks describe the review baseline. Subsequent remediation and its
focused validation are recorded under each finding.
Recommended remediation is limited to these findings, not another configuration
system, new security policy, or unrelated pipeline refactoring.