18 KiB
Configuration Upgrade Review Remediation
The earlier PromptKit findings below retain their original scope and status. The ConfigKit 1.2.0 review and its two new findings are recorded in the final section; they do not reopen or attribute earlier issues to that upgrade.
Scope And Overall Assessment
Review baseline: Notarius 6700ed2, compared with planning revision f5b066b,
the accepted feature roadmap, and
implementation plan. PromptKit v0.12.0 is pinned at
4b8169da286917fde7875425bb32b799b12c6af3; ConfigKit remains v0.5.0, with both
PromptKit /v3 catalogs at v3.20260930.0.
The migration is substantially implemented. Native sources, combined operator documents and imports, recursive overlays, source-relative prompt content, captured engine lifetime, immutable checkpoint identity, and credential-free operator inspection are in place. The provider-neutral execution boundary, prepared execution, retries, scheduling, and durable artifact contracts remain intact. All 16 migrated prompt bodies match the previous definitions after removing their old embedded identity fields; prompt fragments and schemas are unchanged.
Four bounded follow-ups remain below. No production code is changed by this review. No additional runtime security restriction, mandatory credential check, filesystem permission policy, compatibility shim, or release gate was found. Eager final-definition validation and the external keyed format are intentional upstream changes, not accidental strengthening. The previously recorded ConfigKit remediation was completed before this work set; it is not outstanding.
Finding 1 — Compose Registered Application Assets In One Strict Group
Priority: P2. Category: architecture/correctness gap. Status: open.
Evidence:
internal/framework/llm/asset_registry.go:106–139creates one underlay per registered prompt or fallback-profile filesystem.internal/framework/llm/asset_registry_sources_test.go,TestPromptKitConfigurationSourcesKeepsNativeFilesystemGroups, asserts that separate registrations yield separate underlays, encoding this divergence.- The plan's Sources And Engine Ownership section calls for one application
underlay whose inputs each retain their own filesystem context;
docs/internal/llm.md:86–92also describes that behavior.
Separate underlays have recursive overlay semantics. Conflicting application definitions in different registrations therefore silently depend on module registration order instead of receiving native strict-composition diagnostics. An offline probe registered two fallback files with the same profile key and different models. The current inspector accepted them and selected the second model. The exact same filesystem inputs placed in one native underlay correctly failed composition. Distinct filesystem contexts do not require separate layers.
Required changes:
- Gather the native
FilesystemInputsfrom every application prompt and fallback-profile registration into one application underlay. Preserve each input's existing label, filesystem, root, and disabled-import setting. Keep schemas on their separate runtime option and omit an empty application group. - Keep the operator and explicit local-backend contributions as later groups. Preserve their accepted recursive overrides and source precedence. Do not add an application-specific duplicate-key detector or flatten filesystem contexts into an artificial shared root.
- Replace the incidental group-count test with behavioral coverage: conflicting application contributions fail through native strict composition; distinct declarations retain source-relative content; an operator overlay over the resulting application base still succeeds. Reuse existing origin coverage.
Acceptance: application conflicts are not silently resolved by registration order; source-relative imports/content and valid later operator overlays remain unchanged. Runtime, preflight, and reports use the corrected shared construction.
Validation: GOWORK=off go test ./internal/framework/llm ./internal/cli ./internal/modules/....
Finding 2 — Validate Explanation Syntax Before Loading Configuration
Priority: P3. Category: CLI correctness. Status: open.
Evidence: internal/cli/configkit_commands.go:77–106. PromptKit scope loads
Notarius configuration and constructs the inspection engine before validating
the pointer; the default Notarius scope parses the pointer first.
Reproduction, using the native temporary build and a nonexistent config path:
notarius config explain --scope promptkit --config /path/to/missing.yml '/profiles/~2'
The current PromptKit command returns exit 1 with a config-file error. The same malformed pointer returns exit 2 with a valid config, and the Notarius scope returns exit 2 even with the missing config. A bad PromptKit definition or missing content file can likewise mask malformed command syntax. This differs from the selected syntax/runtime distinction in the plan and CLI reference.
Required changes:
- Validate canonical JSON Pointer syntax once, before either scope loads config or constructs an engine. Reuse the existing parser; retain defensive parsing inside the adapter for direct callers.
- Keep the empty root pointer, escaped literal keys, historical-only paths, unknown-path exit 1, and writer-error behavior unchanged.
- Extend CLI coverage with a malformed pointer and unavailable configuration; verify exit 2, no stdout, and no configuration/provider work. No broader argument-parser refactor is needed.
Acceptance: malformed pointers are syntax errors independent of configuration availability for both scopes; valid selectors still load and inspect normally.
Validation: GOWORK=off go test ./internal/cli ./internal/framework/llm.
Finding 3 — Correct Lifecycle And Fallback Documentation
Priority: P3. Category: documentation accuracy. Status: open.
Evidence:
docs/config.md:321–323says prompt and schema definitions are validated at engine construction. Schemas remain preparation-time resources supplied throughWithSchemaFS, not captured configuration definitions.docs/internal/llm.md:71–77anddocs/internal/cli.md:55–61describe preflight as occurring without loading a prompt. Construction now loads and validates effective prompt definitions and captures message files/static templates; it does not prepare a request or generate output.- The comment on
RegisterFallbackProfileFSininternal/framework/llm/asset_registry.go:61–62still says fallback assets are used only when the operator source lacks the ID. Under recursive overlays, their omitted fields remain active beneath a matching operator entry.
Required changes:
- Distinguish constructor-time definition/message capture from preparation-time schema loading and request-variable/artifact resolution. State accurately that preflight needs no transcript, credential value, or provider call; do not promise it performs no prompt-content file reads.
- Describe fallback registration as a lower-precedence application contribution rather than an all-or-nothing lookup fallback.
- While editing this text, clarify that an operator
backends.localdeclaration can supply the backend as well as Notarius's optionallocal_backendobject. The latter is a convenience contribution, not a required second declaration. Link canonical source rules rather than duplicating them. - Change documentation/comments only unless verification exposes a real runtime defect. Do not move schema loading earlier to make the stale prose true.
Acceptance: public and internal guidance consistently explains actual lifecycle, native layering, and operator backend configuration without inventing a stronger preflight guarantee or an additional required config field.
Validation: inspect current constructors/prepared execution and upstream v0.12
contracts; review affected local links; run git diff --check. Reuse unchanged
behavioral checks rather than adding prose-snapshot tests.
Finding 4 — Preserve ADR-0010's Historical Decision Body
Priority: P3. Category: documentation-policy compliance. Status: open.
Evidence: the migration diff rewrites ADR-0010's Decision and Consequences
paragraphs at docs/adr/0010-workload-oriented-llm-profile-defaults.md:26–33
and :47–50, as well as adding the appropriate status/supersession link.
docs/policy/documentation.md requires accepted decision content to remain
immutable; Stage 14 specified a concise status link rather than rewriting the
historical body.
The replacement correctly acknowledges the old behavior, but the historical record should retain its original wording. ADR-0019 already owns the new native overlay decision and accurately limits which part of ADR-0010 it supersedes.
Required change: restore ADR-0010's Decision and Consequences body from f5b066b,
keeping its new partial-supersession status and ADR-0019 link. Leave workload
ownership and ADR-0019 unchanged. Current runtime behavior remains documented
in the canonical current references, not retrofitted into the older decision.
Acceptance: the historical body matches the accepted original, the changed clause's supersession remains discoverable, and no runtime behavior changes.
Validation: compare the body with git show f5b066b:docs/adr/0010-workload-oriented-llm-profile-defaults.md,
verify the ADR link, and run git diff --check.
Review Evidence And Validation
GOWORK=off go test ./...passed.GOWORK=off go vet ./...passed.GOWORK=off go test -race ./internal/framework/llm ./internal/clipassed.GOWORK=off go mod tidy -diffproduced no changes.- Module inspection confirmed the selected ConfigKit, PromptKit, and catalog pins.
- A native build outside the repository succeeded. Credential-free profile and explanation commands and complete/development pipeline validation succeeded.
- A YAML-aware comparison confirmed all 16 migrated prompt definition bodies are unchanged after removing old embedded IDs/versions. Prompt fragments, schemas, and the embedded D&D execution settings are unchanged.
- An offline Go overlay probe, stored entirely outside the repository, confirmed the application-group conflict behavior in Finding 1. CLI invocations reproduced Finding 2 without generation or provider calls.
- The implementation diff passed whitespace checks. No production inference or production model-quality evaluation was performed.
Recommended remediation is limited to these findings, not another configuration system, new security policy, or unrelated pipeline refactoring.
ConfigKit 1.2.0 Review
Reviewed on 2026-11-17 against implementation revision c9744ad, compared with
planning revision 79088a3 and the ConfigKit feature roadmap and implementation
plan. The subsequent roadmap-only clarification in d88a01c does not change
the reviewed runtime. ConfigKit v1.2.0 resolves to
c1c7e87f2cdf77a0cbf3327831ac05578e756b9e.
The production architecture matches the target. Application configuration uses new-path ConfigKit v1.2.0; the unchanged PromptKit v0.12.0 adapter retains old-path ConfigKit v0.5.0. Both use YAML v4.0.0-rc.6 under Go 1.27.0. The CLI describes one exclusive file or ordered native search source, while ConfigKit owns acquisition, imports, collections, inheritance, decoding, diagnostics, and inspection. Notarius supplies domain defaults, recorded normalization, validation, detached runtime conversion, and catalog-dependent resolution.
No new production correctness defect or unwarranted restriction was found. The changed controls, optional import defaults, post-decode CLI assignments, step identity/upsert, search behavior, and higher Go minimum are approved scope. The remaining gaps are two operator-documentation inaccuracies from Stage 9, also missed by Stage 11's final documentation check. Production fixes are not required for either finding.
Finding CK-1 — Distinguish Mapping And Record Inheritance Syntax
Priority: P3. Category: documentation accuracy. Status: open.
Evidence: docs/config.md:143–149 introduces both mapping and identified-record
inheritance, then says a directive names an immediate sibling in the same
mapping. That describes mapping inheritance only. For an identified step record,
the parent belongs to the same sequence and the reference must be an identity
mapping, such as extends: {id: parent}. The reference never shows that form.
An offline CLI probe with a parent step providing references and a child step
providing an artifact lane confirmed that extends: parent fails with
inheritance.invalid, while changing only the directive to
extends: {id: parent} loads successfully and preserves the inherited reference.
TestIdentifiedStepRecordInheritance already exercises the correct form in
internal/core/config/collection_composition_test.go:182–208.
Required changes:
- Explain string-valued mapping-sibling inheritance and identity-mapping sequence-record inheritance separately, including their distinct lookup scopes.
- Add a minimal step-record example or a direct link to the pinned upstream syntax example. Preserve the existing rules for null cancellation, validation, and whole-record upsert; no new application inheritance machinery is needed.
Acceptance: an operator can determine the correct syntax for both a pipeline mapping and a step record from the canonical reference. Any added snippet loads through the current configuration boundary. Reuse the existing behavioral test; do not add a documentation snapshot or change runtime behavior to fit the prose.
Finding CK-2 — Describe Dependency-Ready Step Assembly Order
Priority: P3. Category: documentation accuracy. Status: open.
Evidence: docs/config.md:167–170 says distinct imported step IDs combine
"in source order." ConfigKit distinguishes acquisition/import declaration order
from dependency-ready assembly order. Ready contributions are selected by their
stable source keys, so independent branches can interleave. The feature roadmap
and Stage 8 correctly rely on dependency-ready assembly rather than a Notarius
append pass.
An offline CLI probe declared imports in the order z-first.yml, then
a-second.yml, each contributing a different step and lane. Effective output
placed the step from a-second.yml first, then the step from z-first.yml.
Thus declaration order does not determine the resulting step list. This matters
because Notarius executes steps in the resolved list order and validates their
generated-reference dependencies.
Required changes:
- Replace the source-order claim with the native dependency-ready assembly rule and link to ConfigKit v1.2.0's pinned composition contract for its full ordering semantics. Do not promise declaration order or duplicate the upstream scheduler.
- Explain briefly why the maintained complete example's imported describe-session
step precedes the importing body's extract-events step: the imported dependency
is assembled before its importer. Keep the canonical operator rule in
docs/config.md; other documentation should link rather than redefine it.
Acceptance: the reference does not imply that reordering independent import declarations necessarily reorders steps. The complete example's actual step order and generated-reference handoffs remain unchanged. No custom sorting, merging, or scheduler changes belong in this remediation.
Stage-By-Stage Review Evidence
| Stage | Review result |
|---|---|
| 1 — Dependencies/toolchain | The selected graph builds with Go 1.27.0, both approved ConfigKit versions, PromptKit v0.12.0, and YAML rc.6; tidy reports no diff. |
| 2 — API/handoff | Application imports use the new package, old types remain in the PromptKit boundary, and LoadSource/MainSourcePath retain frozen inspection and detached runtime values. |
| 3 — Normalization/origins | Paths resolve before key movement; unchanged step identities retain leaf records, changed identities retain covering structural records. Existing tests and an additional temporary combined key/path-normalization probe pass. |
| 4 — Collections | The exact step-only identity registration delegates composition to ConfigKit. Real-file tests cover imports, upsert, replace, append, clearing, record inheritance, and source-relative references. |
| 5 — Fixtures | Application fixtures use native controls and assignment timing; old PromptKit fixtures remain correctly scoped. The full suite passes. |
| 6 — CLI discovery | The shared loader describes File/SearchConfig sources once and returns the recorded selected file. Manual probe helpers are removed. |
| 7 — Discovery/failures | Controlled tests cover precedence, filename selection, required exhaustion, terminal failures, reports, symlinks, and debug provenance without relying on host system files. |
| 8 — Examples | Minimal, complete, and development pipelines validate without inference; the supporting fragment preserves module choices, step order, and handoffs. |
| 9 — Operator docs | Search, controls, path bases, CLI timing, and toolchain guidance are updated. CK-1 and CK-2 remain. |
| 10 — Internal/integration docs and notices | The mixed dependency boundary and pinned ConfigKit attribution match the selected graph. Accepted ADRs, policy, and project license/grant were not modified by the sprint. |
| 11 — Final verification | Full tests, vet, tidy, native build, dependency inspection, and provider-free validation/effective/explanation commands pass. The final documentation review missed CK-1 and CK-2. |
ConfigKit Review Validation
GOTOOLCHAIN=go1.27.0 GOWORK=off go test ./...passed.GOTOOLCHAIN=go1.27.0 GOWORK=off go vet ./...passed.GOTOOLCHAIN=go1.27.0 GOWORK=off go test -race ./internal/core/config ./internal/clipassed.GOTOOLCHAIN=go1.27.0 GOWORK=off go mod tidy -diffproduced no changes.- A native build in a temporary directory succeeded. Minimal, complete, and development configuration validation, plus application and PromptKit effective and explanation reports, succeeded without provider calls.
- Selected module inspection confirmed the four approved ConfigKit, PromptKit, and YAML versions. Pinned local upstream documentation and notices were checked.
- Temporary fixtures outside the repository reproduced both documentation gaps and verified combined identifier/reference-path normalization. No production pipeline, credential lookup, or live model-quality evaluation was performed.
- The review document passed
git diff --check; production code and current operator documentation were not changed by this review.
Keep the earlier PromptKit findings separate from these two documentation fixes. The deferred PromptKit/ConfigKit v1.2.1 upgrade and release work remain out of scope.