Files
promptkit/docs/roadmap/remediation.md

295 lines
16 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Configuration Handoff And Repair Review
Reviewed implementation stages 1–17 against
[the implementation plan](implementation.md),
[the feature roadmap](next-release-considerations.md), and accepted ADRs
[0001](../adr/0001-configuration-boundary.md) and
[0002](../adr/0002-shared-configuration-data-model.md).
Implementation baseline: `40f8a6825359b4923645256e230a050dfdb00710`;
reviewed changes begin after `248d0db`.
## Findings
### 1. Narrow The Pre-Handoff Mutation Wording To The Accepted Expectation
Status: complete. Priority: P3. Implementation stage: 16.
Completion: narrowed constructor GoDoc and consumer guidance to the consistency
expectation; reviewed against the roadmap. No runtime behavior changed.
Locations: `engine.go:280` and `docs/consumers/pkg-promptkit.md:132`.
`NewFromConfig` says callers **must not mutate** selected settings between loading
and handoff; the consumer guide repeats that blanket instruction. The accepted
roadmap instead asks callers to hand off before changes that invalidate the
frozen inspection/provenance evidence, and expressly treats this as a usage
expectation rather than a new enforcement mechanism. The published wording
prohibits even value-preserving reassignment or copying, which needs no such
restriction.
Required change: describe the narrower consistency expectation in both owners.
Explain that construction consumes the resolved settings while inspection retains
the completed load's frozen evidence, so callers should keep those consistent at
handoff. Preserve the existing protection against mutation after construction.
No mutation detection, extra validation pass, or new error is needed.
Verification: review both passages against the feature roadmap's ownership
paragraph; ensure they no longer impose an unconditional mutation prohibition.
This is a documentation correction and does not require a new runtime test.
### 2. Keep Application-Field Sensitivity Policy With The Application
Status: complete. Priority: P3. Implementation stage: 16.
Completion: backend and profile guidance now assigns extension sensitivity and
disclosure to applications, retaining the credential/redaction warning. Reviewed
against the accepted extension contract; no runtime changes were needed.
Location: `docs/formats.md:235–236`.
The backend format reference says an unrecognized backend field is application
data and **must not contain secrets**. The accepted extension-point contract
assigns interpretation, validation, and sensitivity rules for those fields to
the application. The implementation follows that contract: inline fields remain
in native inspection, application sensitivity predicates compose with PromptKit's,
and those fields are omitted from resource reports and provider parameters.
The added prohibition unnecessarily narrows the application-owned data contract.
Required change: retain the useful warning that a raw `api_key` extension is not
a PromptKit credential input and is not automatically redacted by PromptKit.
State that the application owns sensitivity declarations and disclosure policy
for its extension fields, rather than prohibiting their contents. Review the
adjacent profile wording for the same distinction between PromptKit credentials
and application-owned data. Do not add automatic credential behavior or a new
validator for application fields.
Verification: compare the revised text with the feature roadmap's
application-specific-fields contract and existing focused-inspection sensitivity
coverage. No change to execution or redaction implementation is needed.
### 3. Preserve Effective Input Metadata Supplied By Application Defaults
Status: complete. Priority: P2. Implementation stages: 5–6 and 13.
Completion: capture reports effective required=true and preserves nullable input
metadata from typed values. Public regressions for Go defaults and recorded
collection defaults failed before the fix and pass afterward, alongside focused
race-enabled presence/normalization tests and offline preflight checks.
Locations: `internal/configuration/load.go:208–216`,
`internal/configuration/prompts.go:174–180`, and
`internal/configuration/resources.go:81–87`.
The new application-owned loading path preserves ConfigKit `Defaults` and
application resolution callbacks, but the existing input-inspection machinery
still equates `FieldInfo.Explicit == false` with an absent effective value.
`finalFieldState` consequently discards metadata for values supplied by Go
defaults or recorded derived defaults. These are valid final values, not missing
configuration.
Reproduced through the public API with a completed ConfigKit load whose
`Definition.Defaults` returns a prompt input equivalent to:
```go
promptkit.PromptInputSettings{
Name: "doc", Required: true,
ContentType: new("text/plain"), Description: new("default description"),
}
```
With a valid prompt and default profile, `NewFromConfig` succeeds. The final
typed input retains those values; native metadata describes their defaulted,
nonnull presence (`Explicit: false`, `Null: false`, default layer), but:
- `InspectPrompt` returns `Required: nil` instead of the effective true value.
- `InspectRequirements` correctly classifies `doc` as required but returns nil
content type and description, contrary to its effective-metadata contract.
- `ValidateRequest` without `doc` fails with `ErrPromptRender` and a missing
required-input error, confirming that execution uses the true value that
resource inspection fails to report.
A second public-API reproduction using an application `Resolve` callback and
`RecordDerivedDefault` for an omitted `required` and `content_type` has the same
result. Source-loaded inputs ordinarily have explicit metadata, which is why
the migrated source fixtures and current handoff tests do not expose this gap.
Required change: capture effective input metadata from the final typed values
and native final metadata without treating source explicitness as a universal
presence test. Preserve the existing ordinary omission/null and explicit
false/empty-string behavior. Keep source provenance in native inspection;
do not inspect raw sources, replay history, reject application defaults, or
rerun resolution to repair the report.
Verification: add public integration regressions for both Go defaults and
recorded application-derived defaults. Compare `InspectPrompt`,
`InspectRequirements`, and preparation/preflight for the same captured input,
and retain existing source-presence cases. Include a collection case if the fix
introduces separate root/collection handling; the current capture logic is shared.
### 4. Avoid Expanding Inheritance Provenance For Input-Presence Flags
Status: complete. Priority: P3. Implementation stage: 6.
Related to finding 3.
Completion: presence capture now uses typed values and `Metadata.IsExplicit`,
retaining only required-field flags. Removed the unused three-state metadata
records and full-origin queries from this phase. Focused race-enabled default,
inherited-input, explicit-presence, normalization, and requirements tests passed.
Locations: `internal/configuration/load.go:173–178` and `208–216`.
Capture calls `Metadata.Lookup` three times per declared input, then retains only
the `Explicit` and `Null` flags. In the selected ConfigKit v1.4.0 implementation,
`Lookup` also materializes and copies the field's complete inheritance-step list.
ConfigKit keeps that evidence compact internally; the full projection is useful
for operator inspection, but unnecessary for these flags. Its public
`Metadata.IsExplicit` method performs the effective-field query without that
projection.
Complexity: for `I` input declarations and inheritance depths `h_i`, these
presence queries perform `O(I + sum(h_i))` provenance work, apart from path
processing. A chain of `N` prompts inheriting a constant-sized input declaration
has depths `0..N-1`, producing quadratic ancestry-step work and cumulative
temporary allocations merely to capture linear-sized presence data. Reading one
metadata snapshot, as selected in the plan, does not eliminate this per-query
expansion.
Refactoring opportunity: combine effective typed values/pointer presence with
`Metadata.IsExplicit` where source explicitness is actually needed. Nullable
content-type and description pointers already distinguish retained values from
absent/null metadata for their public reports. Avoid requesting a complete origin
solely to classify presence, while correcting the default-value handling in
finding 3. Presence classification can be linear in the number of declarations,
plus ordinary path-query costs, independent of inheritance-chain length.
Keep full `Lookup` operations where source origins are genuinely needed for
resource binding. This finding does not propose a different inheritance algorithm,
metadata snapshot policy, inspection contract, or source-origin fallback.
Verification: preserve inherited, omitted/null, explicit-false, empty-string, and
application-default metadata outcomes. Confirm from the call path that the
presence-only phase no longer projects inheritance steps. No timing threshold or
real-world benchmark is needed to establish that algorithmic change.
### 5. Share Immutable Backend Defaults Across Profiles Without Overrides
Status: complete. Priority: P3. Implementation stage: 6.
Completion: profile construction now reads capture-owned backend definitions
directly and shares immutable provider maps when no overlay is needed. Registry
and runtime lookups remain defensive, and overlays still copy before merging.
Added lookup/overlay isolation coverage and updated internal documentation.
The full `go test -race ./...` suite and `go vet ./...` passed with Go 1.27.4.
Locations: `internal/configuration/targets.go:27–29` and `68–78`,
`internal/backend/registry.go:39–48`, and
`internal/domain/smoke_diagnostics.go:124–128`.
Every captured profile selecting a backend calls the registry's defensive-copy
lookup. That lookup deep-copies the backend's provider parameters, and the
profile retains the copied map even when it has no provider-parameter overrides.
Many profiles selecting one backend therefore retain many identical immutable
trees. This is an inherited cost reused by the new capture path, not a newly
introduced regression. The plan selects precomputed execution profiles and
ownership boundaries, not duplication of identical defaults within the owned
snapshot.
Complexity: for `P` profiles sharing one backend with `S` provider-value nodes and
no profile-level provider maps, this step costs `Theta(P*S)` copying time and
retained parameter storage, despite only `O(P+S)` distinct input data. The cost is
quadratic when both dimensions grow together. Existing ID sorting is additional
work and is not the target of this finding.
Refactoring opportunity: obtain an owned backend-default snapshot once per
backend and share its provider map read-only among profiles without overrides.
Keep separate merged maps when a profile needs its own provider parameters.
For the no-override case, default-map copying and storage can become `O(P+S)`;
the existing sorting and public result-copy costs remain unchanged. A small
construction-local cache or use of already owned immutable definitions is
sufficient; a general persistent-map framework is unnecessary.
Preserve defensive `GetBackend` semantics and copies at runtime/caller/client
boundaries. `GetProfile` already detaches the selected target's provider map, so
sharing construction-owned defaults need not expose shared mutable state to
consumers or merge caller requests into retained configuration.
Verification: review copy placement to establish the reduction, and retain
backend/profile mutation isolation, override-merge, prepared-state ownership,
and injected-client mutation coverage. Do not make internal pointer sharing a
public guarantee or introduce benchmark thresholds.
## Initial Review Results And Validation
- Stage 1: ConfigKit v1.4.0, unchanged Go minimum, and commit-pinned attribution
match the selected module and its notices.
- Stages 2–3: configuration and runtime inputs resolve mode-dependent repair
budgets; explicit zero and null/omission, whole-contract replacement, prepared
retention, and existing repair accounting are covered.
- Stages 4–9: shared settings aliases, composed stateless rules, final-metadata
resource capture, native focused inspection, explicit source mounting, and
shared constructor/assembly paths match the target architecture.
- Stages 10–15: migrated tests and maintained offline examples exercise the
changed boundaries, including concurrent loads, cross-filesystem inheritance,
application extensions, ownership, and standalone/handoff equivalence.
- Stages 16–17: canonical documentation is updated, subject to the two wording
corrections above. No additional functional or architectural defect was found.
Fresh checks passed with Go 1.27.4, `GOTOOLCHAIN=local`, and `GOWORK=off`:
`go test -race ./...` and `go vet ./...`. Changed Go files have no `gofmt`
differences, and the implementation diff passes `git diff --check`.
This review changes only this findings document; it does not implement fixes.
## Integration-Focused Follow-Up
The second review revisited stages 1–17 at their interaction boundaries: the
ConfigKit upgrade and native metadata semantics; repair-input conversion and
prepared budgets; shared settings and application callbacks; capture, inspection,
mounting, and constructor wiring; migrated tests and examples; and their contracts
with the existing runtime. Finding 3 records the newly reproduced integration
gap. Findings 1 and 2 remain unchanged.
Additional disposable external-package checks, outside the repository, passed
under the race detector for both `NewEngine` and `NewFromConfig`. They combined
configured default repair with continuation, appended messages, direct session
selection, namespace views, frozen prepared execution, structured output,
numeric-presence settings, usage/observer accounting, capacity release, and
single-pass smoke diagnostics. Both default-metadata reproductions failed their
expected inspection assertions, including under the race detector; those failures
are the evidence for finding 3, not failures in the existing test suite.
Fresh targeted race-enabled tests passed for the public handoff and runtime
contracts, configuration and use-case packages, maintained examples, capacity,
generation, observation, rendering, and validation. No other integration defect
was found. The prior full-suite and vet results remain applicable to unchanged
implementation code. No repository implementation or test files were changed by
this review.
## Algorithmic Review
The third review considered stages 1–17 in order, excluding algorithm choices
expressly selected by the plan. Findings 4 and 5 record two local optimization
opportunities; findings 1–3 remain unchanged. No additional ordinary correctness
bug was found.
- Stages 1–3: checked the selected metadata API costs, constant-time budget
defaulting/conversion, and the unchanged bounded repair/preparation paths.
- Stages 4–6: checked shared data declarations, per-resource normalization and
validation, known-value ownership traversal, presence capture, and profile
construction. Nested resource/input loops traverse distinct data; the two
findings concern hidden work in helper calls rather than loop syntax.
- Stages 7–9: native reporting remains delegated; diagnostic copying is linear
in returned records/locations, and the constructors add no repeated load or
configuration-sized work to ordinary execution.
- Stages 10–15: checked migrated test and example helpers for additional passes
or algorithm changes; they do not introduce another production loader or
runtime resource scan.
- Stages 16–17: checked the findings against the accepted ownership, inspection,
ordering, and configuration-boundary decisions. Deliberate lexical inventories,
native ConfigKit algorithms, and full-prompt repair remain outside reconsideration.
This was static complexity analysis of PromptKit and the relevant calls into the
selected ConfigKit release. No performance measurements or benchmarks were run.
Implementation code is unchanged, so prior functional validation remains
applicable; this pass validates the findings-document diff only.