diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md index 87112128..9c948076 100644 --- a/docs/roadmap/remediation.md +++ b/docs/roadmap/remediation.md @@ -10,14 +10,18 @@ Integration follow-up baseline: `fd134e85e54c6e0f74aab5c77df5bada7c1b1e6a`. The follow-up baseline adds only the initial review findings; application code is unchanged between these revisions. +Algorithmic follow-up baseline: `fdaf39162b46b528fddcd8a4cad18a09e1daf661`. +Only this findings document changed after the application baseline above. + This review followed the 46 stages in [the implementation plan](implementation.md) against the [native migration target](promptkit.md), current owning code, focused tests, maintained examples, and canonical documentation. The migration is substantially complete. Findings 1–2 come from the initial review. Findings -3–5 come from a second, stage-ordered pass through the interaction of the native -migration with existing domain, validation, and diagnostic behavior. None calls -for another compatibility layer or application execution abstraction. This -document records findings, not implemented fixes. +3–5 come from an integration pass. Findings 6–9 and the complexity note on +Finding 1 come from a stage-ordered algorithmic pass. The latter are refactoring +opportunities for consideration, not measured production bottlenecks or new +performance guarantees. None calls for another compatibility layer or application +execution abstraction. This document records findings, not implemented fixes. ## Finding 1: Retain Genuine Native Timeouts During Concurrent Error Selection @@ -69,6 +73,18 @@ application error taxonomy is needed. the existing native timeout fixture where useful. Run the pipeline tests and their concurrency/race checks. +### Algorithmic Follow-up: Select A Minimum Instead Of Sorting + +With `E` collected errors, the current selector comparison-sorts all eligible +records by `(stage, lane, chunk)` solely to return the first. It performs +`O(E log E)` comparisons; stable sorting can also incur additional superlinear +data movement. After eligibility classification, a minimum scan takes `O(E)` +comparisons and constant auxiliary space; error-chain classification costs remain +separate. Keep the first occurrence on equal keys, matching stable-sort behavior. +This can accompany the correctness +fix without changing error precedence or cancellation semantics. Its practical +impact is smaller than the document/cache indexing findings below. + ## Finding 2: Narrow The Documented Empty-Validator-Chain Guarantee Priority: P2. Relevant stages: 11, 38, 39, and 42. @@ -281,6 +297,160 @@ assessment/capture error types. Do not bridge native errors into these retired sentinels or add aliases for compatibility. Verify no current callers remain and run contracts, pipeline, CLI, and module integration tests after removal. +## Finding 6: Use Indexed Source Validation In The Chunk-Plan Hot Path + +Priority: P2. Relevant stages: 16, 24, 28–29, and 37. + +### Evidence And Complexity + +The supporting [source validation code](../../internal/core/source/validation.go) +predates this migration, but the new chunk-plan assessment/cache path repeatedly +uses it. `ValidateDocument`, lines 95–101, validates each unit's self-reference +with `ValidateRef`. That helper resolves both endpoints through the linear +`UnitIndex` scan. For a valid document of `N` units, the second validation pass +alone makes `2 * (1 + ... + N) = N(N+1)` unit-ID comparisons: `Theta(N^2)` work. + +[`ValidateChunkPlan` and `MaterializeChunkPlan`](../../internal/core/source/chunk_plan.go) +also repeatedly scan the source for each of `C` ranges. A materialization thus +performs `O(N^2 + CN)` source-position work in addition to field checking, +annotation processing, metadata ownership, and payload encoding. +The pipeline's +[`validateAndMaterializeChunkPlan`](../../internal/framework/pipeline/chunk_validation.go) +validates once explicitly and then calls a materializer that validates again. +The [new plan path](../../internal/framework/pipeline/runner_chunk_plan.go) +invokes this during assessment, cache encode/decode, and final delivery. Cached +plans therefore retain this deterministic cost even without inference. + +### Potential Refactoring + +Use the existing immutable `source.DocumentIndex` for bulk endpoint lookups. +Build/reuse a unit-position map while validating the document, then check its +self-references in linear expected time. Share resolved positions through a +small private validation/materialization helper so materialization does not +rescan endpoints or repeat the same validation unnecessarily. Keep standalone +public validation/materialization entry points correctly validating their inputs. + +The attainable position-lookup cost is `O(N + C)` expected work with `O(N)` +index space. Field/payload inspection, annotation canonicalization, metadata +ownership, and JSON encoding remain separate costs. This does not promise linear +total output size: permitted overlapping generic ranges can themselves require +superlinear copied output. Repeated materialization of accepted plans can be +reduced where already-owned checked chunks are available, without retaining a +whole-run ledger or removing necessary current-source checks. + +### Preserve And Verify + +Preserve document-order semantics for nonconsecutive/nonmonotonic unit IDs, +duplicate/missing/wrong-source/reversed-reference rejection, annotation handling, +deep ownership, and deterministic chunk IDs. Generic plans still permit their +existing gaps/overlaps; D&D coverage rules remain with scene interpretation and +configured validators. No stricter plan format or new size limit is needed. + +Use the existing source validation/materialization, chunk-plan operation/cache, +and chunk-map tests. Add only missing behavioral regressions; source inspection +or optional synthetic operation counts can verify the scaling change without +wall-clock thresholds or production data. + +## Finding 7: Share A Lazy Source Index Across Artifact Cache Decoders + +Priority: P2. Relevant stages: 11, 21–22, and 27–28. + +### Evidence And Complexity + +[`artifactProducerCache`](../../internal/framework/pipeline/runner_cache.go), +line 17, eagerly creates a full `DocumentIndex` for each model-backed operation +whenever a store collaborator exists. It does so before the producer helper +checks bypass/refresh mode, incoming reuse eligibility, or whether a module takes +a legitimate no-model branch. Cold misses also never use the resulting index. + +For `J` such operations against `N` source units, this adds `Theta(JN)` index +construction work. With a fixed number of lanes and chunks proportional to +transcript length, that cache-specific term becomes quadratic in `N`. +Concurrent operations can retain `O(min(J,W)N)` cache index entries while waiting +for model work, where `W` is the active worker bound. This is separate from +codec decoding and domain/validator input ownership costs. + +### Potential Refactoring + +Supply the cache decoders with one run-scoped, immutable source-position index. +Acquire it lazily only when a decoded hit actually needs evidence checks, using +ordinary concurrency-safe initialization. The cache-specific work becomes +`O(N + R)` for all hits, where `R` is the total number of checked cached source +references, with `O(N)` shared index space. Runs with no such checks need not +construct the cache index at all. + +This is a small in-memory collaborator, not another persistent cache or source +identity framework. Keep the native message-content key, modes, hit validation, +fallback eligibility, and no-semantic-reassessment policy. Finding 3's scene +identity adaptation is a separate usability concern; shared indexing must not +replace it. Other intentional domain/validator snapshots need not be weakened. + +### Preserve And Verify + +Retain wrong-source/missing/reversed citation rejection on hits, normal misses, +refresh/bypass/ineligible/no-model behavior, and canceled-hit delivery. Exercise +concurrent hits with the existing cache/runner tests and race checks. Do not add +a public index-build-count contract or a timing-based release gate. + +## Finding 8: Index Debug Sensitivity Paths Once + +Priority: P3. Relevant stages: 30–31 and 33. + +### Evidence And Complexity + +[`newRunEvidenceRecorder`](../../internal/cli/run_capture.go), lines 64–91, +collects all `H` historical configuration paths, then scans that full list for +each of `P` profiles/backends across root and collection resources. Prefix +matching therefore performs `O(PH)` work, plus path-string processing. With a +roughly fixed number of fields per resource, `H` grows with `P`, yielding a +quadratic resource-count term. + +The caller correctly constructs this recorder only when capture is enabled. +This finding concerns debug startup for large native configurations, not an +unconditional cost on ordinary runs. + +### Potential Refactoring + +Walk metadata paths once and group relevant sensitive suffixes by their exact +resource prefix. Then associate those groups with root/qualified resource IDs in +one resource traversal. This takes `O(H + P)` path/resource visits, plus processing +the path characters and retained masks, rather than rescanning every unrelated +configuration path for every resource. + +Retain native sensitivity classification, correct JSON-pointer segment unescaping, +root/collection distinctions, nested maps/arrays, and nonmutation of native +snapshots. Do not replace exact metadata with credential-like string searches or +broaden omission policy. Use the existing configuration/native capture tests; +add a mixed-resource case only if needed to verify equivalent mask routing. + +## Finding 9: Traverse Dense Chunk Results Without Sorting Their Keys + +Priority: P3. Relevant stages: 11, 25, and 37. + +### Evidence And Complexity + +[`finalizeLaneExtract`](../../internal/framework/pipeline/runner_concurrent.go), +lines 333–349, copies result-map keys to a new slice and sorts them before +assembling the lane. For `C` chunks this costs `O(C log C)` comparisons and +`O(C)` temporary key storage per lane. + +The runner's chunks are already assigned contiguous zero-based indexes by +`MaterializeChunkPlan`; successful finalization occurs only after all of that +lane's jobs return. The required order is therefore known, not arbitrary. + +### Potential Refactoring And Verification + +Traverse the known ordinal range directly for `O(C)` expected map-lookup work +and no temporary key slice, or use an indexed result slice if it simplifies +storage. Keep all artifact/diagnostic ordering, fallback counts, eligibility, +failure handling, cancellation draining, and ready-job scheduling unchanged. +No new chunker restriction is required: dense ordinals are already produced by +the existing materializer, regardless of transcript unit-ID spacing. + +The current runner concurrency and ready-job tests cover out-of-order completion +and public result order. Reuse those checks; do not couple tests to a particular +map/slice representation or add another scheduler. + ## Review Coverage And Verification Owning boundaries were inspected in implementation order: @@ -295,7 +465,7 @@ Owning boundaries were inspected in implementation order: | 35–37 | Removal inventory, maintained examples, assembled recovery and subprocess coverage. | | 38–46 | Successor ADRs and historical-body preservation, canonical references, operations/consumer/migration/internal/integration guides, orientation and final acceptance evidence. | -Checks completed during review: +Checks completed during the initial review: - `GOWORK=off GOPROXY=off go test ./...` passed. - `GOWORK=off GOPROXY=off go vet ./...` passed. @@ -308,7 +478,7 @@ Checks completed during review: - The migration diff contains no changes to model-facing JSON schemas or D&D intrinsic codec schema files. Historical ADR decision bodies are retained; supersession edits are confined to status descriptions. -- Temporary Go-overlay probes reproduced both findings without modifying +- Temporary Go-overlay probes reproduced Findings 1–2 without modifying repository code or tests. Their deliberately failing assertions are review evidence, separate from the passing default suite. @@ -331,6 +501,40 @@ application code and dependencies were unchanged. Additional checks were: These probes live only in a temporary review directory. No production code, test, configuration, schema, or existing canonical documentation was changed. +The algorithmic pass followed the same stage order, inspecting actual loop/call +costs rather than treating nested-loop graph metadata as proof of a bottleneck: + +- Stages 1–7: native loading/delegation, bounded invocation-selection reuse, + resource/input construction, and absence of a second application load. +- Stages 8–11: candidate serialization/isolation, complete-chain traversal, + native helper control flow, and runner/handoff state. +- Stages 12–24: fixed extraction mappings, bounded reconciliation preparation, + interval coalescing, indexed duplicate/group application, direct validators, + deterministic paths, and the source/materialization costs in Finding 6. +- Stages 25–29: constant-work ready-job dispatch, result/error ordering, + filesystem byte traversal, source-index lifetime, and reuse/auxiliary assembly. +- Stages 30–34: incremental capture, sensitivity-path routing, preflight, + command composition, diagnostics, and publication. +- Stages 35–46: retained/removal boundaries, example/assembled coverage, and + documentation consistency with the algorithms and selected policies above. + +Explicitly selected behavior was not reopened: complete validation on every +candidate, fresh normalization application to fixed originals, native growing +history, content-only cache identity, no semantic reassessment on hits, the ready- +job scheduling policy, and self-contained debug records even when content repeats. +The earlier expressly recommended per-handoff output index was also left alone. +Sorting genuinely unordered citations/intervals and collision-safe indexed domain +grouping are not findings merely because they contain loops or comparisons. + +Complexity claims in Findings 6–9 follow from the current source and call sites; +they are not throughput estimates. Position-map costs use ordinary expected +hash-map complexity; retained field/payload processing is accounted for separately. +The existing +offline suites for `internal/core/source`, `internal/framework/pipeline`, +`internal/framework/artifactcache`, `internal/framework/semanticreconcile`, and +`internal/cli` passed again. No production workload or performance data was needed, +and no new benchmark, timing assertion, or permanent gate was added. + No live provider inference was performed. This review does not establish production model quality or performance. Intentional changes such as the hard configuration/CLI cutover, native retry ownership, content-only cache identity,