From 3eacb33eb01363d42268d19faa3321bb521b0549 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 10 Nov 2026 20:12:35 +0000 Subject: [PATCH] Record the algorithmic audit findings --- docs/roadmap/remediation.md | 290 +++++++++++++++++++++++++++++++++++- 1 file changed, 284 insertions(+), 6 deletions(-) diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md index 81c82ff..184709a 100644 --- a/docs/roadmap/remediation.md +++ b/docs/roadmap/remediation.md @@ -11,6 +11,12 @@ provider. A normal grammar correction produced the same transcript and completed both validators on each revision. Differences at the output, credential, and glossary boundaries are recorded in R10–R12. +The algorithmic audit traced the implementation and its pinned library calls +without performance measurements. R13–R20 identify avoidable work within the +adopted design. Fixed transcript chunks, ordered stages, native provider +admission, glossary selection and matching rules, and report schemas remain +the basis of the analysis. + ## Architecture Assessment ConfigKit selects and composes the v3 application configuration, including @@ -30,12 +36,12 @@ validation, source publication, and asynchronous Linux delivery. | Stages | Reviewed behavior | | --- | --- | | 1 and 15 | Dependency and toolchain pins, release profile, launcher, tag verification, and candidate evidence. A disposable ReleaseKit check passed with Go 1.27.3, including race tests, vet, Markdown and module checks, both Linux builds, and version output. The plan records the annotated-tag delivery rehearsal. | -| 2–4 | Native application load, main-relative engine inputs, prompt resources and source accounting, prepared role clients, retries and repairs, and redaction. See R2, R4, and R10. | -| 5–6 | Native app and PromptKit scopes, typed routing, preview, validation, and explicit smoke, including controlled transport tests. See R5. | -| 7–8 | Destination checks, deferred device/pipe delivery, transcript replacement, ordinary reports, optional debug, and reporting modes. See R1, R7, R9, and R11. | -| 9–10 | Native identified glossary composition, typed extensions and selections, recorded defaults, and explicit clearing after CLI assignments. | -| 11–12 | Stage-specific glossary views, late-stage readiness, fixed chunks, ordered mutation, and partial success. See R3, R9, and R12. | -| 13–14 | Stage attribution, report schemas, canonical docs, migration, examples, ADR, and attribution. See R1, R6, R7, and R8. | +| 2–4 | Native application load, main-relative engine inputs, prompt resources and source accounting, prepared role clients, retries and repairs, and redaction. See R2, R4, R10, R13, and R14. | +| 5–6 | Native app and PromptKit scopes, typed routing, preview, validation, and explicit smoke, including controlled transport tests. See R5 and R13. | +| 7–8 | Destination checks, deferred device/pipe delivery, transcript replacement, ordinary reports, optional debug, and reporting modes. The output-path loop compares a growing source list against at most four destinations. See R1, R7, R9, R11, and R18. | +| 9–10 | Native identified glossary composition, typed extensions and selections, recorded defaults, and explicit clearing after CLI assignments. See R15. | +| 11–12 | Stage-specific glossary views, late-stage readiness, fixed chunks, ordered mutation, and partial success. See R3, R9, R12, R15–R17, and R20. | +| 13–14 | Stage attribution, report schemas, canonical docs, migration, examples, ADR, and attribution. The complete stage-by-chunk matrices are part of enabled reports. See R1, R6–R8, R18, and R19. | ## R1 — P2: Keep Failure Records and Counts Consistent @@ -417,3 +423,275 @@ An exempt full name can be corrected when a shorter protected term occurs inside it. A standalone occurrence of the shorter term and a distinct longer protected name retain protection. Other module stages keep their independently selected vocabulary. + +## R13 — P2: Collect Prompt Message Paths From Effective Data + +**Category** + +Repeated configuration history replay in Stages 3, 5, and 6. + +**Evidence and complexity** + +Audita enumerates historical fields and calls `Explain` for every effective +host-backed `content_file`, then reads only the effective YAML scalar. +ConfigKit v1.2.0's implementation of `Explain` replays source applications +and clones configuration checkpoints on each call. With `M` such fields and +`H` source history records, scanning history alone costs at least +`Ω(M × H)`. One file-backed message per source gives quadratic growth in +the number of sources, before accounting for replay and cloning costs. +The native resource commands also do the work because they build a runtime +and then retain only its engine. +[Source-path capture](../../internal/framework/llm/source_paths.go#L40) · +[Engine wiring](../../internal/cli/app_wiring.go#L62) + +**Refactoring opportunity** + +Collect collision paths only for processing. Use one walk of the native +effective projection with per-field origin lookups, or use an upstream +effective-only accessor if it can expose the required path evidence. Keep +the actual load and source-relative resolution with the owning libraries. + +**Acceptance** + +Source accounting for processing uses a single effective pass and origin +lookups without replaying history per message. Inherited and relocated +message files still receive collision protection, including the host context +covered by R2. Inspection retains its native history commands. + +## R14 — P3: Reuse Credential Source Discovery Across Diagnostics + +**Category** + +Repeated catalog scans in Stages 4 and 8. + +**Evidence and complexity** + +`ConfiguredSecrets` lists and inspects every backend and profile each time. +Debug processing calls it per proposal interaction and per validator in each +active section, including deterministic validators that never write a model +interaction. Finalization also recollects it for each warning. With `D` +redaction preparations and `C` catalog resources, this is at least +`Θ(D × C)` resource visits, plus the work of copying each inspection result. +[Credential collection](../../internal/framework/llm/secrets.go#L11) · +[Validator writer setup](../../internal/framework/runner/runner.go#L620) · +[Warning presentation](../../internal/cli/process_output.go#L14) + +**Refactoring opportunity** + +Discover deduplicated credential source names once per captured engine. +Read values at the existing masking boundary, including additional names +supplied by prepared requests and normalized values from R10. Reuse writers +or redactors where they already share a value set, and construct interaction +writers only for consumers that can use them. This reduces catalog discovery +to one pass while retaining the cost of reading the referenced values. + +**Acceptance** + +Increasing the number of sections or warnings does not cause repeated +resource inventory scans. All existing masking behavior remains available, +and PromptKit still controls credential selection and availability. + +## R15 — P3: Avoid Repeated Glossary Selection and Membership Scans + +**Category** + +Redundant list work and quadratic deduplication in Stages 10–12. + +**Evidence and complexity** + +- `resolveSettings` clones root lists before ConfigKit checks whether a stage + already supplied a list. Across `S` stages and `R` root list elements, + this copies `Θ(S × R)` elements even if every stage overrides both lists. +- `Select` walks a collection on every occurrence of its name. Selecting a + collection of `G` entries `K` times visits `Θ(K × G)` entries and + returns only `G`. Readiness also materializes a full rich selection just + to test whether it is empty. +- `Vocabulary` scans a spelling's existing contributor slice for every + insertion. When `K` distinct entries share an alias, the membership tests + for that alias alone take `Θ(K²)` comparisons. + +[Recorded defaults](../../internal/core/config/resolve.go#L21) · +[Glossary selection](../../internal/core/glossary/selection.go#L19) · +[Readiness](../../internal/framework/preflight/preflight.go#L95) · +[Contributor grouping](../../internal/core/glossary/vocabulary.go#L19) + +**Refactoring opportunity** + +Clone only lists that inherit. Skip repeated collection names before walking +their entries, and stop the readiness test at the first populated selection. +Use a membership set keyed by spelling and entry identity while keeping the +ordered output slices. That gives expected constant-time membership per +contribution, excluding the cost of normalizing and hashing the text. + +**Acceptance** + +First occurrence, contributor order, exact collection-scoped IDs, detached +inherited lists, and the distinction between omission and explicit clearing +are preserved. Readiness keeps the same late-stage diagnostics. + +## R16 — P2: Reuse Prepared Glossary Views and Index Validation Context + +**Category** + +Per-section rebuilds and per-candidate glossary scans in Stages 11 and 12. + +**Evidence and complexity** + +The selected stage vocabulary stays fixed, but protected-term validation +compiles its matcher for each active section. Each LLM validator rebuilds the +context matcher again for its section. With `Q` such invocations and +`B` the cost of building a matcher, that is `Θ(Q × B)` preparation for +the same data. Proposal requests also re-encode identical glossary JSON in +each section. Within LLM validation, after matching text, every candidate +scans all `G` selected entries to recover their order. That adds +`Θ(P × G)` work for `P` candidates, even when nothing matches. +[Protected-term validation](../../internal/framework/validators/deterministic.go#L139) · +[LLM validator request](../../internal/framework/validators/llm_validators.go#L252) · +[Context selection](../../internal/framework/validators/llm_context.go#L12) · +[Proposal variables](../../internal/framework/proposal_generation/variables.go#L14) + +**Refactoring opportunity** + +Prepare immutable stage-local matchers and, where used, serialized glossary +variables once. Build an identity-to-entry-position map alongside the context +view. Gather just the matched positions and order those positions before +materializing entries. After indexing, selecting `k` distinct matches per +candidate costs expected `O(k + k log k)` apart from text matching, rather +than a full glossary scan. Sharing the encoding saves local serialization +while leaving the selected glossary payload in each request. + +**Acceptance** + +Rich entries retain selection order and all contributing identities. Scope +and overlap rules remain as specified, including R3 and R12. Matchers and +immutable render inputs are shared safely across sections of the same stage. + +## R17 — P3: Reuse Validation Payload Costs While Fitting and Batching + +**Category** + +Quadratic prefix serialization in the existing budgeting path now used with +stage-specific budgets and shared glossary context. + +**Evidence and complexity** + +`fitValidationContext` removes one optional entry at a time and serializes +the remaining item again. Removing most of `K` similarly sized entries +serializes `Θ(K²)` entry data. `ChunkLLMValidationItems` copies, renumbers, +and marshals the entire tentative batch on each append. A batch that grows +to `N` similarly sized candidates costs `Θ(N²)` in those operations. +Small budgets bound the batch size, but the configured budget may grow and +many glossary entries can match one candidate. +[Context fitting](../../internal/framework/validators/llm_context.go#L43) · +[Batch construction](../../internal/framework/validators/llm_batching.go#L16) · +[Stage budget use](../../internal/framework/validators/llm_validators.go#L74) + +**Refactoring opportunity** + +Keep the existing drop order, greedy batch boundaries, and production +estimator. For that estimator, compose counts of whitespace-delimited runs +and punctuation from encoded components, accounting for join boundaries +and rounding once on the complete payload. Use those costs for prefix +selection and incremental batch construction, then render the selected +payload. Keep exact serialization for estimators whose costs cannot be +composed. This can reduce production work to one pass over the encoded +components and inexpensive checks over the candidate prefixes. + +**Acceptance** + +Representative payloads produce the same retained context, estimates, +oversized-candidate decisions, and batch boundaries. Include local reference +numbering and punctuation or escaped text at encoding boundaries. + +## R18 — P3: Skip Evidence Work for Disabled Outputs + +**Category** + +Unused report rendering and event collection in Stages 8 and 13. + +**Evidence and complexity** + +Preparation renders both initial report bundles and finalization renders +the final bundle even with reporting disabled. For `S` stages and `Q` +chunks, this still builds two `Θ(S × Q)` Markdown matrices at finalization, +in addition to serializing error and recovery records. Processing also +retains all detailed LLM observations, copies them at completion, and sorts +them before testing whether debug artifacts are active. For `E` calls and +events, that is `Θ(E)` storage and up to `O(E log E)` sort comparisons. +Ordinary outcome counts use the role client's terminal event, independently +of that detailed collector. +[Report preparation](../../internal/cli/process_inputs.go#L90) · +[Report delivery](../../internal/cli/process_output.go#L32) · +[Observation wiring](../../internal/cli/process_pipeline.go#L18) · +[Snapshot sorting](../../internal/framework/llm/observations.go#L148) + +**Refactoring opportunity** + +Build file payloads only when ordinary reporting is enabled. Install detailed +observation hooks only for an active debug writer, while retaining terminal +counts and compact completion facts for ordinary reports and CLI summaries. +Coordinate removal of unused utilization bookkeeping with R7. + +**Acceptance** + +Disabled outputs skip their formatting and optional observation costs. +Enabled reports and debug artifacts retain their data, and verbose completion +counts remain available with ordinary reporting disabled. + +## R19 — P3: Index Interrupted Correction Ledger Entries + +**Category** + +Repeated joins in the Stage 13 debug ledger. + +**Evidence and complexity** + +The first pass indexes decisions by proposal for each stage and discards the +index after emitting completed entries. Every interrupted candidate then +searches stage results and scans the matching stage's entire decision list +again. With `I` interrupted candidates, `S` stages, and `D` decisions in +one failed stage, lookup can cost `Θ(I × (S + D))`. Interruption late in a +large stage can make both `I` and `D` grow with candidate count. +[Decision indexing and interrupted fallback](../../internal/framework/processreport/correction_ledger.go#L64) + +**Refactoring opportunity** + +Reuse the first-pass decision index through the interrupted-entry join and +index stage metadata by generated instance. Expected lookup and grouping +become linear in stages, candidates, and decisions visited, plus the +existing final sort. Keep operator IDs as labels. + +**Acceptance** + +Interrupted entries retain the same stage identity, partial validator +decisions, and output order without rescanning decisions for each candidate. + +## R20 — P3: Skip Apply-Time Copies for Sections With No Approved Edits + +**Category** + +Avoidable transcript copying during Stage 12 application. + +**Evidence and complexity** + +The runner invokes application for every section even when its approved +proposal slice is empty. Application clones the section's segments and +categories and builds the segment-ID map before inspecting proposals. For a +stage with `N` segments and no approved edits, that is `Θ(N)` segment +copies and map entries, plus category copies, to preserve identical text. +Across repeated no-op stages this grows with the number of stages times the +transcript size. +[Section application](../../internal/framework/runner/runner.go#L289) · +[Per-section clone and index](../../internal/framework/proposals/apply.go#L42) + +**Refactoring opportunity** + +Retain the existing section when it has no approved proposals. Invoke +application and its protection checks only for sections with candidates. +Keep the runner's work-unit bookkeeping for every section. + +**Acceptance** + +No-edit and all-recoverable-failure runs produce the same normalized +transcript and reports. Sections with approved proposals still use ordered +application and recheck the current text before each edit.