From 93a080059d375a388ebfa713f44fa37c52097c6d Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 11 Nov 2026 04:00:47 +0000 Subject: [PATCH] Plan correction repair implementation --- docs/roadmap/correction-repair.md | 68 +----- docs/roadmap/implementation.md | 367 ++++++++++++++++++++++++++++++ 2 files changed, 372 insertions(+), 63 deletions(-) create mode 100644 docs/roadmap/implementation.md diff --git a/docs/roadmap/correction-repair.md b/docs/roadmap/correction-repair.md index 8a38220..654f5c6 100644 --- a/docs/roadmap/correction-repair.md +++ b/docs/roadmap/correction-repair.md @@ -2,11 +2,8 @@ **Updated:** 2026-11-11 -**Status:** Feature direction and all six decisions accepted; implementation -pending. No open feature decisions remain. Configuration names, concrete schema -fields, and adapter details remain implementation-planning work. This document -owns the target and remaining work, not a release date or implementation status -for the completed library adoption. +**Status:** Feature direction and all six decisions accepted. Implementation +is pending. See the [implementation plan](implementation.md). ## Objective @@ -132,8 +129,7 @@ Use a structured response that distinguishes revision from withdrawal and carrie the requested proposal reference. A revision contains the correction data needed by the existing validators. Empty replacement text remains a possible span deletion; it must not double as a withdrawal signal. The response schema owns -shape validation; Audita owns correspondence and domain acceptance. Exact field -names can be selected with the schema implementation. +shape validation. Audita owns correspondence and domain acceptance. ### Validator-Owned Feedback @@ -168,12 +164,8 @@ request commands expose the configured prompt assets through their normal APIs. PromptKit already supports [appending rendered assistant and corrective messages](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/171fb11f4e967a1da3af57843c6467954b14ba09/docs/consumers/pkg-promptkit.md#append-already-rendered-messages). Use its public preparation and execution APIs to render the module repair -definition and assemble the intended conversation. Appended messages are already -rendered; appending template text alone will not expand its variables. Confirm -the full-history path and per-call response-schema selection with a focused -integration example before settling adapter details. If that exposes a genuine -upstream gap, document a small reproducer and request focused support rather than -introducing another renderer or provider client. +definition and assemble the intended conversation from rendered messages. Use +its per-call schema selection for the revision contract. Maintain the existing distinctions between recoverable model-operation failures, fatal application/invariant failures, and cancellation. Domain revision attempts @@ -182,52 +174,6 @@ structural repairs; account for these separately. Share existing backend admissi and cancellation instead of adding a repair-specific global concurrency limiter. Use the accepted [reporting scope](#ordinary-reports-and-attempt-history). -## Work From the Baseline - -1. **Prove the request path.** Exercise native message composition with an - original multi-proposal response, a rendered module repair instruction, a - revision response schema, and a subsequent branch turn. Confirm operator - prompt replacements and the same behavior with debug disabled. -2. **Carry conversation state through the existing boundaries.** Extend - `framework/structuredoutput`, the LLM adapter, proposal-generation results, - and module/framework contracts as needed to retain and replay the original - request and full response. Preserve source-reference mapping and metadata. - Use compact lineage data and shared immutable initial context where practical; - no particular storage layout or new package hierarchy is prescribed. -3. **Add module repair assets and the response schema.** Extend the prompt - registry, embedded definitions, shared fragments where useful, and - `framework/responseschema`. Provide revision/withdrawal decoding and - correspondence checks without changing ordinary span-deletion semantics. -4. **Adapt feedback collection.** Update validator findings where their current - messages are insufficient for repair. When repair is enabled, collect the - agreed applicable findings and distinguish failures, unavailable decisions, - and checks that could not run. With an effective budget of zero, use the - existing validator filtering. Keep complete validation mandatory for approval. -5. **Implement the bounded branch loop.** Integrate revision calls and full - revalidation into the current stage/section flow. Preserve accepted proposals, - branch isolation, deterministic application, cancellation, and existing - apply-time checks. Unsuccessful branches must not discard unrelated accepted - work. This does not require redesigning stage barriers or chunk scheduling. -6. **Wire configuration and readiness.** Add the accepted - [budget](#semantic-repair-budget-and-configuration-scope) - and [profile](#repair-execution-profile) settings with native presence and stage - inheritance. Keep semantic attempts distinct from structural - `llm.repair_attempts`. Preflight checks any additional required role/prompt - readiness when enabled, without making disabled repair require an unused - profile. Let existing inspection expose resolved settings. -7. **Extend outcomes and documentation.** Implement the accepted reporting scope - through ordinary reporting, correction ledgers, and optional redacted debug - artifacts. Update - the canonical [configuration](../config.md), [pipeline](../internal/pipeline.md), - [module](../internal/modules.md), [validator](../internal/validators.md), - [prompt](../internal/prompts.md), [runtime](../internal/llm-runtime.md), and - [output](../integrations/output-files.md) documentation when behavior lands. - Record significant ownership decisions through the documentation policy's - ADR process if warranted; routine API and template choices need no ADR. - -These are work dependencies for later implementation planning. They do not -mandate separate commits, a general workflow engine, or a new release process. - ## Acceptance and Completion Use controlled model responses and real loading/rendering at integration @@ -287,8 +233,6 @@ the operator configures more attempts. Use a distinct setting from structural Retain existing per-call timeouts and parent cancellation, and document that each semantic attempt may include transport retries and structural repairs. This provides useful feedback-driven revision with a simple finite bound. -The default and scope are accepted; exact configuration names remain an -implementation-planning choice. ### Provider Request Limits @@ -337,8 +281,6 @@ operational errors and recoveries through the existing report framework and keep the current exit-status policy. Editorial rejection and voluntary withdrawal are correction outcomes. Keep conversation text and per-attempt details in optional redacted debug artifacts. -Choose the report location, fields, and any schema version change during -implementation planning. ### Revising the Span or Target Reference diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..abf43e8 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,367 @@ +# Correction Repair Implementation Plan + +Implement the stages in order, one assignment at a time. The [feature +roadmap](correction-repair.md) owns the intended behavior and operator choices. +The implementation baseline is `b036336`, with PromptKit v0.21.0 and ConfigKit +v1.2.0. Start with the [contributor guide](../development.md) and read the +applicable policies and component contracts as the work reaches them. + +This is one end-to-end sprint. Intermediate stages may leave callers or tests +pending their later stages. Carry those changes forward directly and run a +focused check when its boundary is assembled. Complete the assembled checks in +Stage 9. Keep the existing fixed chunks, stage barriers, module scopes, provider +admission, and apply-time safeguards. Decision recovery for unavailable +validator responses remains [deferred](future.md#domain-aware-response-recovery). + +## Stage 1 — Define the repair response and prove native conversation composition + +1. In `internal/framework/responseschema`, register `CorrectionRepairKey` with + schema ID `audita.correction_repair`, name `audita_correction_repair_v1`, and + asset `correction_repair.v1.json`. Use a closed JSON object with three + required fields below. + + - `proposal_ref` is the one-based position in the **original correction + array**, including items the application later rejects. Each turn echoes + the same short reference for that proposal. + - `action` is `revise` or `withdraw`. + - `correction` is a nullable object. Its object shape uses the existing + correction item fields `id`, `original_text`, `corrected_text`, and + `confidence`. The `id` refers to a segment position in the original + section, independent of `proposal_ref`. Keep the existing correction + field constraints, including an empty replacement and a whitespace-only + original span. +2. Decode the response at the repair boundary. Require the expected reference + and the consistent action/payload pair (`revise` with an object, + `withdraw` with null). Treat an inconsistent pair or wrong reference as an + unusable response with the existing `invalid_output` reason. Keep source + span, segment applicability, and confidence threshold decisions with domain + validation. PromptKit owns structural validation and its configured repair + attempts. +3. Add a focused offline test at the PromptKit integration boundary, using the + new schema and a fixture with a multi-message proposal and a separate repair + definition. Use the original proposal ID and variables as the base of each + native request. Render the repair definition with `Engine.Prepare`. Append + the full assistant response from the proposal call, then the rendered + repair instruction using `RunRequest.AppendedMessages`. Set `Validation` + to the repair schema and execute via `PrepareExecution` and `RunPrepared`. + Exercise a second turn by appending the prior instruction and its full + assistant response once, followed by the newly rendered instruction. The + fixture should check the actual messages and schema received by a controlled + transport, including use of a separate repair profile. +4. Use this test to prove that the pinned API can reproduce the initial + prepared prefix and use a different output contract. The native renderer + uses the captured prompt and supplied template variables, and appends the + already-rendered suffix after that definition's messages. Check roles, + contents, and ordering in the prepared request. If the test exposes a + genuine gap, record the failing case and reproducer for a focused upstream + change. Keep rendering and execution with PromptKit. + +## Stage 2 — Add and register the four module repair prompts + +1. Add the definitions `audita/modules.glossary.repair`, + `audita/modules.homophones.repair`, `audita/modules.spoken_word.repair`, + and `audita/modules.grammar.repair` beside the corresponding proposals in + `internal/prompts/assets/prompts`. Register them through `internal/prompts` + and the existing embedded PromptKit underlay. Each built-in module supplies + its repair prompt ID through a small `RepairPromptID() string` capability. + Retain the existing proposal interface for callers that only generate + candidates. +2. Use a user message in each shipped repair definition. Set its native output + contract to JSON schema using `correction_repair.v1.json` so standard + PromptKit inspection can prepare it with its own inputs. Put shared response + instructions in `RepairContract` and expose that fragment through the + existing shared-variable helpers. Render `RepairRequestJSON` from the + current proposal reference, current local segment reference, original and + replacement spans, confidence, and ordered findings. For findings, include + validator key, reason code, and available explanation. Convey an unavailable + assessment as such using safe facts from its decision or operation. +3. Make the instruction unambiguous after the earlier correction-set exchange. + Identify which proposal and current version it addresses, ask for one + revision or withdrawal, and describe the separate proposal and segment + references. Keep each module's existing edit scope, literal matching, + glossary/protection rules, and span deletion semantics. Treat transcript, + earlier model responses, and validator explanations as data in the prompt. +4. Cover schema shape, action and reference correspondence, empty-string + deletion, and real rendering of all four registered assets. Use the standard + inspection path for a replacement definition and a configured message file + through the existing source-loading path. The assembled branch test will + cover execution with that replacement. + +## Stage 3 — Carry the original exchange through proposal generation and the adapter + +1. Use `internal/framework/structuredoutput` for internal conversation data + shared by the role client and runner. Extend the existing structured request + with optional repair replay data. Keep the repair definition as the + requested prompt ID in Audita and carry the original proposal ID, frozen + effective variables, and original prepared messages in a shared read-only + exchange. Retain the full final assistant content used to decode the + original correction set, including when PromptKit structurally repaired + earlier output. Capture the application-level exchange at the existing + prepared execution boundary. +2. Pass the exchange from `GenerateCandidates` through + `ExecuteModuleProposal` and `contracts.ProposalResult` to the section + worker in `runner`. Capture it on a successful proposal response when the + effective repair budget is positive, independently of debug settings. + Reuse the existing request-local segment mapping and keep each proposal's + one-based position in the response aligned with its runner-assigned + proposal index. Hold one initial exchange per section and share it among + branches. Keep the exchange out of ordinary serialized results and release + it when the section's repair work finishes. +3. In `internal/framework/llm/adapter.go`, render the current repair definition + against the repair profile via the same engine and append its rendered + message block to the branch suffix. Form the final request from the + captured proposal prompt ID and variables, the original assistant response, + prior turns from that branch, and the current instruction. Pass the repair + profile, per-call repair schema, timeout, transient retry budget, and + structural repair budget through the existing role client. Preserve + the original message roles and contents. Use the prepared handle that + passed the original-prefix comparison for execution. Treat a preparation + invariant failure as an application failure and discard the unused handle. +4. Return the rendered repair instruction block and the final response for + this turn, so a subsequent attempt appends that exact exchange once. Keep + the repair definition ID in debug attribution alongside native prepared + metadata for the base prompt. Reuse typed failure and terminal count + handling. Retain the existing distinction between caller cancellation, + local setup or invariant errors, and provider errors. An upstream API + rejection of a repair request follows the recoverable branch path even + when PromptKit supplies no usable model output. +5. Exercise the role client with the test from Stage 1 using a real registered + repair asset and an operator replacement. Verify full history, isolated + turns, selected repair backend/profile, and counts for one logical attempt + that uses a native retry or structural repair. Verify capture without + `--debug` and ordinary proposal behavior with repair disabled. + +## Stage 4 — Resolve configuration and wire the repair role + +1. In `internal/core/config`, add `llm.correction_repair_attempts` as a + nonnegative integer, default `1`, and `llm.correction_repair_profile` as an + optional profile selection. Reuse the stage `llm` block. Omitted stage + attempts inherit the effective root value. An explicit stage zero clears + the inherited budget. Use ConfigKit's existing scalar validation and + presence rules, with no extra maximum for the new attempt count. +2. In `resolveSettings`, follow the recorded validation-profile pattern. + Resolve the effective stage proposal profile first. An explicit repair + profile selects that profile. An omitted stage repair profile inherits + an explicit nonnull root repair profile, otherwise it falls back to the + stage's effective proposal profile. An explicit stage null clears the root + override and uses that same fallback. At root, omission or null falls back + to the effective root proposal profile. Preserve recorded origins and + detached profile pointers. Use existing YAML, overlays, and native + application configuration inspection. +3. Extend `CheckReadiness` and the CLI processing wiring in + `internal/cli/process_inputs.go` and `process_pipeline.go`. Check the selected + repair profile and backend only for stages with a positive budget. The + existing captured engine validates registered prompt assets and their + overrides during loading. Retain that path for repair prompts. Allocate + a repair role client in the module run spec only for enabled stages, + using the same runtime, stage execution settings, admission, and optional + observation hooks. Unused repair profile IDs require no readiness check + when repair is disabled. A normalization-only run has no repair role. +4. Test root/stage inheritance including different proposal profiles, null and + explicit overrides, zero and positive budgets, native effective inspection, + and readiness before output preparation. Use the existing configuration + version and presence conventions. + +## Stage 5 — Collect usable findings while preserving approval semantics + +1. Refactor `validateSectionCandidates` in `internal/framework/runner` so the + initial pass and later revisions use the same validation code. Keep the + existing shortlist flow when the effective repair budget is zero. In + feedback mode, track decisions by proposal index and version. Use + the current validator order and batching. Retain approval eligibility only + for candidates that satisfy all required checks. Collect additional findings + separately for candidates already rejected. +2. Make applicability explicit for the built-in checks using the current + candidate and the original section. + + - Shape can evaluate malformed candidates. No-effect can evaluate literal + equality on a usable span, and confidence can evaluate a valid score + independently of target or span placement. + - Original-text presence needs a meaningful target and nonempty span. + Use the existing replacement preview and `ambiguous_original_text` reason + to provide an actionable finding when the span has multiple matches. In + feedback mode, record it before withholding the candidate from semantic + validation. A shape verdict for an unmapped reference already supplies + target feedback. + - Evaluate protection where the existing rule has usable before/after + evidence. Preserve active glossary match boundaries and spoken-word + repetition rules when they require full-segment context. Run the + nonempty-result guard when a replacement can be located, including a + preview that would leave a blank segment. + - Reuse the preview for semantic validators when it supplies usable + before/after text. Continue across independent semantic checks after + an actionable rejection or unavailable decision when the candidate still + supplies that input. Preserve the existing model batching budget. +3. Apply internal decision cardinality to the candidates actually submitted to + each validator. A skipped check supplies no decision. Keep missing, + duplicate, and unusable model decisions as unavailable under existing + reconciliation. Retain recoverable operation evidence. Route fatal errors + and cancellation through the existing peer cancellation and join path. +4. Build feedback from validator decisions and safe optional evidence that the + validator owns. Reuse existing reason codes and messages. Supply specific + evidence where a reason alone gives too little direction, such as the + applicable confidence threshold or a protected spelling when identified. + Use normal LLM decision explanations for semantic judgments. Preserve + debug redaction and avoid putting raw provider errors into feedback. +5. Test two independent meaningful findings after one rejects, a usable + rejected candidate that reaches the semantic checks, invalid target and + ambiguous span handling, and a no-effect candidate with no repair work. + Confirm that an unavailable decision alone remains unavailable and that + the zero-budget pass calls later validators with the existing shortlist. + +## Stage 6 — Run independent repair branches and apply accepted revisions + +1. Extend the section validation worker in `runModulePipeline` after its + initial pass. Keep the approved candidates as they are. For each candidate + with an actionable rejection, initialize an independent branch on the + section exchange. No-effect and unavailable-only outcomes finish without a + branch. When an actionable rejection coexists with unavailable validation, + use the known findings and show the missing assessment as unavailable. +2. Visit repairable candidates in proposal-index order within each section + worker. Reuse existing section concurrency and engine admission. Each + logical repair call consumes one attempt from that proposal's budget. + Track attempts separately from native transient retries, structural + repairs, and revalidation calls. Send the full conversation to the selected + provider without local size checks or truncation. The validation batching + budget still applies to validator calls. End the branch on a provider error + after applicable native retries, including context-window rejection. Leave + other proposals eligible to continue. Keep the existing fatal/cancellation path for app + failures and parent cancellation. +3. Accept a valid withdrawal as a terminal correction outcome. For a + structurally usable revision with a matching proposal reference, map its + segment position using the original section, retaining the original + stage, chunk, and proposal index. Allow a changed target or span inside + that section. Run the full applicable chain from scratch against the + original section. Place a fully approved revision into the original + candidate's application position. A still-actionable rejection can take + another turn if budget remains. Use only the current version's findings + for the next instruction, retaining the earlier turns in that branch's + history. +4. End a rejected revision that has no further actionable finding with a + rejected disposition, and record an actionable rejection at the budget + boundary as exhausted. An unavailable-only revalidation, recoverable + repair-call failure, or unusable response ends the branch as unavailable. + Use the interrupted disposition when fatal failure or cancellation leaves a + branch in progress or prevents an approved revision from reaching + application. Preserve already-terminal dispositions. A revision needs its + own approval, and each response can supply at most one candidate. +5. Merge initial approvals and accepted revisions into the existing section + application path, keeping ascending proposal indices and all apply-time + checks against the text produced by earlier edits. Resolve applied versus + skipped status from the actual application result. Keep the stage-level + commit barrier. Count filtering once for a defective original that remains + excluded, using its original filtering reasons. An approved revision takes + that candidate's place in the existing application accounting. Attribute + defects in a later repair response to its own operation. Keep one terminal + proposal outcome per branch, with superseded versions available as debug + evidence. +6. Test a response with approved and rejected proposals on the same segment, + a successful retargeted revision, a valid span deletion, withdrawal, + exhaustion across two attempts, and overlapping edits where a repaired + candidate passes validation and then skips at application. Check unrelated + approvals survive a recoverable provider failure. Confirm revalidation + revisits a validator that approved an earlier version. + +## Stage 7 — Account for operations and publish compact ordinary outcomes + +1. Add `correction_repair` as an operation kind in + `internal/framework/outcomes`. For each repair call retain its proposal + identity, attempt number, operation counts, and safe terminal reason. + Attribute revalidation calls to the proposal and attempt internally so + their local batch indices can repeat safely. Keep the normal proposal + and initial validation accounting, including proposal response counts. + Extend deterministic report sorting + to visit initial proposal and validation evidence first, then repairs by + proposal, attempt, and validator/batch order. +2. In `processreport`, `reporting`, and the runner's work-unit results, + count repair and revalidation executions as logical operations, including + failed provider requests. Keep existing accounting for pre-call failures. + Use the existing work-unit error paths for recoverable model + failures and the run-level path for fatal or canceled work. Keep failures + from earlier attempts visible when later work succeeds. A usable revision + or withdrawal is a successful model operation. Use existing native retry + and structural-repair reasons where applicable. Add `correction_revised` + to the repair operation that yields + a fully approved revision. This recovery identifies validation success. + Actual application is reflected in the final branch disposition and the + existing correction count. A clean withdrawal or ordinary rejected revision + adds no error or recovery by itself. +3. Add a single optional **Correction repair** section to `report.md`, with + one row per branch that attempted a repair. Use the same stage and chunk + numbering as the matrices, the original internal proposal index, the + semantic attempt count, and a final outcome. Use `applied`, + `approved_skipped`, `withdrawn`, `rejected`, `exhausted`, `unavailable`, + and `interrupted` as applicable. Derive the row from compact work-unit facts + after application or from the available interrupted evidence. Sort by stage, + chunk, and proposal index. Keep the section focused on these fields. +4. Keep the existing three output files, current YAML envelope versions, and + existing YAML fields. Document `correction_repair` and + `correction_revised` as additions to the operation and reason vocabulary. + Keep compact branch rows in the Markdown report. Reporting-disabled runs + still compute the existing CLI counts. Keep debug capture independently + selectable. +5. Verify the report for successful application, approval followed by skip, + withdrawal, exhaustion, provider failure, and partial evidence on fatal + interruption. Check that semantic attempts, actual edits, and PromptKit + retry/repair counts are distinct and that both YAML summaries agree with + the CLI on the counts they share. + +## Stage 8 — Retain optional repair evidence and preserve ledger lineage + +1. Extend the current interaction writers and logical execution observation + data with a repair role, original proposal index, and one-based semantic + attempt number. Use generated stage instance, section, proposal, and + attempt coordinates for repair filenames and include validator and batch + coordinates for revalidation. Original stage labels remain data. Reuse the + existing diagnostics failure and credential redaction path, including the + selected repair role's prepared credential references. +2. Save each repair attempt's rendered conversation, response, feedback, + and validation evidence through existing debug JSON artifacts. Keep these + writes conditional on an active debug directory and keep runtime repair + functional when preparation or writing of optional artifacts fails. +3. Extend `processreport.BuildCorrectionLedger` to emit one final row per + repaired proposal keyed by the original stage instance and proposal + index. Represent the original proposed edit and the final applied edit + separately, including when the revision retargets the proposal. Use + compact optional repair attempt metadata and artifact references to + connect versions and their respective decisions. Keep initial proposal + rejections as evidence within that lineage. Preserve the existing ledger + behavior for untouched candidates and the indexed interrupted join. +4. Test several repairs against a single segment and revalidation batches + that reuse local indices, then verify unique debug artifact names and + unambiguous ledger attribution. Check the terminal row and prior + decisions on interruption. Check redaction with synthetic credentials in + captured interaction material and verify that ordinary reports contain + only safe outcome data. + +## Stage 9 — Validate the assembled command and update canonical documentation + +1. Exercise the normal processing command with controlled provider responses, + real configuration and prompt loading, and the output session. Cover a + mixed proposal set with two independent repair branches, a second turn + on one branch, provider context rejection on the other, full revalidation, + and preserved original approvals. Inspect the wire order and contract. + Run the relevant behavior with debug off and with an operator repair prompt + replacement under the usual underlay. Cover zero budget and independent + report/debug selection. Reuse lower-level cases for individual failure and + disposition variants. +2. Add a controlled cancellation case during a repair or its revalidation. + Confirm active work joins, no completed transcript is published, and + failed reports preserve available partial evidence. Use the existing + subprocess fixtures and output assertions where they express the contract. +3. Update the owning [configuration](../config.md), + [pipeline](../internal/pipeline.md), [modules](../internal/modules.md), + [validators](../internal/validators.md), [prompts](../internal/prompts.md), + [LLM runtime](../internal/llm-runtime.md), and + [output files](../integrations/output-files.md) guides. Explain the default + attempt, per-stage control, profile fallback, complete replay to the + selected repair provider, provider failure behavior, and compact reporting. + Add minimal configuration or preview examples where needed in the existing + maintained material. Update [operations](../operations.md) if the new debug + evidence needs operator guidance. Keep the deferred decision-recovery + entry aligned with the implemented scope. +4. Validate examples and local links. Run the full offline suite and the + assembled CLI build under the [contributor guide](../development.md), + including race detection for the section workers. Check the focused + provider and ledger regressions in the normal suite. The sprint ends + with the implementation and its evidence.