From b904021138dc28765d47ac5f1580f6dd692ba9ac Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 20 Nov 2026 14:42:26 +0000 Subject: [PATCH] Resolve native correction prompts and validator guidance ownership --- docs/roadmap/promptkit.md | 139 +++++++++++++++++++++++++++++++------- 1 file changed, 115 insertions(+), 24 deletions(-) diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md index 218193e0..f2b4531a 100644 --- a/docs/roadmap/promptkit.md +++ b/docs/roadmap/promptkit.md @@ -145,8 +145,9 @@ CommandKit reporters. Pure domain packages retain artifact models, codecs, reference resolution, and deterministic rules. `internal/preflight` retains its separate known-dependency checks. Exact argument grouping and helper signatures can be refined against representative extraction and normalization during -implementation; ownership and placement are settled. Role-specific prompts, -budgets, and fallback policy remain U4–U6. Implementation is pending. +implementation; ownership and placement are settled. U4 selects prompt roles and +guidance ownership below; budgets and fallback policy remain U5/U6. Implementation +is pending. Retain the fixed `input -> chunk -> extract -> merge -> normalize -> output` shape, ordered steps and generated references, typed artifact-family boundaries, @@ -335,10 +336,11 @@ consumers disappear; no replacement filesystem abstraction is needed. Declare response format, validation mode, schema path, and structural-repair budget in native prompt definitions by default. Modules select prompt IDs, supply -inputs, and interpret responses. Initial and correction prompts may reference -the same schema file. Execution, inspection, and prepared-request preview then +inputs, and interpret responses. U4 selects the same schema file for each +initial/correction pair. Execution, inspection, and prepared-request preview then use the same declared contract without repeating schema selection in Go. -Role-specific prompt contents remain U4, and retry defaults remain U5. +U4 selects prompt roles and contracts; exact prose is implementation work, while +retry defaults remain U5. Use a request-level native `OutputContract` only for a demonstrated runtime- dependent contract. It replaces the whole configured contract, so supply the @@ -388,6 +390,84 @@ complete replacement model responses. Normalization may continue to return a reconciliation plan that deterministic code applies to produce the artifact. Per-entity branching is not part of this migration. +#### Prompt Roles And Correction Guidance (U4 Resolved) + +Give each correctable producer an initial native prompt and a domain-specific +correction prompt. Retain existing initial IDs and identify correction IDs +explicitly in the module, for example `notarius/dnd.spells.repair`. Preflight uses +those same declarations to check known prompt dependencies. Shared correction +wording can live in ordinary asset files; distinct domain/operation templates +supply the task and response expectations without a universal application-aware +correction protocol. + +The initial request supplies source material, references, and the task. Enduring +domain principles belong in its system instructions. The correction template +supplies a new user assignment with the current candidate's actionable findings; +PromptKit carries the preceding exchange. Do not rerender the original source or +manually replay prior responses. Review initial wording so later correction +assignments remain valid. Include additional application-derived context +only when needed to interpret the findings. Keep source/reference meanings and +normalization candidate handles fixed throughout the conversation. + +Use the same declared response schema for each initial/correction pair: + +| Role | Response contract | +| --- | --- | +| LLM chunk planning | Complete replacement chunk plan. | +| Extraction | Complete replacement extracted artifact candidate. | +| Normalization | Complete replacement reconciliation plan, applied afresh to the fixed original candidates. | +| LLM validation | The validator's assessment response; no separate semantic correction conversation by default. | + +No correction envelope, patch language, or special repair-result type is needed. +PromptKit retains structural validation and repair inside each native execution. +Notarius assesses the response returned by that public boundary, independently +of how many internal attempts produced it. + +Use the same native default profile for an initial/correction pair. Preserve +explicit application profile selections and run-wide execution overrides across +both requests. An operator may select a different default profile in a correction +prompt, subject to explicit request overrides. Validators retain their own +profile selection. Pass native selections and overrides without copying resolved +settings or adding a Notarius-wide repair-model configuration layer. + +The component that detects a defect owns its actionable correction guidance: + +- Each rejecting validator supplies guidance describing the defect and the + constraints a corrected answer must satisfy. A deterministic validator authors + it alongside the check; an LLM validator requests an actionable explanation in + its prompt and returns it through the application validation contract. +- Preserve the explicit `CorrectionGuidance` boundary separately from operator + diagnostics. Guidance need not prescribe an exact replacement, and validators + do not construct the complete correction prompt. +- Module interpretation and normalization safety checks supply their own guidance + when defects arise before an artifact candidate can be materialized. +- The pipeline collects guidance in configured validator order. The module's + domain-specific template renders it into the correction assignment. Neither + the pipeline helper nor the template duplicates validators' defect-detection + logic. Reason codes and operator messages are not automatically instructions. + +Assess the same candidate through every applicable configured validator before +requesting correction. A rejection does not short-circuit later validators; +collect all actionable findings they return. Reassess every revised candidate +through the complete applicable chain, including earlier approvals and the final +permitted response. The existing validator-owned guidance and ordered-chain +semantics remain useful; replace the hard-coded `CorrectionRequest` message +builder with native template rendering and retain useful ordered aggregation. + +Applicability requires usable input: if interpretation cannot construct a typed +candidate, its module supplies guidance and artifact validators cannot yet run. +Cancellation or fatal execution failures can interrupt assessment. An unavailable +validator assessment is a separate outcome, not a detected defect or an instruction +to change the artifact. Do not convert provider errors into correction guidance; +detailed failure disposition remains U5/U6. + +Ownership, response contracts, and default profile behavior are resolved. Refine +exact template prose, correction IDs, and necessary operation-specific inputs +while implementing representative workflows. Implementation is pending; semantic +budgets and terminal/fallback policies remain U5/U6. + +#### Execution And Assessment Flow + The application flow is: 1. Prepare source context, references, and candidate-handle mappings once. Build @@ -397,8 +477,9 @@ The application flow is: or debug inspection needs one. 2. Consume the native usable response or terminal failure, decode, and apply module interpretation and safety checks. A module can supply actionable - findings before it has a materialized candidate. Run configured artifact - validation on each candidate. + findings before it has a materialized candidate. Assess each candidate through + all applicable configured validators and collect their actionable guidance + before requesting correction, as selected in U4. 3. For actionable correction, call `Engine.Coordinate` with the captured exchange and `AdditionalTurns`. Its callback selects a repair prompt and supplies findings; PromptKit retains the initial exchange and subsequent turns. @@ -410,12 +491,8 @@ The application flow is: failure, or supported safe-fallback policy after exhaustion. Native `stopped` and `exhausted` outcomes do not themselves mean application acceptance. -Repair prompts supply a new user task; enduring principles belong in the original -system instructions. Keep source/reference and candidate-handle meanings stable -throughout the conversation. Avoid replaying the original task and constraints -that permit only an initial response. Accumulated history increases context use; -select the semantic turn budget accordingly. Separate repair profiles or schemas -are supported but not yet selected. +Accumulated history increases context use; select the semantic turn budget +accordingly. Exact budgets remain U5. Remove `single_response_v1`, raw-response `ModelCandidate`/`SemanticCorrection` transport, manual assistant/user replay, and correction-protocol capability @@ -769,7 +846,6 @@ reopen the architecture. IDs retain continuity with the earlier roadmap. | ID | Remaining choice or dependency | Resolve for | | --- | --- | --- | | U1 | Review the message-content hash API/release documented upstream in v0.24.0, including its ConfigKit dependency; verify the final compatible dependency set against the selected baseline. | Track in Phase 1; required to finish cache integration in Phase 5. | -| U4 | Role-specific initial/repair definitions, response contracts, and profile selection within complete replacement and fixed operation inputs. | Phases 3–4. | | U5 | Producer/validator retry-setting migration to native controls and semantic budgets: field names/defaults, non-LLM failure handling, and any justified recovery beyond native exhaustion. | Phases 3–4 and configuration migration. | | U6 | Detailed terminal/fallback policy, especially whether an earlier safe candidate survives a later failed turn. No implicit best-so-far acceptance. | Phases 3–4. | | U7 | Independent worker defaults; application-supplied native backend/queue limits, if any; retired `total_llm` migration and admission-failure handling by role. | Phase 4. | @@ -779,14 +855,22 @@ reopen the architecture. IDs retain continuity with the earlier roadmap. **U2 resolved:** Ordinary embedded assets, prompt-declared output contracts, native schema-source selection, and removal of duplicate schema/filesystem machinery are selected in [the schema design](#prompt-assets-and-response-schemas-u2-resolved). -Implementation remains in Phase 2; role-specific prompts and retry budgets remain -U4/U5. +Implementation remains in Phase 2; U4 selects prompt roles and contracts, while +retry budgets remain U5. **U3 resolved:** Typed stage boundaries, composition-root engine construction, and the shared pipeline producer helper with module functions and a pipeline assessment callback are selected in [the execution design](#module-execution-and-assessment-u3-resolved). Implement in Phase 3 and refine signatures during Phase 4 without reopening -ownership. U4–U6 still own role-specific behavior. +ownership. U4 selects prompt/guidance behavior; U5/U6 retain budget and terminal +policy choices. + +**U4 resolved:** Domain-specific initial/correction prompt pairs, shared response +schemas, default profile behavior, validator-owned guidance, and complete +applicable-chain assessment are selected in +[the prompt and guidance design](#prompt-roles-and-correction-guidance-u4-resolved). +Implement templates and operation-specific inputs in Phases 3–4; exact prose does +not reopen ownership. Budgets and terminal/fallback policy remain U5/U6. **U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight, output, debug policy, and coordinated cutover sequencing are selected above. @@ -808,8 +892,9 @@ major boundary changes in successor ADRs without rewriting accepted decisions. Keep the cache dependency explicit without blocking independent work. - Map affected ADRs, current callers, contracts, and removal inventory. Carry forward Audita evidence without adopting its application wrappers. -- Preserve U4–U9 with their dependent phases; do not reopen resolved module - execution, assets/schemas, CommandKit, debug, cache, or output ownership. +- Preserve U5–U9 with their dependent phases; do not reopen resolved module + execution, prompt/guidance ownership, assets/schemas, CommandKit, debug, cache, + or output ownership. **Completion evidence:** reviewed dependency baseline, upstream dependency, scope, and outstanding choices are explicit. Upstream availability alone does not @@ -834,9 +919,11 @@ without credentials or synthetic requests. `config diff` handles two selections. ### Phase 3: Prove A Representative Native Pipeline -- Implement the resolved U3 collaboration and settle relevant U4–U6 details. +- Implement the resolved U3/U4 collaboration and settle relevant U5/U6 details. Connect module request/interpretation functions to the pipeline assessment callback through the shared producer helper; refine concrete signatures here. + Implement domain-specific initial/correction prompts using the same schema and + default profile, supplied overrides, and validator-owned correction guidance. - Exercise an LLM validator on the same engine and normalization with fixed handles and fresh plan application. Select actual observer/group enrollment. - Run through the new command surface and alias, using the small completion @@ -844,15 +931,18 @@ without credentials or synthetic requests. `config diff` handles two selections. Begin U8/U9 integration without a provisional local message hash. **Completion evidence:** fake-provider workflows exercise accumulated history, +collection of independent validator findings without rejection short-circuiting, full candidate revalidation through the same assessor, final-turn acceptance without duplicate final validation, selected exhaustion/fallback, and useful -failures. The executable publishes artifacts and reports them through native -presentation without the private completion/correction API. +failures. Unavailable assessments remain distinct from actionable defects. The +executable publishes artifacts and reports them through native presentation +without the private completion/correction API. ### Phase 4: Migrate All Roles And Remove Superseded Machinery - Migrate every LLM-backed chunker, extractor, normalizer, and validator; preserve - deterministic paths and ordered validation. Complete U4–U7 across roles. + deterministic paths and ordered validation. Apply U4's prompt/guidance design + across roles and complete U5–U7. - Remove the adapter, copied completion contracts, manual replay/capability checks, outer LLM retries, global scheduler/ceiling, and coupled worker settings. - Finish native command/inspection migration and remove custom dispatch, flags, @@ -905,7 +995,7 @@ Release preparation/publication is governed separately by release policy. | `AssetRegistry`, `PromptAssetManifest`, `ModulePromptFS`, synthetic mounts, flattened schema namespaces, `readonlyfs`, manual fingerprint enumeration | Ordinary embedded content, native source assembly, and module meaning. | | Private response-schema registry/wrappers, byte copies, digest and reporting-only identity metadata, duplicate structural validation | Native prompt output contracts and schema loading; application interpretation and durable artifact contracts. | | `single_response_v1`, raw `ModelCandidate`/`SemanticCorrection`, manual replay, correction capability checks | Typed candidates, actionable findings, native exchanges, assessment. | -| LLM retry loops in `runProducerAttempts`/`executeValidationChain`, module rerun directives | Ordered validators, domain acceptance/rejection/fallback. | +| LLM retry loops in `runProducerAttempts`/`executeValidationChain`, module rerun directives, hard-coded `CorrectionRequest` message builder | Ordered validators, validator/module-owned guidance, native domain correction templates, and domain acceptance/rejection/fallback. | | `Scheduler`, `scheduledClient`, `concurrency.total_llm`, worker coupling | Bounded pipeline work and native backend admission. | | Checkpoint stage/dependency/configuration fingerprints, invocation-wide barriers, manifests, acceptance/provenance envelopes | Small accepted-artifact cache using the upstream message hash, ordinary load checks, explicit controls. | | Inspection wrappers, copied effective parameters, usage summaries/conversions/accumulators, debug completion wrapper, synthetic IDs, profile recorder | Native facts with application scope for diagnostics. | @@ -928,6 +1018,7 @@ retry tests or freeze complete help text and native debug snapshots. | Commands/loaders | Required operands, native alias, supplied-value distinctions, references, and run-wide overrides reach intended handlers/roles; help/version/completion need no config; diff selections remain distinct. | | Preflight/inspection | Same definitions across execution and inspection; missing statically selected resources fail with context; partial profiles alone pass; no provider calls, transcript, or credential requirement for offline preflight. | | Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering; one assessor checks initial/corrected candidates without an extra final pass; deterministic stages and LLM validators need no producer conversation loop. | +| Prompt/guidance contracts | Domain-specific initial/correction pairs use the same response schema and default profile; explicit profile/execution overrides reach both; operator correction-profile defaults work when not overridden; all applicable validators assess the same candidate despite earlier rejection; their actionable findings reach the correction template; module interpretation defects and unavailable assessments remain distinct. | | Continuation/failures | Accumulated history, full revalidation and final-turn assessment; no blanket retries beyond native exhaustion; validator failure differs from rejection; selected fallback, incomplete-validation, and cancellation policy hold. | | Admission/workers | Shared backend pools across roles, independent backends, bounded pipeline work, nested validators, queue exhaustion, and cancellation without leaked capacity or a global ceiling. | | Accepted-artifact cache | Downstream recovery; initial-message keys; corrected final artifact stored late under the initial key; changed messages miss; settings/ID-only changes can hit; no model/revalidation on hits; unusable entries miss; failed/rejected/incomplete work and ineligible derivatives excluded; no required history. |