Resolve native correction prompts and validator guidance ownership
This commit is contained in:
@@ -145,8 +145,9 @@ CommandKit reporters. Pure domain packages retain artifact models, codecs,
|
||||
reference resolution, and deterministic rules. `internal/preflight` retains its
|
||||
separate known-dependency checks. Exact argument grouping and helper signatures
|
||||
can be refined against representative extraction and normalization during
|
||||
implementation; ownership and placement are settled. Role-specific prompts,
|
||||
budgets, and fallback policy remain U4–U6. Implementation is pending.
|
||||
implementation; ownership and placement are settled. U4 selects prompt roles and
|
||||
guidance ownership below; budgets and fallback policy remain U5/U6. Implementation
|
||||
is pending.
|
||||
|
||||
Retain the fixed `input -> chunk -> extract -> merge -> normalize -> output`
|
||||
shape, ordered steps and generated references, typed artifact-family boundaries,
|
||||
@@ -335,10 +336,11 @@ consumers disappear; no replacement filesystem abstraction is needed.
|
||||
|
||||
Declare response format, validation mode, schema path, and structural-repair
|
||||
budget in native prompt definitions by default. Modules select prompt IDs, supply
|
||||
inputs, and interpret responses. Initial and correction prompts may reference
|
||||
the same schema file. Execution, inspection, and prepared-request preview then
|
||||
inputs, and interpret responses. U4 selects the same schema file for each
|
||||
initial/correction pair. Execution, inspection, and prepared-request preview then
|
||||
use the same declared contract without repeating schema selection in Go.
|
||||
Role-specific prompt contents remain U4, and retry defaults remain U5.
|
||||
U4 selects prompt roles and contracts; exact prose is implementation work, while
|
||||
retry defaults remain U5.
|
||||
|
||||
Use a request-level native `OutputContract` only for a demonstrated runtime-
|
||||
dependent contract. It replaces the whole configured contract, so supply the
|
||||
@@ -388,6 +390,84 @@ complete replacement model responses. Normalization may continue to return a
|
||||
reconciliation plan that deterministic code applies to produce the artifact.
|
||||
Per-entity branching is not part of this migration.
|
||||
|
||||
#### Prompt Roles And Correction Guidance (U4 Resolved)
|
||||
|
||||
Give each correctable producer an initial native prompt and a domain-specific
|
||||
correction prompt. Retain existing initial IDs and identify correction IDs
|
||||
explicitly in the module, for example `notarius/dnd.spells.repair`. Preflight uses
|
||||
those same declarations to check known prompt dependencies. Shared correction
|
||||
wording can live in ordinary asset files; distinct domain/operation templates
|
||||
supply the task and response expectations without a universal application-aware
|
||||
correction protocol.
|
||||
|
||||
The initial request supplies source material, references, and the task. Enduring
|
||||
domain principles belong in its system instructions. The correction template
|
||||
supplies a new user assignment with the current candidate's actionable findings;
|
||||
PromptKit carries the preceding exchange. Do not rerender the original source or
|
||||
manually replay prior responses. Review initial wording so later correction
|
||||
assignments remain valid. Include additional application-derived context
|
||||
only when needed to interpret the findings. Keep source/reference meanings and
|
||||
normalization candidate handles fixed throughout the conversation.
|
||||
|
||||
Use the same declared response schema for each initial/correction pair:
|
||||
|
||||
| Role | Response contract |
|
||||
| --- | --- |
|
||||
| LLM chunk planning | Complete replacement chunk plan. |
|
||||
| Extraction | Complete replacement extracted artifact candidate. |
|
||||
| Normalization | Complete replacement reconciliation plan, applied afresh to the fixed original candidates. |
|
||||
| LLM validation | The validator's assessment response; no separate semantic correction conversation by default. |
|
||||
|
||||
No correction envelope, patch language, or special repair-result type is needed.
|
||||
PromptKit retains structural validation and repair inside each native execution.
|
||||
Notarius assesses the response returned by that public boundary, independently
|
||||
of how many internal attempts produced it.
|
||||
|
||||
Use the same native default profile for an initial/correction pair. Preserve
|
||||
explicit application profile selections and run-wide execution overrides across
|
||||
both requests. An operator may select a different default profile in a correction
|
||||
prompt, subject to explicit request overrides. Validators retain their own
|
||||
profile selection. Pass native selections and overrides without copying resolved
|
||||
settings or adding a Notarius-wide repair-model configuration layer.
|
||||
|
||||
The component that detects a defect owns its actionable correction guidance:
|
||||
|
||||
- Each rejecting validator supplies guidance describing the defect and the
|
||||
constraints a corrected answer must satisfy. A deterministic validator authors
|
||||
it alongside the check; an LLM validator requests an actionable explanation in
|
||||
its prompt and returns it through the application validation contract.
|
||||
- Preserve the explicit `CorrectionGuidance` boundary separately from operator
|
||||
diagnostics. Guidance need not prescribe an exact replacement, and validators
|
||||
do not construct the complete correction prompt.
|
||||
- Module interpretation and normalization safety checks supply their own guidance
|
||||
when defects arise before an artifact candidate can be materialized.
|
||||
- The pipeline collects guidance in configured validator order. The module's
|
||||
domain-specific template renders it into the correction assignment. Neither
|
||||
the pipeline helper nor the template duplicates validators' defect-detection
|
||||
logic. Reason codes and operator messages are not automatically instructions.
|
||||
|
||||
Assess the same candidate through every applicable configured validator before
|
||||
requesting correction. A rejection does not short-circuit later validators;
|
||||
collect all actionable findings they return. Reassess every revised candidate
|
||||
through the complete applicable chain, including earlier approvals and the final
|
||||
permitted response. The existing validator-owned guidance and ordered-chain
|
||||
semantics remain useful; replace the hard-coded `CorrectionRequest` message
|
||||
builder with native template rendering and retain useful ordered aggregation.
|
||||
|
||||
Applicability requires usable input: if interpretation cannot construct a typed
|
||||
candidate, its module supplies guidance and artifact validators cannot yet run.
|
||||
Cancellation or fatal execution failures can interrupt assessment. An unavailable
|
||||
validator assessment is a separate outcome, not a detected defect or an instruction
|
||||
to change the artifact. Do not convert provider errors into correction guidance;
|
||||
detailed failure disposition remains U5/U6.
|
||||
|
||||
Ownership, response contracts, and default profile behavior are resolved. Refine
|
||||
exact template prose, correction IDs, and necessary operation-specific inputs
|
||||
while implementing representative workflows. Implementation is pending; semantic
|
||||
budgets and terminal/fallback policies remain U5/U6.
|
||||
|
||||
#### Execution And Assessment Flow
|
||||
|
||||
The application flow is:
|
||||
|
||||
1. Prepare source context, references, and candidate-handle mappings once. Build
|
||||
@@ -397,8 +477,9 @@ The application flow is:
|
||||
or debug inspection needs one.
|
||||
2. Consume the native usable response or terminal failure, decode, and apply
|
||||
module interpretation and safety checks. A module can supply actionable
|
||||
findings before it has a materialized candidate. Run configured artifact
|
||||
validation on each candidate.
|
||||
findings before it has a materialized candidate. Assess each candidate through
|
||||
all applicable configured validators and collect their actionable guidance
|
||||
before requesting correction, as selected in U4.
|
||||
3. For actionable correction, call `Engine.Coordinate` with the captured exchange
|
||||
and `AdditionalTurns`. Its callback selects a repair prompt and supplies
|
||||
findings; PromptKit retains the initial exchange and subsequent turns.
|
||||
@@ -410,12 +491,8 @@ The application flow is:
|
||||
failure, or supported safe-fallback policy after exhaustion. Native `stopped`
|
||||
and `exhausted` outcomes do not themselves mean application acceptance.
|
||||
|
||||
Repair prompts supply a new user task; enduring principles belong in the original
|
||||
system instructions. Keep source/reference and candidate-handle meanings stable
|
||||
throughout the conversation. Avoid replaying the original task and constraints
|
||||
that permit only an initial response. Accumulated history increases context use;
|
||||
select the semantic turn budget accordingly. Separate repair profiles or schemas
|
||||
are supported but not yet selected.
|
||||
Accumulated history increases context use; select the semantic turn budget
|
||||
accordingly. Exact budgets remain U5.
|
||||
|
||||
Remove `single_response_v1`, raw-response `ModelCandidate`/`SemanticCorrection`
|
||||
transport, manual assistant/user replay, and correction-protocol capability
|
||||
@@ -769,7 +846,6 @@ reopen the architecture. IDs retain continuity with the earlier roadmap.
|
||||
| ID | Remaining choice or dependency | Resolve for |
|
||||
| --- | --- | --- |
|
||||
| U1 | Review the message-content hash API/release documented upstream in v0.24.0, including its ConfigKit dependency; verify the final compatible dependency set against the selected baseline. | Track in Phase 1; required to finish cache integration in Phase 5. |
|
||||
| U4 | Role-specific initial/repair definitions, response contracts, and profile selection within complete replacement and fixed operation inputs. | Phases 3–4. |
|
||||
| U5 | Producer/validator retry-setting migration to native controls and semantic budgets: field names/defaults, non-LLM failure handling, and any justified recovery beyond native exhaustion. | Phases 3–4 and configuration migration. |
|
||||
| U6 | Detailed terminal/fallback policy, especially whether an earlier safe candidate survives a later failed turn. No implicit best-so-far acceptance. | Phases 3–4. |
|
||||
| U7 | Independent worker defaults; application-supplied native backend/queue limits, if any; retired `total_llm` migration and admission-failure handling by role. | Phase 4. |
|
||||
@@ -779,14 +855,22 @@ reopen the architecture. IDs retain continuity with the earlier roadmap.
|
||||
**U2 resolved:** Ordinary embedded assets, prompt-declared output contracts,
|
||||
native schema-source selection, and removal of duplicate schema/filesystem
|
||||
machinery are selected in [the schema design](#prompt-assets-and-response-schemas-u2-resolved).
|
||||
Implementation remains in Phase 2; role-specific prompts and retry budgets remain
|
||||
U4/U5.
|
||||
Implementation remains in Phase 2; U4 selects prompt roles and contracts, while
|
||||
retry budgets remain U5.
|
||||
|
||||
**U3 resolved:** Typed stage boundaries, composition-root engine construction,
|
||||
and the shared pipeline producer helper with module functions and a pipeline
|
||||
assessment callback are selected in [the execution design](#module-execution-and-assessment-u3-resolved).
|
||||
Implement in Phase 3 and refine signatures during Phase 4 without reopening
|
||||
ownership. U4–U6 still own role-specific behavior.
|
||||
ownership. U4 selects prompt/guidance behavior; U5/U6 retain budget and terminal
|
||||
policy choices.
|
||||
|
||||
**U4 resolved:** Domain-specific initial/correction prompt pairs, shared response
|
||||
schemas, default profile behavior, validator-owned guidance, and complete
|
||||
applicable-chain assessment are selected in
|
||||
[the prompt and guidance design](#prompt-roles-and-correction-guidance-u4-resolved).
|
||||
Implement templates and operation-specific inputs in Phases 3–4; exact prose does
|
||||
not reopen ownership. Budgets and terminal/fallback policy remain U5/U6.
|
||||
|
||||
**U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight,
|
||||
output, debug policy, and coordinated cutover sequencing are selected above.
|
||||
@@ -808,8 +892,9 @@ major boundary changes in successor ADRs without rewriting accepted decisions.
|
||||
Keep the cache dependency explicit without blocking independent work.
|
||||
- Map affected ADRs, current callers, contracts, and removal inventory. Carry
|
||||
forward Audita evidence without adopting its application wrappers.
|
||||
- Preserve U4–U9 with their dependent phases; do not reopen resolved module
|
||||
execution, assets/schemas, CommandKit, debug, cache, or output ownership.
|
||||
- Preserve U5–U9 with their dependent phases; do not reopen resolved module
|
||||
execution, prompt/guidance ownership, assets/schemas, CommandKit, debug, cache,
|
||||
or output ownership.
|
||||
|
||||
**Completion evidence:** reviewed dependency baseline, upstream dependency, scope,
|
||||
and outstanding choices are explicit. Upstream availability alone does not
|
||||
@@ -834,9 +919,11 @@ without credentials or synthetic requests. `config diff` handles two selections.
|
||||
|
||||
### Phase 3: Prove A Representative Native Pipeline
|
||||
|
||||
- Implement the resolved U3 collaboration and settle relevant U4–U6 details.
|
||||
- Implement the resolved U3/U4 collaboration and settle relevant U5/U6 details.
|
||||
Connect module request/interpretation functions to the pipeline assessment
|
||||
callback through the shared producer helper; refine concrete signatures here.
|
||||
Implement domain-specific initial/correction prompts using the same schema and
|
||||
default profile, supplied overrides, and validator-owned correction guidance.
|
||||
- Exercise an LLM validator on the same engine and normalization with fixed
|
||||
handles and fresh plan application. Select actual observer/group enrollment.
|
||||
- Run through the new command surface and alias, using the small completion
|
||||
@@ -844,15 +931,18 @@ without credentials or synthetic requests. `config diff` handles two selections.
|
||||
Begin U8/U9 integration without a provisional local message hash.
|
||||
|
||||
**Completion evidence:** fake-provider workflows exercise accumulated history,
|
||||
collection of independent validator findings without rejection short-circuiting,
|
||||
full candidate revalidation through the same assessor, final-turn acceptance
|
||||
without duplicate final validation, selected exhaustion/fallback, and useful
|
||||
failures. The executable publishes artifacts and reports them through native
|
||||
presentation without the private completion/correction API.
|
||||
failures. Unavailable assessments remain distinct from actionable defects. The
|
||||
executable publishes artifacts and reports them through native presentation
|
||||
without the private completion/correction API.
|
||||
|
||||
### Phase 4: Migrate All Roles And Remove Superseded Machinery
|
||||
|
||||
- Migrate every LLM-backed chunker, extractor, normalizer, and validator; preserve
|
||||
deterministic paths and ordered validation. Complete U4–U7 across roles.
|
||||
deterministic paths and ordered validation. Apply U4's prompt/guidance design
|
||||
across roles and complete U5–U7.
|
||||
- Remove the adapter, copied completion contracts, manual replay/capability checks,
|
||||
outer LLM retries, global scheduler/ceiling, and coupled worker settings.
|
||||
- Finish native command/inspection migration and remove custom dispatch, flags,
|
||||
@@ -905,7 +995,7 @@ Release preparation/publication is governed separately by release policy.
|
||||
| `AssetRegistry`, `PromptAssetManifest`, `ModulePromptFS`, synthetic mounts, flattened schema namespaces, `readonlyfs`, manual fingerprint enumeration | Ordinary embedded content, native source assembly, and module meaning. |
|
||||
| Private response-schema registry/wrappers, byte copies, digest and reporting-only identity metadata, duplicate structural validation | Native prompt output contracts and schema loading; application interpretation and durable artifact contracts. |
|
||||
| `single_response_v1`, raw `ModelCandidate`/`SemanticCorrection`, manual replay, correction capability checks | Typed candidates, actionable findings, native exchanges, assessment. |
|
||||
| LLM retry loops in `runProducerAttempts`/`executeValidationChain`, module rerun directives | Ordered validators, domain acceptance/rejection/fallback. |
|
||||
| LLM retry loops in `runProducerAttempts`/`executeValidationChain`, module rerun directives, hard-coded `CorrectionRequest` message builder | Ordered validators, validator/module-owned guidance, native domain correction templates, and domain acceptance/rejection/fallback. |
|
||||
| `Scheduler`, `scheduledClient`, `concurrency.total_llm`, worker coupling | Bounded pipeline work and native backend admission. |
|
||||
| Checkpoint stage/dependency/configuration fingerprints, invocation-wide barriers, manifests, acceptance/provenance envelopes | Small accepted-artifact cache using the upstream message hash, ordinary load checks, explicit controls. |
|
||||
| Inspection wrappers, copied effective parameters, usage summaries/conversions/accumulators, debug completion wrapper, synthetic IDs, profile recorder | Native facts with application scope for diagnostics. |
|
||||
@@ -928,6 +1018,7 @@ retry tests or freeze complete help text and native debug snapshots.
|
||||
| Commands/loaders | Required operands, native alias, supplied-value distinctions, references, and run-wide overrides reach intended handlers/roles; help/version/completion need no config; diff selections remain distinct. |
|
||||
| Preflight/inspection | Same definitions across execution and inspection; missing statically selected resources fail with context; partial profiles alone pass; no provider calls, transcript, or credential requirement for offline preflight. |
|
||||
| Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering; one assessor checks initial/corrected candidates without an extra final pass; deterministic stages and LLM validators need no producer conversation loop. |
|
||||
| Prompt/guidance contracts | Domain-specific initial/correction pairs use the same response schema and default profile; explicit profile/execution overrides reach both; operator correction-profile defaults work when not overridden; all applicable validators assess the same candidate despite earlier rejection; their actionable findings reach the correction template; module interpretation defects and unavailable assessments remain distinct. |
|
||||
| Continuation/failures | Accumulated history, full revalidation and final-turn assessment; no blanket retries beyond native exhaustion; validator failure differs from rejection; selected fallback, incomplete-validation, and cancellation policy hold. |
|
||||
| Admission/workers | Shared backend pools across roles, independent backends, bounded pipeline work, nested validators, queue exhaustion, and cancellation without leaked capacity or a global ceiling. |
|
||||
| Accepted-artifact cache | Downstream recovery; initial-message keys; corrected final artifact stored late under the initial key; changed messages miss; settings/ID-only changes can hit; no model/revalidation on hits; unusable entries miss; failed/rejected/incomplete work and ineligible derivatives excluded; no required history. |
|
||||
|
||||
Reference in New Issue
Block a user