From e73716773040235f1921385030ab9767e54b0de9 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 20 Nov 2026 15:34:05 +0000 Subject: [PATCH] Set semantic turn defaults and delegate structural repair upstream --- docs/roadmap/promptkit.md | 174 +++++++++++++++++++++++++++----------- 1 file changed, 125 insertions(+), 49 deletions(-) diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md index f2b4531a..6e5867a5 100644 --- a/docs/roadmap/promptkit.md +++ b/docs/roadmap/promptkit.md @@ -146,8 +146,8 @@ reference resolution, and deterministic rules. `internal/preflight` retains its separate known-dependency checks. Exact argument grouping and helper signatures can be refined against representative extraction and normalization during implementation; ownership and placement are settled. U4 selects prompt roles and -guidance ownership below; budgets and fallback policy remain U5/U6. Implementation -is pending. +guidance ownership below; U5 selects retry controls and the correction budget. +Remaining failure and fallback policies stay in U5/U6. Implementation is pending. Retain the fixed `input -> chunk -> extract -> merge -> normalize -> output` shape, ordered steps and generated references, typed artifact-family boundaries, @@ -194,10 +194,11 @@ a singleton flag and use the last successfully parsed value. | Capture destination | Application `--debug-dir PATH`, with the semantics below. | Keep worker counts and semantic budgets in configuration, adjustable through -application `--set`; their fields/defaults remain open. Use ckprompt's reusable -profile, session, and execution flags, including model, temperature, timeout, -reasoning effort, and transient retries, for run-wide overrides across producers, -validators, and continuations. Omission preserves normal selection; explicitly +application `--set`. U5 selects `max_additional_turns`; worker defaults remain U7. +Use ckprompt's reusable profile, session, and execution flags, including model, +temperature, timeout, reasoning effort, and transient retries, for run-wide +overrides across producers, validators, and continuations. Omission preserves +normal selection; explicitly empty reasoning effort clears inheritance. Use native `--llm-profile` and `--llm-session-id`, retiring the separate reasoning-clearing control. Native transient retries remain distinct from semantic additional-turn budgets. @@ -275,7 +276,8 @@ ConfigKit continues to own loading and composition under Retire `promptkit.profile_file`, `profile_dir`, and `local_backend` in favor of native companion sources; do not introduce a separate `promptkit.config_file`. Retire `concurrency.total_llm` and revise retry/worker settings as described below. -Their replacement defaults and detailed migration treatment remain open. +U5 selects `max_additional_turns` and removes application structural-repair +controls; worker defaults and retired global-limit migration remain U7. Use one source assembly for runtime, preflight, and inspection: @@ -334,21 +336,23 @@ Remove synthetic mounts, file copying, flattened schema namespaces, and module asset manifests. Retire `readonlyfs` when its prompt-mounting and asset-flattening consumers disappear; no replacement filesystem abstraction is needed. -Declare response format, validation mode, schema path, and structural-repair -budget in native prompt definitions by default. Modules select prompt IDs, supply -inputs, and interpret responses. U4 selects the same schema file for each -initial/correction pair. Execution, inspection, and prepared-request preview then -use the same declared contract without repeating schema selection in Go. -U4 selects prompt roles and contracts; exact prose is implementation work, while -retry defaults remain U5. +Declare response format, validation mode, and schema path in native prompt +definitions by default. Omit structural-repair attempt settings in bundled +prompts to inherit PromptKit's upstream default under U5. Modules select prompt +IDs, supply inputs, and interpret responses. U4 selects the same schema file for +each initial/correction pair. Execution, inspection, and prepared-request preview +then use the same declared contract without repeating schema selection in Go. +U4 selects prompt roles and contracts; exact prose is implementation work. +U5 selects budget ownership and defaults. Use a request-level native `OutputContract` only for a demonstrated runtime- dependent contract. It replaces the whole configured contract, so supply the -intended format, validation mode, schema, and repair policy together. Do not -introduce a wrapper or per-call registry solely to reproduce Audita's approach. -Operator prompt replacements must still produce responses the selected module -can interpret; that module boundary does not require duplicating each prompt's -schema selection in Go. +intended format, validation mode, and schema together while leaving structural +repair at its native default. Do not inject a repair count merely to construct +the override, or introduce a wrapper or per-call registry solely to reproduce +Audita's approach. Operator prompt replacements must still produce responses the +selected module can interpret; that module boundary does not require duplicating +each prompt's schema selection in Go. Default to native `WithSchemaFS(assets.FS(), ".")`. Schema paths are relative to that selected root, for example `dnd/spells/schemas/dnd_spells_llm.v1.json`. @@ -459,12 +463,12 @@ candidate, its module supplies guidance and artifact validators cannot yet run. Cancellation or fatal execution failures can interrupt assessment. An unavailable validator assessment is a separate outcome, not a detected defect or an instruction to change the artifact. Do not convert provider errors into correction guidance; -detailed failure disposition remains U5/U6. +terminal/fallback disposition remains U6. Ownership, response contracts, and default profile behavior are resolved. Refine exact template prose, correction IDs, and necessary operation-specific inputs -while implementing representative workflows. Implementation is pending; semantic -budgets and terminal/fallback policies remain U5/U6. +while implementing representative workflows. Implementation is pending; U5 selects +semantic budgets and U6 retains terminal/fallback policy choices. #### Execution And Assessment Flow @@ -481,8 +485,9 @@ The application flow is: all applicable configured validators and collect their actionable guidance before requesting correction, as selected in U4. 3. For actionable correction, call `Engine.Coordinate` with the captured exchange - and `AdditionalTurns`. Its callback selects a repair prompt and supplies - findings; PromptKit retains the initial exchange and subsequent turns. + and the resolved `max_additional_turns` as native `AdditionalTurns`. Its callback + selects a repair prompt and supplies findings; PromptKit retains the initial + exchange and subsequent turns. 4. Interpret every delivered revised response and rebuild from the fixed original inputs. Run the complete configured validation chain again for changed candidates, including validators that previously approved. Assess the final @@ -491,8 +496,8 @@ The application flow is: failure, or supported safe-fallback policy after exhaustion. Native `stopped` and `exhausted` outcomes do not themselves mean application acceptance. -Accumulated history increases context use; select the semantic turn budget -accordingly. Exact budgets remain U5. +Accumulated history increases context use; U5's allowance bounds additional +semantic correction while stopping as soon as the candidate is accepted. Remove `single_response_v1`, raw-response `ModelCandidate`/`SemanticCorrection` transport, manual assistant/user replay, and correction-protocol capability @@ -502,16 +507,71 @@ findings and explicit fallback information. Retain useful ordered collection and deduplication of actionable findings, rendered through a native repair prompt. Reason codes and operator messages are not automatically model instructions. -Transport retries, structural repair, and semantic additional turns remain -separate controls. Preserve the ability to override native transport retries, -using the upstream default when omitted. Remove blanket outer LLM retries after -native transport/structural budgets are exhausted, including repeated LLM -validator invocations in `executeValidationChain`. Keep the ordered validator -chain and aggregation. A validator execution failure is distinct from rejection -of a producer candidate. Existing producer/validator `retries` settings cannot be -mechanically renamed; decide their migration and non-LLM failure handling -explicitly. Use native `OperationTimeout` when that operation scope is needed, -without adding an arbitrary default or replacing broader application deadlines. +#### Retry Controls And Correction Budget (U5 Controls Resolved) + +Delegate transport retries and structural repair to PromptKit. Notarius selects +only the allowance for application-directed semantic correction: + +| Concern | Control and selected default | +| --- | --- | +| Temporary provider failures | Native PromptKit transient retries; inherit the upstream default when omitted and preserve native execution overrides. | +| Structurally invalid model output | Native output-contract policy; bundled prompts and application requests inherit PromptKit's upstream structural-repair default. | +| Actionable domain defects | Application `max_additional_turns`, passed directly to native `AdditionalTurns`; application default `2`. | + +`max_additional_turns` is a nonnegative maximum, excluding the initial producer +response. Resolve producer-binding override, then pipeline override, then the +application default of `2`. Omission inherits; explicit `0` disables semantic +correction. It applies to LLM-backed producers. Validators do not receive a +separate semantic continuation budget, and deterministic stages do not acquire +conversation machinery. Keep the field in configuration, adjustable through the +existing application `--set` mechanism, without another dedicated CLI flag. + +For example, a producer binding can override its pipeline allowance: + +```yaml +extract: + module: dnd/spells + max_additional_turns: 2 +``` + +Two additional turns are an allowance, not mandatory executions. An accepted +initial candidate needs none; an accepted first correction needs only one. A +second turn is available when actionable defects remain and continuation is +appropriate. Assess every delivered response, including the final permitted turn, +and stop on acceptance or a terminal condition. PromptKit enforces the turn +limit; do not add an application counter or retry loop. Internal transport +attempts and structural repairs do not consume this semantic-turn allowance. +The default is an application policy choice, not a measured claim about how +frequently corrections will be needed. + +For structural repair, omit `output.repair_attempts` from all bundled initial, +correction, and validator definitions. Remove Notarius's pipeline/binding/ +validator `structured_output_repair_attempts` fields and resolution plumbing. +Do not replace them with a hard-coded default, including zero, in request +construction or a native output-contract override. Explicit operator +customization remains available through native PromptKit prompt configuration. + +Notarius deliberately inherits future upstream default changes. An anticipated +PromptKit default change is not a migration prerequisite, and its timing or +chosen count is not a Notarius contract. Notarius still declares response format, +validation mode, and schema; PromptKit owns the mechanics of obtaining a valid +response. At the public boundary, handle a usable result or terminal failure, +including a returned result reporting unsuccessful structural validation. +Do not restart the operation to extend PromptKit's exhausted repair budget. + +Remove blanket outer LLM retries, including repeated LLM validator invocations in +`executeValidationChain`. Keep ordered candidate assessment and distinguish a +validator execution failure from a domain rejection. Use native `OperationTimeout` +when that operation scope is needed, without adding an arbitrary default or +replacing broader application deadlines. Cancellation stops work. + +Retire the broad producer/validator `retries` settings for LLM execution rather +than automatically renaming them. Migration guidance maps intent: a producer +allowance intended for semantic correction becomes `max_additional_turns`; native +transport controls and prompt-owned structural customization cover their own +concerns. Do not promise equivalent total provider-attempt counts. Generic retry +handling for non-LLM operations remains the outstanding U5 choice; detailed +terminal and fallback dispositions remain U6. Implementation is pending. Keep source grounding, reference resolution, candidate isolation, domain invariants, and reconciliation safety checks. Unknown or overlapping handles @@ -846,7 +906,7 @@ reopen the architecture. IDs retain continuity with the earlier roadmap. | ID | Remaining choice or dependency | Resolve for | | --- | --- | --- | | U1 | Review the message-content hash API/release documented upstream in v0.24.0, including its ConfigKit dependency; verify the final compatible dependency set against the selected baseline. | Track in Phase 1; required to finish cache integration in Phase 5. | -| U5 | Producer/validator retry-setting migration to native controls and semantic budgets: field names/defaults, non-LLM failure handling, and any justified recovery beyond native exhaustion. | Phases 3–4 and configuration migration. | +| U5 | Decide generic retry handling for non-LLM operations. LLM controls, `max_additional_turns` naming/default/precedence, structural-default delegation, and intent-based LLM setting migration are resolved below. | Phases 3–4 and configuration migration. | | U6 | Detailed terminal/fallback policy, especially whether an earlier safe candidate survives a later failed turn. No implicit best-so-far acceptance. | Phases 3–4. | | U7 | Independent worker defaults; application-supplied native backend/queue limits, if any; retired `total_llm` migration and admission-failure handling by role. | Phase 4. | | U8 | Cache directory configuration and reuse/bypass/refresh controls (names, defaults, scope); old-checkpoint treatment and unusable-entry diagnostics. Hash inputs, lookup timing, accepted value, and ownership are settled. | Design with Phase 3; finish in Phase 5. | @@ -855,22 +915,29 @@ reopen the architecture. IDs retain continuity with the earlier roadmap. **U2 resolved:** Ordinary embedded assets, prompt-declared output contracts, native schema-source selection, and removal of duplicate schema/filesystem machinery are selected in [the schema design](#prompt-assets-and-response-schemas-u2-resolved). -Implementation remains in Phase 2; U4 selects prompt roles and contracts, while -retry budgets remain U5. +Implementation remains in Phase 2; U4 selects prompt roles and contracts, and U5 +selects native repair delegation and the application correction allowance. **U3 resolved:** Typed stage boundaries, composition-root engine construction, and the shared pipeline producer helper with module functions and a pipeline assessment callback are selected in [the execution design](#module-execution-and-assessment-u3-resolved). Implement in Phase 3 and refine signatures during Phase 4 without reopening -ownership. U4 selects prompt/guidance behavior; U5/U6 retain budget and terminal -policy choices. +ownership. U4/U5 select prompt/guidance and budget behavior; remaining failure +and terminal-policy choices stay in U5/U6. **U4 resolved:** Domain-specific initial/correction prompt pairs, shared response schemas, default profile behavior, validator-owned guidance, and complete applicable-chain assessment are selected in [the prompt and guidance design](#prompt-roles-and-correction-guidance-u4-resolved). Implement templates and operation-specific inputs in Phases 3–4; exact prose does -not reopen ownership. Budgets and terminal/fallback policy remain U5/U6. +not reopen ownership. U5 selects budgets; terminal/fallback policy remains U6. + +**U5 controls resolved:** `max_additional_turns` defaults to `2`, with binding over +pipeline precedence and explicit zero disabling correction. Native transient and +structural-repair defaults remain upstream-owned; bundled prompts and application +requests do not pin structural repair. The [retry-control design](#retry-controls-and-correction-budget-u5-controls-resolved) +also selects intent-based LLM setting migration and removal of outer LLM retries. +Non-LLM retry handling remains open; U6 owns terminal/fallback decisions. **U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight, output, debug policy, and coordinated cutover sequencing are selected above. @@ -893,8 +960,8 @@ major boundary changes in successor ADRs without rewriting accepted decisions. - Map affected ADRs, current callers, contracts, and removal inventory. Carry forward Audita evidence without adopting its application wrappers. - Preserve U5–U9 with their dependent phases; do not reopen resolved module - execution, prompt/guidance ownership, assets/schemas, CommandKit, debug, cache, - or output ownership. + execution, prompt/guidance ownership, retry controls/defaults, assets/schemas, + CommandKit, debug, cache, or output ownership. **Completion evidence:** reviewed dependency baseline, upstream dependency, scope, and outstanding choices are explicit. Upstream availability alone does not @@ -905,7 +972,8 @@ of the migration. - Implement the resolved U2 design, update available dependencies, introduce configuration version 6, convert native embedded/companion resources, and use - native schema loading with embedded/default or operator-selected roots. + native schema loading with embedded/default or operator-selected roots. Omit + bundled structural-repair settings and remove their application override fields. - Build the small CommandKit application, shared ConfigKit/engine loaders, standard suites, pipeline declarations/native alias, and `internal/preflight`. - Add maintained embedded-only and operator-override examples. Remove duplicated @@ -919,11 +987,14 @@ without credentials or synthetic requests. `config diff` handles two selections. ### Phase 3: Prove A Representative Native Pipeline -- Implement the resolved U3/U4 collaboration and settle relevant U5/U6 details. +- Implement the resolved U3/U4 collaboration and U5 controls; settle remaining + U5/U6 failure-policy details. Connect module request/interpretation functions to the pipeline assessment callback through the shared producer helper; refine concrete signatures here. Implement domain-specific initial/correction prompts using the same schema and default profile, supplied overrides, and validator-owned correction guidance. + Resolve `max_additional_turns` with omission/zero distinctions and pass it to + native coordination; inherit native transport and structural-repair defaults. - Exercise an LLM validator on the same engine and normalization with fixed handles and fresh plan application. Select actual observer/group enrollment. - Run through the new command surface and alias, using the small completion @@ -942,7 +1013,7 @@ without the private completion/correction API. - Migrate every LLM-backed chunker, extractor, normalizer, and validator; preserve deterministic paths and ordered validation. Apply U4's prompt/guidance design - across roles and complete U5–U7. + and U5's selected controls across roles; settle remaining U5–U7 choices. - Remove the adapter, copied completion contracts, manual replay/capability checks, outer LLM retries, global scheduler/ceiling, and coupled worker settings. - Finish native command/inspection migration and remove custom dispatch, flags, @@ -996,6 +1067,7 @@ Release preparation/publication is governed separately by release policy. | Private response-schema registry/wrappers, byte copies, digest and reporting-only identity metadata, duplicate structural validation | Native prompt output contracts and schema loading; application interpretation and durable artifact contracts. | | `single_response_v1`, raw `ModelCandidate`/`SemanticCorrection`, manual replay, correction capability checks | Typed candidates, actionable findings, native exchanges, assessment. | | LLM retry loops in `runProducerAttempts`/`executeValidationChain`, module rerun directives, hard-coded `CorrectionRequest` message builder | Ordered validators, validator/module-owned guidance, native domain correction templates, and domain acceptance/rejection/fallback. | +| Application `structured_output_repair_attempts` fields/resolution and explicit bundled `output.repair_attempts` | Native upstream structural-repair default, with operator customization through native prompt configuration. | | `Scheduler`, `scheduledClient`, `concurrency.total_llm`, worker coupling | Bounded pipeline work and native backend admission. | | Checkpoint stage/dependency/configuration fingerprints, invocation-wide barriers, manifests, acceptance/provenance envelopes | Small accepted-artifact cache using the upstream message hash, ordinary load checks, explicit controls. | | Inspection wrappers, copied effective parameters, usage summaries/conversions/accumulators, debug completion wrapper, synthetic IDs, profile recorder | Native facts with application scope for diagnostics. | @@ -1019,6 +1091,7 @@ retry tests or freeze complete help text and native debug snapshots. | Preflight/inspection | Same definitions across execution and inspection; missing statically selected resources fail with context; partial profiles alone pass; no provider calls, transcript, or credential requirement for offline preflight. | | Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering; one assessor checks initial/corrected candidates without an extra final pass; deterministic stages and LLM validators need no producer conversation loop. | | Prompt/guidance contracts | Domain-specific initial/correction pairs use the same response schema and default profile; explicit profile/execution overrides reach both; operator correction-profile defaults work when not overridden; all applicable validators assess the same candidate despite earlier rejection; their actionable findings reach the correction template; module interpretation defects and unavailable assessments remain distinct. | +| Retry controls | `max_additional_turns` resolves binding, pipeline, then application default `2`; explicit zero disables correction; acceptance stops early and the final allowed turn is assessed. Bundled declarations/application overrides leave structural repair unspecified, native operator customization remains effective, and tests do not freeze upstream default attempt counts. | | Continuation/failures | Accumulated history, full revalidation and final-turn assessment; no blanket retries beyond native exhaustion; validator failure differs from rejection; selected fallback, incomplete-validation, and cancellation policy hold. | | Admission/workers | Shared backend pools across roles, independent backends, bounded pipeline work, nested validators, queue exhaustion, and cancellation without leaked capacity or a global ceiling. | | Accepted-artifact cache | Downstream recovery; initial-message keys; corrected final artifact stored late under the initial key; changed messages miss; settings/ID-only changes can hit; no model/revalidation on hits; unusable entries miss; failed/rejected/incomplete work and ineligible derivatives excluded; no required history. | @@ -1053,8 +1126,11 @@ Provide practical release guidance for operators and subprocess callers: - Version 6 and native resources, embedded-only versus companion examples in appropriate directories, imports, qualified IDs, replacement/inheritance, native local backends, removed catalogs/versions, and separate source versions. -- Native admission and worker defaults, removed global limit, retry-budget - migration, accumulated-history costs, and selected role failure policy. +- Native admission and worker defaults, removed global limit, `max_additional_turns` + default/precedence/zero behavior, and intent-based migration from LLM `retries`. + Explain upstream-owned structural defaults, removal of application repair + controls and bundled counts, native operator customization, accumulated-history + costs, and selected role failure policy. - Accepted-artifact caching, message-based misses, settings-only reuse, refresh/ bypass for changed interpretation or validation, and old-checkpoint handling. - Artifact delivery without mandatory manifests/receipts; preserve intrinsic