diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md index 6e5867a5..fb104307 100644 --- a/docs/roadmap/promptkit.md +++ b/docs/roadmap/promptkit.md @@ -147,7 +147,7 @@ separate known-dependency checks. Exact argument grouping and helper signatures can be refined against representative extraction and normalization during implementation; ownership and placement are settled. U4 selects prompt roles and guidance ownership below; U5 selects retry controls and the correction budget. -Remaining failure and fallback policies stay in U5/U6. Implementation is pending. +Remaining terminal and fallback policies stay in U6. Implementation is pending. Retain the fixed `input -> chunk -> extract -> merge -> normalize -> output` shape, ordered steps and generated references, typed artifact-family boundaries, @@ -507,7 +507,7 @@ findings and explicit fallback information. Retain useful ordered collection and deduplication of actionable findings, rendered through a native repair prompt. Reason codes and operator messages are not automatically model instructions. -#### Retry Controls And Correction Budget (U5 Controls Resolved) +#### Retry Controls And Correction Budget (U5 Resolved) Delegate transport retries and structural repair to PromptKit. Notarius selects only the allowance for application-directed semantic correction: @@ -565,13 +565,28 @@ validator execution failure from a domain rejection. Use native `OperationTimeou when that operation scope is needed, without adding an arbitrary default or replacing broader application deadlines. Cancellation stops work. -Retire the broad producer/validator `retries` settings for LLM execution rather -than automatically renaming them. Migration guidance maps intent: a producer -allowance intended for semantic correction becomes `max_additional_turns`; native -transport controls and prompt-owned structural customization cover their own -concerns. Do not promise equivalent total provider-attempt counts. Generic retry -handling for non-LLM operations remains the outstanding U5 choice; detailed -terminal and fallback dispositions remain U6. Implementation is pending. +Remove generic retries for non-LLM operations as well. Parsing, chunk +materialization, deterministic merging, normalization-plan application, and +encoding run without an outer attempt loop. Repeating those operations against +unchanged inputs ordinarily cannot correct a defect. Return operation errors +through normal pipeline failure handling; apply the selected terminal policy to +a rejected deterministic candidate instead of regenerating the identical value. +Cancellation stops work. + +A demonstrated transient I/O recovery need belongs with that specific operation, +where repeatability and failure conditions are understood. Do not retain a generic +stage retry setting or framework in anticipation of filesystem or future remote- +storage failures. After a fix, an operator can rerun the pipeline using the +accepted-artifact cache for successful predecessors while cheap deterministic +preparation runs again. + +Retire producer and validator `retries` fields entirely, together with the generic +outer attempt loop. Migration guidance maps intent: a producer allowance intended +for semantic correction becomes `max_additional_turns`; native transport controls +and prompt-owned structural customization cover their own concerns. Remove +non-LLM retry settings without a replacement generic allowance. Do not promise +equivalent total provider-attempt counts. U5 is resolved; detailed terminal and +fallback dispositions remain U6. Implementation is pending. Keep source grounding, reference resolution, candidate isolation, domain invariants, and reconciliation safety checks. Unknown or overlapping handles @@ -906,7 +921,6 @@ reopen the architecture. IDs retain continuity with the earlier roadmap. | ID | Remaining choice or dependency | Resolve for | | --- | --- | --- | | U1 | Review the message-content hash API/release documented upstream in v0.24.0, including its ConfigKit dependency; verify the final compatible dependency set against the selected baseline. | Track in Phase 1; required to finish cache integration in Phase 5. | -| U5 | Decide generic retry handling for non-LLM operations. LLM controls, `max_additional_turns` naming/default/precedence, structural-default delegation, and intent-based LLM setting migration are resolved below. | Phases 3–4 and configuration migration. | | U6 | Detailed terminal/fallback policy, especially whether an earlier safe candidate survives a later failed turn. No implicit best-so-far acceptance. | Phases 3–4. | | U7 | Independent worker defaults; application-supplied native backend/queue limits, if any; retired `total_llm` migration and admission-failure handling by role. | Phase 4. | | U8 | Cache directory configuration and reuse/bypass/refresh controls (names, defaults, scope); old-checkpoint treatment and unusable-entry diagnostics. Hash inputs, lookup timing, accepted value, and ownership are settled. | Design with Phase 3; finish in Phase 5. | @@ -922,8 +936,8 @@ selects native repair delegation and the application correction allowance. and the shared pipeline producer helper with module functions and a pipeline assessment callback are selected in [the execution design](#module-execution-and-assessment-u3-resolved). Implement in Phase 3 and refine signatures during Phase 4 without reopening -ownership. U4/U5 select prompt/guidance and budget behavior; remaining failure -and terminal-policy choices stay in U5/U6. +ownership. U4/U5 select prompt/guidance and retry behavior; remaining terminal +and fallback-policy choices stay in U6. **U4 resolved:** Domain-specific initial/correction prompt pairs, shared response schemas, default profile behavior, validator-owned guidance, and complete @@ -932,12 +946,14 @@ applicable-chain assessment are selected in Implement templates and operation-specific inputs in Phases 3–4; exact prose does not reopen ownership. U5 selects budgets; terminal/fallback policy remains U6. -**U5 controls resolved:** `max_additional_turns` defaults to `2`, with binding over +**U5 resolved:** `max_additional_turns` defaults to `2`, with binding over pipeline precedence and explicit zero disabling correction. Native transient and structural-repair defaults remain upstream-owned; bundled prompts and application -requests do not pin structural repair. The [retry-control design](#retry-controls-and-correction-budget-u5-controls-resolved) -also selects intent-based LLM setting migration and removal of outer LLM retries. -Non-LLM retry handling remains open; U6 owns terminal/fallback decisions. +requests do not pin structural repair. The [retry-control design](#retry-controls-and-correction-budget-u5-resolved) +also removes generic non-LLM retries, all producer/validator `retries` fields, +and the generic outer attempt loop. Any demonstrated transient I/O recovery stays +with its specific operation. Implement in Phases 3–4; U6 owns terminal/fallback +decisions. **U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight, output, debug policy, and coordinated cutover sequencing are selected above. @@ -959,7 +975,7 @@ major boundary changes in successor ADRs without rewriting accepted decisions. Keep the cache dependency explicit without blocking independent work. - Map affected ADRs, current callers, contracts, and removal inventory. Carry forward Audita evidence without adopting its application wrappers. -- Preserve U5–U9 with their dependent phases; do not reopen resolved module +- Preserve U6–U9 with their dependent phases; do not reopen resolved module execution, prompt/guidance ownership, retry controls/defaults, assets/schemas, CommandKit, debug, cache, or output ownership. @@ -987,8 +1003,8 @@ without credentials or synthetic requests. `config diff` handles two selections. ### Phase 3: Prove A Representative Native Pipeline -- Implement the resolved U3/U4 collaboration and U5 controls; settle remaining - U5/U6 failure-policy details. +- Implement the resolved U3–U5 collaboration and retry policy; settle U6's + terminal/fallback details. Connect module request/interpretation functions to the pipeline assessment callback through the shared producer helper; refine concrete signatures here. Implement domain-specific initial/correction prompts using the same schema and @@ -1013,9 +1029,10 @@ without the private completion/correction API. - Migrate every LLM-backed chunker, extractor, normalizer, and validator; preserve deterministic paths and ordered validation. Apply U4's prompt/guidance design - and U5's selected controls across roles; settle remaining U5–U7 choices. + and U5's retry policy across roles; settle remaining U6/U7 choices. - Remove the adapter, copied completion contracts, manual replay/capability checks, - outer LLM retries, global scheduler/ceiling, and coupled worker settings. + generic outer attempt loop, all producer/validator `retries` fields, global + scheduler/ceiling, and coupled worker settings. - Finish native command/inspection migration and remove custom dispatch, flags, encoders, verbosity handling, and obsolete asset registration. Retire the emptied `internal/framework/llm` package after moving engine assembly to the @@ -1023,8 +1040,9 @@ without the private completion/correction API. **Completion evidence:** every role uses the shared native engine and selected admission policy. No production path depends on the private completion API, -latest-response-only correction, global ceiling, or legacy CLI infrastructure. -Obsolete tests retire with the machinery; surviving domain requirements remain. +latest-response-only correction, generic stage retries, global ceiling, or legacy +CLI infrastructure. Obsolete tests retire with the machinery; surviving domain +requirements remain. ### Phase 5: Complete Cache, Output, And Capture Integration @@ -1066,7 +1084,7 @@ Release preparation/publication is governed separately by release policy. | `AssetRegistry`, `PromptAssetManifest`, `ModulePromptFS`, synthetic mounts, flattened schema namespaces, `readonlyfs`, manual fingerprint enumeration | Ordinary embedded content, native source assembly, and module meaning. | | Private response-schema registry/wrappers, byte copies, digest and reporting-only identity metadata, duplicate structural validation | Native prompt output contracts and schema loading; application interpretation and durable artifact contracts. | | `single_response_v1`, raw `ModelCandidate`/`SemanticCorrection`, manual replay, correction capability checks | Typed candidates, actionable findings, native exchanges, assessment. | -| LLM retry loops in `runProducerAttempts`/`executeValidationChain`, module rerun directives, hard-coded `CorrectionRequest` message builder | Ordered validators, validator/module-owned guidance, native domain correction templates, and domain acceptance/rejection/fallback. | +| Generic `runProducerAttempts` outer loop, repeated validator executions in `executeValidationChain`, all producer/validator `retries` fields, module rerun directives, hard-coded `CorrectionRequest` message builder | Ordinary deterministic execution, native LLM execution/continuation, ordered validators, validator/module-owned guidance, native domain correction templates, and domain acceptance/rejection/fallback. | | Application `structured_output_repair_attempts` fields/resolution and explicit bundled `output.repair_attempts` | Native upstream structural-repair default, with operator customization through native prompt configuration. | | `Scheduler`, `scheduledClient`, `concurrency.total_llm`, worker coupling | Bounded pipeline work and native backend admission. | | Checkpoint stage/dependency/configuration fingerprints, invocation-wide barriers, manifests, acceptance/provenance envelopes | Small accepted-artifact cache using the upstream message hash, ordinary load checks, explicit controls. | @@ -1092,6 +1110,7 @@ retry tests or freeze complete help text and native debug snapshots. | Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering; one assessor checks initial/corrected candidates without an extra final pass; deterministic stages and LLM validators need no producer conversation loop. | | Prompt/guidance contracts | Domain-specific initial/correction pairs use the same response schema and default profile; explicit profile/execution overrides reach both; operator correction-profile defaults work when not overridden; all applicable validators assess the same candidate despite earlier rejection; their actionable findings reach the correction template; module interpretation defects and unavailable assessments remain distinct. | | Retry controls | `max_additional_turns` resolves binding, pipeline, then application default `2`; explicit zero disables correction; acceptance stops early and the final allowed turn is assessed. Bundled declarations/application overrides leave structural repair unspecified, native operator customization remains effective, and tests do not freeze upstream default attempt counts. | +| Non-LLM failure handling | Deterministic errors propagate without generic retries; rejected deterministic candidates reach terminal policy without regeneration; cancellation stops work. Producer/validator `retries` fields and the generic outer attempt loop are retired. | | Continuation/failures | Accumulated history, full revalidation and final-turn assessment; no blanket retries beyond native exhaustion; validator failure differs from rejection; selected fallback, incomplete-validation, and cancellation policy hold. | | Admission/workers | Shared backend pools across roles, independent backends, bounded pipeline work, nested validators, queue exhaustion, and cancellation without leaked capacity or a global ceiling. | | Accepted-artifact cache | Downstream recovery; initial-message keys; corrected final artifact stored late under the initial key; changed messages miss; settings/ID-only changes can hit; no model/revalidation on hits; unusable entries miss; failed/rejected/incomplete work and ineligible derivatives excluded; no required history. | @@ -1128,7 +1147,8 @@ Provide practical release guidance for operators and subprocess callers: native local backends, removed catalogs/versions, and separate source versions. - Native admission and worker defaults, removed global limit, `max_additional_turns` default/precedence/zero behavior, and intent-based migration from LLM `retries`. - Explain upstream-owned structural defaults, removal of application repair + Explain retirement of all producer/validator `retries` fields and generic non-LLM + retries, upstream-owned structural defaults, removal of application repair controls and bundled counts, native operator customization, accumulated-history costs, and selected role failure policy. - Accepted-artifact caching, message-based misses, settings-only reuse, refresh/