From d17d261d623b1a2a51a30f3de78fbfae1d11c79c Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Thu, 19 Nov 2026 15:27:08 +0000 Subject: [PATCH] Rewrite roadmap for PromptKit-native architecture --- .../promptkit-architecture-discussion.md | 25 +- docs/roadmap/promptkit.md | 734 +++++++++++++----- 2 files changed, 539 insertions(+), 220 deletions(-) diff --git a/docs/roadmap/promptkit-architecture-discussion.md b/docs/roadmap/promptkit-architecture-discussion.md index 9478a1f3..4260d7cb 100644 --- a/docs/roadmap/promptkit-architecture-discussion.md +++ b/docs/roadmap/promptkit-architecture-discussion.md @@ -1,13 +1,15 @@ -# PromptKit Architecture Discussion +# PromptKit Architecture Discussion Record ## Purpose And Status -Working notepad for reviewing the ownership boundaries in the -[PromptKit v0.23.2 adoption roadmap](promptkit.md). Work through the topics -below in order, recording questions, evidence, decisions, and follow-up work. -Items 1–6 record selected architectural directions. Detailed design and the -read-only filesystem and schema decisions remain open. Item 7 records the -completed Audita comparison and its implications for these decisions. +The decisions in items 1–6 have been incorporated into the +[PromptKit-native architecture migration roadmap](promptkit.md). That roadmap +is the implementation plan and owns current unresolved choices, sequencing, +acceptance criteria, and implementation status. + +This document retains the discussion, rationale, and evidence, including item 7's +completed Audita comparison. Item-level questions and follow-ups below record +the discussion as it developed; they are not a second implementation plan. The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit capabilities as fully as practical. Application ownership of a policy does not @@ -15,12 +17,9 @@ necessarily require application ownership of its implementation. Evaluate remaining machinery against this goal, including opportunities for upstream features where PromptKit does not yet meet an application need. -No application changes have been implemented through this notepad. The adoption -roadmap still needs reconciliation with the selected direction, including its -adapter-preservation, latest-response-only, cross-backend scheduler preservation, -and broad checkpoint-fingerprint language. Significant changes to accepted -architecture need successor ADRs under the -[documentation policy](../policy/documentation.md#architecture-decision-records). +No application changes have been implemented through this discussion or the +roadmap rewrite. Significant changes to accepted architecture need successor ADRs +under the [documentation policy](../policy/documentation.md#architecture-decision-records). Current boundaries are described in [Architecture](../policy/architecture.md) and [LLM Runtime Internals](../internal/llm.md). diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md index 6bf21311..9d2945fe 100644 --- a/docs/roadmap/promptkit.md +++ b/docs/roadmap/promptkit.md @@ -1,245 +1,565 @@ -# PromptKit v0.23.2 Adoption +# PromptKit-Native Architecture Migration -Open questions about the retained application machinery are tracked in the -[architecture discussion notepad](promptkit-architecture-discussion.md). -Those proposals remain pending until reconciled into this roadmap. +## Purpose And Status -## Purpose +Make Notarius fully PromptKit-native: use upstream configuration, execution, +continuation, admission, inspection, and accounting directly, and retain the +application work of producing, assessing, storing, and reusing typed artifacts. +Historical duplication of PromptKit behavior is not a requirement to preserve it. -Replace Notarius's PromptKit v0.12 integration with a native v0.23.2 -integration. Delegate configuration composition, resource inspection, request -preparation, and model execution to PromptKit and ConfigKit. Keep application -machinery where it implements a distinct Notarius decision, such as typed -artifact handling, stage acceptance, cross-backend scheduling, or checkpoint -reuse. +This is the implementation roadmap for the agreed directions in items 1–6 of +[the architecture discussion record](promptkit-architecture-discussion.md). +That document retains the discussion and Audita evidence. This roadmap owns the +selected target, unresolved choices, implementation sequence, and acceptance +criteria. All implementation phases below are pending; rewriting the roadmap +has not changed application behavior. -This is a hard cutover for both the Notarius application configuration and -PromptKit resource definitions. Provide migration guidance and maintained -examples; support only the new formats in the resulting application. This -roadmap defines the PromptKit cutover. Operator interface standardization -belongs to the subsequent CommandKit adoption. +The core migration is a hard cutover for application configuration and PromptKit +resource definitions. Supply migration guidance and maintained examples; support +only the new formats in the resulting application. CommandKit adoption remains a +subsequent delivery phase with its ownership accounted for throughout this plan. +Release preparation and publication are separate work. -## Reviewed Baseline +## Baseline And Dependency Target -Notarius at `65b8e8f0a2d60041c18a29a3c33afc5c689468df` uses application -configuration version 5, Go 1.27, new-path ConfigKit v1.2.0, and old-path -PromptKit v0.12.0. The latter brings in a separate old-path ConfigKit v0.5.0 -and two external provider catalogs. Notarius and PromptKit currently share -the YAML v4 release candidate. +At Notarius revision `12eeb6369e40746b2c18c55397a289a1e1b5908d`, the application +uses configuration version 5 and Go 1.27. Its dependencies include old-path +PromptKit v0.12.0 and ConfigKit v0.5.0, new-path ConfigKit v1.2.0, go-yaml +v4.0.0-rc.6, and two external PromptKit provider catalogs. -The target is `go.maximumdirect.net/promptkit@v0.23.2`, published at -`382ccfb14b9630fd8f48c8364dbe2b6159637d71`. The tagged [release -notes][release], [migration guide][migration], [format reference][formats], -and [Go consumer guide][consumer] are the primary upstream references. +The initial dependency target remains `go.maximumdirect.net/promptkit@v0.23.2` +(tagged revision `382ccfb14b9630fd8f48c8364dbe2b6159637d71`), with shared +ConfigKit v1.2.1, stable go-yaml v4, and compatible validation dependencies. +The Go consumer minimum remains 1.27. The tagged [release notes][release], +[migration guide][migration], [format reference][formats], and +[Go consumer guide][consumer] supply the upstream baseline. -The main baseline seams are: +Confirm the final compatible tagged dependency set before implementation. +A demonstrated unmet application need may justify an upstream request and a +later target; the agreed design does not currently require a new PromptKit +feature. In particular, neither a cross-backend global limit nor a comprehensive +execution fingerprint is a prerequisite. Remove obsolete imports and catalog +dependencies when unused; do not retain a compatibility fork or permanent +`replace` directive. Update attribution under the [OSS policy](../policy/oss.md). -- `internal/core/config` resolves `promptkit.profile_file` and - `profile_dir` and interprets the optional `local_backend` object. - `promptkit_profiles.go` builds the operator layers for execution and - inspection; `asset_registry.go` supplies the embedded layer. -- `assets/` and `examples/llm-profiles/` use keyed resource definitions. - Completion requests and diagnostics carry a PromptKit prompt version. -- `internal/framework/llm` does split preparation and execution to retain - debug material; it projects a backend-completed profile report and walks - effective YAML to discover versioned prompts for checkpoint identity. -- The pipeline's producer-attempt loop coordinates domain and stage policy. - The scheduled LLM client enforces the application-wide limit independently - of any PromptKit backend policy. +Audita's reviewed v0.23.2 integration demonstrates native source assembly, +continuation, observations, and CommandKit wiring. Adopt those mechanisms +without copying its `RoleClient`, structured-output interface, or opaque exchange +wrapper. Its per-proposal repair branches and lack of checkpoints do not define +Notarius's artifact and recovery policies. -## Target End State +## Target Architecture -### Dependency And Application Configuration Cutover +### Application Structure And Ownership -Use the vanity PromptKit import path and shared ConfigKit v1.2.1, stable -go-yaml v4, and the resulting compatible validation dependencies. Remove the -old-path imports and catalog dependencies once unused. Use tagged modules -without a compatibility fork or permanent replacement. The consumer minimum -remains Go 1.27. Update attribution for the selected and removed components -under the [OSS policy](../policy/oss.md). +| Part | Responsibility | +| --- | --- | +| Application composition | Load configuration and native sources; construct one shared PromptKit engine for producers, validators, and continuation runs. | +| Artifact-family modules | Prepare domain inputs, select prompt IDs and response requirements, interpret responses, and apply deterministic domain rules. | +| Pipeline execution | Connect generation to configured validation and terminal policy; manage ordered dependencies, typed handoffs, bounded workers, and stable result ordering. | +| PromptKit | Prepare and execute requests; maintain captured exchanges and conversations; enforce native retry, repair, turn, and admission budgets; report execution facts. | +| Application state and records | Store artifacts, apply checkpoint reuse policy, preserve acceptance provenance, and attach pipeline context to execution facts. | +| ConfigKit and CommandKit | ConfigKit owns configuration mechanics; CommandKit and ckconfig/ckprompt own standardized commands and presentation. Notarius supplies loaders, request builders, report content, and outcome mapping. | -Introduce application configuration version 6 as the only supported application -version. Version 6 retains unrelated pipeline structure, discovery, and -resolution behavior. ConfigKit continues to own application loading and -composition under [ADR-0020](../adr/0020-delegate-configuration-mechanics-to-configkit.md). -Retire `profile_file`, `profile_dir`, and `local_backend` in favor of native -PromptKit companion files. There is no separate `promptkit.config_file` selector. -Preserve the ability to override transport retries through the PromptKit engine -option; leave the upstream default in effect when the operator does not -override it. +Use native PromptKit requests, results, exchanges, errors, and observers in +model-executing application code. Remove the private provider-neutral completion +API. Pure domain models, codecs, source-reference resolution, and deterministic +transformations need no PromptKit dependency. Application interfaces continue to +express artifact production, typed validation, output encoding, and state access. -### Companion Files And Embedded Defaults +A small shared application function connects module request construction and +response interpretation to pipeline validation and terminal policy. It calls +native `Run` and `Coordinate` and returns a typed application outcome. It owns no +replacement completion API, conversation protocol, or general retry engine. +Exact signatures and helper placement remain open. The current +`internal/framework/llm` package may shrink to engine assembly and focused helpers +or cease to be a separate package. -Adopt Audita's companion-file convention. Use the directory of the main -Notarius configuration actually selected by ConfigKit, whether explicit or -discovered. Load the optional `prompts.yml` and recursive `.yml` and `.yaml` -files under `prompts/` beside it as one higher-priority operator underlay -using native ConfigKit imports. Do not add a working-directory fallback -for companion discovery. Configurations in the same directory share those -companions. Missing companions are fine; loading errors follow upstream -semantics rather than silently falling back to embedded assets. Native -imports and source-relative resource paths govern any files reached from the -companions. The two companion locations are strict peers; filenames do not -set precedence between conflicting operator records. +Retain the fixed `input -> chunk -> extract -> merge -> normalize -> output` +shape, ordered steps and generated references, typed artifact-family boundaries, +and deterministic public ordering. Keep modules independent of physical output, +cache, and debug roots. Native adoption does not turn the pipeline into a general +workflow engine or redesign durable D&D artifacts. -Use one source assembly for runtime, preflight, and inspection: PromptKit's -bundled OpenRouter underlay, Notarius's embedded underlay, then the operator -underlay when present. Embed all Notarius default prompts and profiles under -`collections.notarius` in the application underlay. Operator companion files -can replace individual defaults by defining complete records with the same -local IDs in that same collection. They can also define new resources and -local backends using the native format. +### Configuration, Sources, And Prompt Assets -The bundled OpenRouter inventory differs from the former external catalogs -and supplies no backend-local concurrency limit. It does not implicitly -select a model for any Notarius request. Migrate deployments that use removed -profiles or backends to intentionally selected native records. The -application-wide LLM ceiling remains independent of backend-local settings. +Introduce application configuration version 6 as the only supported version. +Preserve unrelated discovery, pipeline structure, and resolution behavior. +ConfigKit continues to own loading and composition under +[ADR-0020](../adr/0020-delegate-configuration-mechanics-to-configkit.md). +Retire `promptkit.profile_file`, `profile_dir`, and `local_backend` in favor of +native companion sources; do not introduce a separate `promptkit.config_file`. +Retire `concurrency.total_llm` and revise retry/worker settings as described below. +Their replacement defaults and detailed migration treatment remain open. -### Native Prompt And Profile Resources +Use one source assembly for runtime, preflight, and inspection: -Convert maintained prompts, profiles, examples, and fixtures to identified -lists, with `config:` for source controls and native sibling inheritance -(`extends: {id: parent}`). Under native default composition, a later same-ID -record replaces the earlier record in full. Explain complete replacements, -distinct inherited variants, and the new backend fallback semantics for -profile nulls. -The optional PromptKit source specification header is version 2; it is not the -Notarius application version. Do not translate legacy resource files, -synthesize aliases, or reconstruct old recursive same-ID overlays. +1. PromptKit's bundled OpenRouter underlay. +2. Notarius's embedded native prompts and profiles under `collections.notarius`. +3. An optional, higher-priority operator underlay containing `prompts.yml` and + recursive `.yml`/`.yaml` files under `prompts/`, beside the main application + configuration actually selected by ConfigKit. -Select prompts by ID alone and remove the obsolete PromptKit version selector -and metadata from Notarius's request, debug, and reconciliation identity -plumbing. Review affected diagnostic fields and document deliberate changes. -Keep the current local resource IDs and use their qualified forms in -lookups and references, for example `notarius/dnd.spells` and -`notarius/dnd-extraction`. Update prompt profile references, application -profile selections, and inspection examples to use the qualified IDs. -The `dnd-extraction` profile retains its current model and settings and -continues to reference the unqualified bundled `openrouter` backend. A -profile that extends it belongs in the same collection and uses its local -sibling ID in `extends`. Retain message roles, bytes, order, cache boundaries, -and intended output/repair behavior while changing the declaration format. -Keep module-owned asset selection and embedded filesystem contexts intact. -Verify maintained private response schemas work under upstream's -self-contained-schema rule; this is not a redesign of durable D&D artifacts. +Anchor companion discovery to that selected main source, whether explicit or +discovered. Configurations in the same directory share companions; there is no +working-directory fallback. The two companion locations are strict peers, so +filenames do not resolve conflicting operator records. Missing companions are +fine; loading errors do not silently fall back to embedded defaults. Preserve +native imports and source-relative resource paths. -### Inspection And Reuse Identity +Native declarations become the sole prompt assembly mechanism. Keep the embedded +content tree and module ownership of prompt meaning, response requirements, and +domain interpretation. Remove prompt/fallback-profile registration in +`AssetRegistry`, per-module `PromptAssetManifest` plumbing, `ModulePromptFS`, +shared-name maps, synthetic `sharedassets` mounts, and manual prompt enumeration +for checkpoint fingerprints. Schema-related registry responsibilities remain +subject to the explicit schema decision below. -Use PromptKit's captured configuration reports for effective configuration -and provenance. Use its resource inventory and inspection APIs for exact -identities, presence-aware resource fields, and captured prompt hashes, rather -than deriving an inventory from rendered configuration YAML. Make only the -changes needed to keep the existing `config effective`, `config explain`, and -`config profile` commands usable after the cutover. Report PromptKit's -resolved profile resource rather than rebuilding the former -`effective_model_params` view or designing a combined profile/backend report. -Retain the existing disclosure boundary, including redaction of -credential-shaped provider parameters. Defer new operator commands and report -surfaces to the CommandKit work. +Convert maintained resources, examples, and fixtures to native identified lists, +`config:` source controls, and sibling inheritance (`extends: {id: parent}`). +Same-ID replacement supplies a complete record; distinct inherited variants can +customize an existing resource. Explain native profile-null/backend fallback +semantics. The optional PromptKit source header is version 2, independently of +Notarius's application version. Do not translate legacy resource formats or +synthesize compatibility aliases. -Retain a private, versioned Notarius checkpoint identity based on the engine's -captured state. Use native resource enumeration and prompt hashes, keeping -the effective execution inputs needed to distinguish reuse safely. A diff of -effective values alone cannot detect edits to a referenced message file. -Version the fingerprint for the cutover so obsolete identity inputs cannot -cause a false hit; do not reread mutable sources or expose the digest inputs. -Keep run manifests based on the profiles actually used, not the inventory. +Select prompts by ID alone; remove obsolete PromptKit version selectors and +related request/debug metadata. Retain current local resource IDs and use exact +qualified references, such as `notarius/dnd.spells` and +`notarius/dnd-extraction`. Keep the extraction profile's existing model/settings +and its reference to the bundled unqualified `openrouter` backend. Inherited +application profiles use sibling local IDs in the same collection. The bundled +inventory differs from the old external catalogs and selects no model for an +application request; document intentional replacements for removed catalog IDs. +The bundled backend supplies no concurrency limit. Application/operator limits +need explicit native configuration; application-supplied defaults remain U7. -### Offline Preflight +Preserve prompt prose, message ordering, roles, and cache boundaries where their +meaning carries forward. Review and deliberately revise initial/repair prompts +for accumulated conversation history; preserving every old byte or retry message +is not a migration requirement. Shared content should use native filesystem +contexts and imports rather than reconstructed virtual trees. -Keep `config validate` focused on validating the resolved Notarius application -configuration. With `--pipeline`, also resolve the selected stages and lanes -against the module catalog. For an LLM-backed selection, load the same -effective PromptKit definitions used by `run`, even when the pipeline does -not explicitly select a profile. Let PromptKit validate and capture those -definitions, including companion sources and referenced message files. -Check the prompt and profile IDs known from pipeline resolution, including a -prompt's default profile when the prompt is known and its default is the -effective selection. Do not simulate stages to discover runtime selections. +PromptKit owns private model-response validation and structural repair. Native +schemas resolve against the configured schema root, separately from message-file +contexts, and v0.23.2 requires self-contained schemas loaded during preparation. +A request override replaces the complete output contract; it must retain the +intended schema, validation mode, and repair policy. A prompt hash alone does not +establish schema equality. -Preflight requires no transcript, credential values, or provider calls. Its -success confirms configuration and known resource selection, not that every -concrete request is runnable. A valid PromptKit profile can be partial. Let -PromptKit check the completed target, required credentials, inputs, and schema -during request preparation. Use `InspectRequirements` to explain recognized -dependencies when useful; use `Prepare` or `ValidateRequest` for an actual -request when no-generation validation is needed. Do not construct synthetic -requests or reimplement target resolution for offline preflight. +The custom read-only filesystem, schema root/path selection, module-supplied +request contracts versus prompt-declared contracts, wrapper removal, and useful +schema metadata remain open. An ordinary embedded schema filesystem, as used by +Audita, is a candidate rather than a selected requirement. Durable artifact +schemas and their validators remain distinct from private response schemas. -### Native Execution And Retained Pipeline Policy +### Native Production, Continuation, And Assessment -Use ordinary `Engine.Run` for structured completion and use `OnPrepared` -where opt-in debug capture needs the actual prepared request. The callback -runs after admission, before generation, and is not delivered for failed -preparation or admission. Keep the callback lightweight and use native -observations for safe terminal execution facts, including attempts, repairs, -and available token usage. Where a diagnostic needs totals over selected -executions, use `ExecutionGroup` rather than inventing a second accumulator. -Use `OperationTimeout` where a completion calls for that scope of budget; -do not add a new default deadline or replace application deadlines that also -cover work outside the PromptKit call. +Start with one conversation per producer operation: a chunk-plan generation, +one lane's extraction of one chunk, or one lane's normalization pass. Use +complete replacement model responses. Normalization may continue to return a +reconciliation plan that deterministic code applies to produce the artifact. +Per-entity branching is not part of this migration. -PromptKit owns provider admission, transport retries, and structural repair. -Keep the Notarius cross-backend scheduler because it serves a different -budget from backend or pipeline worker limits. Keep the adapter's typed -decode, actual-run provenance, and provider-neutral error and diagnostic -mapping. PromptKit execution types stay within that integration boundary. -In particular, preserve the distinction between invalid structured -output, provider failure, capacity rejection, and cancellation. A configured -repair override still selects a complete upstream output contract; it must not -accidentally drop the selected schema or validation mode. +The application flow is: -Notarius owns stage attempts, validation and fallback decisions, and -publication of accepted artifacts. Keep its latest-response-only correction -policy using native appended messages, as defined by -[ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and -[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md). -`Exchange` and `Coordinate` are for workflows that benefit from retained -conversation history. Adopting them for stage retries would change what the -model sees and which work consumes an attempt. They are not needed to execute -the existing retry policy. +1. Prepare source context, references, and candidate-handle mappings once. Build + the native initial request and call `Engine.Run` with exchange capture when + continuation is enabled. Capture is independent of detailed debug recording. +2. Check native structural validation, decode, and apply module interpretation + and safety checks. A module can supply actionable findings before it has a + materialized candidate. Run configured artifact validation on each candidate. +3. For actionable correction, call `Engine.Coordinate` with the captured exchange + and `AdditionalTurns`. Its callback selects a repair prompt and supplies + findings; PromptKit retains the initial exchange and subsequent turns. +4. Interpret every delivered revised response and rebuild from the fixed original + inputs. Run the complete configured validation chain again for changed + candidates, including validators that previously approved. Assess the final + permitted response before declaring exhaustion. +5. Stop on acceptance or a terminal condition. Apply application rejection, + failure, or supported safe-fallback policy after exhaustion. Native `stopped` + and `exhausted` outcomes do not themselves mean application acceptance. -## Migration And Verification +Repair prompts supply a new user task; enduring principles belong in the original +system instructions. Keep source/reference and candidate-handle meanings stable +throughout the conversation. Avoid replaying the original task and constraints +that permit only an initial response. Accumulated history increases context use; +select the semantic turn budget accordingly. Separate repair profiles or schemas +are supported but not yet selected. -Update the maintained minimal and complete D&D examples to distinguish -embedded-only operation from companion configuration. Use separate example -directories if necessary to demonstrate those deployments independently. -Document how to move existing PromptKit sources into the companion layout -or import them from it, declare a local backend, and replace or extend an -embedded resource. Explain qualified IDs, retired catalog IDs, the changed -profile report, and any new checkpoint cold misses. Update the canonical -configuration, CLI, operations, and integration guides when behavior lands. -Record the changed decisions about source precedence and inspection in a -successor to -[ADR-0019](../adr/0019-adopt-native-promptkit-configuration.md) rather than -rewriting its accepted decision text. Release preparation is separate. +Remove `single_response_v1`, raw-response `ModelCandidate`/`SemanticCorrection` +transport, manual assistant/user replay, and correction-protocol capability +checks. Replace `runProducerAttempts`' LLM retry loop with native coordination and +application assessment. Adapt `NormalizeRetry` and `producerRetryDirective` to +findings and explicit fallback information. Retain useful ordered collection and +deduplication of actionable findings, rendered through a native repair prompt. +Reason codes and operator messages are not automatically model instructions. -Verify the dependency graph and offline examples after the cutover. Use -focused behavior tests for the new format and intentional rejection of the -old one, main-relative companion discovery, and native relative imports. -Verify that a companion can replace a qualified embedded prompt or profile -without changing unrelated defaults, and that preflight, inspection, and -runtime use the same effective definitions. Verify that pipeline preflight -catches invalid effective definitions and unknown selected IDs without -requiring inputs or credentials, and does not reject a profile merely for -being partial. Exercise preparation, generation, structural repair, and -semantic retries through the maintained fake-provider integration. Check -that real message or target changes affect checkpoint reuse, and that session -routing, cache controls, diagnostics, and durable outputs keep their existing -meaning. Live provider checks are useful for deployment confidence, not a -requirement of the offline suite. +Transport retries, structural repair, and semantic additional turns remain +separate controls. Preserve the ability to override native transport retries, +using the upstream default when omitted. Remove blanket outer LLM retries after +native transport/structural budgets are exhausted, including repeated LLM +validator invocations in `executeValidationChain`. Keep the ordered validator +chain and aggregation. A validator execution failure is distinct from rejection +of a producer candidate. Existing producer/validator `retries` settings cannot be +mechanically renamed; decide their migration and non-LLM failure handling +explicitly. Use native `OperationTimeout` when that operation scope is needed, +without adding an arbitrary default or replacing broader application deadlines. -## Follow-Up: CommandKit +Keep source grounding, reference resolution, candidate isolation, domain +invariants, and reconciliation safety checks. Unknown or overlapping handles +remain invalid. A supported safe fallback may retain safe merges while leaving +unsafe groups separate; it must pass configured artifact validation and terminal +policy. After structural-repair exhaustion it can be assessed directly without +restarting the same model operation. Whether an earlier safe candidate can +survive a later failed turn remains open; do not silently introduce best-so-far +acceptance. -The planned follow-up adopts `go.maximumdirect.net/commandkit` with its -`ckconfig` and `ckprompt` integrations and replaces Notarius-owned -configuration and PromptKit commands where applicable. That work owns the -standardized CLI and decisions about exposing resource lists, configuration -diffs, request previews, and explicit provider smoke diagnostics. Leave -those commands and the CommandKit dependencies to the follow-up. This -deferral does not change offline pipeline preflight, internal use of native -inspection and observation APIs, or existing run diagnostics. +Preserve rejection versus unavailable/incomplete validation. A structurally +valid candidate may advance under the existing `warn_continue` policy with +explicit incomplete-validation provenance, but it and its affected derivatives +cannot be loaded from or published to reusable checkpoint/cache state. A +completed rejection never advances. Cancellation cannot make an undelivered +response accepted. Keep conversation state in process, outside ordinary +manifests and checkpoints; explicitly requested detailed debug is separate. + +### Admission And Pipeline Workers + +Remove `Scheduler`, `scheduledClient`, and custom global queue/permit plumbing. +Retire the separate cross-backend ceiling and the default/maximum extraction +worker coupling to `concurrency.total_llm`. Keep bounded extraction workers and +bounded lane continuations with independent defaults to be selected. + +Use one shared native engine across all roles. Native capacity belongs to the +engine and backend ID: profiles selecting the same ID share its pool; different +backend IDs or separate engines do not automatically share limits, even for the +same endpoint. Endpoint-only profiles and backends without a positive limit +remain unrestricted by native backend admission. + +For limited backends, v0.23.2 bounds active model attempts by `concurrency_limit` +and admitted operations by that limit plus `queue_capacity`. Admission spans +preparation, retries, and repairs; active permits are released during retry waits. +A completed continuation turn releases admission before assessment, allowing a +nested validator to use the same backend. Configure native limits at the intended +capacity boundary and handle `ErrCapacityExceeded` through application policy. + +Aggregate concurrency across independent backends may exceed the former global +ceiling. That change is intentional. Bounded pipeline workers are application +work controls, not a substitute cross-backend LLM ceiling. Defaults, retired-field +migration, and role-specific admission-failure dispositions remain open. + +### Checkpoint Reuse For Successful Work + +Checkpoints support recovery after a downstream failure: fixing a later stage +should not require rerunning earlier successful work. Use this conceptual key: + +```text +logical stage identity + input-data identity + prompt ID, where applicable +``` + +Scope logical identity to the relevant step/lane/stage and artifact kind, without +hashing complete configuration or implementations. Inputs include actual source +or chunk data, reference material, and consumed upstream artifact content. +Deterministic stages need no synthetic prompt ID. Content digests can represent +inputs and verify payload integrity without fingerprinting the execution setup. + +| Change | Intended reuse behavior | +| --- | --- | +| Fix a later failed stage | Reuse successful predecessors with matching logical work and inputs. | +| Edit prompt content while keeping its ID | Reuse; explicit recomputation applies the edit. | +| Change model, generation settings, implementation, or validation policy | Reuse otherwise usable completed work; explicit recomputation applies improvements. | +| Change the prompt ID or consumed source/reference/upstream data | No matching checkpoint. | +| Request explicit recomputation | Recompute the selected work and its dependents. | + +Keep recording separate from reuse. The existing controls are +[`--resume` and `--recompute-step`](../cli.md#run); migrate their compatibility +behavior to this policy. A matching key does not make an unreadable, corrupted, +or incompatible artifact usable. Retain acceptance/completion status, integrity, +and decode/consumer compatibility checks. Exact schema hash equality is not +required for a compatible artifact. + +Only accepted, completely validated work is reusable. Preserve original +execution and acceptance provenance; do not rerun semantic validators solely +because their configuration changed or claim the old artifact passed a new +policy. Retain the exclusion of rejected, structurally invalid, and +validation-incomplete work and its ineligible derivatives. The accepted tradeoff +is reuse of results produced by prompts, settings, or code that have improved. + +Remove whole PromptKit configuration, catalog/library versions, model settings, +prompt-content hashes, and component-policy fingerprints as matching criteria. +Native hashes may remain diagnostic metadata. Revise storage/lookup scoping so +an invocation-wide pipeline/configuration digest cannot still block recovery. +There is no need for an upstream semantic-equality feature, persisted conversation, +native prepared-execution handle, or temporary conservative fingerprint. + +Keep the separate source-addressed canonical chunk-plan cache policy, including +current validation of materialized chunks. Concrete checkpoint identifiers, +input representation, old-record transition, and incompatible-payload reporting +remain open. + +### Inspection, Bookkeeping, And Presentation + +Use native configuration/resource inspection rather than effective-YAML parsing, +a profile-inspector wrapper, or copied effective-model-parameter types. A profile +can be partial; its inspection is not the completed execution target. Use actual +prepared/result data for execution provenance, including request overrides, and +record the profiles/targets used rather than the available catalog. + +Keep application pipeline preflight for modules, typed artifact compatibility, +reference bindings, and known prompt/profile selections. `config validate` checks +resolved application configuration; with a pipeline selection, load the same +native definitions used by execution, including a known prompt's default profile +when effective. Do not simulate stages to discover dynamic selections. Preflight +needs no transcript, credential values, or provider call. Use native requirement +inspection for explanation and concrete `Prepare`/`ValidateRequest` for completed +target, input, schema, and credential checks when requested. Partial profiles +alone are not preflight failures. + +Use ordinary `Run` with `OnPrepared` for needed actual request details and optional +debug capture. Remove forced `PrepareExecution`/`RunPrepared` solely for recording. +`OnPrepared` runs after preparation/admission and before generation; it is absent +on preparation/admission failure and cannot be a terminal record. Native +observations supply terminal facts even when a failed call returns no `RunResult`. +Keep callbacks lightweight and synchronize shared observation state as required. + +Use native `RunID`/`ParentRunID` plus application step/lane/chunk/validator scope. +Remove duplicate `LLMExecutionSummary`, usage types, enum conversions, cloning, +independent accumulators, debug completion wrappers, and redundant synthetic +call IDs. Replace adapter profile-recording/forwarding with only the projection +needed at application boundaries. Preserve native failure distinctions while +Notarius chooses domain outcomes and exit-status mapping. + +Use `ExecutionGroup` for totals over selected executions. Enroll the initial run, +continuation runs, and nested validator executions explicitly as applicable. +Preserve terminal coverage and usage completeness: unknown usage is not zero. +Avoid counting both attempt and terminal aggregates or adding overlapping groups. +Provider execution success remains separate from domain acceptance. + +Retain application records of acceptance, rejection, validation completeness, +fallback, generation versus reuse, and pipeline outcome. Reused artifacts retain +original provenance; the current invocation records reuse without counting old +token usage as current work. Keep small deliberate projections for the +[published manifest](../integrations/json-output.md) and +[run-result receipt](../integrations/run-result.md), with explicit compatibility +decisions. Do not serialize complete upstream runtime types as durable contracts. + +Notarius retains debug placement, opt-in capture, retention, and disclosure +handling, including credential redaction. Native callbacks provide the data; +ordinary provenance retains only needed fields, and sensitive response/history +content stays out of ordinary records. Debug is not checkpoint input. + +CommandKit with ckconfig/ckprompt is the selected owner of standardized +configuration/resource commands, requirements, validation, preview, opt-in smoke +diagnostics, formatting, visibility, and progress presentation. Notarius supplies +loaders, request builders, report content, pipeline meaning, and outcome mapping. +Keep machine-output receipts deliberate and presentation delivery failures +distinguishable from operation outcomes. Until that follow-up lands, make only +the changes needed to keep existing inspection commands usable through native +APIs; do not build temporary custom commands or a new reporting framework. + +## Unresolved Choices + +The directions above are selected. The following details are not. Resolve and +record each choice before implementing the work that depends on it; independent +work need not wait for every choice. Candidates here are not implicit defaults. + +| ID | Choice still needed | Resolve for | +| --- | --- | --- | +| U1 | Final compatible tagged PromptKit/ConfigKit/dependency set; any specific unmet need requiring an upstream request. v0.23.2 is the initial target, not a restriction on useful upstream work. | Phase 1; revisit if the target or upstream needs change. | +| U2 | Remaining custom read-only filesystem responsibilities; schema root/path and assembly, prompt-declared versus per-request contracts, removable wrappers, and metadata useful for manifests/artifact compatibility. | Phase 2 source and schema assembly. | +| U3 | Concrete module/pipeline signatures and placement of engine assembly, request construction, interpretation, assessment, and focused helpers. | Phase 3 representative workflow, refined during Phase 4. | +| U4 | Initial and repair prompt definitions, response contracts, and profile selection within complete replacement and fixed operation inputs. | Phases 3–4 for each producer role. | +| U5 | Mapping producer/validator retry settings to native controls; names, defaults, compatibility, non-LLM failure handling, and any explicitly justified recovery beyond native exhaustion. | Phases 3–4 and configuration migration. | +| U6 | Detailed terminal/fallback dispositions, especially whether any earlier safe candidate survives a later failed turn. | Phases 3–4; no implicit best-so-far acceptance. | +| U7 | Independent worker defaults, native backend/queue defaults where supplied, retired `total_llm` migration, and admission-failure handling by role. | Phase 4 shared-engine cutover. | +| U8 | Logical checkpoint/prompt identifiers, consumed-input representation, storage scoping, old-record transition, and load-compatibility reporting. | Phase 5 recovery migration. | +| U9 | Minimal durable fields and their compatibility treatment; observer/group enrollment, application scope, and debug write/disclosure handling. | Design with Phase 3, complete in Phase 5. | +| U10 | CommandKit/ckconfig/ckprompt versions, command exposure and compatibility, and application wiring for the follow-up. | Phase 7; do not recreate its machinery earlier. | + +## Implementation Sequence + +Each phase is pending. Phases describe reviewable outcomes and dependencies, +not separate releases. The supported core result is the hard cutover; any +intermediate migration scaffolding must retire before that result is complete. +Update affected canonical documentation as behavior lands. Record significant +boundary decisions in successor ADRs without rewriting accepted decision text. + +### Phase 1: Establish The Migration Baseline + +- Confirm U1 against the tagged APIs and dependency graph. Carry forward the + Audita evidence without treating its wrappers as requirements. +- Map the changed architecture and ADRs, and identify concrete decisions needed + for the next phase. Keep unresolved entries visible until actually settled. +- Identify the production callers and old contracts covered by the removal + inventory below; use that inventory to track complete cutover coverage. + +**Completion evidence:** selected dependency baseline and decision/implementation +scope are recorded; unresolved choices have a clear dependent phase. No new +upstream feature is assumed necessary without an identified requirement. + +### Phase 2: Assemble Native Sources And A Shared Engine + +- Resolve U2, update dependencies, and implement version 6 and companion-source + composition. Convert embedded resources and establish the schema loading path. +- Construct the shared native engine and use the same assembly for execution, + preflight, and inspection. Remove duplicate prompt assembly as callers migrate. +- Add maintained embedded-only and operator-override examples with the new format. + +**Completion evidence:** offline preparation proves resource selection, shared +content/schema resolution, source precedence, and operator replacement. Invalid +sources fail rather than silently selecting defaults. Preflight needs no provider +credentials or synthetic stage execution. + +### Phase 3: Prove Native Production And Assessment + +- Resolve U3 and the relevant U4–U6/U9 choices for a representative extraction + workflow. Connect native generation, domain interpretation, configured + validation, and continuation through the small application function. +- Exercise a configured LLM validator on the same engine. Attach native observers, + application scope, and selected execution-group membership from the start. +- Exercise normalization's fixed candidate handles and fresh plan application, + including the selected terminal/fallback policy. + +**Completion evidence:** fake-provider workflows demonstrate accumulated history, +full revalidation, final-turn acceptance, correct exhaustion/failure handling, +and typed application outcomes without the private completion/correction API. + +### Phase 4: Migrate All Roles And Retire Runtime Machinery + +- Apply the established flow to every LLM-backed chunker, extractor, normalizer, + and validator. Preserve deterministic stage paths and ordered validation. +- Complete U4–U7 across roles. Migrate retry controls, remove the global scheduler, + and give bounded pipeline workers independent configuration. +- Remove the adapter, copied completion contracts, correction-protocol checks, + outer LLM retry loops, and redundant asset registration once unused. + +**Completion evidence:** all production roles share native execution and admission; +no application path still depends on the retired completion API, global ceiling, +or latest-response-only correction. Native failures and cancellation settle +through the selected role policy. Tests for obsolete machinery retire with it. + +### Phase 5: Simplify Recovery And Execution Records + +- Resolve U8–U9. Implement logical-work/input/prompt checkpoint matching and revise + storage scoping so unrelated configuration changes cannot defeat recovery. +- Implement and document old-checkpoint handling, artifact usability checks, + original acceptance provenance, and explicit recomputation behavior. +- Complete native inspection, observation/group accounting, minimal durable + projections, and optional debug capture; remove the duplicate bookkeeping. + +**Completion evidence:** a downstream-failure/fix/resume scenario reuses successful +predecessors; settings changes do not spuriously invalidate them, consumed-data +changes do, and reused work does not count as current execution. Published records +follow the selected compatibility policy and preserve failure/usage distinctions. + +### Phase 6: Verify And Document The Core Cutover + +- Complete the acceptance checks below and the removal inventory. Verify the + dependency graph contains no unused old-path PromptKit/ConfigKit or catalogs. +- Finish migration guidance, maintained examples, attribution, successor ADRs, + and canonical documentation updates. Reconcile affected future-work notes. +- Keep existing inspection commands usable with native reports. Document that + the standardized CommandKit command surface remains the subsequent phase. + +**Completion evidence:** the core native architecture, configuration migration, +checkpoint policy, diagnostics, and maintained offline workflows are verified; +remaining work is explicitly the CommandKit follow-up or separately identified +future work, not hidden adapter or retry compatibility machinery. + +### Phase 7: Adopt CommandKit Presentation + +- Resolve U10 and adopt CommandKit with ckconfig/ckprompt. Supply application and + engine loaders plus request builders for the selected diagnostic commands. +- Replace applicable Notarius-owned configuration/PromptKit commands and standard + presentation while preserving deliberate machine-output contracts and domain + outcome mapping. Keep provider smoke diagnostics explicitly opt-in. + +**Completion evidence:** standard commands use upstream integrations and the same +native source assembly as execution; application preflight and published receipts +retain their intended meaning. This phase completes the presentation target and +is not a prerequisite for completing the core PromptKit cutover. + +## Removal Inventory + +| Retire or replace | Retained application responsibility | +| --- | --- | +| `StructuredLLMClient`, private structured-completion request/response types, adapter conversions | Native request construction and typed domain interpretation. | +| Prompt/profile `AssetRegistry` plumbing, `PromptAssetManifest`, `ModulePromptFS`, synthetic shared mounts, manual fingerprint enumeration | Embedded content and module semantics; schema responsibilities resolved under U2. | +| `single_response_v1`, raw `ModelCandidate`/`SemanticCorrection`, manual replay, correction capability checks | Typed candidates, actionable findings, native exchanges, and assessment. | +| LLM retry portions of `runProducerAttempts` and `executeValidationChain`, module rerun directives | Ordered validators, acceptance/rejection/fallback policy, and useful provenance. | +| `Scheduler`, `scheduledClient`, `concurrency.total_llm`, worker coupling | Bounded pipeline work and native per-backend admission. | +| Broad configuration/prompt/catalog/implementation checkpoint fingerprints and invocation-wide lookup barriers | Logical work, consumed-input identity, load checks, and explicit recomputation. | +| Inspection wrappers, copied effective parameters, `LLMExecutionSummary` and usage conversions/accumulators, debug completion wrapper, redundant call IDs, adapter profile recorder | Native facts with application scope and selected durable projections. | +| Custom standard inspection commands/presentation in the CommandKit phase | Application loaders, request builders, report content, and outcome mapping. | + +## Acceptance And Verification + +Use focused behavioral tests with real deterministic collaborators, a native +engine, and injected clients or a fake HTTP provider. Test domain interpretation +and assessment directly; pipeline tests can use fake stages. Replace tests that +preserve removed mechanics rather than reproducing a private LLM interface for +testing. Default tests remain offline and independent of real credentials. + +| Area | Required evidence | +| --- | --- | +| Configuration and assets | Version 6/new resources work; legacy formats are rejected intentionally; main-relative companions and native imports work independently of working directory; peer conflicts fail; same-ID replacement affects only the intended default; ordering/cache controls and selected schema contracts survive assembly. | +| Preflight and inspection | Runtime, inspection, and preflight share effective definitions; unknown statically selected IDs and invalid definitions fail offline; partial profiles alone pass; completed target checks occur on concrete requests; actual-run overrides appear in provenance. | +| Native artifact workflow | Representative extraction and normalization produce typed candidates, invoke ordered validators, and preserve source/reference identity, safe plan application, artifact-schema checks, and stable output ordering. | +| Continuation and failures | Multiple turns accumulate history with fixed input/handle meanings; every revised candidate and final allowed response is assessed; native transport/structural exhaustion is not silently retried outside PromptKit; validator failure differs from rejection; selected safe fallback and cancellation policy hold. | +| Admission and workers | All roles share the selected backend pool; independent backends remain independent; pipeline work stays bounded; queue exhaustion, nested validators, and cancellation settle without leaked capacity or a retained global ceiling. | +| Checkpoints | Successful predecessors survive downstream fixes and prompt/model/validation-setting changes; changed prompt IDs or consumed source/reference/upstream data miss; recompute forces selected work/dependents; corrupt/incompatible or unaccepted/incomplete artifacts and ineligible derivatives are unusable; compatible schema changes do not require exact hash equality. | +| Reuse provenance | Original acceptance and execution provenance survive resume without new semantic validation claims or historical token usage counted as current; recording remains separate from reuse; canonical chunk-plan caching retains its distinct policy. | +| Execution records | Initial/continuation/nested validator accounting preserves terminal coverage and unknown-versus-zero usage without double counting; failures without a result remain observable; native success is distinct from acceptance; debug is opt-in and ordinary records exclude sensitive response/history content. | +| Published behavior | Maintained minimal and complete D&D workflows, session routing, artifact output, manifest/receipt compatibility, and selected migration diagnostics work through the assembled CLI. CommandKit-phase tests cover its own command/presentation changes. | + +Run focused checks while implementing and the repository-wide commands for the +completed application cutover, following the [testing policy](../policy/testing.md): + +```sh +go test ./... +go vet ./... +go build ./cmd/notarius +``` + +Check maintained examples, dependency removal, links, and attribution alongside +those commands. Live provider checks and human model-quality evaluation can +inform deployment and prompt refinement; they are not requirements of the +deterministic offline suite. Planning/documentation completion does not claim +that these implementation acceptance checks have already passed. + +## Migration Guidance And Documentation + +Maintain examples that distinguish embedded-only operation from companion +configuration, using separate directories when needed. Explain moving/importing +old sources, qualified IDs, complete replacement versus inheritance, native local +backends, removed catalogs and prompt versions, and the separate application and +PromptKit source versions. Document worker/global-limit changes, retry-budget +migration, accumulated conversation costs, and the selected failure policy. + +Explain successful-work checkpoint reuse, explicit recomputation to apply prompt +or setting improvements, consumed-data misses, original validation provenance, +and the chosen old-record transition. Document changed inspection and durable +fields deliberately rather than promising incidental old adapter output. + +Update the canonical [configuration](../config.md), [CLI](../cli.md), +[operations](../operations.md), [internal overview](../internal/overview.md), +[LLM](../internal/llm.md), [pipeline](../internal/pipeline.md), +[modules](../internal/modules.md), [state](../internal/state.md), and affected +integration/consumer guides when their behavior changes. Update the +[architecture policy](../policy/architecture.md) to make native PromptKit an +explicit execution dependency and remove the superseded scheduler, retry, +asset-registration, and fingerprint constraints. + +Record significant ownership and contract changes in successor ADRs, including +those affecting [asset assembly](../adr/0011-centralize-llm-assets.md), +[validation retries](../adr/0014-feedback-aware-validation-retries.md), +[module-requested retries](../adr/0016-feedback-aware-module-requested-retries.md), +and [PromptKit configuration](../adr/0019-adopt-native-promptkit-configuration.md). +Also record the changed admission and checkpoint matching decisions. Preserve +accepted decision text and update supersession status as appropriate under the +[documentation policy](../policy/documentation.md#architecture-decision-records). +The discussion record supplies rationale and evidence; this roadmap tracks +implementation status until the corresponding behavior lands. [release]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.2/docs/releases/v0.23.2.md [migration]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.2/docs/consumers/configuration-migration.md