Consolidate migration planning into the master roadmap

This commit is contained in:
2026-11-20 13:06:52 +00:00
parent f883316d7a
commit 2d69c26813
6 changed files with 124 additions and 2166 deletions

View File

@@ -63,7 +63,9 @@ Releases](docs/release.md).
for maintainers.
- [Developer guide](docs/development.md) — contributor orientation and
validation guidance.
- [Future work](docs/roadmap/future.md) — unimplemented ideas and priorities.
- [Native migration roadmap](docs/roadmap/promptkit.md) — the planned PromptKit,
ConfigKit, and CommandKit cutover.
- [Future work](docs/roadmap/future.md) — separate backlog of ideas and priorities.
Notarius is licensed under the [GPL v3.0 or later](COPYRIGHT).

View File

@@ -48,7 +48,7 @@ implementing changes. It owns implementation discipline and local commit rules.
| Output, cache, resume, or debug artifacts | [Run State Internals](internal/state.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. |
| External input formats, artifact schemas, or durable output files | [Integration Contracts](integrations/) | Integration documents define external and durable data contracts. |
| Repository-specific release commands or diagnostics | [Source Releases](release.md) | The runbook supplies publication commands and recovery procedures. |
| Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. |
| Proposed or unimplemented behavior | [Native migration roadmap](roadmap/promptkit.md) and [separate backlog](roadmap/future.md) | The master roadmap owns the native cutover; the backlog holds other future work. Implemented behavior belongs in its canonical documentation. |
For an existing subsystem, also inspect its focused tests and the package-local
types and contracts before changing behavior.

File diff suppressed because it is too large Load Diff

View File

@@ -1,30 +1,18 @@
# Future Work
Current Notarius behavior is documented in the canonical README, CLI,
configuration, operations, internal, and integration docs. This roadmap records
future work only. Items are ordered roughly by current value and specificity,
not as committed release dates.
## Near-Term Validation And LLM Reliability
PromptKit now owns structural output repair within one completion. Notarius
owns stage candidates, validator chains, semantic rejection policy, bounded
feedback-aware stage retries, validation provenance, and reusable-state
eligibility, and the separation of actionable process warnings from quality
diagnostics. The remaining near-term work applies those completed foundations
to domain review and empirical evaluation.
Near-term reliability work should now be selected from the concrete evaluation
and extension opportunities below. The retry, validation, and subprocess
foundations described above are implemented current behavior.
configuration, operations, internal, and integration docs. This backlog records
future work outside the [native migration roadmap](promptkit.md), which owns
PromptKit/ConfigKit/CommandKit adoption, continuation, admission, cache, output,
and debug decisions. Items below are ordered roughly by value and specificity,
not as committed release dates or prerequisites for that migration.
## Near-Term D&D Pipeline
### Evaluate Spell Extraction And Normalization
- Evaluate ordinary extraction retries and the completed normalization path
against a human-reviewed transcript set before and after adopting the shared
PromptKit repair and Notarius validation-retry policies above.
- Evaluate extraction, normalization, and correction quality against a
human-reviewed transcript set before and after the native migration.
- Maintain a small set of human-reviewed transcripts and outputs for prompt,
validator, and normalizer development. Treat model-quality review as an
iterative human evaluation aid, not a deterministic correctness gate.
@@ -46,8 +34,8 @@ The sections below keep broader extensions deferred.
### Large-Collection Semantic Reconciliation
- Evaluate deterministic candidate blocking only after representative registry
inputs exceed the active roadmap's bounded single-request limits. Blocking
should use cheap, explainable signals to form plausible comparison sets while
inputs exceed practical single-request capacity. Blocking should use cheap,
explainable signals to form plausible comparison sets while
preserving the possibility that a duplicate appears outside a lexical name
match.
- Define correctness for candidates that appear in more than one block,
@@ -61,16 +49,15 @@ The sections below keep broader extensions deferred.
### Operator-Selected Semantic Policies
- Consider allowing an operator to select an approved semantic-policy prompt
for a typed reconciliation module without replacing the shared protocol,
response schema, or deterministic safety rules.
- Define the trusted asset source, configuration syntax, compatibility checks,
startup validation, provenance, prompt fingerprinting, checkpoint effects,
and support boundary before exposing the option.
- Prefer selection among registered, typed-policy-compatible prompt assets over
arbitrary filesystem prompt paths. Do not add this flexibility until an
operator workflow requires it; artifact-family-owned policy remains simpler
and safer for the initial implementation.
- Consider a dedicated semantic-policy selection option for a typed
reconciliation module when a concrete operator workflow needs more than the
migration's [native prompt replacement](promptkit.md#configuration-sources-and-prompt-assets).
- Use native prompt IDs and sources while retaining typed interpretation and
deterministic safety rules. Define any additional configuration and domain
compatibility checks from that workflow; do not recreate asset registration,
prompt fingerprints, or checkpoint dependency tracking.
- Reuse the selected [accepted-artifact cache policy](promptkit.md#accepted-artifact-cache)
when evaluating policy changes and the need for explicit refresh.
### Broader Reconciliation Inputs And Module Selection
@@ -89,9 +76,8 @@ The sections below keep broader extensions deferred.
artifact.
- Add production LLM-backed validators only when a concrete review policy
benefits from model judgment and deterministic checks are insufficient.
- Add validator diagnostics and timing summaries if operators need more detail
than the current [durable output bundle](../integrations/json-output.md)
provides.
- Consider additional validator diagnostic views only if a concrete need remains
after the selected [native observations and debug capture](promptkit.md#inspection-and-execution-bookkeeping).
- Add validator compatibility metadata if deployments need config-time proof
that a validator is suitable for a particular stage, module, or artifact
kind.
@@ -133,10 +119,11 @@ occurred.
Define compatibility and migration policy before generated-reference chains
must span multiple schema versions or long-lived historical artifacts. The
policy should address stable identifier semantics, which schema changes permit
checkpoint reuse, when an older artifact may be decoded or adapted, and when a
producer or all dependents must be recomputed. Do not add a general migration
framework until an actual contract change requires one.
policy should address stable identifier semantics, when an older artifact may
be decoded or adapted, and when consumers need regenerated artifacts. For cached
artifacts, use the [selected reuse policy](promptkit.md#accepted-artifact-cache)
and explicit refresh where needed. Do not add a general migration framework
until an actual contract change requires one.
### Artifact-family-oriented physical packaging
@@ -170,7 +157,6 @@ section only after a concrete workflow, contract, and priority emerge.
### Workspace And Storage
- Default-idempotent run behavior with an explicit force override.
- Remote workspace storage.
- Workspace garbage collection and archival policies.
- Cross-machine checkpoint reuse.
- Cross-machine sharing of accepted-artifact cache entries.

View File

@@ -1,842 +0,0 @@
# PromptKit Architecture Discussion Record
## Purpose And Status
The decisions in items 1–6, the subsequent CommandKit discussion, and the
simplified cache/output designs are incorporated into the
[master native migration roadmap](promptkit.md). It owns the selected target,
unresolved choices, sequencing, acceptance criteria, and implementation status.
The [CommandKit discussion record](commandkit-migration-discussion.md) retains
the related CLI, preflight, output, debug, and migration rationale. The master
roadmap replaces earlier checkpoint matching/provenance proposals with the
accepted-artifact cache and retires mandatory manifests, extensive receipts,
and durable execution ledgers. The upstream message-content hash remains pending;
incorporation does not claim it is already available or implemented.
This document retains the discussion, rationale, and evidence, including item 7's
completed Audita comparison. Item-level questions and follow-ups below record
the discussion as it developed; they are not a second implementation plan.
The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit
capabilities as fully as practical. Application ownership of a policy does not
necessarily require application ownership of its implementation. Evaluate
remaining machinery against this goal, including opportunities for upstream
features where PromptKit does not yet meet an application need.
No application changes have been implemented through this discussion or the
roadmap rewrite. Significant changes to accepted architecture need successor ADRs
under the [documentation policy](../policy/documentation.md#architecture-decision-records).
Current boundaries are described in [Architecture](../policy/architecture.md)
and [LLM Runtime Internals](../internal/llm.md).
## 1. LLM API And Diagnostic Boundary
**Discussion / decision:** Direction agreed: remove the Notarius PromptKit
adapter and compose the application directly from native PromptKit execution,
typed artifact operations, and pipeline orchestration. A small application
function connects generation to configured validation and acceptance. The design
below incorporates items 2–6 and the Audita review; exact signatures and migration
details remain open. No application changes have been implemented.
### Background And Intent
Notarius and related applications, including Audita, originally developed their
own LLM interfaces or used the OpenAI SDK directly. PromptKit emerged by
extracting their duplicated mechanics into a shared library. Notarius adopted
an early version and predates many of the library's modern capabilities.
The maintainer clarified that duplication of PromptKit capabilities, including
slight behavioral differences, does not reflect an intentional requirement for
an independent Notarius implementation. Existing wrappers, tests, and historical
decisions are migration evidence, not sufficient reasons to preserve that
duplication. Identify actual application needs before choosing what to retain.
Missing reusable capabilities may justify upstream feature requests. The
discussion is not constrained to what v0.23.2 can replace immediately; any
resulting change to the adoption target remains to be selected.
### Application Structure
| Part | Responsibility |
| --- | --- |
| Application composition | Load configuration, assemble native prompt sources, and construct one shared PromptKit engine for producers, validators, and continuation runs. |
| Artifact-family modules | Supply prompt inputs and response requirements, interpret responses, and apply domain rules. |
| Pipeline execution | Connect generation to configured validation, decide acceptance or fallback, and manage dependencies and bounded workers. |
| PromptKit | Prepare and execute requests, maintain conversations, enforce native budgets and admission, and report execution facts. |
| Application records and CLI | Store artifacts and accepted-artifact cache entries, attach pipeline meaning to execution facts, and present results through CommandKit. |
Engine construction becomes ordinary application wiring using item 2's embedded
defaults and operator sources. Profile selection is expressed through native
requests. U3 now places shared loaders and engine construction in `internal/cli`
and the producer helper beside pipeline assessment. Retire the emptied
`internal/framework/llm` package rather than retaining a constructor-only package.
The [master roadmap](promptkit.md#module-execution-and-assessment-u3-resolved) owns
the resolved design; implementation is pending.
Use native requests, results, exchanges, errors, and observers directly in
model-executing application code. Pure domain models, codecs, source-reference
resolution, and deterministic transformations need no PromptKit dependency.
Retain application interfaces for extracting artifacts, validating typed
candidates, encoding output, and loading accepted cached artifacts. Deterministic
stages continue to operate without conversation machinery.
### Connect Modules To Pipeline Assessment
The current [spell extractor](../../internal/modules/dnd/extract/spells/extractor.go)
prepares inputs, invokes the private completion interface, interprets a response,
and returns a copied raw model candidate to the
[producer retry loop](../../internal/framework/pipeline/producer_attempts.go).
Replace that arrangement with direct collaboration between module interpretation,
pipeline assessment, and native execution:
1. The module prepares source material, references, and any candidate-handle
mappings once, and constructs the native initial request.
2. Application execution calls `Engine.Run`, capturing an exchange when
continuation is enabled.
3. The module interprets the structurally valid response into a typed candidate
or actionable domain findings.
4. The pipeline runs configured candidate validation and decides whether to
accept, request correction, or apply terminal policy.
5. If correction is needed, native `Coordinate` invokes the same interpretation
and assessment for revised responses. The module supplies the repair request;
PromptKit maintains history and enforces the turn budget.
The resolved U3 design uses a small shared helper in `internal/framework/pipeline`.
Modules supply native initial/correction requests and response interpretation;
the pipeline supplies an operation-scoped assessment callback and relevant
cache/debug collaborators. Local variables or closures hold operation state.
Use the same assessor for initial and revised candidates without an additional
final pass. Native preparation supports cache lookup before execution; hits load
the accepted artifact directly. The helper returns a typed application outcome
and saves only final accepted artifacts under the initial hash.
Keep deterministic stages on their ordinary typed production/validation path.
LLM validators call the shared engine directly and return findings without an
automatic producer correction loop. No replacement completion API, execution-plan
language, or copied conversation model is selected.
Item 3 owns the detailed continuation and assessment rules, including final-turn
assessment, failure handling, complete replacement responses, and safe fallback.
For normalization, a revised reconciliation plan is applied to the original
inputs to produce a fresh candidate. LLM-backed validators can call the same
native engine and return application validation findings; they do not need the
producer's conversation machinery.
### Removal, Records, And Testing
Retire `StructuredLLMClient`, the private structured-completion request/response
family, adapter conversions, and opaque correction transport. Items 3, 4, and 6
specify removal of the outer LLM retry loop, scheduler, debug completion wrapper,
profile-recording plumbing, and duplicate execution accounting. Do not recreate
that wrapper stack around native calls.
Attach pipeline scope through native callbacks and observers. Use native
`ExecutionGroup` totals for operational diagnostics and optional debug artifacts,
following item 6. The small completion report follows the
[CommandKit output contract](commandkit-migration-discussion.md#4-reports-machine-output-and-outcomes).
Item 5 now selects a simple accepted-artifact cache, without required durable
acceptance or execution-provenance records.
Audita demonstrates shared engine assembly, native continuation, and CommandKit
integration. Its `RoleClient`, structured-output interface, and opaque exchange
wrapper are not part of this design; item 7 records the comparison's evidence
and limits.
Test domain interpretation and assessment directly. Exercise model workflows
with a real PromptKit engine and its injectable client or a fake HTTP provider.
Pipeline orchestration can still use fake stage implementations. Testing does
not require preserving a private completion interface.
### Remaining Implementation Work
- Implement the resolved U3 ownership and placement; refine function signatures
and argument grouping against representative extraction and normalization.
These implementation details do not reopen the selected boundaries.
- Implement item 2's resolved asset/schema design and settle the remaining
migration details recorded under items 3–6, including durable compatibility.
- Evaluate any remaining upstream gap against an actual application requirement,
rather than preserving historical machinery by default.
**Follow-up:** Reconcile the adoption roadmap with this design and items 2–6.
Record the significant boundary change in successor ADRs. Update architecture
and implementation documentation when the change lands, explicitly replacing
the current transport-neutral completion boundary with PromptKit as the execution
dependency of application workflows. Keep domain operations and application
outcomes as the meaningful Notarius-owned boundaries.
## 2. Prompt Assets And Schema Assembly
**Discussion / decision:** Agreed and incorporated as resolved U2 in the
[master roadmap](promptkit.md#prompt-assets-and-response-schemas-u2-resolved).
The filesystem and schema questions initially deferred here are now settled.
Implementation is pending; the master roadmap owns the detailed design and checks.
Use ordinary embedded assets, native prompt declarations, and native schema
loading. Keep the content tree and module ownership of prompt meaning and domain
interpretation. Shared files use real native filesystem paths. Remove duplicate
asset registration, per-module manifests, synthetic mounts, flattened schema
namespaces, and the custom read-only filesystem that served that assembly.
Prompt definitions select output contracts by default. Initial and correction
prompts can share schema files. Use native per-request replacement only for a
demonstrated runtime-dependent contract, without an application wrapper or schema
registry. Audita demonstrates ordinary embedded schema loading and per-call
contracts; its per-call selection is not required for every Notarius operation.
The default schema source is the embedded asset tree. Operator-selected external
schema directories use native whole-root replacement, not an application overlay.
Prompt underlays do not implicitly overlay schema files. PromptKit owns private
response-schema loading, structural checks, and repair; Notarius keeps typed
interpretation, domain validation, and distinct durable artifact schemas/codecs.
Retire response-schema hashes and copied metadata used only by the adapter,
checkpoints, or reporting. Preserve actual artifact contracts and native diagnostic
snapshots. Neither asset nor schema metadata adds another cache fingerprint.
The spell extractor illustrated the duplication: prompt YAML declared ordered
messages and a schema while a Go manifest separately listed, mounted, registered,
and hashed assets. Native declarations now supply the selected assembly model.
**Follow-up:** Implement the master roadmap's Phase 2 and retire tests of removed
mounting/registry machinery. Verify shared paths, operator replacements, native
schema sources, and any necessary complete request-contract replacement.
## 3. Retry And Semantic Correction Ownership
**Discussion / decision:** Direction agreed: adopt Audita's native continuation
approach. PromptKit owns captured exchanges, accumulated conversation history,
continuation execution, and turn-budget enforcement. Notarius supplies semantic
assessment, corrective findings, and acceptance or fallback decisions. Use native
PromptKit types directly, consistent with item 1; Audita's adapter and opaque
exchange wrapper are not part of the selected direction.
The flow and removal/adaptation direction below are agreed. Begin with one
conversation per producer operation and complete replacement model responses.
Retain application assessment, safe-fallback decisions, artifact validation,
and cache eligibility. Detailed response contracts, budget migration, and
the open policy questions below remain to be resolved. No application changes
have been implemented.
### Agreed Flow
For cacheable operations, item 5's initial preparation and lookup precede this
flow. The steps below describe execution on a miss.
1. Prepare the source context, references, and any request-local candidate-handle
mapping once for the producer operation. Run its initial native request with
exchange capture when continuation is enabled. Capture is independent of
detailed debug recording.
2. Handle the native call's terminal success or failure, decode a usable response,
and perform the module's deterministic interpretation and safety checks. Run
the configured validation chain on a materialized candidate. A module can
identify actionable defects before a candidate is ready for that chain.
3. If correction is needed, call `Engine.Coordinate` with the initial exchange
and a configured `AdditionalTurns` budget. The callback selects a native
repair prompt and supplies actionable findings as variables. PromptKit retains
the original exchange and each subsequent repair turn.
4. Assess every delivered response, including the last permitted turn. Rebuild
the typed candidate from the fixed operation inputs and the revised response;
do not repeatedly apply revisions to an already modified artifact. A changed
candidate receives the complete configured validation chain again, including
validators that approved an earlier version.
5. Stop on acceptance or a terminal condition. After exhaustion, Notarius decides
whether to reject, fail, or evaluate an explicitly supported safe fallback.
Native `stopped` and `exhausted` outcomes describe coordination, not acceptance.
Cancellation never turns an undelivered response into an accepted candidate.
Start with one conversation per producer operation: a chunk-plan generation,
one lane's extraction of one chunk, or one lane's normalization pass. This aligns
with Notarius's whole-output validation boundary. Audita's per-proposal branches
are useful evidence, but per-entity repair would require its own combination and
validation design. It is not necessary to adopt native continuation.
Repair prompts should supply a new user task, with enduring principles in the
original system instructions. Avoid replaying the initial task or embedding
initial-response-only constraints in system messages. Begin with complete
replacement model responses; for normalization, the response can remain a
reconciliation plan that Notarius applies to construct the durable artifact.
A different repair schema or profile is supported but not yet selected. Keep
source context and candidate-handle meanings stable throughout the conversation.
Accumulated history increases context and token use across turns; the semantic
turn budget should reflect that tradeoff.
### Machinery To Remove Or Adapt
| Current machinery | Agreed direction |
| --- | --- |
| `CorrectionProtocolSingleResponseV1`, raw-response `ModelCandidate`, and `SemanticCorrection` | Retire the latest-response-only transport protocol and its copy/validation plumbing. Keep the typed domain candidate and use native `RunResult` / `Exchange` for continuation state. |
| Adapter code that rebuilds the original request and appends an assistant/user pair | Remove with the adapter. Native continuation assembles history; native repair definitions render findings. |
| `runProducerAttempts` combining operational, structural, module-requested, and validator-semantic retries | Replace its LLM retry loop with native coordination plus application assessment and terminal disposition. Retain useful stage provenance, not a second turn counter or general retry engine. |
| Outer LLM retries after transport or structural repair exhaustion | Remove by default and settle the failure after PromptKit's budgets are exhausted. A semantic turn addresses actionable domain defects; it should not silently restart a failed provider execution. Any additional recovery policy needs an explicit application purpose. |
| `executeValidationChain`'s repeated LLM validator invocations | Keep the ordered chain and result aggregation; move transport and response-shape recovery into each validator's native execution. A validator execution failure remains distinct from a rejected producer candidate. |
| `NormalizeRetry` and `producerRetryDirective` | Replace requests to rerun the whole module with assessment findings and explicit fallback information. Preserve the domain meaning of a safe partial result. |
| `CorrectionRequest`'s hard-coded retry-message construction | Keep useful ordered collection/deduplication of actionable findings, and render the repair assignment through a native prompt. Operator messages and reason codes remain diagnostic data, not automatic model instructions. |
| Correction capability declarations and preparation checks | Retire checks tied to `single_response_v1`; prepare the actual native initial/repair definitions and response requirements. Deterministic operations need no synthetic exchange. |
Native transient retries, structural repair, and semantic additional turns remain
distinct controls, but Notarius need not implement all three loops. Existing
producer and validator `retries` settings cannot be mechanically renamed: their
current budgets cover different behavior. Non-LLM failure handling should be
reviewed separately rather than routed through conversation coordination.
### Validation And Safe Fallback Still Have Application Meaning
Keep source grounding, reference resolution, domain invariants, reconciliation
plan checks, typed application rules, and candidate isolation. In particular,
normalization must still reject unknown or overlapping candidate handles and
apply only safe groups. A final fallback can retain safe merges while leaving
unsafe groups separate; it must go through configured artifact validation and
terminal policy before it can advance. After a native terminal failure covered
by the selected fallback policy, a supported deterministic fallback can be assessed
directly instead of first
restarting the same model operation through a feedback-free module retry. Which
earlier safe candidate, if any, may survive a later failed turn remains an explicit
policy question; do not introduce implicit best-so-far acceptance.
Retain the distinction between a rejection and unavailable validation. Under the
current `warn_continue` policy, incomplete validation can advance a structurally
valid candidate but prevents accepted-artifact cache reuse/publication, including
for derived results. Continuation adoption does not itself change that policy
or safe-fallback eligibility. Keep raw conversation state in process and
out of cached artifacts; explicit detailed debug remains separate. Item 5
supersedes checkpoint-specific publication and provenance machinery.
PromptKit owns everything within an individual request execution, including
structural validation, internal retries, and repairs. Notarius consumes the
public result or failure and owns domain interpretation, validation, and any
application-directed correction passes. Do not inspect or reproduce internal
PromptKit attempts to decide cache eligibility; cache only the final accepted
Notarius artifact after its complete application process succeeds.
PromptKit should own validation and repair of the private model response.
Notarius's serialized schema validator currently receives the codec-produced
artifact and its artifact schema, which can differ from the model response.
Do not remove that boundary merely because both use JSON Schema. Audit actual
duplicate response checks individually; artifact validation and Go/domain
interpretation are not automatically redundant. Item 2's resolved native asset
and schema design preserves this distinction.
Evidence: [producer attempt loop](../../internal/framework/pipeline/producer_attempts.go),
[validation executor](../../internal/framework/pipeline/validation_executor.go),
[typed artifact validation](../../internal/framework/pipeline/runner_typed.go),
[NPC normalizer](../../internal/modules/dnd/normalize/npcregistry/normalizer.go),
[semantic reconciliation](../../internal/framework/semanticreconcile/engine.go),
and the native continuation evidence in item 7.
### Open Implementation And Policy Questions
- Exact repair prompt definitions, response schemas, and profile selection within
the agreed complete-replacement approach.
- Migration of producer and validator `retries` settings to native controls,
including configuration names, defaults, and compatibility.
- Detailed terminal/fallback handling, including whether an earlier safe
candidate may survive a later failed turn.
- Failure handling for non-LLM operations and any justified recovery beyond
native execution budgets.
- Item 5's remaining cache implementation details and migration to the selected
CommandKit output contract.
**Follow-up:** Resolve the remaining details above, then reconcile the adoption
roadmap. Record the changed architecture in a successor to
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and
[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md).
Migration checks should cover accumulated history, fixed handle mappings,
revalidation, final-turn acceptance, native budget exhaustion, safe fallback,
validator failure, cancellation, and cache eligibility. Replace tests of
the retired message-reconstruction protocol rather than preserving its mechanics.
## 4. Global Scheduling And Admission
**Discussion / decision:** Direction agreed: remove Notarius's LLM scheduler and
retire the separate cross-backend ceiling. Rely on PromptKit's native backend
admission and capacity limits while keeping bounded Notarius pipeline workers.
This deliberately changes the existing global-limit contract; reproducing that
ceiling in PromptKit is not a migration prerequisite. No application changes
have been implemented.
### Agreed Ownership
| Concern | Owner |
| --- | --- |
| Bounded extraction jobs and lane continuations | Notarius pipeline workers |
| Backend admission, active model attempts, queues, cancellation, and permit release | PromptKit |
| Application disposition after admission or execution failure | Notarius application policy |
For limited backends, PromptKit v0.23.2 bounds active model attempts by
`concurrency_limit` and admitted operations by that limit plus `queue_capacity`.
An operation holds admission
across preparation, retries, and structural repair, while active permits cover
individual model attempts and are released during retry waits. A completed
continuation turn releases admission before application assessment, so its
callback can invoke an LLM validator using the same backend.
The current Notarius wrapper holds a global permit around the complete
structured call, including backend waiting and repairs. Removing it eliminates
that additional queue and avoids making independent backends share those
permits. Independent native backend limits may allow aggregate concurrency above
the former global ceiling; that is an accepted behavior change. Pipeline worker
bounds remain application work controls, not a replacement global LLM ceiling.
### Removal And Configuration Consequences
- Remove `Scheduler`, `scheduledClient`, their production wiring, and custom
queue/cancellation/permit bookkeeping.
- Retire `concurrency.total_llm`. Give pipeline worker settings an independent
default and remove the extraction-worker maximum derived from `total_llm`.
- Use one shared execution engine for producers, validators, and repair turns.
Native capacity belongs to an engine and selected backend ID; separate engine
instances have independent pools.
- Configure native limits at the intended backend capacity boundary. Profiles
selecting the same backend ID share its pool; separate IDs pointing to the
same endpoint do not automatically share capacity. Endpoint-only profiles and
backends without a positive limit remain unrestricted by backend admission.
- Handle native `ErrCapacityExceeded` as an admission failure through application
policy, consistently with item 3's removal of blanket execution retries.
- Retire scheduler implementation tests with the removed code. Retain focused
integration checks for shared native admission across roles, independent
backends, bounded pipeline work, queue exhaustion, and cancellation.
### Remaining Details
Select independent worker defaults, native backend/queue defaults where the
application supplies them, and the configuration migration treatment of the
retired field. Specify how admission failures are reported and handled for each
application role. These details do not reopen the selected ownership boundary.
A future demonstrated need for capacity shared across several backend IDs could
justify an upstream PromptKit feature request. No such feature is required for
this agreed design, and it does not justify retaining the Notarius scheduler.
Evidence: [scheduled client](../../internal/framework/llm/scheduled_client.go),
[scheduler](../../internal/framework/llm/scheduler.go),
[pipeline workers](../../internal/framework/pipeline/runner_concurrent.go), and
[native capacity behavior](../../../promptkit/docs/internal/capacity.md).
**Follow-up:** Reconcile the adoption roadmap's scheduler-preservation language
and record the changed ownership in an ADR. Update architecture, configuration,
operations, and implementation documentation when the migration lands. Continue
with item 5's accepted-artifact cache discussion.
## 5. Accepted-Artifact Cache
**Discussion / decision:** Agreed. Rename and replace the checkpoint subsystem
with a simple cache surrounding the complete Notarius producer operation.
Compute its key early from the initial prepared request's message contents;
store its value late, after Notarius has accepted the final artifact. This
supersedes both comprehensive execution fingerprints and the earlier
stage/input/prompt-ID matching proposal. Implementation is pending.
### Purpose And Key
The cached value is an accepted answer to the original task. After a downstream
failure, rerunning the pipeline can reuse that artifact without repeating the
expensive model work that produced it. Cheap deterministic work elsewhere in
the pipeline does not need a general stage-checkpoint framework.
Use a PromptKit-provided `MessageContentHash` (proposed name) over the initial
request's ordered messages, including each message's role and exact text. Exclude
model settings, session IDs, execution IDs, timestamps, and provider cache hints.
Do not add whole-configuration, stage, prompt-ID, library-version, or validation-
policy fingerprints to this matching rule.
Changing the actual initial messages changes the key. Changing a prompt ID,
model, or setting without changing those messages does not. Fixing deterministic
interpretation or validation may therefore require explicit refresh/bypass to
replace a previously accepted artifact; a hit does not claim fresh validation.
This is a concrete upstream hash requirement. The reviewed PromptKit v0.23.3
`RenderedPromptHash` includes session identity and provider cache hints, while
hashing the entire `PreparedRun` would include volatile execution metadata.
Neither is the selected key. Expose the content hash through native preparation
rather than implementing another message-hashing policy in Notarius. Confirm
the supplying upstream API/release before cache implementation; do not claim
the proposed field already exists or silently substitute a broader fingerprint.
### Compute Early, Store After Acceptance
```text
Prepare initial request -> obtain message-content hash -> look up cache/<hash>.json
Hit: decode the accepted Notarius artifact and continue the pipeline
Miss: execute the complete Notarius operation
-> native PromptKit calls as needed
-> domain interpretation, validation, and correction passes
-> on final acceptance, save the artifact under the initial hash
```
Lookup occurs before model execution. Use native preparation to inspect the
initial request; preserve that prepared request for execution on a miss where
the native API supports it. This is a reason to prepare before deciding whether
to execute, distinct from preparing solely to obtain logging details.
Retain the initial hash throughout application-directed correction. Save the
final accepted artifact under that hash, not under a later correction request's
hash. A subsequent run can then reuse the accepted answer without replaying the
failed turns that led to it.
The value is the final Notarius artifact, serialized through its existing codec.
For normalization, it is the accepted normalized artifact after plan application,
not the model's proposed reconciliation plan. Do not store an intermediate raw
response, `RunResult`, exchange, or prepared-execution handle as the cache value.
Only publish after the whole Notarius process succeeds, including configured
domain validation and any application-directed retries/correction passes.
Rejected, failed, and validation-incomplete results are not cacheable. Retain
the existing exclusion of derivatives of validation-incomplete work without
restoring checkpoint manifests or fingerprint graphs to implement it.
A hit loads the accepted artifact directly. Ordinary decoding and usability
checks apply; absent or unusable entries are misses. Do not execute a model
request, reconstruct native results/exchanges, replay corrections, or repeat
semantic validation merely because the value came from cache. Do not count
historical model work as current execution.
### PromptKit Ownership Boundary
PromptKit owns the mechanics within each request execution, including structural
response validation, internal retries, and repairs. Notarius handles a usable
response or terminal failure at the public call boundary, then applies its own
domain interpretation, validation, correction, and acceptance process.
Internal PromptKit attempts are not inputs to the cache design. Native debug
observations may expose evidence about them, but Notarius does not inspect or
reproduce those mechanics to determine cache eligibility. The upstream addition
needed here is the message-content hash; no cached-response injection, replay,
exchange restoration, or PromptKit-owned write-through cache is required.
### Storage And Removal
Use a small filesystem lookup/write facility with `cache/<hash>.json` containing
the accepted artifact. Ordinary temporary-file/rename publication suffices to
avoid exposing partial entries. Cache lookup must not require original output
or debug directories, acceptance ledgers, execution provenance, or a separate
versioned checkpoint envelope. Debug capture can record the current hit and
loaded artifact without requiring historical metadata in the cache file.
Retire checkpoint-specific identities, invocation-wide lookup barriers,
stage/dependency fingerprints, manifests, and recovery bookkeeping. Do not
preserve that machinery solely to cache cheap deterministic stages. The existing
standalone canonical chunk-plan cache remains a separate concern.
The old `--resume`/`--recompute-step` conventions are migration context, not a
compatibility requirement. Select straightforward cache reuse/bypass/refresh
controls and legacy-file handling during implementation planning; do not infer
new flag spellings or a migration framework from this decision.
**Follow-up:** The master roadmap now includes this design and the upstream hash
requirement. During implementation, verify that a downstream failure leaves
earlier accepted artifacts reusable, corrected answers are stored under the
initial hash, changed messages miss, settings-only changes can hit, and a hit makes no model calls or
new semantic-validation claims. Keep output/report contract decisions separate.
## 6. Inspection And Execution Bookkeeping
**Discussion / decision:** Direction agreed: use native PromptKit inspection
and execution facts, with CommandKit owning standardized operator presentation.
Remove duplicate LLM reporting and accounting machinery. Notarius retains
application context, domain decisions, artifact delivery, completion-summary
content, and optional debug capture. The CommandKit output decision below removes
mandatory durable execution provenance. No application changes have been
implemented.
### Ownership
| Responsibility | Owner |
| --- | --- |
| Configuration provenance, resource inspection, and request preparation | ConfigKit and PromptKit through their native APIs. |
| Standard configuration/prompt commands, preview and smoke commands, report formatting and visibility | CommandKit and its ckconfig/ckprompt integrations. |
| Provider attempts, transport retries, structural repairs, execution failures, and token observations | PromptKit observations and results; `ExecutionGroup` for selected execution totals. |
| Step/lane/chunk/validator attribution and pipeline progress meaning | Notarius. |
| Acceptance, rejection, fallback, accepted-artifact reuse, and pipeline outcome | Notarius. |
| Artifact storage, completion-summary content, and optional debug storage | Notarius. |
### Native Inspection And Operator Presentation
Remove the profile-inspector wrapper, copied effective-model-parameter types,
and local reconstruction of upstream configuration. Use native inspection for
configured resources and native prepared/result data for the target actually
executed, including request overrides. Profile inspection can describe a partial
configured profile; it is not evidence of the completed execution target.
Adopt ckconfig and ckprompt for standard inspection, requirements, validation,
preview, and opt-in smoke commands. Notarius supplies configuration/engine
loaders and application request builders. Avoid building temporary custom
commands or report machinery that the planned CommandKit adoption would replace.
Retain application pipeline preflight: resolve modules, artifact compatibility,
references, and known prompt/profile selections. Use native definition checks
without simulating stage execution or requiring provider credentials. Concrete
request preparation remains the place to check the completed target, inputs,
schema, and credential requirements.
CommandKit supplies formatting, visibility, progress presentation, and delivery
error handling. Notarius supplies report content, domain outcomes, and exit-status
mapping. Follow the small structured completion report and native text template
selected in [CommandKit item 4](commandkit-migration-discussion.md#4-reports-machine-output-and-outcomes);
do not preserve the old receipt shape or independent encoders.
### Native Execution Facts
- Use ordinary `Run`, with `OnPrepared` when actual prepared details are needed
for provenance or optional debug capture. Remove unconditional
`PrepareExecution`/`RunPrepared` use solely to obtain those details.
- Use native observations for terminal execution facts, including failures that
return no `RunResult`. `OnPrepared` is conditional and is not delivered after
preparation or admission failure; it cannot serve as a completion record.
- Use native `RunID`/`ParentRunID` with application scope to correlate executions.
Remove the private debug completion wrapper and duplicate synthetic call-ID
machinery where native identity and application attribution suffice.
- Remove parallel `LLMExecutionSummary`, token-usage types, enum conversions,
cloning, and independent attempt/usage accumulators. Use native types internally
and project selected fields only at an application reporting or durable boundary.
- Remove the adapter's `LLMProfileRecorder` and provider-forwarding plumbing.
Native snapshots provide the profiles and targets actually used for optional
diagnostics; do not reconstruct a durable profile inventory.
- Use `ExecutionGroup` for totals over selected executions. Explicitly enroll the
initial run, continuation runs, and nested validator runs as applicable.
Preserve terminal-coverage and token-usage completeness; unknown usage is not
zero. Do not count both attempt observations and terminal aggregates, or add
overlapping groups as though they were disjoint work.
Application execution code should attach scope and observers, run and assess the
native result, and record its disposition and execution references. This does
not require a replacement general-purpose LLM interface. A successful native
execution can still produce a domain-rejected candidate; provider success and
application acceptance remain separate facts.
### Application Records And Durable Boundaries
Attach the application context needed to explain a pipeline in optional debug
capture: which stage/lane/chunk produced an artifact, which validators assessed
it, whether continuation or a safe fallback was used, and whether the artifact
was generated now or reused. PromptKit cannot infer that meaning from provider
execution alone. These diagnostic facts do not require consumer-facing histories.
Under the revised item 5, cache files contain accepted artifacts without required
historical execution or acceptance records. A current debug record can identify
reuse and show the loaded artifact without counting earlier token usage as
current work or claiming new validation.
The selected CommandKit output contract delivers artifacts as files and a small
completion report through native rendering. It does not require a run manifest,
metadata bundle, separately persisted receipt, or durable execution ledger.
Existing [manifest](../integrations/json-output.md) and
[receipt](../integrations/run-result.md) contracts document migration impact;
their field inventories are not requirements to recreate. Preserve intrinsic
domain artifact content and reuse existing outcome facts for the completion
summary and debug overview.
Keep application debug artifact placement, opt-in capture, and lifecycle under
[CommandKit item 5](commandkit-migration-discussion.md#5-debug-capture-and-optional-diagnostics).
Native callbacks supply execution data; they do not own Notarius's storage policy.
Detailed prepared messages and execution history belong in that optional capture,
not a mandatory consumer metadata bundle.
Evidence: [current inspection wrapper](../../internal/framework/llm/promptkit_inspection.go),
[adapter bookkeeping](../../internal/framework/llm/promptkit_client.go),
[debug completion wrapper](../../internal/framework/pipeline/debug.go),
[native PromptKit APIs](../../../promptkit/docs/consumers/pkg-promptkit.md),
[ckconfig integration](../../../commandkit/integrations/ckconfig/README.md),
[ckprompt integration](../../../commandkit/integrations/ckprompt/README.md), and
[CommandKit reporting](../../../commandkit/docs/consumers/guide.md).
### Remaining Implementation Work
Select application scope and observer/group enrollment for the agreed diagnostics.
Implement the selected completion report and retire superseded manifest, receipt,
and execution-ledger machinery. Coordinate command migration and debug capture
with the incorporated design in the master roadmap. The core
bookkeeping ownership change needs no new PromptKit capability; any remaining
upstream request should identify a concrete unmet requirement.
**Follow-up:** Reconcile the adoption roadmap with items 1–6 and record the
significant boundary changes in successor ADRs. Update current-behavior
documentation when implemented. Verify failure reporting without a result,
continuation/validator accounting, unknown usage, actual-target provenance,
cache reuse accounting, and optional debug capture during implementation.
## 7. Compare Audita's PromptKit Workflow
**Review status:** Completed on 2026-11-18 against the clean sibling checkout
`../audita`, revision `da5b8913d1206a8e8f0a784b2322760385548a60`.
Audita uses PromptKit v0.23.2, ConfigKit v1.2.1, CommandKit v0.7.0,
ckconfig v0.9.0, and ckprompt v0.10.0. These are observations of that revision,
not new Notarius contracts. The source links below assume the sibling checkout.
### Embedded Defaults And Operator Replacement
- One captured engine is assembled from PromptKit's default catalog, the
embedded Audita underlay, and a higher-priority operator underlay.
`FilesystemInputs` supplies eleven native prompt definitions in
`collections.audita`: four proposal prompts, four repair prompts, and three
validator prompts.
- Operator sources are optional `prompts.yml` and recursively selected YAML
under `prompts/`, anchored to the application main source's recorded context.
Both locations feed one operator underlay and are peers; conflicting peer
definitions fail rather than receiving filename-based precedence.
- Replacement is by resource identity, not filename. For example, a complete
prompt record with local ID `modules.grammar.repair` under
`collections.audita.prompts` replaces `audita/modules.grammar.repair` while
other embedded definitions remain available. Native same-ID replacement is
whole-record replacement, not recursive merging of individual prompt fields.
- Native imports and message-file context are retained. Tests show that an
existing engine retains captured message content after a file edit and a new
engine sees the edit. They also verify inherited message-file resolution,
main-relative discovery independent of the working directory, and peer
conflict rejection.
- A diagnostic embedded-only construction can classify a loading error; it
does not silently fall back to defaults after an operator-source failure.
Evidence: [runtime assembly](../../../audita/internal/framework/llm/runtime.go),
[source tests](../../../audita/internal/framework/llm/engine_sources_test.go),
and [assembly failure tests](../../../audita/internal/framework/llm/runtime_assembly_test.go).
### Prompt Content And Schema Ownership Are Separate
Audita uses ordinary embedded filesystems for prompt and schema assets. Its
small prompt registry records IDs and embedded paths; it does not reconstruct
Notarius-style per-module virtual prompt trees. Shared instruction fragments are
supplied as template variables. Processing supplies the application-owned values
for those shared names, but an operator replacement decides whether its template
references them at all. They are not forcibly appended to replacement messages.
The processing request independently supplies a complete native output contract:
JSON Schema validation, the schema path selected for that operation, and its
structural-repair budget. The three schemas describe a correction set, a repair
decision, and validator decisions. PromptKit receives their canonical embedded
filesystem via `WithSchemaFS`. Thus operator control of prompt wording does not
change the processing code's expected response shape. A test replaces a prompt
with a text/no-validation declaration and still exercises the processing
request's JSON-schema repair contract.
Audita still keeps a response-schema key/metadata/hash registry and application
decoding checks. Its use of native loading does not prove that every schema
wrapper is necessary, nor decide Notarius's deferred schema design. It supplies
a concrete option: module-owned response requirements expressed directly through
native per-request contracts. Standalone native preview/preparation can use the
prompt declaration's contract; processing supplies its own override.
Evidence: [prompt registry](../../../audita/internal/prompts/registry.go),
[schema registry](../../../audita/internal/framework/responseschema/registry.go),
`prepareCall` and `OutputContract` in the
[processing adapter](../../../audita/internal/framework/llm/adapter.go), and
`TestOperatorPromptOverridesUsePerCallSchemaAndOptionalSharedText` in
[adapter tests](../../../audita/internal/framework/llm/adapter_test.go).
### Native Continuation With Application Assessment
The implemented flow is:
1. Generate a section's proposal set, capturing a native immutable exchange when
correction repair is enabled. This retains the actual proposal messages and
response; it does not require debug recording.
2. Apply application validation. An actionable rejected proposal starts its own
branch from the original proposal exchange. Unavailable-only findings and
no-effect corrections do not start repair. Existing approvals remain separate.
3. Call `Engine.Coordinate` with that exchange and an additional-turn budget.
Each follow-up selects a distinct repair prompt, repair profile, and repair
response schema. The request carries the original proposal reference, current
correction, ordered findings, and available before/after text previews.
4. PromptKit assembles history and progresses the conversation. A second turn
retains the first repair instruction and response. A sibling branch starts
from the original exchange and does not inherit another branch's repairs.
Audita does not rerender the original proposal or manually replay messages.
5. Audita decodes each delivered decision. Withdrawal ends the branch without
applying that correction. A revision undergoes the full application validation
chain again, including validators that approved its earlier version. Approval
stops the branch; actionable rejection can request another turn.
6. The final permitted response is assessed before exhaustion is determined.
Requesting another turn at that boundary does not allocate another logical
call. Recoverable provider/invalid-output failures finish the branch as
unavailable; fatal errors or parent cancellation interrupt processing.
Canceled, undelivered responses are diagnostic evidence only, never approvals.
Transport retries and structural repair operate inside each PromptKit execution;
they are separate from the semantic branch's additional-turn budget. Tests show
one semantic repair turn containing multiple provider attempts for structural
repair. The application retains acceptance, withdrawal, application ordering,
and reporting; PromptKit retains conversation and execution mechanics.
Audita's prompt roles support this model deliberately: system instructions carry
durable principles, the initial user message carries the initial module task,
and the repair prompt supplies a new user assignment. Adopting continuation in
Notarius would require reviewing the complete conversation and response-format
transitions, rather than mechanically appending to prompts written for fresh
replacement requests.
Evidence: [native coordination bridge](../../../audita/internal/framework/llm/coordinate.go),
[domain repair flow](../../../audita/internal/framework/runner/repair.go),
[wire-level conversation tests](../../../audita/internal/framework/llm/conversation_test.go),
[runner repair tests](../../../audita/internal/framework/runner/repair_test.go),
and [prompt guidance](../../../audita/docs/internal/prompts.md).
### Other Boundaries And Limits Of The Comparison
- Audita still has `RoleClient`, a structured-call interface, an opaque exchange
wrapper, and application diagnostic types. Adopt the demonstrated native
mechanisms without assuming Notarius should copy those wrappers; item 1's
adapter-removal direction remains in force.
- Proposal, validator, and repair roles share one engine and its per-backend
admission. The runner separately bounds section work. This demonstrates useful
separation of application work scheduling and provider capacity. Item 4 now
selects that ownership and retires Notarius's independent cross-backend ceiling.
- Ordinary calls use `Run`, `OperationTimeout`, and `OnPrepared`. Observations
feed a native `ExecutionGroup`; application records add logical-call and
stage/section/proposal attribution. Nested validator executions enroll
explicitly. Counts preserve terminal coverage and token-usage completeness.
- CommandKit's ckconfig and ckprompt integrations supply configuration and prompt
commands using Audita's engine construction, including request preparation,
validation, and smoke diagnostics.
- Audita edits independently assessable transcript corrections; Notarius accepts
typed stage artifacts and has cross-stage dependencies. Audita branches at an
individual correction within a proposal response. Notarius must deliberately
choose its correction unit rather than assume per-entity branching.
- Audita has no resume/checkpoint/manifest system. It cannot settle Notarius's
cache design; item 5 records that application decision independently.
Evidence: [LLM runtime](../../../audita/docs/internal/llm-runtime.md),
[observations](../../../audita/internal/framework/llm/observations.go),
[CLI integrations](../../../audita/internal/cli/app_wiring.go), and
[component scope](../../../audita/docs/internal/overview.md).
### Implications For The Remaining Discussion
Item 2's native underlay direction has a working application precedent. Audita
also offers evidence for ordinary embedded filesystems and per-request schema
selection. U2 now selects ordinary embedded assets and prompt-declared contracts
by default, reserving per-request replacement for demonstrated runtime needs.
The subsequent item 3 discussion selects native continuation and accumulated
conversation history. Keeping domain validation local does not require keeping
conversation assembly or the turn loop local. Item 3 owns the agreed Notarius
flow and the remaining repair-prompt, response-contract, and policy details;
the comparison alone did not settle those details.
Item 4 now records agreement to use native admission with bounded application
workers. The revised item 5 selects an accepted-artifact cache and a focused
upstream message-content hash instead of comprehensive execution fingerprints.
Item 6 adopts native execution observations and CommandKit presentation;
the [CommandKit output decision](commandkit-migration-discussion.md#4-reports-machine-output-and-outcomes)
selects artifact files and a small native completion report.
The two requested capabilities already work with v0.23.2; no new PromptKit
feature is needed merely to provide replaceable defaults or domain-directed
continuation. Any remaining upstream request should identify the specific unmet
requirement after reconsidering Notarius's historical behavior.
### Verification
Read the production composition, adapter, coordinator, domain repair callbacks,
prompt/schema definitions, and focused behavior tests. The following checks
passed from the Audita checkout:
```sh
go test ./internal/framework/llm ./internal/prompts ./internal/framework/responseschema ./internal/framework/runner ./internal/framework/proposal_generation
go test ./cmd/audita -run '^TestSubprocess(CorrectionRepairConversation|CancellationDuringCorrectionRepair|RepairDisabledAndOutputSelections)$' -count=1
```
The subprocess checks exercise assembled configuration, real native continuation
messages, operator replacement, final-turn assessment, sibling isolation,
repair-disabled behavior, and cancellation using a local fake provider. These
checks establish implementation behavior, not live-model quality. No live model
calls or application source changes were made. Broader migration decisions and
changes to the adoption roadmap remain for discussion.

View File

@@ -8,13 +8,19 @@ presentation directly. Retain the application work of producing, assessing,
publishing, and reusing typed artifacts. Historical duplication of library
behavior is not a requirement to preserve it.
This roadmap incorporates the [PromptKit architecture discussion](promptkit-architecture-discussion.md)
and all six items of the [CommandKit discussion](commandkit-migration-discussion.md),
including the subsequent accepted-artifact cache and artifact-first output
decisions. The notepads retain rationale and Audita evidence; this document owns
the selected target, unresolved choices, implementation sequence, removal
inventory, and acceptance criteria. All implementation phases are pending.
Recording these decisions has not changed application behavior.
This is the single migration plan for PromptKit, ConfigKit, CommandKit,
accepted-artifact caching, and artifact-first output. It owns the selected target,
remaining choices, implementation sequence, removal inventory, acceptance
criteria, and supporting evidence. The earlier discussion documents are retired;
Git history preserves the discussion. All implementation phases are pending.
Recording these decisions has not changed application behavior. Unrelated future
work remains in the [separate backlog](future.md).
Notarius predates many modern PromptKit capabilities: the library grew out of
mechanics shared by Notarius and related applications. Historical duplication
and slight behavioral differences do not express an intentional application
requirement. Evaluate remaining machinery against actual domain needs, and
consider upstream features when a reusable capability is missing.
Complete one coordinated cutover for configuration, PromptKit resources, and the
CLI. Adopt native public conventions fully; preserving old CLI behavior is a
@@ -23,6 +29,10 @@ parallel parsers, or compatibility wrappers for legacy behavior. CommandKit is
part of the main migration, not a later follow-up. Release preparation and
publication remain separate work.
Adopting the kits together lets standard inspection and execution share the same
loaders and native facts. It avoids building temporary command and reporting
adapters while those boundaries are already being replaced.
## Baseline And Dependency Target
At Notarius revision `12eeb6369e40746b2c18c55397a289a1e1b5908d`, the application
@@ -38,22 +48,27 @@ The selected reviewed baseline is:
| `go.maximumdirect.net/ckconfig` | `v0.9.0` |
| `go.maximumdirect.net/ckprompt` | `v0.10.0` |
| `go.maximumdirect.net/configkit` | `v1.2.1` |
| `go.maximumdirect.net/promptkit` | `v0.23.3`, pending the message-content hash addition below |
| `go.maximumdirect.net/promptkit` | `v0.23.3`, subject to U1's hash-capable release review |
Keep Go's consumer minimum at 1.27, adopt stable go-yaml v4, and resolve compatible
validation dependencies. PromptKit v0.23.3 supersedes the initial v0.23.2 target
for its timeout/cancellation fixes. These are selected versions, not a claim
that they are the latest releases. The local tag review and integration guidance
are recorded in [CommandKit item 1](commandkit-migration-discussion.md#1-adoption-scope-and-dependency-baseline).
The tagged [release notes][release], [migration guide][migration],
[format reference][formats], and [Go consumer guide][consumer] supply the baseline.
validation dependencies. The selected CommandKit integration releases document
compatibility with PromptKit v0.23.2. Select v0.23.3 explicitly for its timeout
attribution and cancellation/error identity fixes; that patch requires no
configuration or request API migration from v0.23.2 and retains its dependencies.
The original review used local release tags and tagged dependency manifests;
remote release lookup was unavailable, so these are selected versions, not a
claim that they are the latest releases. See the
[integration version guidance](../../../commandkit/docs/consumers/guide.md#module-selection)
and tagged [release notes][release], [migration guide][migration],
[format reference][formats], and [Go consumer guide][consumer].
The cache requires the proposed native `MessageContentHash`. The
[upstream request](../../../promptkit/docs/roadmap/notarius-request.md) is pending;
v0.23.3's `RenderedPromptHash` is not a substitute. Confirm the supplying API and
tagged release, then verify the assembled dependency set. Independent composition
and pipeline work can proceed while that addition is pending; the completed
cache must use the upstream capability. Do not add a local hash-policy fork.
The cache requires native `MessageContentHash`; v0.23.3's `RenderedPromptHash`
is not a substitute. The sibling PromptKit repository has retired the original
feature request, and its [v0.24.0 release notes](../../../promptkit/docs/releases/v0.24.0.md)
document the addition and a ConfigKit v1.3.0 dependency. U1 remains open to review
the supplying API/release and verify a compatible assembled dependency set before
changing the selected baseline. Independent work can proceed; completed cache
integration must use the upstream capability. Do not add a local hash-policy fork.
Neither a cross-backend global limit nor a comprehensive execution fingerprint
is required. Remove obsolete imports and catalog dependencies when unused;
avoid permanent `replace` directives and update [attribution](../oss/attribution.md)
@@ -480,8 +495,8 @@ Prepare initial request -> obtain MessageContentHash -> look up cache/<hash>.jso
-> on final acceptance, store the artifact under the INITIAL hash
```
Use the proposed upstream hash of ordered message roles and exact text. Include
message boundaries and order, excluding model/settings, session/execution IDs,
Use the upstream message-content hash of ordered message roles and exact text.
Include message boundaries and order, excluding model/settings, session/execution IDs,
timing, provider cache hints, and other non-content metadata. Add no stage,
prompt-ID, library-version, whole-configuration, schema-policy, or implementation
fingerprint. Actual input or schema text rendered into messages participates as
@@ -499,7 +514,7 @@ prepared snapshot for execution on a miss using `PrepareExecution`/`RunPrepared`
where supported, and discard the handle on a hit. `OnPrepared` execution hooks
alone cannot short-circuit execution. v0.23.3's `RenderedPromptHash` includes
session identity and cache hints; it is not the requested hash. The upstream
request in U1 must land before completing this cache integration.
API/release review in U1 must finish before completing this cache integration.
Keep the initial key throughout corrections. Cache the final Notarius artifact
through its existing codec, not an intermediate response, `RunResult`, exchange,
@@ -548,7 +563,9 @@ diagnostics; split preparation has a separate purpose for cache lookup. Native
observations provide terminal facts even without a returned `RunResult`.
`OnPrepared` is absent on preparation/admission failures and is not a completion
record. Keep synchronous callbacks lightweight, compose capture/accounting
observers, and synchronize shared state where operations overlap.
observers, and synchronize shared state where operations overlap. `OnPrepared`
runs while admission is held; persist retained evidence at normal operation
boundaries.
Use native `RunID`/`ParentRunID` with application step/lane/chunk/validator scope.
Remove `LLMExecutionSummary`, duplicate usage types, conversions, cloning,
@@ -751,7 +768,7 @@ reopen the architecture. IDs retain continuity with the earlier roadmap.
| ID | Remaining choice or dependency | Resolve for |
| --- | --- | --- |
| U1 | Supplying PromptKit API/release for the requested message-content hash; verify the final compatible dependency set against the selected baseline. The upstream request is pending. | Track in Phase 1; required to finish cache integration in Phase 5. |
| U1 | Review the message-content hash API/release documented upstream in v0.24.0, including its ConfigKit dependency; verify the final compatible dependency set against the selected baseline. | Track in Phase 1; required to finish cache integration in Phase 5. |
| U4 | Role-specific initial/repair definitions, response contracts, and profile selection within complete replacement and fixed operation inputs. | Phases 3–4. |
| U5 | Producer/validator retry-setting migration to native controls and semantic budgets: field names/defaults, non-LLM failure handling, and any justified recovery beyond native exhaustion. | Phases 3–4 and configuration migration. |
| U6 | Detailed terminal/fallback policy, especially whether an earlier safe candidate survives a later failed turn. No implicit best-so-far acceptance. | Phases 3–4. |
@@ -787,7 +804,7 @@ major boundary changes in successor ADRs without rewriting accepted decisions.
### Phase 1: Establish The Migration Baseline
- Record the selected dependency set and track U1 against the upstream request.
- Record the selected dependency set and review U1 against the upstream release.
Keep the cache dependency explicit without blocking independent work.
- Map affected ADRs, current callers, contracts, and removal inventory. Carry
forward Audita evidence without adopting its application wrappers.
@@ -795,8 +812,9 @@ major boundary changes in successor ADRs without rewriting accepted decisions.
execution, assets/schemas, CommandKit, debug, cache, or output ownership.
**Completion evidence:** reviewed dependency baseline, upstream dependency, scope,
and outstanding choices are explicit. This phase does not claim that the hash
already exists or that the migration is implemented.
and outstanding choices are explicit. Upstream availability alone does not
establish compatibility of the assembled Notarius dependencies or completion
of the migration.
### Phase 2: Build Native Composition, Sources, And Preflight
@@ -869,7 +887,8 @@ including failures after publication and quiet-mode capture.
- Complete the acceptance checks and removal inventory, including dependency
cleanup, maintained examples, and the assembled command surface.
- Finish release migration guidance, attribution, successor ADRs, and affected
canonical documentation. Reconcile future-work notes and notepad status.
canonical documentation. Update this roadmap's status and reconcile the
separate future-work backlog.
- Confirm no compatibility scaffold, deferred CommandKit adoption, local hash
policy, or obsolete metadata requirement remains in the finished migration.
@@ -968,7 +987,52 @@ Record significant changes in successor ADRs, including
[PromptKit configuration](../adr/0019-adopt-native-promptkit-configuration.md),
admission, caching, and CLI/reporting ownership. Preserve accepted decision text
and update supersession status under [documentation policy](../policy/documentation.md#architecture-decision-records).
Notepads retain discussion history; this roadmap owns implementation status.
This roadmap owns implementation status; Git history retains earlier discussions.
## Evidence And Reading
These sources support the selected design, not a claim of completed Notarius
implementation. Sibling links assume the local repositories are available; the
tagged PromptKit links below preserve the reviewed baseline.
### Native Library Guidance
- [CommandKit consumer guide](../../../commandkit/docs/consumers/guide.md) and
[CLI conventions](../../../commandkit/docs/cli.md): module selection, native
aliases/inputs, shared loaders, reporting modes, outcomes, and delivery failures.
- [ckconfig](../../../commandkit/integrations/ckconfig/README.md) and
[ckprompt](../../../commandkit/integrations/ckprompt/README.md): standard suites
supplied with application loaders and request builders.
- [ConfigKit](../../../ConfigKit/README.md): native loading and inspection.
- [PromptKit consumer guide][consumer] and
[capacity behavior](../../../promptkit/docs/internal/capacity.md): native
preparation/execution, continuation, observations, and admission.
### Audita Comparison
The comparison was completed on 2026-11-18 against Audita revision
`da5b8913d1206a8e8f0a784b2322760385548a60`, using PromptKit v0.23.2,
ConfigKit v1.2.1, CommandKit v0.7.0, ckconfig v0.9.0, and ckprompt v0.10.0.
The observations below concern that revision; linked sibling files may evolve.
| Evidence | Lesson carried into this roadmap |
| --- | --- |
| [Engine assembly](../../../audita/internal/framework/llm/runtime.go), [source tests](../../../audita/internal/framework/llm/engine_sources_test.go), and [assembly failure tests](../../../audita/internal/framework/llm/runtime_assembly_test.go) | One captured engine, native underlays and main-source-relative companions, whole-record replacement, preserved file contexts, strict peer conflicts, and no silent fallback on operator-source errors. |
| [Adapter tests](../../../audita/internal/framework/llm/adapter_test.go) | Ordinary embedded schemas and complete per-request output contracts work with operator prompt replacement. U2 chooses prompt-declared defaults for Notarius; Audita's schema registry and per-call overrides are not requirements to copy. |
| [Native coordination bridge](../../../audita/internal/framework/llm/coordinate.go), [repair flow](../../../audita/internal/framework/runner/repair.go), [conversation tests](../../../audita/internal/framework/llm/conversation_test.go), and [repair tests](../../../audita/internal/framework/runner/repair_test.go) | Captured exchanges, accumulated history, complete candidate revalidation, final-turn assessment, and canceled/undelivered response handling. |
| [Prompt guidance](../../../audita/docs/internal/prompts.md) | Durable system instructions, an initial user task, and subsequent repair assignments suit native continuation. |
| [Observations](../../../audita/internal/framework/llm/observations.go) and [CLI composition](../../../audita/internal/cli/app_wiring.go) | Native execution totals with application scope and explicit nested-validator enrollment; standard CommandKit suites using shared loaders. |
Audita repairs individual transcript corrections in independent branches;
Notarius's selected unit is a whole producer operation and typed artifact.
Audita's wrappers, response-schema registry, per-proposal policies, and absence
of a cache do not define Notarius requirements.
At the reviewed revision, focused LLM, prompt/schema, runner, and proposal-generation
tests passed, together with subprocess checks for correction conversations,
cancellation during repair, and repair-disabled/output selections using a local
fake provider. These historical checks do not establish live-model quality or
completion of Notarius's migration acceptance checks.
[release]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/releases/v0.23.3.md
[migration]: https://forge.maximumdirect.net/base-layer/promptkit/src/tag/v0.23.3/docs/consumers/configuration-migration.md