Files
notarius/docs/roadmap/commandkit-migration-discussion.md

1253 lines
70 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CommandKit Migration Discussion
## Purpose And Status
Discussion record for the decisions incorporated into the
[master native migration roadmap](promptkit.md). That roadmap now owns the
selected design, remaining choices, implementation phases, removal inventory,
and acceptance criteria. This notepad retains discussion detail and evidence;
it is not a second implementation checklist.
**Direction agreed and incorporated:** Make a full cutover to native CommandKit,
ConfigKit, and PromptKit workflows in one coordinated migration. Adopt upstream
public conventions fully. CLI backward compatibility is a non-goal; provide
release migration guidance instead.
All six items below are incorporated, including the accepted-artifact cache,
small completion report and native text template, debug inventory/depth,
`/tmp/notarius-debug` default, `latest`, and explicit-empty directory behavior.
No application or dependency changes have been implemented through this
planning work. Implementation remains pending.
The roadmap resolves U10's CommandKit scope, tracks the pending upstream
message-content hash in U1, replaces U8's checkpoint design, and narrows U9 to
implementation details within the agreed output/debug contracts. U2 now resolves
asset and schema assembly. Role-specific correction/fallback policy, worker
defaults, and concrete cache controls/transition remain visible there. Item-level
directions below record the accepted design; the roadmap is the authority for
sequencing and current status.
## Agreed Proposal
### Guiding Rule: Full Native Cutover
Backward compatibility with the existing CLI conventions is a non-goal. Use
CommandKit's native command, input, help/version, reporting, error, and outcome
conventions as the design baseline. This mirrors the PromptKit-native direction.
Existing command names, flags, aliases, parsing behavior, report shapes, and exit
conventions are migration evidence, not requirements to preserve. Do not add
legacy aliases, compatibility shims, parallel parsers, or reporting wrappers
solely to reproduce the old CLI. Any application-specific behavior should serve
an actual Notarius need and fit the native conventions.
Provide migration guidance in release documentation, explaining the new commands
and conventions and the actions operators or subprocess callers must take.
Documenting the transition replaces a requirement to keep the old CLI working.
Update canonical CLI and consumer documentation when the behavior lands.
Application responsibilities such as producing artifacts, reporting publication
outcomes, and managing cache/debug storage remain meaningful. Their existing
CLI expression is not protected by an implicit compatibility exception. This
guiding rule does not itself choose stored artifact formats. The revised cache
decision below selects simple artifact storage rather than checkpoint envelopes.
### Why Adopt Together
- Source assembly and preflight are already changing. CommandKit integrations
can consume the same application and native engine loaders.
- ckconfig and ckprompt can replace standard inspection commands directly,
avoiding an intermediate adaptation of commands that would soon be replaced.
- Native execution facts and application outcomes can feed the intended reporting
facilities while duplicate LLM bookkeeping is removed.
- Configuration and command migration can be documented and verified as one
coherent transition.
The added work is principally CLI composition and contract migration: dispatch,
flag parsing, configuration selection, reporting, and process-status mapping.
Choose their native forms deliberately and document the transition, rather than
spending migration effort recreating existing CLI conventions.
### Ownership And Scope
| Concern | Selected owner |
| --- | --- |
| Command hosting, standard input/help mechanics, common report formatting and visibility | CommandKit. |
| Standard configuration and PromptKit inspection commands | ckconfig and ckprompt, supplied with application loaders and request builders. |
| Configuration composition and model execution | ConfigKit and PromptKit, using the native boundaries already selected. |
| Pipeline execution and listing, pipeline-specific preflight, artifact publication, accepted-artifact caching, and domain outcomes | Notarius application handlers and framework. |
| Completion-summary content, debug artifact lifecycle, and application outcome meaning | Notarius, expressed through CommandKit's reporting and outcome conventions. |
Adopt the standard capability suite selected in item 1, including upstream
commands and recommendations beyond the existing Notarius surface. Items 2–4
select application inputs, composition, and reporting; items 5–6 select debug
capture, verification, and migration. The earlier extensive receipt/manifest
proposal is superseded by item 4's artifact-first contract.
Pipeline and domain code should remain independent of CommandKit. Native
PromptKit observations supply execution facts; Notarius attaches application
meaning and selects report content. CommandKit supplies presentation mechanics.
Two application needs already identified for deliberate treatment are:
- The current `run --json` publishes a receipt after output publication. Item 4
replaces that contract with artifact files and a small completion summary,
rendered natively as text, YAML, or JSON. No automatic metadata bundle or
separately persisted report is required.
- Item 5 uses native `--debug` for diagnostic verbosity and for capture when an
effective debug directory is available, including an application default.
Capture selection remains independent of the final reporting mode.
### Intended Sequencing
1. Incorporate items 1–2's selected dependency baseline, standard capabilities,
and native application command design, bringing U10 forward.
2. Introduce item 3's CommandKit composition alongside native source assembly.
Connect shared loaders, standard inspection, and the focused application
preflight package.
3. Develop native execution and accounting against the intended CLI boundary,
with pipeline and domain tests independent of CommandKit.
4. Complete command and reporting migration before final end-to-end verification
and migration documentation.
Item 6 records the incorporation decision. The master roadmap now owns the
resulting six-phase sequence; implementation is pending.
## 1. Adoption Scope And Dependency Baseline
**Discussion / decision:** Versions and standard capability scope agreed. Adopt
the set below, including the move to PromptKit v0.23.3. Items 2–3 select
application commands and composition; items 4–5 select reporting and debug
capture principles. These are selected migration targets; the application
dependencies have not changed.
### Selected Versions
| Module | Version | Role |
| --- | --- | --- |
| `go.maximumdirect.net/commandkit` | `v0.7.0` | Native command declarations, inputs, help/version, reporting, and completion. |
| `go.maximumdirect.net/ckconfig` | `v0.9.0` | Standard configuration commands and ConfigKit diagnostic classification. |
| `go.maximumdirect.net/ckprompt` | `v0.10.0` | Standard PromptKit inspection, validation, preparation, and smoke diagnostics. |
| `go.maximumdirect.net/configkit` | `v1.2.1` | Shared configuration dependency for the application and integrations. |
| `go.maximumdirect.net/promptkit` | `v0.23.3` | Native model execution, including corrected timeout/cancellation reporting. |
The CommandKit integration releases form a documented compatible set using
PromptKit v0.23.2. Select v0.23.3 explicitly for its timeout attribution and
cancellation/error identity fixes. It retains the same dependencies and requires
no configuration or request API migration from v0.23.2. The Go consumer minimum
remains 1.27. Verify the assembled dependency set through the migration's normal
integration checks and update attribution when the dependency changes land.
The version review used locally available release tags and tagged dependency
manifests. Remote release lookup was unavailable, so it did not establish whether
newer tags existed remotely. These are deliberate selected versions, not a claim
that every module is the latest published release.
Evidence: [integration version guidance](../../../commandkit/docs/consumers/guide.md#module-selection)
and [PromptKit v0.23.3 release notes](../../../promptkit/docs/releases/v0.23.3.md).
### Selected Standard Capabilities
| Area | Adopt in the cutover |
| --- | --- |
| Core CLI | Native help, `version`, typed flags and operands, supplied-value semantics, standard outcomes, and command declaration/dispatch. |
| Reporting | Native `--format` with text/YAML/JSON, `--quiet`, `--verbose`, `--debug`, and `--progress`, including upstream visibility and failure behavior. |
| Configuration | `config scopes`, `validate`, `show`, `sources`, `explain`, and `diff`, for application and PromptKit configuration. |
| Resource discovery | PromptKit listings for prompts, profiles, backends, namespaces, and collections, with native filters and selectors. |
| Resource inspection | Native prompt, profile, and backend reports, plus requirements discovery. |
| Concrete request checks | Requirements validation and prepared-request preview using native inputs, variables, profiles, and execution overrides. |
| Provider diagnostics | Reachability, authentication, text-inference, structured-output, and aggregate smoke tests. |
| Convenience features | Bash, Zsh, and Fish completion, useful declaration hints, and upstream PromptKit convenience aliases. |
Use the [upstream CLI conventions](../../../commandkit/docs/cli.md) directly.
Providing the standard capability suite means supplying application loaders and
callbacks, not recreating those commands locally. Upstream convenience aliases
are selected native features; this does not reintroduce legacy Notarius aliases.
### Wiring Boundaries And Follow-Up
- Keep configuration validation, requirements discovery, and concrete request
validation distinct. Pipeline preflight remains application-owned. Concrete
preparation may read inputs and schemas and check credentials even though it
makes no model call.
- Install smoke commands for explicit invocation. Their availability does not
introduce provider calls during startup, ordinary validation, or offline tests.
- Enable application configuration overrides through ConfigKit. The selected
PromptKit configuration scope does not support arbitrary `--set` overrides;
use supported request-level controls rather than implementing a local equivalent.
- Retain application handlers for pipeline execution and discovery, with native
commands, inputs, and result presentation as selected in items 2–5.
The standard capability suite needs no identified upstream addition. The later
cache decision does require a native message-content hash, which is not the
existing v0.23.3 rendered-prompt hash. Confirm the supplying API/release before
implementing cache lookup; the version table remains the reviewed baseline, not
a claim that the new hash is already available. The master roadmap now records
this baseline, the pending upstream requirement, the resolved CommandKit scope,
and the application wiring/presentation decisions. Superseded durable-record
requirements have been removed from that plan.
## 2. Native Command Surface And Inputs
**Discussion / decision:** Command structure and input conventions agreed. Use a
small application-owned `pipeline` family alongside the standard `config` and
`promptkit` families. These are migration targets, not implemented commands.
### Command Structure
```text
notarius pipeline list
notarius pipeline show <pipeline-id>
notarius pipeline validate <pipeline-id>
notarius pipeline run <pipeline-id> --input-file <path>
notarius run <pipeline-id> --input-file <path>
notarius config scopes
notarius config validate|show|sources|explain|diff ...
notarius promptkit list ...
notarius promptkit show ...
notarius promptkit requirements show|validate ...
notarius promptkit test <kind>
notarius version
notarius completion bash|zsh|fish
notarius help [command...]
```
The alternatives separated by `|` above summarize the installed operations;
they are not literal shell commands.
| Application command | Responsibility |
| --- | --- |
| `pipeline list` | Discover configured pipelines. |
| `pipeline show` | Describe the resolved ordered steps, lanes, modules, reference slots, and configured selections. |
| `pipeline validate` | Perform offline application preflight using the intended selections and overrides. |
| `pipeline run` | Execute the selected pipeline and publish artifacts. |
| `run` | Native convenience alias for `pipeline run`. |
Move pipeline-specific validation out of `config validate --pipeline ...`.
Configuration validation, pipeline composition checks, and concrete PromptKit
request validation remain distinct. Pipeline preflight does not synthesize stage
requests or require provider credentials. PromptKit's concrete validation may
read inputs and schemas and check credentials without making a model call.
**Naming decision:** Keep `pipeline run` as the canonical command and install
`run` as a native convenience alias using
`commandkit.Alias("run", "pipeline run")`. Both routes use the same handler,
operands, flags, validation, and reporting. CommandKit supplies alias-aware help
and diagnostics; no forwarding handler or additional parser is needed.
The grouped family and root shortcut are application usability choices, not an
upstream naming requirement. The alias uses the new input conventions and does
not preserve legacy flags or behavior. Use `notarius run` in quick-start examples
and document both routes in the command reference when the migration lands.
See [native alias behavior](../../../commandkit/docs/cli.md#application-owned-aliases).
### Input Conventions
Use a required pipeline-ID operand and a required `--input-file PATH` for runs.
Use native typed declarations and supplied-value semantics, including explicit
empty, false, and zero values. Apply defaults only when the relevant value is
omitted; application validation still decides whether a supplied value is valid.
| Concern | Selected convention |
| --- | --- |
| Configuration | Native `--config` and `--overlay`; name the application scope `app`. |
| Lane selection | Repeatable `--lane ID`; omission selects the normal pipeline lane set. Replace comma-separated `--only`. |
| Output placement | Application-owned `--output-dir PATH`. |
| Chunk-plan caching | `--chunk-cache auto\|bypass\|refresh`. |
| Accepted-artifact cache | Replace checkpoint-centric controls with cache reuse/bypass/refresh behavior; concrete CLI spelling remains to be selected. `--resume` and `--recompute-step` are migration context, not compatibility requirements. |
| Presentation | Native `--format`, `--quiet`, `--verbose`, `--debug`, and `--progress`. |
| Configuration-valued tuning | Supported application `--set POINTER=JSON` overrides, including worker and semantic-budget settings once their configuration design is settled. |
Adopt native parsing rules: flags follow the complete command path and may be
interspersed with operands; `--` ends option parsing. Singleton flags use the
last successfully parsed occurrence, with every occurrence checked. Repeatable
flags preserve occurrence order and treat commas as ordinary value content.
Use filesystem and choice completion hints where applicable.
Keep worker counts and semantic continuation budgets in configuration rather
than introducing additional dedicated CLI controls now. Their field names,
defaults, and domain meaning remain master-roadmap decisions. Item 3 selects
shared loading and override wiring across application and standard commands.
### Native LLM Controls
Use ckprompt's reusable flag declarations for applicable profile, session, and
execution controls on `pipeline run`, including model, temperature, timeout,
reasoning effort, and transient retries. Run-wide overrides reach all LLM-backed
producers and validators, including continuation calls. Without an override,
normal binding/pipeline/prompt selection applies.
| Reasoning input | Meaning |
| --- | --- |
| Omitted | Inherit the selected settings. |
| `--llm-reasoning-effort high` | Replace the inherited setting. |
| `--llm-reasoning-effort ""` | Clear the inherited setting. |
Remove the separate `--clear-reasoning-effort` control. Native transient retries
remain distinct from application semantic continuation budgets. Individual
prompt selection and named prompt input injection belong on standard PromptKit
commands; pipeline modules construct their own requests and response contracts.
Use native `--llm-profile` and `--llm-session-id` spellings, without inventing
another LLM input parser.
### Application Reference Inputs
Retain repeatable `--reference SELECTOR=PATH` and `--without-reference SELECTOR`.
Keep useful pipeline-wide, lane, chunk, and stage-specific selectors, such as
`party=party.txt` and `npc-registry.extract.party=npc-party.txt`. CommandKit owns
repeatable input parsing and typed conversion; Notarius owns selector expansion,
specificity, required slots, and external/generated-reference conflict checks.
These selectors describe pipeline relationships and are not interchangeable
with PromptKit's individual-request `--llm-input-file` bindings.
### Preparing Pipeline Inspection Content
**Selected design:** Use ConfigKit for the composed, decoded
configuration, defaults, imports, overlay, inheritance, overrides, and native
provenance/redacted inspection. Feed that loaded configuration through the same
application resolution used by validation and execution to obtain selected
lanes, registered modules, validator chains, and reference relationships.
CommandKit renders a small application report of that resolved composition.
ConfigKit does not know Notarius's module catalog or pipeline semantics. Reuse
the existing resolver rather than writing a second resolution path for `show`.
Generic effective configuration and source explanations remain available through
`config show` and `config explain`. ConfigKit's safe inspection facilities do not
automatically redact an arbitrary application report; select appropriate report
content rather than dumping the raw typed configuration. Item 3 selects loader
wiring; item 4 selects the artifact-first output and completion-report contract.
### Removals And Remaining Work
Retire custom dispatch, argument-count handling, assembled help, version-command
parsing, standard configuration/LLM flag parsing, comma-separated lane parsing,
and the separate reasoning-clearing machinery. Replace `config effective` with
native `config show`, `config profile` with `promptkit show profile`, and bespoke
`--json` switches with `--format json`.
Application handlers still validate meaningful input combinations and translate
typed inputs into pipeline selections. Native `--debug` supplies reporting
behavior and conditional artifact capture as selected in item 5.
Items 3–4 select loader behavior, the small completion report, and native outcome
presentation. The master roadmap limits U9 to implementation details within
these decisions.
Evidence: [CommandKit command/input conventions](../../../commandkit/docs/cli.md#registered-command-syntax),
[reusable PromptKit flags](../../../commandkit/docs/consumers/guide.md#request-builders-and-loading),
[ConfigKit loading and inspection](../../../ConfigKit/README.md), and the current
[Notarius configuration boundaries](../internal/configuration.md).
Use the current [CLI reference](../cli.md) for migration guidance, not as a
compatibility constraint.
## 3. Application Composition, Loaders, And Preflight
**Discussion / decision:** Agreed. Fully adopt native ConfigKit, PromptKit, and
CommandKit workflows through a small composition root, shared loaders, and
invocation-local state. Follow substantially the same composition/loading pattern
as Audita while retaining Notarius's pipeline, artifact, validation, recovery,
and publication responsibilities. Centralize application prompt/profile
dependency checks in a focused `internal/preflight` package. Implementation is
pending.
### Native Application Composition
One application constructor installs the pipeline declarations and native `run`
alias, a ckconfig suite with `app` and `promptkit` scopes, the selected ckprompt
capabilities, and native help/version/completion/reporting. Supply ordinary
callbacks and typed inputs; do not introduce another CLI framework or LLM wrapper.
Constructing the application registers declarations without loading configuration,
building an engine, or initializing run storage. Help, version, and completion
remain independent of runtime configuration and provider work. Application
handlers acquire only what their operation requires.
### Shared Configuration And Engine Loaders
Use two shared loading responsibilities across application and standard commands:
1. **Application configuration:** Select explicit `--config`, then the existing
environment/discovery policy; apply native `--overlay` and supported
application `--set` overrides. ConfigKit owns composition, imports,
inheritance, presence, provenance, and inspection. Supply application
defaults, normalization, and structural validation through its extension
points. Retain the native ConfigKit result and selected main-source context;
derive runtime values where needed without copying inspection machinery into
another result abstraction.
2. **PromptKit engine:** Build a native `*promptkit.Engine` from that loaded
application's source context and the agreed native source assembly. Runtime,
preflight, and inspection use the same builder and definitions.
Preserve supplied-value information throughout loading. An explicitly empty
configuration path receives deliberate validation rather than silently falling
through to discovery. Across application and PromptKit commands, `--config`
selects the Notarius application configuration; the PromptKit scope inspects the
engine assembled for that application selection.
Retain the master roadmap's [source assembly](promptkit.md#configuration-sources-and-prompt-assets):
PromptKit's bundled underlay, embedded Notarius resources, then optional operator
companions beside the main application source actually selected by ConfigKit.
Use native imports and filesystem contexts. There is no second source search or
working-directory fallback; strict composition, missing-companion handling, and
load-error behavior remain as selected in the master roadmap. Do not reinterpret
the application document as a standalone PromptKit configuration or add another
inspection-only assembly path.
Application `--set` overrides flow through ConfigKit's load. Native request
execution overrides remain request controls; they do not mutate captured engine
configuration. Preserve the selected PromptKit configuration scope's lack of
arbitrary `--set` support.
### Invocation State And Engine Ownership
Use CommandKit's supported invocation state to retain configuration and an engine
when callbacks need them repeatedly. Application handlers use the same loader
functions with small invocation-owned state. One pipeline execution shares one
native engine across producers, validators, and continuations, preserving native
backend admission limits.
Do not assume one invocation always means one configuration selection. Native
`config diff` acquires left and right selections with potentially different
configuration, overlays, and overrides while sharing invocation state. Load the
sides independently or distinguish selections in any local reuse. A single
unqualified cached result would be incorrect. No persistent loader cache or
cryptographic configuration identity is needed.
### Shared Pipeline Resolution And Command Work
Reuse the existing application resolver with loaded configuration, selected
pipeline/lanes/reference changes, applicable overrides, and the module catalog.
It owns module/options validation, typed artifact compatibility, validator-chain
selection, ordered dependencies, and reference-binding compatibility.
| Command | Required application work |
| --- | --- |
| `pipeline list` | Load application configuration and enumerate pipelines. |
| `pipeline show` | Resolve the selected pipeline and report its composition. |
| `pipeline validate` | Resolve the pipeline and check its known native resource dependencies offline. |
| `pipeline run` / `run` | Use the same resolution and preflight checks, then prepare and execute the pipeline. |
| `config … --scope app` | Supply the native ConfigKit result directly to the integration. |
| PromptKit inspection commands | Acquire the shared engine assembly and delegate to native inspection. |
Run handlers call shared application functions directly; they do not invoke the
validation command or parse its output. `pipeline show` can describe configured
prompt/profile selections without constructing an engine just to expand them
into execution settings or performing every execution-readiness check.
### Prompt Dependencies And Offline Preflight
Notarius knows which operations need prompts; PromptKit knows whether those
definitions resolve in the assembled engine. Check every statically known prompt
dependency of the selected pipeline, including LLM-backed validators and any
separately selected correction prompts. A deterministic stage may require none,
an LLM-backed stage may require several, and multiple stages may share one.
Configuration can select the required set. For data-dependent selections, check
what is knowable before execution and resolve the final choice at runtime.
Share prompt identifiers and selection logic with execution through module or
operation declarations. Do not maintain a separate hardcoded preflight inventory.
These small dependency declarations identify required resources; they do not
restore asset manifests, prompt filesystem registration, message assembly, or
schema registries. Native sources remain the owners of prompt definitions.
Use the same engine and intended namespaces as execution to inspect known prompts
and applicable profile selections, including a known prompt's default profile
when effective. Partial profiles alone are not failures. Report missing
dependencies with pipeline, step/lane, stage or validator, and resource context
where applicable.
Offline preflight does not read the transcript, materialize reference contents,
synthesize stage requests, require credential values, or contact providers.
Engine assembly may read configuration and prompt resources. Availability does
not certify prompt domain correctness or guarantee that future requests pass
schema, credential, or provider checks. Concrete `Prepare`/`ValidateRequest`
remains the boundary for completed request validation, and smoke commands remain
explicit provider operations.
### Dedicated Preflight Package
Add a small application-level `internal/preflight` package accepting the resolved
pipeline, relevant module declarations, and the native engine. It checks the
pipeline's known prompt/profile dependencies and returns contextual findings or
errors for the command boundary to present.
Keep configuration validation in ConfigKit/application definitions, structural
resolution in the pipeline resolver, and concrete preparation in PromptKit.
Preflight does not discover configuration, register production modules, construct
engines, prepare synthetic requests, initialize storage, or own CLI reporting.
The composition root supplies its collaborators. Module dependency declarations
remain in existing module contracts; modules do not import the preflight package.
Exact function signatures and finding representation remain implementation choices.
### Request Construction, Storage, And Removals
Use native request builders directly for standard PromptKit commands where
sufficient. Add an application callback only for a demonstrated application need,
preserving supplied operator values. Pipeline modules construct their own domain
requests; share concrete request-building functions when useful without
simulating pipeline execution to populate an inspection command.
Output publication, artifact caching, debug capture, and execution records belong
to the run path. Loaders do not initialize those resources; item 5 selects debug
allocation timing. Remove command-specific loading paths, copied configuration
and resource inspection, eager runtime setup, and duplicated preflight
orchestration. No broader application/package reorganization is required by this
decision.
Evidence: [native suite and loader ownership](../../../commandkit/docs/consumers/guide.md#configkit-scopes),
[PromptKit integration callbacks](../../../commandkit/docs/consumers/guide.md#request-builders-and-loading),
[Audita composition](../../../audita/internal/cli/app_wiring.go), and the
[existing configuration ownership decision](../adr/0020-delegate-configuration-mechanics-to-configkit.md).
## 4. Reports, Machine Output, And Outcomes
**Discussion / decision:** Agreed. Deliver artifacts as files and use CommandKit
to render a small structured completion report on stdout, with a native text
template for the ordinary successful case. Additional operational diagnostics
use stderr; detailed investigation uses optional debug capture. This supersedes
the earlier extensive receipt, manifest, and durable-execution-record proposals.
Keep native outcome and delivery-failure handling. Implementation is pending.
### Accepted-Artifact Cache And PromptKit Ownership
**Agreed follow-up:** Rename and replace the checkpoint subsystem with the
[accepted-artifact cache](promptkit-architecture-discussion.md#5-accepted-artifact-cache).
That decision owns the matching and storage policy; this section records its
CommandKit and execution-boundary implications.
The cached value is an accepted answer to the original task. Compute the key
from the initial prepared request using a proposed upstream `MessageContentHash`
over ordered message roles and exact text, excluding settings, session/execution
IDs, timestamps, and provider cache hints. Do not hash the entire prepared
request or introduce another Notarius-owned fingerprint policy.
Look up `cache/<hash>.json` before model execution. A hit loads the final accepted
Notarius artifact directly, subject only to ordinary decoding/usability checks.
It does not execute a model request, reconstruct a result/exchange, replay
corrections, or repeat semantic validation merely because the artifact was cached.
On a miss, execute the complete Notarius operation: native PromptKit calls,
domain interpretation and validation, and any application-directed correction
passes. Only after final acceptance save the artifact under the initial hash,
even if correction required several turns. For normalization, cache the accepted
normalized artifact after applying the plan, not the model's proposed plan.
Rejected, failed, and validation-incomplete work is not published to the cache.
PromptKit owns structural response validation, internal retries, repairs, and
other mechanics inside each request execution. Notarius consumes a usable
response or terminal failure at the public boundary and owns domain acceptance.
Internal PromptKit attempts do not determine cache eligibility. Native debug
evidence about those attempts does not transfer their ownership to Notarius.
The upstream requirement is message-content hashing. It does not require native
cached-response injection, replay, exchange restoration, or write-through
response caching. A small Notarius filesystem lookup/write facility stores the
accepted artifact without checkpoint manifests, dependency fingerprints,
acceptance ledgers, or required execution provenance. A cache hit is not a new
model execution or a claim of validation under a changed policy.
Concrete cache flags and legacy-checkpoint handling remain implementation-planning
choices. The output/report discussion must not reintroduce cache metadata solely
to populate a report. The master roadmap now incorporates this replacement policy.
### Artifacts Are The Deliverable
Publish the accepted domain artifacts as files in the output directory. Their
contents and encoding are independent of CLI report formatting: `--format`
controls the completion summary, not the artifact payloads. Keep artifact
discovery understandable without requiring consumers to read execution metadata.
Do not require a run manifest, metadata bundle, or persisted completion receipt
alongside the artifacts. Preserve information intrinsic to the domain artifacts
and their existing schemas; do not add model, retry, validation-history, or
execution inventories to those payloads merely because it is available.
Detailed operational evidence belongs in item 5's optional debug bundle.
### Small Structured Completion Report And Native Text Template
Submit one small application value through `Reporter.Report`. Use these fields
for an ordinary successful run:
| Field | Meaning |
| --- | --- |
| `pipeline` | Selected pipeline identifier. |
| `artifact_count` | Number of deliverable artifacts produced, including reused artifacts; excludes internal intermediate results, cache entries, and debug files. |
| `output_directory` | Absolute path to the directory containing the published artifacts. |
| `debug_directory` | Optional absolute path to the completed run-specific debug bundle; omitted when capture is disabled. |
For example, without debug capture:
```json
{
"pipeline": "dnd-session",
"artifact_count": 6,
"output_directory": "/work/results/run-123"
}
```
Supply a short native `text/template` to render that same value as:
```text
Success: dnd-session produced 6 artifacts in /work/results/run-123
```
CommandKit automatically renders a string value as plain text. Here the value
is structured so JSON/YAML callers receive named fields instead of a success
sentence they must parse. The native text template provides the concise human
presentation; without it, a structured value would default to YAML. Use native
`--format text|yaml|json` selection and serializers, without separate encoders or
independent human/machine report construction.
When a completed run includes rejected work or permitted incomplete validation,
qualify the completion message clearly. Include the rejection count and/or
incomplete-validation fact in the same structured value when applicable. Do not
present unqualified success that conceals those outcomes, or make their visibility
depend on verbose/debug mode. These are small outcome qualifications, not
per-producer histories or a general metadata envelope.
No report-version field, run-ID field, token totals, profile inventory, warning
statistics, validation-summary collection, or durable execution ledger is required
by this completion contract. The old receipt is migration context, not a shape
to preserve. Standard configuration and PromptKit commands retain their native
report values and behavior without a universal Notarius wrapper.
### Optional Machine Output And Saved Reports
Operators and subprocess callers select YAML or JSON through the native format
flag. The completion report is enough to find the artifacts and recognize any
qualified outcome. Consumers interested only in artifacts need no metadata file.
Save a report with ordinary shell redirection:
```sh
notarius run dnd-session --input-file transcript.json --format json > run-report.json
```
Do not add a separate report-file flag or automatically persist a duplicate report
for this design. Keep stdout as one completion report, with additional diagnostics
on stderr, so `--format json --verbose` still produces valid report JSON.
### Publication, Streams, And Machine Consumption
The run command executes the pipeline, publishes the output bundle, completes
required run finalization, submits its successful report, and returns a native
outcome. The report describes an actual published result. Item 5 requires
requested capture to finalize successfully before a successful run report is
submitted and defines how capture failures preserve publication facts.
Use one primary run-result report, with progress and diagnostics separate. Wire
CommandKit's result destination to stdout and its diagnostic destination to
stderr. Subprocess consumers request `--format json`, capture streams separately,
check the exit status, and decode the successful run report. A report-write
failure may leave partial stdout even though publication already succeeded.
These run-consumption rules do not replace the native report behavior of other
commands, including configuration commands that can report failed validation.
### Native Visibility, Diagnostics, And Debug Location
Wire CommandKit's result destination to stdout and diagnostic destination to
stderr. Use `Info`, `Warn`, `Debug`, `Progress`, and native outcome explanations
for their respective categories; do not mix these messages into a JSON/YAML
completion report.
| Mode | Stdout | Stderr |
| --- | --- | --- |
| Ordinary | Brief completion summary. | Failure explanation, if needed. |
| `--verbose` | Same completion report. | Additional operational information, warnings, and any failure explanation. |
| `--quiet` | Nothing. | Failure explanation, if needed. |
| `--debug` | Completion report, including capture location when enabled. | Native debug diagnostics, information, warnings, progress, and any failure explanation. |
Adopt native reporting-mode ordering, explicit progress selection, and failure
visibility. The table describes mode defaults; `--progress` can explicitly enable
progress outside quiet mode. Machine callers expecting a report do not request
`--quiet`. Quiet suppression is not a delivery failure; native rendering errors
can still fail a quiet invocation. Artifact-quality advisories keep their domain
classification rather than becoming process warnings solely for visibility.
When capture completes, the text template adds a line such as:
```text
Debug capture: /tmp/notarius-debug/run-123
```
Use `debug_directory` in the structured value for the same location. Provide the
path rather than mirroring the debug bundle to stdout. An explicitly empty debug
directory preserves native debug diagnostics but creates no capture or capture
path in the completion report. Item 5 owns capture controls and lifecycle.
### Native Outcomes And Domain Acceptance
Adopt the standard process-status conventions:
| Status | Meaning |
| --- | --- |
| `0` | The command completed successfully. |
| `1` | Operational failure, cancellation, publication failure, or presentation failure. |
| `2` | Invalid invocation or identifiable invalid configuration input. |
| `3` | Native `config diff --check` found differences. |
Use ckconfig's structured configuration-error classification. Do not classify
every configuration failure as operational or every preflight failure as invalid
input. Invalid operator selections and missing bundled application resources can
have different causes; preserve that distinction and native error identity.
Command completion remains separate from domain acceptance. A successfully
published run can contain recorded rejections or validation-incomplete results
allowed by application policy; those facts qualify the completion report and do
not automatically fail the command. A terminal pipeline failure does fail the
command. Native status zero also does not replace reading standard PromptKit
reports for partial discovery or unsupported smoke checks.
Application handlers supply useful native outcome explanations and retain the
underlying error through native cause handling. CommandKit presents the terminal
result. On failure, provide a useful explanation and nonzero status rather than
requiring a separate structured error-report protocol. `--format` controls the
completion report, not native diagnostics. If artifacts were published before a
later failure, include their location in the failure explanation. Remove duplicate
terminal-error printing and custom outcome wrappers.
### Delivery Failures, Progress, And Accounting
Use CommandKit's native separation of operation causes from output and diagnostic
delivery failures. If publication succeeds but stdout fails, the final command
status reflects presentation failure while the published artifacts remain a
fact. Do not rewrite pipeline history as failed execution or automatically rerun
model work because report delivery failed.
Notarius owns stage/lane/chunk progress, domain acceptance, and cache reuse.
PromptKit supplies native execution observations and `ExecutionGroup` totals.
These facts support native operational diagnostics and optional debug capture;
they do not require a permanent consumer execution report. Explicitly enroll the
relevant initial, continuation, and validator executions without double counting.
Cache reuse does not add historical token usage to current execution totals, and
unavailable usage remains distinguishable from zero.
Remove bespoke format selection, parallel receipt encoders, custom verbosity
handling, duplicate execution accounting, and superseded terminal-reporting
machinery. Retire automatic reporting-only metadata bundles, mandatory run
manifests, per-producer provenance inventories, and durable execution-ledger
requirements. Do not preserve or rebuild those structures to populate the small
completion value. Use existing application outcome facts for its contents and
the debug overview. Artifact schemas remain domain contracts; detailed debug
representations remain diagnostic data.
Evidence: [native reporting](../../../commandkit/docs/consumers/guide.md#invocation-reporting),
[visibility and delivery](../../../commandkit/docs/cli.md#reporting), and
[outcome classification](../../../commandkit/docs/cli.md#outcome-status).
The current [run-result contract](../integrations/run-result.md) and
[CLI failure mapping](../internal/cli.md#failure-mapping-and-terminal-reporting)
document migration impact. Item 6 of the
[PromptKit discussion record](promptkit-architecture-discussion.md#6-inspection-and-execution-bookkeeping)
retains the native bookkeeping boundary. The master roadmap now incorporates
this contract and retires superseded durable-record requirements.
## 5. Debug Capture And Optional Diagnostics
**Discussion / decision:** Controls, native capture hooks, lifecycle, failure
policy, and optional diagnostics agreed. Use `--debug` for native verbosity and
durable capture with `/tmp/notarius-debug` as the default root. Operator-owned
data handling and convenient diagnostic representation are agreed. Replace the
summary/trace hierarchy with a compact run index, configuration and composition
snapshots, and operation records; maintain a `latest` symlink at the debug root.
The capture inventory and depth below are agreed: one default depth retaining
every application-visible turn, useful inputs and outputs, and application
assessments, with native snapshots and convenient readable views. An explicitly
empty debug directory disables capture while preserving native reporting-mode
selection, resolving verbosity without capture. Exact filenames and serialized
application fields remain implementation choices. Implementation is pending.
### Shared Debug Control With Conditional Capture
Attach a native application boolean binding to CommandKit's shared `--debug`
option on the pipeline run command. Capture is enabled when that boolean is true
and an effective debug directory is available. `--debug-dir PATH` overrides
`debug.directory` in application configuration, whose default is
`/tmp/notarius-debug`. A directory alone does not enable capture. Do not add a
separate `--capture-debug` flag.
**Default-directory decision:** Use the absolute root `/tmp/notarius-debug`,
with each run's bundle beneath it. Temporary diagnostic output belongs under
`/tmp` or an operator-selected location under `/var`, rather than beside a
configuration file likely to live under `/etc`. Replace the old configuration-
relative `./notarius-debug` default. Precedence is CLI override, configured value,
then application default. Nonempty relative configuration paths retain their
source-relative interpretation; nonempty relative CLI paths use the invocation
directory.
Omitting both directory overrides selects the default, so `--debug` enables
verbosity and capture without requiring a path.
**Explicit-empty decision:** An explicitly empty `--debug-dir ""` or
`debug.directory: ""` disables capture while preserving native debug verbosity
and reporting-mode ordering. Omission inherits the configured value or default;
an explicit empty value is meaningful and must not fall back to either. CLI
selection overrides configuration, including when the CLI value is empty. A
nonempty CLI directory can therefore re-enable capture over an empty configured
directory when `--debug` is enabled. No additional flag is needed for verbosity
without capture.
Recognize an empty value before resolving relative paths so it cannot become the
invocation or configuration directory. Disabled capture creates no bundle and
leaves `latest` untouched. A nonempty but unusable destination follows the
requested-capture failure policy when capture is enabled; it does not silently
disable capture or select a fallback directory.
Determine capture from the application boolean and destination, not from
`Invocation.ReportingMode()`. A later quiet or verbose selection can change
presentation without clearing the debug input. Explicit `--debug=false` follows
native boolean binding semantics and disables capture.
| Configured directory | Invocation flags | Reporting selection | Capture |
| --- | --- | --- | --- |
| Any | No `--debug`, with any directory selection | Selected native mode | None. |
| Omitted | `--debug` | Debug | `/tmp/notarius-debug`. |
| `/var/tmp/notarius` | `--debug` | Debug | `/var/tmp/notarius`. |
| Any | `--debug --debug-dir ""` | Debug | None. |
| `""` | `--debug` | Debug | None. |
| `""` | `--debug --debug-dir /tmp/investigation` | Debug | `/tmp/investigation`. |
| Omitted | `--debug --quiet` | Quiet | `/tmp/notarius-debug`. |
| Omitted | `--quiet --debug` | Debug | `/tmp/notarius-debug`. |
| Any | `--debug --quiet --debug-dir ""` | Quiet | None. |
**Upstream ordering clarification:** CommandKit uses the last enabled reporting
mode. Thus `--debug --quiet` suppresses command output while retaining capture
when the effective directory is nonempty; `--quiet --debug` selects debug
verbosity. Preserve that native ordering rather than making quiet unconditional.
Quiet still shows failure explanations, and capture does not bypass result
suppression. On success the normal run report includes the bundle location when
capture was enabled; failure explanations include an allocated bundle path when
available.
### Native Evidence And A Small Application Recorder
Use native facilities directly, attaching application step/lane/chunk/validator
context where useful:
| Facility | Available evidence |
| --- | --- |
| `OnPrepared` | Prepared request details when execution reaches that point. |
| `Observer` | Execution identities, attempt/retry/repair metadata, timing, usage completeness, and terminal metadata. |
| Returned `RunResult` | Available model output and native validation result. |
| `Exchange.Details()` | Explicit inspection of captured conversation history. |
| Application decisions | Domain acceptance/rejection, fallback, validation completeness, and cache reuse. |
Compose capture and accounting observers. Remove the debug completion wrapper,
copied execution summaries, synthetic call identities, and duplicate terminal
reporting. Keep callbacks lightweight and synchronize shared capture state:
callbacks are synchronous, can overlap across operations, and `OnPrepared` runs
while admission is held. Retain needed details and persist them at normal
operation boundaries without requiring an entire pipeline's content in memory.
Native snapshots may repeat conversation prefixes when that is the convenient
representation; do not introduce a custom delta format merely to remove them.
Capture only available evidence. Preparation/admission failures may have no
`OnPrepared` notification; observer events have metadata, not response bodies;
operational failures may return no `RunResult`. Do not present the bundle as a
complete provider transcript. Use ordinary `Run` with native hooks, without
forcing split preparation/execution solely for recording. The cache's lookup
before execution is a separate reason to use native preparation; `OnPrepared`
notifications alone do not provide cache short-circuiting.
Conversation capture required for continuation remains in-process execution
state. Persisting its history requires enabled debug capture. Cached values are
accepted domain artifacts, not conversation state or provenance records. The
separate artifact-first output and completion-report contract is selected in item 4.
### Capture Lifecycle And Failure Policy
For enabled capture, allocate the bundle after application configuration loads
and the destination and run identity are known, before engine construction and
pipeline preflight. Argument and configuration-load failures use native
diagnostics and may have no bundle. Loaders remain free of storage allocation.
After creating the run directory, update `<debug-root>/latest` to a relative
symlink targeting that run ID before engine construction or preflight starts.
This points to the newly started debug session, including one that later fails;
it is not a last-successful-run selector. Runs without capture leave it alone.
Do not update it again at completion, so an older overlapping run cannot take
the link back when it finishes. Concurrent starts use the last successful link
update; individual run directories remain the actual storage locations.
Treat link-update failures under the requested-capture failure policy below.
One application-owned recorder records initial context, accepts native execution
evidence and application decisions, and finalizes once with the known pipeline,
publication, and capture outcomes. Debug remains separate from output/cache,
never supplies cache reuse, and is not automatically deleted.
Requested capture is required work:
- Allocation failure stops before expensive model work.
- Recording failure is surfaced at the next normal application boundary.
- Finalization failure is an operational command failure.
- An existing pipeline failure remains primary; capture failures are additional
information and preserve the underlying causes.
- Already-published artifacts and already-written diagnostic data remain
available. Capture failure does not cause model retries.
Observer callbacks cannot return errors. Retain a recording error for application
code to check at an ordinary boundary rather than adding an exception or retry
framework. Submit the successful run report only after publication and requested
capture finalization succeed. If finalization fails after publication, identify
the published output and available debug location in the native failure
explanation; preserve the successful publication as a fact.
### Debug Data And Operator Responsibility
Debug artifacts are expected to contain internal application data, including
source material, rendered requests, model output, and correction history where
useful for diagnosis. If the application data is sensitive, the debug artifacts
are sensitive in the same way. Operators secure them through the ordinary means
used for application inputs and outputs, such as filesystem permissions. Do not
require a special debug-only security or retention model, or use sensitivity
concerns to limit useful diagnostic content.
Do not directly print or serialize values explicitly marked as secrets by
ConfigKit, such as resolved API keys. Use native secret-aware configuration
inspection and preserve upstream credential omissions. This boundary does not
authorize scanning or rewriting LLM output, prompts, source content, or arbitrary
strings for suspected sensitive values. Do not carry forward heuristic content
scrubbing or generic secret-detection machinery as a migration requirement.
Separate debug/output/cache lifecycles and no automatic debug deletion remain
application organization choices. Choose capture contents for diagnostic value,
size, and duplication, rather than promising a shareable or sanitized bundle.
### Native Optional Diagnostics
Use the selected standard commands through the shared loaders and native inputs:
| Command family | Purpose |
| --- | --- |
| `config …`, `promptkit list/show …` | Inspect configuration and stored resources. |
| `pipeline validate` | Check application composition and known dependencies offline. |
| `promptkit requirements show` | Discover request requirements. |
| `promptkit requirements validate`, `promptkit show prepared` | Validate or preview a concrete request without a model call. |
| `promptkit test …` | Explicit provider diagnostics. |
Application request builders are added only for actual application needs.
Inspection/diagnostic commands use their native reports and do not automatically
allocate pipeline debug bundles; operators can redirect reports for retention.
Neither debug verbosity nor enabled capture initiates additional provider probes.
Smoke reports retain native support, applicability, and result distinctions;
status zero alone does not establish that a check passed.
### Selected Bundle Organization
**Guiding principle:** Capture the most convenient debug representation. Optimize
for straightforward capture and useful inspection, rather than maximum
conciseness, completeness, or deduplication. Native values, complete snapshots,
and self-contained operation records are appropriate even when fields or content
repeat.
"Record each fact once and reference it where needed" remains a practical
guideline for avoiding independently maintained decisions and accounting, not a
strict storage constraint. Use shared application outcome facts and native
execution evidence. Repeating that evidence in useful representations does not
require another source of truth or justify custom compression/reconstruction.
Organize the bundle around the run and its operations, replacing `summary/` and
`trace/` as separate categories. Use a compact overview for navigation and keep
detailed evidence with its operation. Summary/trace are not security classes.
Illustrative layout: `latest` is the selected link name; other file names and
exact serialized fields remain implementation choices:
```text
<debug-root>/
latest -> <run-id>/
<run-id>/
run.json
config/
application.yml
promptkit.yml
pipeline.json
source.json
operations/
<operation-id>/
...
```
| Component | Purpose |
| --- | --- |
| Run overview | Identify the invocation, outcome, published output, and available diagnostic records. |
| Configuration snapshots | Preserve effective settings through native inspection. |
| Resolved pipeline | Describe selected steps, lanes, modules, validators, and reference relationships. |
| Parsed source | Preserve the generic source document, unit identities, and content produced by the input stage. |
| Operation records | Connect diagnostic evidence with its stage, chunk, lane, or validator. |
The overview provides navigation, with terminal facts drawn from the same
application outcome used for the CLI report. Reference detailed records where
useful; include convenient summary fields without maintaining a second execution
ledger. Configuration and resolved composition normally need one snapshot each.
Use native execution IDs with application scope to connect records.
Reused work has a loaded-artifact snapshot and current cache-hit decision, not
a fabricated model execution. Historical provenance is not required in cache
files or reconstructed solely for debug capture.
Retain enough available context to investigate a run that fails before publishing
output. Do not depend entirely on successfully published artifacts or automatically
copy every original input and published artifact. Retain complete native
conversation snapshots for successive turns as selected below. Do not build a
competing transcript model or deduplication system to eliminate repeated history.
### Selected Capture Inventory And Depth
**Resolved:** Use one default capture depth preserving every application-visible
turn, its useful inputs and outputs, and the decisions made about it. An operator
should be able to inspect a failed operation without reconstructing deltas,
finding a cache entry, or extracting model text from escaped JSON strings.
Convenience governs both capture and inspection; direct native serialization
and a simple readable rendering of the same values can both be useful.
#### Run Context
Retain the run overview, native application and PromptKit configuration
snapshots, resolved pipeline, and parsed source identified above. Include
application version, run/pipeline identity, timing, outcome, publication location,
and navigation to operation records in the overview. Record effective invocation
selections so request overrides remain visible even when they do not change
engine configuration.
The parsed source snapshot retains the generic document's unit identities and
content independently of the original input file. Do not automatically archive
every original configuration, template, or input file; effective snapshots and
executed requests supply the default evidence. If input parsing fails, retain
the input location and available parsing diagnostics; a parsed-source snapshot
may not exist.
#### Operation Inputs
An operation is meaningful application work: chunk planning, extraction for one
lane/chunk, merging, normalization, or validation. Identify its stage, lane,
chunk, role, and relationships to other operations as applicable. Retain an input
snapshot containing the actual values consumed:
- Relevant source/chunk content.
- Loaded reference content used by the operation.
- Incoming artifacts.
- Fixed original candidates and candidate-handle mappings for normalization.
- Prompt input bindings and application variables where applicable.
Include this content even when it also appears elsewhere in the bundle. An
extraction operation should expose its source and reference context directly.
Use existing application values and artifact codecs at meaningful boundaries;
do not invent a parallel representation of the pipeline's entire internal state.
#### Model Executions And Conversations
For each initial execution, continuation, and LLM validator execution, retain
the following evidence when available:
| Evidence | Representation |
| --- | --- |
| Prepared request | Native `PreparedRun` snapshot, including effective settings, messages, and exposed output-contract/schema details. |
| Execution events | Native metadata events in occurrence order for that execution, including attempts, retries, timing, and usage availability. |
| Returned result | Native `RunResult` snapshot. |
| Model response | Exact `RawOutput` in a separate text file that opens directly in an editor. |
| Conversation | Full `Exchange.Details()` snapshot. |
| Readable messages | Simple text or Markdown rendering of native messages, with role labels and unescaped content. |
Use indented JSON for structured native snapshots and existing artifact formats
for domain payloads. Readable messages are a straightforward rendering of native
data, not an independently maintained transcript model. Use the completed
exchange when available and prepared messages when execution fails before
returning a result; label which representation is shown.
Keep full conversation snapshots for successive turns, including rejected turns.
Repeated history is acceptable and makes a later turn independently inspectable.
Enable native exchange capture for debug-enabled executions where needed,
including single-turn calls. Continuation's in-process exchange requirements
remain independent of whether durable debug capture is enabled.
#### Interpretation, Assessment, And Stage Artifacts
Retain beside each response:
- The interpreted candidate, serialized through the existing candidate codec
before semantic acceptance.
- Decode, interpretation, or safety-check failures when no candidate could be
constructed.
- Each configured validator's result, findings, and actionable correction guidance.
- The application's disposition: accepted, rejected, continued, failed, or
accepted under the selected fallback/incomplete-validation policy.
- Links to LLM validator executions that informed the assessment.
For normalization, retain the model's reconciliation plan and the candidate
produced by applying it to the fixed original inputs. Preserve constructed
correction inputs even if the next execution fails during preparation; successful
preparation captures the actual rendered correction in its native messages.
Retain deterministic stage outputs too: chunk plans and materialized chunks,
merged artifacts, and normalized artifacts. For cache reuse, retain the loaded
artifact and current hit decision without requiring historical provenance. These
records do not fabricate a current execution or new validation of reused work.
#### Failed Runs And Native Availability
Persist evidence as operations advance under the selected recorder lifecycle;
do not wait for final publication. On failure, retain available evidence and
record where execution stopped, the returned error, and the application outcome.
Distinguish no returned result from a returned result that failed validation.
A completed response returned through coordination's `Undelivered` field is
captured as unassessed, without implying acceptance.
Internal transport retries and structural repairs receive metadata capture at
the depth PromptKit exposes. Intermediate response bodies are not required by
this inventory. Operational errors may return no `RunResult`; record that
absence clearly. This scope does not require a provider wrapper or an upstream
feature request. Evaluate a later demonstrated gap on its actual diagnostic need.
Keep selected content in full, without arbitrary truncation or a second
capture-level switch. Preserve the agreed ConfigKit-marked secret omissions and
native credential omissions; do not scan or scrub application content. Exact
filenames and serialized application fields remain implementation choices;
the inventory and capture depth are resolved.
Evidence: [native shared debug binding](../../../commandkit/docs/consumers/guide.md#invocation-reporting),
[reporting-mode ordering](../../../commandkit/docs/cli.md#modes-and-visibility),
[PromptKit observations](../../../promptkit/docs/consumers/pkg-promptkit.md#observe-an-ordinary-run),
and [exchange inspection](../../../promptkit/docs/consumers/pkg-promptkit.md#capture-and-continue-a-conversation).
The current [debug lifecycle](../internal/state.md#debug-bundles) and
[operator contract](../operations.md#debug-bundles) provide migration context.
## 6. Verification, Migration, And Roadmap Incorporation
**Discussion / decision:** Agreed. Complete one coordinated native cutover, with
CommandKit composition introduced early, verification focused on Notarius's
integration and domain responsibilities, and practical release migration guidance.
These decisions are incorporated into the existing master roadmap; no replacement
document was needed. Implementation is pending.
### Focused Verification Of Application Behavior
Use real CommandKit, ConfigKit, and PromptKit components in representative
assembled workflows, with a fake at the provider boundary. Keep the default
suite deterministic, offline, and independent of real credentials. Following the
[testing policy](../policy/testing.md), assign each behavior an appropriate test
owner and avoid duplicating upstream parser, renderer, or retry coverage.
Select a small set of workflows that collectively establish:
- Execution, inspection, and preflight resolve the same intended configuration
and prompt definitions, including application overrides and companion sources.
- Help, version, and completion work without application configuration; offline
preflight makes no provider calls.
- `run` and `pipeline run` reach the same application behavior, with the selected
flags, supplied-value distinctions, and overrides mapped correctly.
- Reports, diagnostics, and exit outcomes follow item 4, distinguishing domain
rejection, command failure, and presentation failure after publication.
- A downstream failure followed by another invocation reuses accepted artifacts
under the revised cache policy. Final corrected artifacts are stored under the
initial message hash; changed messages miss, settings-only changes can hit,
and hits neither make model calls nor repeat domain validation. Rejected or
validation-incomplete work is never stored as an accepted cache value.
- Debug capture follows item 5's directory selection, reporting-mode independence,
`latest` update timing, useful evidence, and allocation/write/finalization
failure behavior.
Use subprocess tests where the actual executable boundary adds protection:
stdout/stderr separation, process exit status, and representative machine output.
Test domain interpretation, validation, continuation, and recovery at their
appropriate package boundaries. Preserve meaningful native-execution acceptance
checks from the master roadmap without repeating them throughout CLI tests.
### Retire Tests With Superseded Machinery
Review tests of the private adapter, global scheduler, manual continuation
protocol, broad checkpoint fingerprints, and superseded CLI contracts for removal
as those implementations retire. Where an application requirement survives, test
that requirement through its new boundary. Do not mechanically reproduce old
parser, formatter, or wrapper tests against the new implementation.
Avoid freezing complete upstream help output, incidental diagnostic wording, or
every field of native debug snapshots. Debug verification should establish useful
evidence and correct failure handling while allowing representations to evolve
for convenience. Do not require exhaustive provider-attempt reconstruction or
strict serialization deduplication.
Reconcile older debug-sensitivity language, special-handling requirements, and
heuristic redaction behavior/tests with item 5. Avoid directly disclosing values
marked as secrets by ConfigKit and preserve native omissions. Do not carry forward
heuristic scrubbing of prompts, source content, or model output, or special
debug-data handling requirements. Operators secure debug artifacts as they do
ordinary application inputs and outputs.
### Migration Guidance And Documentation
Provide release guidance for operators and subprocess callers, with a practical
old-to-new migration map covering:
- Commands and flags, including the native `run` alias, configuration inspection,
and `--format`.
- Application and PromptKit configuration changes, source discovery, profiles,
and companion files.
- Native backend admission, independent worker settings, and revised retry controls.
- Accepted-artifact cache reuse, explicit refresh/bypass, and the selected
treatment of old checkpoint files.
- Machine reports, exit outcomes, and the distinction between successful command
execution and accepted domain results.
- Debug capture under `/tmp/notarius-debug`, `latest`, and the interaction between
capture and reporting modes.
CLI backward compatibility remains a non-goal; migration guidance does not imply
compatibility shims or preservation of old command behavior. Legacy checkpoint
handling and published-artifact compatibility remain explicit application
decisions, without assuming that old checkpoint envelopes must survive.
Maintain runnable examples for embedded defaults and operator overrides. Update
canonical CLI, configuration, operations, integration/consumer, and internal
documentation as the corresponding behavior lands. Record major architectural
changes in successor ADRs, preserving accepted decision text under the
[documentation policy](../policy/documentation.md#architecture-decision-records).
Keep future behavior in roadmap documents until implemented.
### Incorporation Into The Master Roadmap
**Completed:** The [master roadmap](promptkit.md#implementation-sequence) now
integrates CommandKit throughout six phases, beginning with native composition
and shared loaders, proving a representative pipeline through the new CLI, and
finishing cache/output/capture integration before final verification. The old
separate Phase 7 is retired.
Its dependency baseline includes the selected versions and pending upstream hash
request. U10 is resolved; U8 now concerns cache operation/transition, and U9
concerns implementation details within the selected output/debug contracts.
Earlier receipt/manifest inventories are retired, and U2 resolves asset/schema
assembly. Remaining correction, fallback, worker, and cache-transition choices
are explicit in its
[unresolved-choice table](promptkit.md#unresolved-choices).
The master roadmap owns sequencing and implementation status. This notepad and
the PromptKit discussion remain discussion records, not competing implementation
checklists.
### Completion Checks
Use focused checks while implementing and the existing repository-wide
`go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` checks for the
completed cutover. Check the affected maintained examples, documentation links,
dependency attribution, and removal inventory as part of this migration.
Live provider smoke tests remain explicitly optional and outside the deterministic
default suite. These change-specific migration checks do not create additional
permanent release gates; publication continues to follow the
[release policy](../policy/release.md). Recording these decisions does not claim
that implementation acceptance checks have already passed.
## Evidence And Reading
These sources support the discussion and selected decisions; they do not
constitute a completed implementation audit. Sibling links assume the local
repositories remain available.
- [Notarius CLI implementation overview](../internal/cli.md) and
[configuration internals](../internal/configuration.md).
- [CommandKit consumer guide](../../../commandkit/docs/consumers/guide.md), including
module selection, typed inputs, raw/formatted output, reporting, and loaders.
- [ckconfig integration](../../../commandkit/integrations/ckconfig/README.md) and
[ckprompt integration](../../../commandkit/integrations/ckprompt/README.md).
- [Audita command composition](../../../audita/internal/cli/app_wiring.go), showing
application handlers and upstream integrations in one CommandKit application.