From 3c73ccfad03cd6ec1933d07c1a2f6eb7c782df46 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 20 Nov 2026 00:48:29 +0000 Subject: [PATCH] Record CommandKit verification and migration plan --- .../commandkit-migration-discussion.md | 161 +++++++++++++++--- 1 file changed, 136 insertions(+), 25 deletions(-) diff --git a/docs/roadmap/commandkit-migration-discussion.md b/docs/roadmap/commandkit-migration-discussion.md index 71f2ca4d..e0aa4ed1 100644 --- a/docs/roadmap/commandkit-migration-discussion.md +++ b/docs/roadmap/commandkit-migration-discussion.md @@ -26,7 +26,9 @@ selects native composition, shared loaders, and application preflight. Item 4 selects native reporting, machine output, and outcomes. Item 5 selects conditional debug capture through the shared `--debug` flag, native diagnostics, and a run/ operation bundle with a `latest` symlink. Capture depth, exact run-report fields, -and verification remain open below. +and other concrete record details remain open below. Item 6 selects verification, +migration guidance, and incorporation into the master roadmap; that incorporation +is still pending. The PromptKit architecture and checkpoint decisions remain in force. This discussion develops the CommandKit portion of that migration and its interactions @@ -85,8 +87,9 @@ spending migration effort recreating existing CLI conventions. Adopt the standard capability suite selected in item 1, including upstream commands and recommendations beyond the existing Notarius surface. Items 2–4 -select application inputs, composition, and reporting; the remaining discussion -determines debug capture, verification, and concrete record details. +select application inputs, composition, and reporting; items 5–6 select debug +capture principles, verification, and migration. Concrete record details remain +open as identified below. Pipeline and domain code should remain independent of CommandKit. Native PromptKit observations supply execution facts; Notarius attaches application @@ -114,8 +117,8 @@ Two application needs already identified for deliberate treatment are: 4. Complete command and reporting migration before final end-to-end verification and migration documentation. -This is the agreed sequencing direction, not a replacement phase plan. Concrete -work units and their placement in the master roadmap follow the discussion below. +Item 6 selects the replacement phase sequence for incorporation into the master +roadmap. The master roadmap has not yet been updated. ## 1. Adoption Scope And Dependency Baseline @@ -886,28 +889,136 @@ The current [debug lifecycle](../internal/state.md#debug-bundles) and ## 6. Verification, Migration, And Roadmap Incorporation -**Discussion / decision:** Pending. +**Discussion / decision:** Agreed. Complete one coordinated native cutover, with +CommandKit composition introduced early, verification focused on Notarius's +integration and domain responsibilities, and practical release migration guidance. +Incorporate these decisions into the existing master roadmap; another replacement +document is unnecessary. Implementation and master-roadmap incorporation are pending. -- Which focused command and subprocess tests protect the new native command, - output, outcome, receipt, and debug behavior? Which tests of superseded CLI - contracts and parser/formatter implementation can retire? -- Which representative assembled workflows prove that execution, inspection, - and preflight use the same intended configuration without duplicating lower-level - coverage? Keep the default suite offline and independent of real credentials. -- How should CommandKit steps replace the master roadmap's deferred Phase 7, - and which migration examples, documentation, and ADRs need coordinated updates? -- What release-documentation guidance will show operators and subprocess callers - how to migrate their workflows and integrations to the new conventions? -- Reconcile older debug-sensitivity language, special-handling requirements, and - heuristic redaction behavior/tests with item 5's clarified boundary. Do not - reintroduce them as implicit requirements while migrating documentation or code. +### Focused Verification Of Application Behavior -After discussion, reconcile the master roadmap's U1/U9/U10 entries, sequencing, -removals, and acceptance criteria, including its treatment of existing inspection -commands and machine-output compatibility. Carry forward the explicit non-goal -of CLI backward compatibility and release migration guidance. Preserve other -unresolved choices explicitly. Then mark this document as the discussion record -and keep implementation status in the master roadmap. +Use real CommandKit, ConfigKit, and PromptKit components in representative +assembled workflows, with a fake at the provider boundary. Keep the default +suite deterministic, offline, and independent of real credentials. Following the +[testing policy](../policy/testing.md), assign each behavior an appropriate test +owner and avoid duplicating upstream parser, renderer, or retry coverage. + +Select a small set of workflows that collectively establish: + +- Execution, inspection, and preflight resolve the same intended configuration + and prompt definitions, including application overrides and companion sources. +- Help, version, and completion work without application configuration; offline + preflight makes no provider calls. +- `run` and `pipeline run` reach the same application behavior, with the selected + flags, supplied-value distinctions, and overrides mapped correctly. +- Reports, diagnostics, and exit outcomes follow item 4, distinguishing domain + rejection, command failure, and presentation failure after publication. +- A downstream failure followed by resume reuses successful predecessors under + the simpler checkpoint policy, with explicit recomputation and consumed-input + changes handled as selected in the master roadmap. +- Debug capture follows item 5's directory selection, reporting-mode independence, + `latest` update timing, useful evidence, and allocation/write/finalization + failure behavior. + +Use subprocess tests where the actual executable boundary adds protection: +stdout/stderr separation, process exit status, and representative machine output. +Test domain interpretation, validation, continuation, and recovery at their +appropriate package boundaries. Preserve meaningful native-execution acceptance +checks from the master roadmap without repeating them throughout CLI tests. + +### Retire Tests With Superseded Machinery + +Review tests of the private adapter, global scheduler, manual continuation +protocol, broad checkpoint fingerprints, and superseded CLI contracts for removal +as those implementations retire. Where an application requirement survives, test +that requirement through its new boundary. Do not mechanically reproduce old +parser, formatter, or wrapper tests against the new implementation. + +Avoid freezing complete upstream help output, incidental diagnostic wording, or +every field of native debug snapshots. Debug verification should establish useful +evidence and correct failure handling while allowing representations to evolve +for convenience. Do not require exhaustive provider-attempt reconstruction or +strict serialization deduplication. + +Reconcile older debug-sensitivity language, special-handling requirements, and +heuristic redaction behavior/tests with item 5. Avoid directly disclosing values +marked as secrets by ConfigKit and preserve native omissions. Do not carry forward +heuristic scrubbing of prompts, source content, or model output, or special +debug-data handling requirements. Operators secure debug artifacts as they do +ordinary application inputs and outputs. + +### Migration Guidance And Documentation + +Provide release guidance for operators and subprocess callers, with a practical +old-to-new migration map covering: + +- Commands and flags, including the native `run` alias, configuration inspection, + and `--format`. +- Application and PromptKit configuration changes, source discovery, profiles, + and companion files. +- Native backend admission, independent worker settings, and revised retry controls. +- Checkpoint reuse, explicit recomputation, and the selected treatment of existing + checkpoint files. +- Machine reports, exit outcomes, and the distinction between successful command + execution and accepted domain results. +- Debug capture under `/tmp/notarius-debug`, `latest`, and the interaction between + capture and reporting modes. + +CLI backward compatibility remains a non-goal; migration guidance does not imply +compatibility shims or preservation of old command behavior. Existing checkpoint +and published-artifact compatibility remain explicit application decisions about +stored work and consumer data, separate from CLI syntax. + +Maintain runnable examples for embedded defaults and operator overrides. Update +canonical CLI, configuration, operations, integration/consumer, and internal +documentation as the corresponding behavior lands. Record major architectural +changes in successor ADRs, preserving accepted decision text under the +[documentation policy](../policy/documentation.md#architecture-decision-records). +Keep future behavior in roadmap documents until implemented. + +### Incorporation Into The Master Roadmap + +Replace the deferred CommandKit Phase 7 with adoption throughout the existing +master roadmap. Use this phase sequence: + +1. Establish the agreed dependency baseline and remaining contract decisions. +2. Build native application composition, shared loaders, standard commands, and + the dedicated preflight package alongside native source assembly. +3. Prove a representative native pipeline through the actual command surface, + including reporting and useful debug capture. +4. Migrate remaining roles and remove the adapter, scheduler, retry machinery, + and custom CLI infrastructure. +5. Complete checkpoint migration, durable records, and capture behavior. +6. Verify the assembled application and finish migration documentation. + +Each phase delivers a reviewable result with appropriate verification. Tests and +documentation proceed throughout; the final phase checks integration and +completeness. The phases do not require separate releases or an intermediate +compatibility framework. + +Resolve U1's dependency baseline and U10's CommandKit scope from items 1–4. Narrow +U9 to remaining durable-field, accounting, and capture details within the selected +boundaries. Reconcile removal inventory and acceptance criteria, including old +inspection-command and machine-output compatibility assumptions. Preserve +unresolved schema handling, role-specific retry/fallback policy, worker defaults, +checkpoint transition choices, and other concrete choices until actually settled. + +Once incorporated, the master roadmap owns implementation sequencing and status; +both migration notepads remain discussion records. Do not maintain competing +implementation checklists in the notepads. + +### Completion Checks + +Use focused checks while implementing and the existing repository-wide +`go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` checks for the +completed cutover. Check the affected maintained examples, documentation links, +dependency attribution, and removal inventory as part of this migration. + +Live provider smoke tests remain explicitly optional and outside the deterministic +default suite. These change-specific migration checks do not create additional +permanent release gates; publication continues to follow the +[release policy](../policy/release.md). Recording these decisions does not claim +that implementation acceptance checks have already passed. ## Evidence And Reading