diff --git a/docs/roadmap/commandkit-migration-discussion.md b/docs/roadmap/commandkit-migration-discussion.md index c7db053b..f12e7eac 100644 --- a/docs/roadmap/commandkit-migration-discussion.md +++ b/docs/roadmap/commandkit-migration-discussion.md @@ -25,15 +25,15 @@ Item 2 selects the application command structure and input conventions. Item 3 selects native composition, shared loaders, and application preflight. Item 4 selects native reporting, machine output, and outcomes. Item 5 selects conditional debug capture through the shared `--debug` flag, native diagnostics, and a run/ -operation bundle with a `latest` symlink. Item 6 selects verification, migration -guidance, and incorporation into the master roadmap; that incorporation is still -pending. +operation bundle with a `latest` symlink. Its capture inventory and depth are +resolved below. Item 6 selects verification, migration guidance, and incorporation +into the master roadmap; that incorporation is still pending. -The major architectural decisions are settled. Remaining choices are the exact -capture inventory and depth (item 5), run-report fields/versioning and durable -record formats (item 4/U9), and explicit-empty debug-directory handling plus any -demonstrated need for verbosity without capture (item 5). Function signatures, -bundle filenames other than `latest`, and similar implementation details do not +The major architectural decisions are settled. Remaining choices are run-report +fields/versioning and durable record formats (item 4/U9), and explicit-empty +debug-directory handling plus any demonstrated need for verbosity without capture +(item 5). Function signatures, bundle filenames other than `latest`, and similar +implementation details do not reopen the selected architecture. Broader PromptKit choices remain with the master roadmap. @@ -663,9 +663,12 @@ durable capture with `/tmp/notarius-debug` as the default root. Operator-owned data handling and convenient diagnostic representation are agreed. Replace the summary/trace hierarchy with a compact run index, configuration and composition snapshots, and operation records; maintain a `latest` symlink at the debug root. -Convenience guides capture depth; the precise inventory and serialized fields -remain under discussion. Explicit-empty directory handling and any demonstrated -need for verbosity without capture also remain open. Implementation is pending. +The capture inventory and depth below are agreed: one default depth retaining +every application-visible turn, useful inputs and outputs, and application +assessments, with native snapshots and convenient readable views. Exact filenames +and serialized application fields remain implementation choices. Explicit-empty +directory handling and any demonstrated need for verbosity without capture +remain open. Implementation is pending. ### Shared Debug Control With Conditional Capture @@ -850,6 +853,7 @@ exact serialized fields remain implementation choices: application.yml promptkit.yml pipeline.json + source.json operations/ / ... @@ -860,6 +864,7 @@ exact serialized fields remain implementation choices: | Run overview | Identify the invocation, outcome, published output, and available diagnostic records. | | Configuration snapshots | Preserve effective settings through native inspection. | | Resolved pipeline | Describe selected steps, lanes, modules, validators, and reference relationships. | +| Parsed source | Preserve the generic source document, unit identities, and content produced by the input stage. | | Operation records | Connect diagnostic evidence with its stage, chunk, lane, or validator. | The overview provides navigation, with terminal facts drawn from the same @@ -867,29 +872,127 @@ application outcome used for the CLI report. Reference detailed records where useful; include convenient summary fields without maintaining a second execution ledger. Configuration and resolved composition normally need one snapshot each. Use native execution IDs with application scope to connect records. -Reused work has a checkpoint/reuse decision and reference, not a fabricated model -execution. +Reused work has a restored-payload snapshot and reuse decision linked to original +provenance, not a fabricated model execution. Retain enough available context to investigate a run that fails before publishing output. Do not depend entirely on a successful output manifest or automatically -copy every original input and published artifact. Prefer the native conversation -representation when convenient, including complete per-turn snapshots if those -are easier to capture and inspect. Do not build a competing transcript model or -deduplication system to eliminate repeated history. +copy every original input and published artifact. Retain complete native +conversation snapshots for successive turns as selected below. Do not build a +competing transcript model or deduplication system to eliminate repeated history. -### Open Discussion: Capture Depth +### Selected Capture Inventory And Depth -The organization and emphasis on convenience are selected; precise evidence -and granularity remain open. Capture readily available, useful evidence without -making exhaustive reconstruction or every provider attempt a prerequisite: +**Resolved:** Use one default capture depth preserving every application-visible +turn, its useful inputs and outputs, and the decisions made about it. An operator +should be able to inspect a failed operation without reconstructing deltas, +finding a checkpoint, or extracting model text from escaped JSON strings. +Convenience governs both capture and inspection; direct native serialization +and a simple readable rendering of the same values can both be useful. -- Which prepared request details, outputs, intermediate artifacts, validation - decisions, and correction content to retain. -- Which native conversation snapshots make troubleshooting most convenient. -- How much failed-run context to retain independently, and when references to - existing inputs, checkpoints, or published artifacts suffice. -- Which desired evidence is available through native public hooks and which - gaps would justify an upstream feature request. +#### Run Context + +Retain the run overview, native application and PromptKit configuration +snapshots, resolved pipeline, and parsed source identified above. Include +application version, run/pipeline identity, timing, outcome, publication location, +and navigation to operation records in the overview. Record effective invocation +selections so request overrides remain visible even when they do not change +engine configuration. + +The parsed source snapshot retains the generic document's unit identities and +content independently of the original input file. Do not automatically archive +every original configuration, template, or input file; effective snapshots and +executed requests supply the default evidence. If input parsing fails, retain +the input location and available parsing diagnostics; a parsed-source snapshot +may not exist. + +#### Operation Inputs + +An operation is meaningful application work: chunk planning, extraction for one +lane/chunk, merging, normalization, or validation. Identify its stage, lane, +chunk, role, and relationships to other operations as applicable. Retain an input +snapshot containing the actual values consumed: + +- Relevant source/chunk content. +- Loaded reference content used by the operation. +- Incoming artifacts. +- Fixed original candidates and candidate-handle mappings for normalization. +- Prompt input bindings and application variables where applicable. + +Include this content even when it also appears elsewhere in the bundle. An +extraction operation should expose its source and reference context directly. +Use existing application values and artifact codecs at meaningful boundaries; +do not invent a parallel representation of the pipeline's entire internal state. + +#### Model Executions And Conversations + +For each initial execution, continuation, and LLM validator execution, retain +the following evidence when available: + +| Evidence | Representation | +| --- | --- | +| Prepared request | Native `PreparedRun` snapshot, including effective settings, messages, and exposed output-contract/schema details. | +| Execution events | Native metadata events in occurrence order for that execution, including attempts, retries, timing, and usage availability. | +| Returned result | Native `RunResult` snapshot. | +| Model response | Exact `RawOutput` in a separate text file that opens directly in an editor. | +| Conversation | Full `Exchange.Details()` snapshot. | +| Readable messages | Simple text or Markdown rendering of native messages, with role labels and unescaped content. | + +Use indented JSON for structured native snapshots and existing artifact formats +for domain payloads. Readable messages are a straightforward rendering of native +data, not an independently maintained transcript model. Use the completed +exchange when available and prepared messages when execution fails before +returning a result; label which representation is shown. + +Keep full conversation snapshots for successive turns, including rejected turns. +Repeated history is acceptable and makes a later turn independently inspectable. +Enable native exchange capture for debug-enabled executions where needed, +including single-turn calls. Continuation's in-process exchange requirements +remain independent of whether durable debug capture is enabled. + +#### Interpretation, Assessment, And Stage Artifacts + +Retain beside each response: + +- The interpreted candidate, serialized through the existing candidate codec + before semantic acceptance. +- Decode, interpretation, or safety-check failures when no candidate could be + constructed. +- Each configured validator's result, findings, and actionable correction guidance. +- The application's disposition: accepted, rejected, continued, failed, or + accepted under the selected fallback/incomplete-validation policy. +- Links to LLM validator executions that informed the assessment. + +For normalization, retain the model's reconciliation plan and the candidate +produced by applying it to the fixed original inputs. Preserve constructed +correction inputs even if the next execution fails during preparation; successful +preparation captures the actual rendered correction in its native messages. + +Retain deterministic stage outputs too: chunk plans and materialized chunks, +merged artifacts, and normalized artifacts. For checkpoint reuse, retain the +restored payload and reuse decision linked to original provenance. These records +do not fabricate a current execution or new validation of reused work. + +#### Failed Runs And Native Availability + +Persist evidence as operations advance under the selected recorder lifecycle; +do not wait for final publication. On failure, retain available evidence and +record where execution stopped, the returned error, and the application outcome. +Distinguish no returned result from a returned result that failed validation. +A completed response returned through coordination's `Undelivered` field is +captured as unassessed, without implying acceptance. + +Internal transport retries and structural repairs receive metadata capture at +the depth PromptKit exposes. Intermediate response bodies are not required by +this inventory. Operational errors may return no `RunResult`; record that +absence clearly. This scope does not require a provider wrapper or an upstream +feature request. Evaluate a later demonstrated gap on its actual diagnostic need. + +Keep selected content in full, without arbitrary truncation or a second +capture-level switch. Preserve the agreed ConfigKit-marked secret omissions and +native credential omissions; do not scan or scrub application content. Exact +filenames and serialized application fields remain implementation choices; +the inventory and capture depth are resolved. Evidence: [native shared debug binding](../../../commandkit/docs/consumers/guide.md#invocation-reporting), [reporting-mode ordering](../../../commandkit/docs/cli.md#modes-and-visibility), @@ -1008,10 +1111,12 @@ completeness. The phases do not require separate releases or an intermediate compatibility framework. Resolve U1's dependency baseline and U10's CommandKit scope from items 1–4. Narrow -U9 to remaining durable-field, accounting, and capture details within the selected -boundaries. Reconcile removal inventory and acceptance criteria, including old -inspection-command and machine-output compatibility assumptions. Preserve -unresolved schema handling, role-specific retry/fallback policy, worker defaults, +U9 to remaining durable-field and accounting choices, and debug implementation +details within item 5's resolved inventory and depth. Preserve its separate +open directory-control choices. Reconcile removal inventory and acceptance +criteria, including old inspection-command and machine-output compatibility +assumptions. Preserve unresolved schema handling, role-specific retry/fallback +policy, worker defaults, checkpoint transition choices, and other concrete choices until actually settled. Once incorporated, the master roadmap owns implementation sequencing and status;