diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md index 18873f56..d759ac07 100644 --- a/docs/roadmap/promptkit.md +++ b/docs/roadmap/promptkit.md @@ -927,14 +927,18 @@ independent accumulators, debug completion wrappers, synthetic call IDs, and adapter profile-recording/forwarding. Native snapshots serve actual-target inspection; do not reconstruct a durable profile inventory. -Use `ExecutionGroup` for selected diagnostic totals, explicitly enrolling initial, -continuation, and nested validator executions without overlapping double counts. -Preserve terminal coverage and unknown-versus-zero usage. Reuse adds no historical -usage to current totals. These facts support diagnostics and capture, not a -mandatory consumer execution ledger. Notarius's application outcome supplies -publication and fallback qualifications for the completion summary; assessment, -failure, and reuse details support optional diagnostics and debug capture without -independently maintained reporting state. +Use one native `ExecutionGroup` per invocation for diagnostic totals. Explicitly +enroll every actual initial, continuation, and nested LLM validator execution once, +with application step/lane/chunk/validator context attached to observations. +Use native aggregation directly, without application-maintained totals or a +hierarchy of overlapping groups. Preserve terminal coverage and unknown-versus-zero +usage. Cache hits create no execution and add no historical usage to current totals. +Concrete observer composition, enrollment plumbing, and context attachment remain +implementation choices within this contract. These facts support diagnostics and +capture, not a mandatory consumer execution ledger. Notarius's application outcome +supplies publication and fallback qualifications for the completion summary; +assessment, failure, and reuse details support optional diagnostics and debug +capture without independently maintained reporting state. ### Artifact Delivery And Completion Output @@ -946,6 +950,28 @@ need no run manifest, metadata bundle, persisted receipt, or execution history t use the artifacts. Retire reporting-only metadata and mandatory manifests instead of rebuilding them with native values. +Preserve the existing run-specific destination and domain artifact paths: + +```text +// + lanes/.json + chunk-map.json # when explicitly enabled and available + evidence-context.json # when explicitly enabled and available +``` + +Keep the existing lane filename mapping and intrinsic payload schemas. Remove +the mandatory `index.json`, `manifest.json`, and reporting-only `rejected.json`, +`warnings.json`, and `diagnostics.json` sidecars. Their relevant diagnostic content +belongs in ordinary diagnostics and optional debug capture; retain intrinsic domain +content in the artifacts themselves. `output_directory` identifies the absolute +run-specific directory, not the configured output root. + +Consumers check process status, optionally decode the small completion report, and +read documented artifact paths beneath the published directory. For configurable +pipelines, the pipeline configuration identifies expected lanes. Migration guidance +replaces receipt/index traversal with those paths, without introducing a replacement +discovery manifest or a payload-schema change. + Submit one small structured value through `Reporter.Report`: | Field | Meaning | @@ -954,6 +980,7 @@ Submit one small structured value through `Reporter.Report`: | `artifact_count` | Deliverable artifacts, including reused artifacts; excludes intermediates, cache entries, and debug files. | | `output_directory` | Absolute published artifact directory. | | `debug_directory` | Optional absolute completed run-specific capture directory; omitted when capture is disabled. | +| `fallback_count` | Number of producer operations that emitted a fallback; omitted when zero. | Supply a native text template for the ordinary successful case: @@ -970,10 +997,19 @@ or validation-history collection is required. When a completed run used a fallback, qualify the human summary and structured value with that fact, even if native diagnostic verbosity suppresses individual -warnings. For example, a small fallback count can accompany the completion fields; -exact field names remain implementation details. Do not present the run as if all -operations obtained normally validated results. Detailed reasons belong in optional -diagnostics/debug capture, without per-producer histories in the completion report. +warnings. Count each producer operation that actually emitted a fallback once; +do not count rejected attempts, individual validator failures, or downstream +artifacts affected by that fallback. JSON/YAML include `fallback_count` only when +positive. The native text template can render: + +```text +Success with fallback: dnd-session produced 6 artifacts in /work/results/run-123 +2 operations used fallback. +``` + +Do not present the run as if all operations obtained normally validated results. +Detailed reasons belong in optional diagnostics/debug capture, without per-producer +histories in the completion report. Quiet mode continues to suppress stdout. Wire results to stdout and diagnostics to stderr: @@ -1071,9 +1107,12 @@ never cause model retries. Finalize before submitting the success report. Replace separate summary/trace categories with a compact run overview, configuration/composition snapshots, parsed source, and per-operation records. -Use native execution IDs with application scope for navigation. `latest` is the -selected link name; other filenames and serialized application fields remain -implementation choices. +Organize operation directories by meaningful pipeline scope, with native execution +IDs within those operations. Provide a readable run overview and clearly named +structured snapshots and text views. `latest` is the selected link name; exact +debug filenames, directory nesting, and serialized application fields are explicitly +delegated to implementation. Evaluate them by ease of investigating an operation; +do not freeze them as a public format or introduce another debug framework. | Capture | Selected contents and depth | | --- | --- | @@ -1128,7 +1167,10 @@ reopen the architecture. IDs retain continuity with the earlier roadmap. | ID | Remaining choice or dependency | Resolve for | | --- | --- | --- | | U1 | Review the message-content hash API/release documented upstream in v0.24.0, including its ConfigKit dependency; verify the final compatible dependency set against the selected baseline. | Track in Phase 1; required to finish cache integration in Phase 5. | -| U9 | Observer/group enrollment and application scope; concrete conditional report fields and debug filenames/serialization within the agreed contracts; publication-layout/consumer migration details preserving intrinsic artifact schemas. | Implementation details in Phases 3–5; no new receipt/manifest inventory or capture-policy decision required. | + +U1 is the only remaining unresolved dependency decision. Observer wiring, debug +filenames, and other routine implementation details are delegated under the +resolved designs below; all implementation phases remain pending. **U2 resolved:** Ordinary embedded assets, prompt-declared output contracts, native schema-source selection, and removal of duplicate schema/filesystem @@ -1186,6 +1228,17 @@ explicit `--cache`, module-binding `cache`, pipeline `cache`, then `auto`. results without reading. Cache I/O is best effort; legacy records remain untouched and are not imported. Implement in Phases 3–5, subject to U1's hash dependency. +**U9 resolved:** [Execution bookkeeping](#inspection-and-execution-bookkeeping) +uses one native execution group per invocation, with all actual initial, +continuation, and LLM validator executions enrolled once and application scope +attached. [Completion output](#artifact-delivery-and-completion-output) adds only +optional positive `fallback_count`, counting producer operations that emitted +fallbacks. Preserve run-specific domain artifact paths and schemas; retire index, +manifest, and reporting-only sidecars and migrate consumers to documented paths. +Observer wiring and [debug filenames/layout](#inventory-and-depth) are explicitly +implementation choices, not unresolved architecture or stable debug contracts. +Implement in Phases 3–5. + **U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight, output, debug policy, and coordinated cutover sequencing are selected above. The former later CommandKit phase is retired. CLI compatibility is a non-goal; @@ -1206,9 +1259,8 @@ major boundary changes in successor ADRs without rewriting accepted decisions. Keep the cache dependency explicit without blocking independent work. - Map affected ADRs, current callers, contracts, and removal inventory. Carry forward Audita evidence without adopting its application wrappers. -- Preserve U9 with its dependent phases; do not reopen resolved module - execution, prompt/guidance ownership, retry controls/defaults, assets/schemas, - worker/admission policy, CommandKit, debug, cache, or output ownership. +- Carry resolved U2–U10 designs into their dependent phases without reopening + ownership. Leave observer wiring and debug filenames to implementation. **Completion evidence:** reviewed dependency baseline, upstream dependency, scope, and outstanding choices are explicit. Upstream availability alone does not @@ -1243,11 +1295,12 @@ without credentials or synthetic requests. `config diff` handles two selections. Resolve `max_additional_turns` with omission/zero distinctions and pass it to native coordination; inherit native transport and structural-repair defaults. - Exercise an LLM validator on the same engine and normalization with fixed - handles and fresh plan application. Select actual observer/group enrollment. + handles and fresh plan application. Wire U9's single invocation execution group + with application scope for initial, continuation, and validator executions. - Run through the new command surface and alias, using the small completion report, native outcomes/streams, and useful debug capture from the outset. - Begin the resolved U8 cache integration and U9 details without a provisional - local message hash. + Begin the resolved U8 cache integration and U9 reporting/capture implementation + without a provisional local message hash. **Completion evidence:** fake-provider workflows exercise accumulated history, collection of independent validator findings without rejection short-circuiting, @@ -1286,7 +1339,8 @@ requirements remain. - Resolve U1's supplying release and implement U8's unified lookup at initial preparation, accepted-artifact/plan storage under that hash, directory and mode precedence, and best-effort cache I/O. Retire both legacy cache mechanisms. -- Complete the artifact-first publication/report contract and U9 details. Remove +- Complete U9's artifact-first publication/report contract. Preserve domain paths + and schemas, add `fallback_count`, and retire index/reporting sidecars. Remove checkpoint fingerprint graphs, mandatory manifests/receipts, durable execution ledgers, and duplicate accounting; do not retain them to populate summaries. - Complete capture inventory, per-operation evidence, directory/empty-value @@ -1299,8 +1353,11 @@ affected downstream work skip cache lookup/storage; unaffected accepted work remains reusable. Chunk plans use the same cache, rematerialize against current source, and no longer pin one partition across changed messages. Mode precedence, cache I/O failures, and leaving both legacy stores untouched are verified. -Artifacts need no metadata bundle. Machine output and debug capture obey the -selected contracts, including failures after publication and quiet-mode capture. +Artifacts need no metadata bundle and consumers use documented domain paths. +Machine output and debug capture obey the selected contracts, including omitted +zero/accurate positive fallback counts, failures after publication, and quiet-mode +capture. Observer wiring and debug organization satisfy the selected scope and +inventory without becoming separate public contracts. ### Phase 6: Verify And Document The Coordinated Cutover @@ -1333,7 +1390,7 @@ Release preparation/publication is governed separately by release policy. | Separate source-keyed chunk-plan store, canonical-plan provenance/publication policy, nested legacy cache configuration, `--resume`, `--recompute-step`, separate chunk-cache flag | Unified message-content lookup/storage, domain chunk-plan interpretation/materialization, `cache_directory`, pipeline/module `cache`, and invocation-wide `--cache`. | | Inspection wrappers, copied effective parameters, usage summaries/conversions/accumulators, debug completion wrapper, synthetic IDs, profile recorder | Native facts with application scope for diagnostics. | | Custom dispatch/help/version/parsing, command-specific loading, copied inspection, legacy aliases/flags, parallel encoders and verbosity handling | Native declarations/loaders, pipeline handlers, domain input checks. | -| Mandatory run manifests, reporting-only metadata bundles, extensive receipts and durable execution ledgers | Domain artifact files, one small native completion report, optional debug evidence. | +| Mandatory `index.json`/`manifest.json`, reporting-only `rejected.json`/`warnings.json`/`diagnostics.json`, extensive receipts and durable execution ledgers | Preserved domain artifact paths and schemas, one small native completion report with optional `fallback_count`, ordinary diagnostics, and optional debug evidence. | | Summary/trace hierarchy, heuristic debug-content scrubbing, custom transcript/deduplication machinery | Convenient run/operation capture and native secret/credential omissions. | ## Acceptance And Verification @@ -1360,8 +1417,8 @@ retry tests or freeze complete help text and native debug snapshots. | Accepted-artifact cache | Downstream recovery; initial-message keys; corrected final artifact stored late under the initial key; changed messages miss; settings/ID-only changes can hit; no model/revalidation on hits; unusable entries miss. Failed/rejected/incomplete work is not stored; fallbacks and affected downstream work skip lookup/storage while unaffected accepted predecessors remain reusable; no required history. | | Chunk-plan reuse | Final accepted plan stored under the initial request hash after materialization/validation/correction; hits rematerialize with structural/source-reference usability checks and no renewed semantic validation. Changed messages can select a new partition; deterministic chunkers run without synthetic cache requests. | | Cache controls and migration | Root-level directory default/explicit selection; mode precedence CLI, module binding, pipeline, then `auto`. `auto` reads/writes; `bypass` does no I/O or directory creation; `refresh` only replaces entries after new acceptance. Unusable entries miss; cache I/O failures preserve successful pipeline results. Both legacy stores remain untouched and unread; no conversion or separate canonical-plan mechanism. | -| Output/outcomes | Artifact files independent of report format; native text/JSON/YAML summary and qualified outcomes; no mandatory metadata bundle; quiet suppression, verbose stderr, debug path; native nonzero failures and useful explanations. Publication survives later capture/report failure without model reruns. | -| Execution observations | Initial/continuation/validator totals preserve unknown-versus-zero usage without double counts; result-less failures observable; reuse does not count historical work; native execution success remains distinct from acceptance. | +| Output/outcomes | Preserved run-specific domain paths, lane filename mapping, and payload schemas; optional chunk-map/evidence-context publication remains. No index/manifest/reporting sidecars. Native text/JSON/YAML summary includes positive `fallback_count` and omits zero; count emitting producer operations, not attempts, validator failures, or affected derivatives. Quiet suppression, verbose stderr, debug path, and useful nonzero failures hold. Publication survives later capture/report failure without model reruns. | +| Execution observations | One native invocation group enrolls actual initial/continuation/LLM validator executions once, with application scope; native totals preserve unknown-versus-zero usage without application accumulators or overlapping-group aggregation. Result-less failures remain observable; cache hits add no executions/history; native execution success remains distinct from acceptance. | | Debug controls/lifecycle | Default/CLI/config/explicit-empty paths and native mode ordering; quiet capture; allocation before engine/preflight; start-time relative `latest`, no completion retarget; disabled runs leave it untouched; capture failures preserve primary causes and publication facts. | | Debug evidence | Available input/source/native snapshots, readable text, all visible turns, assessments, normalization and deterministic outputs, cache hits, and failed-run evidence; no invented provider transcript; ConfigKit secrets/native credentials omitted, no heuristic content scrubbing. | | End-to-end migration | Maintained minimal and complete D&D workflows, session routing, reference behavior, raw artifacts, reports, and migration examples work through the new executable. No postponed CommandKit phase. | @@ -1408,8 +1465,11 @@ Provide practical release guidance for operators and subprocess callers: Explain refresh/bypass, best-effort I/O, fallback/affected-work exclusion, removal of both legacy mechanisms without importing/deleting records, and retirement of canonical-source partition reuse. -- Artifact delivery without mandatory manifests/receipts; preserve intrinsic - artifact schemas and document any explicitly selected format change. +- Artifact delivery through preserved run-specific domain paths and lane filename + mapping, without index/manifest/reporting sidecars or receipt traversal. Document + `fallback_count` and its omission/counting rules; consumers use the published + directory and pipeline-defined lanes. Preserve intrinsic artifact schemas and + document any explicitly selected format change. - Debug root/default/empty behavior, `latest`, native mode ordering, failure behavior, and ordinary operator responsibility for application data.