Record conditional debug capture and diagnostic design
This commit is contained in:
@@ -23,8 +23,9 @@ Item 1 selects the dependency versions and standard capabilities, including
|
||||
PromptKit v0.23.3 in place of the master roadmap's initial v0.23.2 target.
|
||||
Item 2 selects the application command structure and input conventions. Item 3
|
||||
selects native composition, shared loaders, and application preflight. Item 4
|
||||
selects native reporting, machine output, and outcomes. Exact run-report fields,
|
||||
debug capture, and verification choices remain open below.
|
||||
selects native reporting, machine output, and outcomes. Item 5 selects conditional
|
||||
debug capture through the shared `--debug` flag and native diagnostics. Debug
|
||||
content/organization, exact run-report fields, and verification remain open below.
|
||||
|
||||
The PromptKit architecture and checkpoint decisions remain in force. This
|
||||
discussion develops the CommandKit portion of that migration and its interactions
|
||||
@@ -96,9 +97,9 @@ Two application needs already identified for deliberate treatment are:
|
||||
replaces its presentation with one compact application report rendered natively
|
||||
for humans and machines. Exact fields/versioning remain with U9; reports must
|
||||
describe the actual publication and domain outcome.
|
||||
- The current `--debug` requests durable artifacts. Retain the distinction between
|
||||
artifact capture and reporting verbosity, while choosing native controls and
|
||||
meanings rather than requiring that flag to retain its old behavior.
|
||||
- Item 5 uses native `--debug` for diagnostic verbosity and for capture when a
|
||||
debug directory is supplied through CLI or configuration. Capture selection
|
||||
remains independent of the final reporting mode.
|
||||
|
||||
### Intended Sequencing
|
||||
|
||||
@@ -329,7 +330,7 @@ native `config show`, `config profile` with `promptkit show profile`, and bespok
|
||||
|
||||
Application handlers still validate meaningful input combinations and translate
|
||||
typed inputs into pipeline selections. Native `--debug` supplies reporting
|
||||
behavior; a distinct artifact-capture control remains to be selected in item 5.
|
||||
behavior and conditional artifact capture as selected in item 5.
|
||||
Items 3–4 own loader behavior, exact report content, and outcome presentation.
|
||||
|
||||
Evidence: [CommandKit command/input conventions](../../../commandkit/docs/cli.md#registered-command-syntax),
|
||||
@@ -549,8 +550,9 @@ migration evidence, not a requirement to reproduce it.
|
||||
|
||||
The run command executes the pipeline, publishes the output bundle, completes
|
||||
required run finalization, submits its successful report, and returns a native
|
||||
outcome. The report describes an actual published result. Item 5 will settle the
|
||||
treatment of requested debug-finalization failures.
|
||||
outcome. The report describes an actual published result. Item 5 requires
|
||||
requested capture to finalize successfully before a successful run report is
|
||||
submitted and defines how capture failures preserve publication facts.
|
||||
|
||||
Use one primary run-result report, with progress and diagnostics separate. Wire
|
||||
CommandKit's result destination to stdout and its diagnostic destination to
|
||||
@@ -641,18 +643,160 @@ retains the bookkeeping boundary; U9 still owns concrete durable record choices.
|
||||
|
||||
## 5. Debug Capture And Optional Diagnostics
|
||||
|
||||
**Discussion / decision:** Pending.
|
||||
**Discussion / decision:** Controls, native capture hooks, lifecycle, failure
|
||||
policy, and optional diagnostics agreed. Use `--debug` for native verbosity and
|
||||
conditional durable capture. The summary/trace distinction, concrete capture
|
||||
inventory, and bundle organization remain under discussion. Implementation is
|
||||
pending.
|
||||
|
||||
- How should artifact-capture controls fit alongside native quiet/verbose/debug
|
||||
reporting selections without requiring the existing `--debug` behavior?
|
||||
- How should requested debug artifact writes, failures, disclosure, and retention
|
||||
connect to command completion and native callbacks?
|
||||
- For selected preview, request-validation, or smoke capabilities, what native
|
||||
request builders and explicit execution choices should the application supply?
|
||||
### Shared Debug Control With Conditional Capture
|
||||
|
||||
Keep debug creation and physical storage in Notarius. Ordinary records exclude
|
||||
sensitive response/history content, and provider smoke diagnostics remain
|
||||
explicitly opt-in. Native inspection alone must not become a model call.
|
||||
Attach a native application boolean binding to CommandKit's shared `--debug`
|
||||
option on the pipeline run command. Capture is enabled when that boolean is true
|
||||
and a debug directory is defined by `--debug-dir PATH` or `debug.directory` in
|
||||
application configuration. The CLI directory overrides the configured directory.
|
||||
A directory alone does not enable capture. Do not add a separate
|
||||
`--capture-debug` flag.
|
||||
|
||||
Make the configured directory optional, removing the implicit
|
||||
`./notarius-debug` default. With no supplied destination, `--debug` supplies
|
||||
native diagnostics without creating files. Preserve normal CLI/configuration
|
||||
path resolution; detailed empty-value handling can be settled with the
|
||||
configuration migration.
|
||||
|
||||
Determine capture from the application boolean and destination, not from
|
||||
`Invocation.ReportingMode()`. A later quiet or verbose selection can change
|
||||
presentation without clearing the debug input. Explicit `--debug=false` follows
|
||||
native boolean binding semantics and disables capture.
|
||||
|
||||
| Invocation flags | Reporting selection | Capture |
|
||||
| --- | --- | --- |
|
||||
| `--debug`, no CLI/configured directory | Debug | None. |
|
||||
| `--debug`, with CLI/configured directory | Debug | Enabled. |
|
||||
| `--debug --quiet`, no CLI/configured directory | Quiet | None. |
|
||||
| `--debug --quiet`, with CLI/configured directory | Quiet | Enabled. |
|
||||
| `--quiet --debug`, with CLI/configured directory | Debug | Enabled. |
|
||||
|
||||
**Upstream ordering clarification:** CommandKit uses the last enabled reporting
|
||||
mode. Thus `--debug --quiet` suppresses command output while retaining capture;
|
||||
`--quiet --debug` selects debug verbosity. Preserve that native ordering rather
|
||||
than making quiet unconditional. Quiet still shows failure explanations, and
|
||||
capture does not bypass result suppression. On success the normal run report
|
||||
includes the bundle location; failure explanations include an allocated bundle
|
||||
path when available.
|
||||
|
||||
### Native Evidence And A Small Application Recorder
|
||||
|
||||
Use native facilities directly, attaching application step/lane/chunk/validator
|
||||
context where useful:
|
||||
|
||||
| Facility | Available evidence |
|
||||
| --- | --- |
|
||||
| `OnPrepared` | Prepared request details when execution reaches that point. |
|
||||
| `Observer` | Execution identities, attempt/retry/repair metadata, timing, usage completeness, and terminal metadata. |
|
||||
| Returned `RunResult` | Available model output and native validation result. |
|
||||
| `Exchange.Details()` | Explicit inspection of captured conversation history. |
|
||||
| Application decisions | Domain acceptance/rejection, fallback, validation completeness, and checkpoint reuse. |
|
||||
|
||||
Compose capture and accounting observers. Remove the debug completion wrapper,
|
||||
copied execution summaries, synthetic call identities, and duplicate terminal
|
||||
reporting. Keep callbacks lightweight and synchronize shared capture state:
|
||||
callbacks are synchronous, can overlap across operations, and `OnPrepared` runs
|
||||
while admission is held. Retain needed details and persist them at normal
|
||||
operation boundaries rather than buffering an entire pipeline or repeatedly
|
||||
writing every accumulated conversation prefix.
|
||||
|
||||
Capture only available evidence. Preparation/admission failures may have no
|
||||
`OnPrepared` notification; observer events have metadata, not response bodies;
|
||||
operational failures may return no `RunResult`. Do not present the bundle as a
|
||||
complete provider transcript. Use ordinary `Run` with native hooks, without
|
||||
forcing split preparation/execution solely for recording.
|
||||
|
||||
Conversation capture required for continuation remains in-process execution
|
||||
state. Persisting its history requires enabled debug capture. Normal manifests,
|
||||
checkpoints, and run reports retain their minimal projections independently of
|
||||
debug selection.
|
||||
|
||||
### Capture Lifecycle And Failure Policy
|
||||
|
||||
For enabled capture, allocate the bundle after application configuration loads
|
||||
and the destination and run identity are known, before engine construction and
|
||||
pipeline preflight. Argument and configuration-load failures use native
|
||||
diagnostics and may have no bundle. Loaders remain free of storage allocation.
|
||||
|
||||
One application-owned recorder records initial context, accepts native execution
|
||||
evidence and application decisions, and finalizes once with the known pipeline,
|
||||
publication, and capture outcomes. Debug remains separate from output/cache,
|
||||
never supplies checkpoint reuse, and is not automatically deleted.
|
||||
|
||||
Requested capture is required work:
|
||||
|
||||
- Allocation failure stops before expensive model work.
|
||||
- Recording failure is surfaced at the next normal application boundary.
|
||||
- Finalization failure is an operational command failure.
|
||||
- An existing pipeline failure remains primary; capture failures are additional
|
||||
information and preserve the underlying causes.
|
||||
- Already-published artifacts and already-written diagnostic data remain
|
||||
available. Capture failure does not cause model retries.
|
||||
|
||||
Observer callbacks cannot return errors. Retain a recording error for application
|
||||
code to check at an ordinary boundary rather than adding an exception or retry
|
||||
framework. Submit the successful run report only after publication and requested
|
||||
capture finalization succeed. If finalization fails after publication, identify
|
||||
the published output and available debug location in the native failure
|
||||
explanation; preserve the successful publication as a fact.
|
||||
|
||||
### Disclosure And Retention
|
||||
|
||||
Retain the separate debug root, restrictive debug permissions, and explicit
|
||||
operator retention policy. Captured content may include source material, rendered
|
||||
requests, generated output, and correction history. Credential redaction does not
|
||||
make that material safe to share. Do not record credential values, authorization
|
||||
headers, or environment dumps. Use upstream safe inspection where available and
|
||||
deliberate content selection elsewhere. No summary or directory label should
|
||||
imply that arbitrary diagnostic content is non-sensitive.
|
||||
|
||||
### Native Optional Diagnostics
|
||||
|
||||
Use the selected standard commands through the shared loaders and native inputs:
|
||||
|
||||
| Command family | Purpose |
|
||||
| --- | --- |
|
||||
| `config …`, `promptkit list/show …` | Inspect configuration and stored resources. |
|
||||
| `pipeline validate` | Check application composition and known dependencies offline. |
|
||||
| `promptkit requirements show` | Discover request requirements. |
|
||||
| `promptkit requirements validate`, `promptkit show prepared` | Validate or preview a concrete request without a model call. |
|
||||
| `promptkit test …` | Explicit provider diagnostics. |
|
||||
|
||||
Application request builders are added only for actual application needs.
|
||||
Inspection/diagnostic commands use their native reports and do not automatically
|
||||
allocate pipeline debug bundles; operators can redirect reports for retention.
|
||||
Neither debug verbosity nor enabled capture initiates additional provider probes.
|
||||
Smoke reports retain native support, applicability, and result distinctions;
|
||||
status zero alone does not establish that a check passed.
|
||||
|
||||
### Open Discussion: Summary And Trace
|
||||
|
||||
The current `summary/` and `trace/` layout is not selected as the new design.
|
||||
Discuss what diagnostic questions the bundle should answer, which application
|
||||
records and native evidence it needs, and how to avoid duplicate snapshots or
|
||||
execution accounting. Keep these choices open:
|
||||
|
||||
- Whether a separate summary is useful, and whether it is a compact index/final
|
||||
outcome rather than another copy of execution records.
|
||||
- Whether summary/trace are useful conceptual categories or directory boundaries.
|
||||
- Which prepared details, outputs, conversation snapshots, and application
|
||||
decisions to persist, and at what granularity.
|
||||
- How self-contained a failed-run bundle should be, given that it may have no
|
||||
published output bundle, and when to reference existing artifacts instead of
|
||||
copying them.
|
||||
|
||||
Evidence: [native shared debug binding](../../../commandkit/docs/consumers/guide.md#invocation-reporting),
|
||||
[reporting-mode ordering](../../../commandkit/docs/cli.md#modes-and-visibility),
|
||||
[PromptKit observations](../../../promptkit/docs/consumers/pkg-promptkit.md#observe-an-ordinary-run),
|
||||
and [exchange inspection](../../../promptkit/docs/consumers/pkg-promptkit.md#capture-and-continue-a-conversation).
|
||||
The current [debug lifecycle](../internal/state.md#debug-bundles) and
|
||||
[operator contract](../operations.md#debug-bundles) provide migration context.
|
||||
|
||||
## 6. Verification, Migration, And Roadmap Incorporation
|
||||
|
||||
|
||||
Reference in New Issue
Block a user