From 3fb8cb73929c22e07c8ecd302a9bb19fb5e1733c Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Thu, 19 Nov 2026 14:59:56 +0000 Subject: [PATCH] Record inspection and execution bookkeeping ownership --- .../promptkit-architecture-discussion.md | 136 +++++++++++++++--- 1 file changed, 116 insertions(+), 20 deletions(-) diff --git a/docs/roadmap/promptkit-architecture-discussion.md b/docs/roadmap/promptkit-architecture-discussion.md index 9109b390..c245a819 100644 --- a/docs/roadmap/promptkit-architecture-discussion.md +++ b/docs/roadmap/promptkit-architecture-discussion.md @@ -5,11 +5,9 @@ Working notepad for reviewing the ownership boundaries in the [PromptKit v0.23.2 adoption roadmap](promptkit.md). Work through the topics below in order, recording questions, evidence, decisions, and follow-up work. -Items 1–5 record selected architectural directions. Detailed design, -read-only filesystem and schema decisions, and the remaining topics are still -under discussion. -Item 7 records the completed Audita comparison and its implications for the -remaining decisions. +Items 1–6 record selected architectural directions. Detailed design and the +read-only filesystem and schema decisions remain open. Item 7 records the +completed Audita comparison and its implications for these decisions. The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit capabilities as fully as practical. Application ownership of a policy does not @@ -445,22 +443,119 @@ unusable or unaccepted stored artifacts. Continue with item 6. ## 6. Inspection And Execution Bookkeeping -Extend the roadmap's use of native inspection, `Run`, `OnPrepared`, observations, -and `ExecutionGroup` to reconsider surrounding wrappers and copied report types. -Notarius retains stage correlation, artifact provenance, storage, and disclosure -policy; CommandKit is the planned owner of standardized operator presentation. +**Discussion / decision:** Direction agreed: use native PromptKit inspection +and execution facts, with CommandKit owning standardized operator presentation. +Remove duplicate LLM reporting and accounting machinery. Notarius retains +application context, domain decisions, durable artifacts, and their provenance. +No application changes have been implemented. -Questions: +### Ownership -- Which reports and accounting mechanisms can use upstream results directly? -- Which application projections remain necessary for durable contracts? -- Can the PromptKit migration avoid building temporary machinery that the - CommandKit adoption would immediately replace? +| Responsibility | Owner | +| --- | --- | +| Configuration provenance, resource inspection, and request preparation | ConfigKit and PromptKit through their native APIs. | +| Standard configuration/prompt commands, preview and smoke commands, report formatting and visibility | CommandKit and its ckconfig/ckprompt integrations. | +| Provider attempts, transport retries, structural repairs, execution failures, and token observations | PromptKit observations and results; `ExecutionGroup` for selected execution totals. | +| Step/lane/chunk/validator attribution and pipeline progress meaning | Notarius. | +| Acceptance, rejection, fallback, checkpoint reuse, and pipeline outcome | Notarius. | +| Artifact storage, durable provenance, receipts, and optional debug storage/disclosure | Notarius. | -**Discussion / decision:** Pending. +### Native Inspection And Operator Presentation -**Follow-up:** Coordinate this topic with item 1 and the roadmap's CommandKit -follow-up so report and adapter decisions are made once. +Remove the profile-inspector wrapper, copied effective-model-parameter types, +and local reconstruction of upstream configuration. Use native inspection for +configured resources and native prepared/result data for the target actually +executed, including request overrides. Profile inspection can describe a partial +configured profile; it is not evidence of the completed execution target. + +Adopt ckconfig and ckprompt for standard inspection, requirements, validation, +preview, and opt-in smoke commands. Notarius supplies configuration/engine +loaders and application request builders. Avoid building temporary custom +commands or report machinery that the planned CommandKit adoption would replace. + +Retain application pipeline preflight: resolve modules, artifact compatibility, +references, and known prompt/profile selections. Use native definition checks +without simulating stage execution or requiring provider credentials. Concrete +request preparation remains the place to check the completed target, inputs, +schema, and credential requirements. + +CommandKit supplies formatting, visibility, progress presentation, and delivery +error handling. Notarius supplies report content, domain outcomes, and exit-status +mapping. Preserve deliberate machine-output contracts rather than treating every +receipt as a generic formatted report. + +### Native Execution Facts + +- Use ordinary `Run`, with `OnPrepared` when actual prepared details are needed + for provenance or optional debug capture. Remove unconditional + `PrepareExecution`/`RunPrepared` use solely to obtain those details. +- Use native observations for terminal execution facts, including failures that + return no `RunResult`. `OnPrepared` is conditional and is not delivered after + preparation or admission failure; it cannot serve as a completion record. +- Use native `RunID`/`ParentRunID` with application scope to correlate executions. + Remove the private debug completion wrapper and duplicate synthetic call-ID + machinery where native identity and application attribution suffice. +- Remove parallel `LLMExecutionSummary`, token-usage types, enum conversions, + cloning, and independent attempt/usage accumulators. Use native types internally + and project selected fields only at an application reporting or durable boundary. +- Replace the adapter's `LLMProfileRecorder` and provider-forwarding plumbing with + the application projection needed for actual-run provenance. Do not substitute + a catalog inventory for the profiles and targets actually used. +- Use `ExecutionGroup` for totals over selected executions. Explicitly enroll the + initial run, continuation runs, and nested validator runs as applicable. + Preserve terminal-coverage and token-usage completeness; unknown usage is not + zero. Do not count both attempt observations and terminal aggregates, or add + overlapping groups as though they were disjoint work. + +Application execution code should attach scope and observers, run and assess the +native result, and record its disposition and execution references. This does +not require a replacement general-purpose LLM interface. A successful native +execution can still produce a domain-rejected candidate; provider success and +application acceptance remain separate facts. + +### Application Records And Durable Boundaries + +Keep the facts needed to explain a pipeline: which stage/lane/chunk produced an +artifact, which validators assessed it, whether continuation or a safe fallback +was used, and whether the artifact was generated now or reused. PromptKit cannot +infer that meaning from provider execution alone. + +Under item 5, a reused artifact retains its original execution and acceptance +provenance. The current invocation records the reuse decision without counting +the earlier execution's token usage as current work or claiming new validation. + +Retain small, deliberate application projections for the +[published manifest](../integrations/json-output.md) and +[run-result receipt](../integrations/run-result.md), with explicit compatibility +decisions for changed fields. Durable contracts should not become raw upstream +type dumps or require parallel runtime representations of every native result. + +Keep application debug artifact placement, opt-in capture, retention, and +disclosure handling. Native callbacks supply execution data; they do not own +Notarius's storage policy. Detailed prepared messages belong in the selected +debug capture path, while ordinary provenance can retain only the needed fields. + +Evidence: [current inspection wrapper](../../internal/framework/llm/promptkit_inspection.go), +[adapter bookkeeping](../../internal/framework/llm/promptkit_client.go), +[debug completion wrapper](../../internal/framework/pipeline/debug.go), +[native PromptKit APIs](../../../promptkit/docs/consumers/pkg-promptkit.md), +[ckconfig integration](../../../commandkit/integrations/ckconfig/README.md), +[ckprompt integration](../../../commandkit/integrations/ckprompt/README.md), and +[CommandKit reporting](../../../commandkit/docs/consumers/guide.md). + +### Remaining Implementation Work + +Select the minimal application scope and durable fields, observer/group +enrollment, and compatibility treatment for existing manifests, receipts, and +debug artifacts. Coordinate command migration with the roadmap's CommandKit +follow-up. The core ownership change needs no new PromptKit capability; any +remaining upstream request should identify a concrete unmet requirement. + +**Follow-up:** Reconcile the adoption roadmap with items 1–6 and record the +significant boundary changes in successor ADRs. Update current-behavior +documentation when implemented. Verify failure reporting without a result, +continuation/validator accounting, unknown usage, actual-target provenance, +checkpoint reuse accounting, and optional debug capture during implementation. ## 7. Compare Audita's PromptKit Workflow @@ -619,9 +714,10 @@ flow and the remaining repair-prompt, response-contract, and policy details; the comparison alone did not settle those details. Item 4 now records agreement to use native admission with bounded application -workers. Item 6 can draw on Audita's observations and CommandKit integration. -Item 5 now selects simple application-owned recovery matching and removes the -upstream execution-fingerprint prerequisite. +workers. Item 5 selects simple application-owned recovery matching and removes +the upstream execution-fingerprint prerequisite. Item 6 adopts native execution +observations and CommandKit presentation, retaining application attribution, +domain outcomes, and durable provenance. The two requested capabilities already work with v0.23.2; no new PromptKit feature is needed merely to provide replaceable defaults or domain-directed continuation. Any remaining upstream request should identify the specific unmet