From d1d23947291daff7a522fb5075aa69dda8f020f5 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 18 Nov 2026 15:28:38 +0000 Subject: [PATCH] Define the offline preflight boundary --- docs/roadmap/promptkit.md | 52 ++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 25 deletions(-) diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md index 3f2d6596..a60c6e3e 100644 --- a/docs/roadmap/promptkit.md +++ b/docs/roadmap/promptkit.md @@ -133,13 +133,6 @@ the existing operator disclosure boundary, including redaction of credential-shaped provider parameters. Do not recreate backend default resolution to produce the former `effective_model_params` view. -Distinguish configuration and resource preflight from request readiness. -Use `InspectRequirements` to explain recognized dependencies when helpful, -not as proof that a request is executable. Use `Prepare` or -`ValidateRequest` when concrete request inputs are available and their -validation is actually needed. A routine config-only check should remain -credential-free and offline, without synthetic requests. - Retain a private, versioned Notarius checkpoint identity based on the engine's captured state. Use native resource enumeration and prompt hashes, keeping the effective execution inputs needed to distinguish reuse safely. A diff of @@ -148,6 +141,27 @@ Version the fingerprint for the cutover so obsolete identity inputs cannot cause a false hit; do not reread mutable sources or expose the digest inputs. Keep run manifests based on the profiles actually used, not the inventory. +### Offline Preflight + +Keep `config validate` focused on validating the resolved Notarius application +configuration. With `--pipeline`, also resolve the selected stages and lanes +against the module catalog. For an LLM-backed selection, load the same +effective PromptKit definitions used by `run`, even when the pipeline does +not explicitly select a profile. Let PromptKit validate and capture those +definitions, including companion sources and referenced message files. +Check the prompt and profile IDs known from pipeline resolution, including a +prompt's default profile when the prompt is known and its default is the +effective selection. Do not simulate stages to discover runtime selections. + +Preflight requires no transcript, credential values, or provider calls. Its +success confirms configuration and known resource selection, not that every +concrete request is runnable. A valid PromptKit profile can be partial. Let +PromptKit check the completed target, required credentials, inputs, and schema +during request preparation. Use `InspectRequirements` to explain recognized +dependencies when useful; use `Prepare` or `ValidateRequest` for an actual +request when no-generation validation is needed. Do not construct synthetic +requests or reimplement target resolution for offline preflight. + ### Native Execution And Retained Pipeline Policy Use ordinary `Engine.Run` for structured completion and use `OnPrepared` @@ -201,9 +215,12 @@ focused behavior tests for the new format and intentional rejection of the old one, main-relative companion discovery, and native relative imports. Verify that a companion can replace a qualified embedded prompt or profile without changing unrelated defaults, and that preflight, inspection, and -runtime use the same effective definitions. Exercise preparation, -generation, structural repair, and semantic retries through the maintained -fake-provider integration. Check that real message or target changes affect +runtime use the same effective definitions. Verify that pipeline preflight +catches invalid effective definitions and unknown selected IDs without +requiring inputs or credentials, and does not reject a profile merely for +being partial. Exercise preparation, generation, structural repair, and +semantic retries through the maintained fake-provider integration. Check +that real message or target changes affect checkpoint reuse, and that session routing, cache controls, diagnostics, and durable outputs keep their existing meaning. Live provider checks are useful for deployment confidence, not a requirement of the offline suite. @@ -215,21 +232,6 @@ for deployment confidence, not a requirement of the offline suite. ## Open Decisions -### How Far Offline Preflight Goes - -**Recommendation:** Check that the captured configuration is valid and that -selected resource IDs resolve. Make clear that a valid profile can be partial -and that concrete request readiness is checked during preparation. This -preserves a useful offline preflight without duplicating target resolution or -requiring credentials or source data. - -**Alternative:** Preserve offline checking of completed targets by pursuing a -dedicated upstream API that resolves targets without artifacts or credentials. -That would catch more missing endpoint/model settings before a run and could -serve other consumers, but it would add an upstream dependency to this work or -require a subsequent release. Reimplementing completion rules in Notarius -would risk a discrepancy with actual execution. - ### Operator Diagnostics In This Upgrade **Recommendation:** Migrate existing inspection commands and use native