From 3e1abc7fbbb5fea1a457e62c70584fde041d96ab Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Thu, 19 Nov 2026 14:49:36 +0000 Subject: [PATCH] Record simpler checkpoint reuse policy --- .../promptkit-architecture-discussion.md | 137 ++++++++++++++---- 1 file changed, 108 insertions(+), 29 deletions(-) diff --git a/docs/roadmap/promptkit-architecture-discussion.md b/docs/roadmap/promptkit-architecture-discussion.md index ede70fa8..9109b390 100644 --- a/docs/roadmap/promptkit-architecture-discussion.md +++ b/docs/roadmap/promptkit-architecture-discussion.md @@ -5,7 +5,7 @@ Working notepad for reviewing the ownership boundaries in the [PromptKit v0.23.2 adoption roadmap](promptkit.md). Work through the topics below in order, recording questions, evidence, decisions, and follow-up work. -Items 1–4 record selected architectural directions. Detailed design, +Items 1–5 record selected architectural directions. Detailed design, read-only filesystem and schema decisions, and the remaining topics are still under discussion. Item 7 records the completed Audita comparison and its implications for the @@ -19,9 +19,9 @@ features where PromptKit does not yet meet an application need. No application changes have been implemented through this notepad. The adoption roadmap still needs reconciliation with the selected direction, including its -adapter-preservation, latest-response-only, and cross-backend scheduler -preservation language. Significant changes to accepted architecture need -successor ADRs under the +adapter-preservation, latest-response-only, cross-backend scheduler preservation, +and broad checkpoint-fingerprint language. Significant changes to accepted +architecture need successor ADRs under the [documentation policy](../policy/documentation.md#architecture-decision-records). Current boundaries are described in [Architecture](../policy/architecture.md) and [LLM Runtime Internals](../internal/llm.md). @@ -102,8 +102,9 @@ Candidates for removal: individual modules; schema responsibilities remain under review. - `ModulePromptFS`, `PromptAssetManifest`, shared-name lookup tables, synthetic `sharedassets` mounts. -- Manual prompt-file enumeration for fingerprints, replaced by captured native - prompt identities subject to the broader identity design in item 5. +- Manual prompt-file enumeration for checkpoint fingerprints, retired under + item 5's simpler reuse policy. Native prompt hashes remain available for + inspection and diagnostics. Keep prompt prose, ordered messages, cache controls, workload profiles, private response schema documents, and domain interpretation. Durable artifact schemas @@ -123,22 +124,24 @@ Deferred decisions: - Schema assembly, root/path selection, wrapper removal, and schema metadata. Supplying the embedded filesystem as a native schema root remains a candidate, not a selected design. -- How schema identity contributes to manifests and checkpoint reuse. +- Schema metadata for manifests and artifact compatibility. Item 5 settles that + checkpoint matching does not require exact schema identity. Implementation questions: - Can native sources represent the existing shared fragments and module assets? - Which registry, remounting, and hashing helpers could disappear? -- What private schema identity is still needed for checkpoint reuse? +- What schema metadata remains useful for inspection and artifact loading under + item 5's simpler reuse policy? - How do we preserve message ordering and cache controls while simplifying assembly? **Follow-up:** Continue with item 3, then revisit the deferred filesystem and schema decisions with any new clarity. Confirm native source contexts for shared -fragments and operator replacements. Resolve metadata and reuse identity with -items 5–6. Verify the resulting definitions through native preparation and -fake-provider execution; filesystem remounting tests can retire with the removed -implementation. The agreed direction has not yet been implemented. +fragments and operator replacements. Apply item 5's reuse policy and resolve +remaining metadata with item 6. Verify the resulting definitions through native +preparation and fake-provider execution; filesystem remounting tests can retire +with the removed implementation. The agreed direction has not yet been implemented. ## 3. Retry And Semantic Correction Ownership @@ -260,8 +263,8 @@ and the native continuation evidence in item 7. candidate may survive a later failed turn. - Failure handling for non-LLM operations and any justified recovery beyond native execution budgets. -- Item 2's deferred filesystem and schema assembly choices, and items 5–6's - execution identity and durable bookkeeping design. +- Item 2's deferred filesystem and schema assembly choices, item 5's remaining + checkpoint implementation details, and item 6's durable bookkeeping design. **Follow-up:** Resolve the remaining details above, then reconcile the adoption roadmap. Record the changed architecture in a successor to @@ -346,24 +349,99 @@ with item 5's execution identity and checkpoint reuse discussion. ## 5. Execution Identity And Checkpoint Reuse -Consider moving identity for PromptKit-owned execution semantics upstream. -Notarius would combine opaque upstream identities with its source, module, -reference, schema, and policy identities and retain checkpoint eligibility and -storage decisions. +**Discussion / decision:** Direction agreed: resuming preserves successful stage +results for the same logical work and data; explicit recomputation applies +changed execution behavior. Use a simple Notarius-owned matching policy instead +of comprehensive execution fingerprints. No application changes have been +implemented. -Questions: +### Purpose And Matching Rule -- Which identity can be computed before a stage runs, and which needs a concrete - prepared request? -- Which changes should invalidate reuse, and at what granularity? -- Should a conservative application fingerprint be an interim implementation - rather than the architectural destination? -- Does this justify upstream work and a later adoption target? +Checkpoints preserve a successful producer's result when a dependent stage fails. +After fixing the failed stage, an operator should be able to retry it without +paying to rerun the successful producer. Matching every configuration parameter +or implementation revision would undermine that recovery purpose. -**Discussion / decision:** Pending. +The agreed conceptual key is: -**Follow-up:** Review PromptKit's deferred semantic execution-target fingerprint -proposal. This capability is not supplied by the v0.23.2 adoption target. +```text +logical stage identity + input-data identity + prompt ID, where applicable +``` + +Scope the checkpoint to the relevant step/lane/stage and artifact kind so that +shared prompt IDs do not mix unrelated results. These are logical identifiers, +not hashes of implementations or complete configuration. Deterministic stages +need no synthetic prompt identity. + +Input data includes the source or chunk being processed, reference material, +and upstream artifacts actually consumed. Matching only the original source +would miss changes to generated dependencies. Content digests can compactly +identify these inputs and check stored payload integrity without requiring an +exact fingerprint of the execution environment. + +| Change | Reuse behavior | +| --- | --- | +| Fix the implementation of a later failed stage | Preserve earlier successful stages with matching work and inputs. | +| Edit prompt text while retaining its ID | Reuse; explicit recomputation applies the edit. | +| Change model, generation settings, or implementation/validation policy | Reuse otherwise compatible completed work; explicit recomputation applies the change. | +| Change the prompt ID or data consumed by the stage | No matching checkpoint under this rule. | +| Explicitly request recomputation | Recompute the selected work and its dependents. | + +The existing controls are `--resume` and `--recompute-step`, documented in the +[CLI reference](../cli.md#run) and +[selective recompute workflow](../operations.md#selective-recompute). Their +current strict compatibility checks still need migration to this agreed policy. +The accepted tradeoff is that a checkpoint may preserve a successful result +produced by code, prompts, or settings that have since improved. + +### Matching, Loading, And Provenance + +A matching key selects a checkpoint; it does not make an unreadable or +incompatible artifact usable. Retain completion/acceptance status, integrity +checks, and the ability to decode and consume the stored artifact. Exact schema +hash equality is not required when the artifact remains compatible. + +Only accepted, completely validated results are reusable. Rejected, structurally +invalid, or validation-incomplete results and their ineligible derivatives +remain outside reusable state. Acceptance on resume refers to the earlier +execution: do not automatically rerun semantic validators merely because their +configuration changed, or claim the artifact passed the new validation policy. +Preserve original provenance; explicit recomputation applies changed behavior. + +Useful execution metadata can remain available for explanation without becoming +a matching requirement. Neither complete conversation history nor native +prepared-execution handles are needed to locate and reuse a completed artifact. +The separate source-addressed canonical chunk-plan policy remains distinct. + +### Simplification And Ownership + +- Remove the broad PromptKit configuration fingerprint from checkpoint matching, + including effective-configuration parsing and whole-catalog enumeration. +- Remove library/catalog versions, model settings, prompt-content hashes, and + component-policy fingerprints as automatic invalidators under this policy. +- Keep straightforward input/dependency identity, successful-stage records, + artifact loading checks, storage, and explicit recomputation in Notarius. +- Keep PromptKit hashes available for native inspection and diagnostics without + making them checkpoint prerequisites. + +Comprehensive upstream execution identity is no longer required for this +design. It does not require a later PromptKit release or a temporary conservative +configuration fingerprint. Item 5 supersedes the earlier proposal to make an +upstream semantic-equality feature part of the migration's checkpoint design. + +### Remaining Implementation Work + +Define the concrete logical stage/prompt identifiers and consumed-input +representation. Revise checkpoint lookup and storage scoping so that an +invocation-wide pipeline/configuration digest cannot still prevent reuse of +successful predecessors. Decide how existing checkpoint records transition to +the simpler format and how incompatible payloads are reported. + +**Follow-up:** Reconcile the adoption roadmap and record the changed reuse +contract in an ADR. Update current-behavior documentation when implemented. +Verify recovery after a downstream fix, reuse across prompt/settings changes, +misses after consumed-data changes, explicit recomputation, and rejection of +unusable or unaccepted stored artifacts. Continue with item 6. ## 6. Inspection And Execution Bookkeeping @@ -542,7 +620,8 @@ the comparison alone did not settle those details. Item 4 now records agreement to use native admission with bounded application workers. Item 6 can draw on Audita's observations and CommandKit integration. -Item 5 remains independent upstream/application identity work. +Item 5 now selects simple application-owned recovery matching and removes the +upstream execution-fingerprint prerequisite. The two requested capabilities already work with v0.23.2; no new PromptKit feature is needed merely to provide replaceable defaults or domain-directed continuation. Any remaining upstream request should identify the specific unmet