Record native continuation direction and retry simplifications
This commit is contained in:
@@ -5,7 +5,7 @@
|
||||
Working notepad for reviewing the ownership boundaries in the
|
||||
[PromptKit v0.23.2 adoption roadmap](promptkit.md). Work through the topics
|
||||
below in order, recording questions, evidence, decisions, and follow-up work.
|
||||
Items 1 and 2 record selected architectural directions. Detailed design,
|
||||
Items 1–3 record selected architectural directions. Detailed design,
|
||||
read-only filesystem and schema decisions, and the remaining topics are still
|
||||
under discussion.
|
||||
Item 7 records the completed Audita comparison and its implications for the
|
||||
@@ -141,29 +141,120 @@ implementation. The agreed direction has not yet been implemented.
|
||||
|
||||
## 3. Retry And Semantic Correction Ownership
|
||||
|
||||
Consider narrowing outer stage retries once PromptKit has exhausted its own
|
||||
transport and structural-repair budgets. Notarius would retain application
|
||||
assessment, semantic correction, fallback, and acceptance decisions.
|
||||
**Discussion / decision:** Direction selected: adopt Audita's native continuation
|
||||
approach. PromptKit owns captured exchanges, accumulated conversation history,
|
||||
continuation execution, and turn-budget enforcement. Notarius supplies semantic
|
||||
assessment, corrective findings, and acceptance or fallback decisions. Use native
|
||||
PromptKit types directly, consistent with item 1; Audita's adapter and opaque
|
||||
exchange wrapper are not part of the selected direction.
|
||||
|
||||
Questions:
|
||||
The following is a proposed application of that direction. Correction scope,
|
||||
response contracts, budget migration, and terminal policy details still need
|
||||
agreement.
|
||||
|
||||
- Which operational failures justify repeating an entire stage?
|
||||
- Should semantic correction preserve only the latest response or accumulate
|
||||
conversation history?
|
||||
- How should retries for non-LLM operations differ from LLM execution retries?
|
||||
- Would a native orchestration API reduce machinery without obscuring typed
|
||||
validation and terminal disposition?
|
||||
### Proposed Flow
|
||||
|
||||
**Discussion / decision:** Pending.
|
||||
1. Prepare the source context, references, and any request-local candidate-handle
|
||||
mapping once for the producer operation. Run its initial native request with
|
||||
exchange capture when continuation is enabled. Capture is independent of
|
||||
detailed debug recording.
|
||||
2. Inspect native structural validation, decode the response, and perform the
|
||||
module's deterministic interpretation and safety checks. Run the configured
|
||||
validation chain on a materialized candidate. A module can identify actionable
|
||||
defects before a candidate is ready for that chain.
|
||||
3. If correction is needed, call `Engine.Coordinate` with the initial exchange
|
||||
and a configured `AdditionalTurns` budget. The callback selects a native
|
||||
repair prompt and supplies actionable findings as variables. PromptKit retains
|
||||
the original exchange and each subsequent repair turn.
|
||||
4. Assess every delivered response, including the last permitted turn. Rebuild
|
||||
the typed candidate from the fixed operation inputs and the revised response;
|
||||
do not repeatedly apply revisions to an already modified artifact. A changed
|
||||
candidate receives the complete configured validation chain again, including
|
||||
validators that approved an earlier version.
|
||||
5. Stop on acceptance or a terminal condition. After exhaustion, Notarius decides
|
||||
whether to reject, fail, or evaluate an explicitly supported safe fallback.
|
||||
Native `stopped` and `exhausted` outcomes describe coordination, not acceptance.
|
||||
Cancellation never turns an undelivered response into an accepted candidate.
|
||||
|
||||
**Follow-up:** Evaluate behavior changes against
|
||||
Start with one conversation per producer operation: a chunk-plan generation,
|
||||
one lane's extraction of one chunk, or one lane's normalization pass. This aligns
|
||||
with Notarius's whole-output validation boundary. Audita's per-proposal branches
|
||||
are useful evidence, but per-entity repair would require its own combination and
|
||||
validation design. It is not necessary to adopt native continuation.
|
||||
|
||||
Repair prompts should supply a new user task, with enduring principles in the
|
||||
original system instructions. Avoid replaying the initial task or embedding
|
||||
initial-response-only constraints in system messages. A complete replacement
|
||||
model response is a simple starting point; for normalization that can remain a
|
||||
replacement reconciliation plan, rather than a model-generated durable artifact.
|
||||
A different repair schema or profile is supported but not yet selected. Keep
|
||||
source context and candidate-handle meanings stable throughout the conversation.
|
||||
Accumulated history increases context and token use across turns; the semantic
|
||||
turn budget should reflect that tradeoff.
|
||||
|
||||
### Machinery To Remove Or Adapt
|
||||
|
||||
| Current machinery | Proposed treatment |
|
||||
| --- | --- |
|
||||
| `CorrectionProtocolSingleResponseV1`, raw-response `ModelCandidate`, and `SemanticCorrection` | Retire the latest-response-only transport protocol and its copy/validation plumbing. Keep the typed domain candidate and use native `RunResult` / `Exchange` for continuation state. |
|
||||
| Adapter code that rebuilds the original request and appends an assistant/user pair | Remove with the adapter. Native continuation assembles history; native repair definitions render findings. |
|
||||
| `runProducerAttempts` combining operational, structural, module-requested, and validator-semantic retries | Replace its LLM retry loop with native coordination plus application assessment and terminal disposition. Retain useful stage provenance, not a second turn counter or general retry engine. |
|
||||
| Outer LLM retries after transport or structural repair exhaustion | Prefer settling the failure after PromptKit's budgets are exhausted. A semantic turn addresses actionable domain defects; it should not silently restart a failed provider execution. Any additional recovery policy needs a concrete application reason. |
|
||||
| `executeValidationChain`'s repeated LLM validator invocations | Keep the ordered chain and result aggregation; move transport and response-shape recovery into each validator's native execution. A validator execution failure remains distinct from a rejected producer candidate. |
|
||||
| `NormalizeRetry` and `producerRetryDirective` | Replace requests to rerun the whole module with assessment findings and explicit fallback information. Preserve the domain meaning of a safe partial result. |
|
||||
| `CorrectionRequest`'s hard-coded retry-message construction | Keep useful ordered collection/deduplication of actionable findings, and render the repair assignment through a native prompt. Operator messages and reason codes remain diagnostic data, not automatic model instructions. |
|
||||
| Correction capability declarations and preparation checks | Retire checks tied to `single_response_v1`; prepare the actual native initial/repair definitions and response requirements. Deterministic operations need no synthetic exchange. |
|
||||
|
||||
Native transient retries, structural repair, and semantic additional turns remain
|
||||
distinct controls, but Notarius need not implement all three loops. Existing
|
||||
producer and validator `retries` settings cannot be mechanically renamed: their
|
||||
current budgets cover different behavior. Non-LLM failure handling should be
|
||||
reviewed separately rather than routed through conversation coordination.
|
||||
|
||||
### Validation And Safe Fallback Still Have Application Meaning
|
||||
|
||||
Keep source grounding, reference resolution, domain invariants, reconciliation
|
||||
plan checks, typed application rules, and candidate isolation. In particular,
|
||||
normalization must still reject unknown or overlapping candidate handles and
|
||||
apply only safe groups. A final fallback can retain safe merges while leaving
|
||||
unsafe groups separate; it must go through configured artifact validation and
|
||||
terminal policy before it can advance. After native structural-repair exhaustion,
|
||||
a supported deterministic fallback can be assessed directly instead of first
|
||||
restarting the same model operation through a feedback-free module retry. Which earlier safe
|
||||
candidate, if any, may survive a later failed turn remains an explicit policy
|
||||
question; do not introduce implicit best-so-far acceptance.
|
||||
|
||||
Retain the distinction between a rejection and unavailable validation. Under the
|
||||
current `warn_continue` policy, incomplete validation can advance a structurally
|
||||
valid candidate but prevents checkpoint reuse/publication, including for derived
|
||||
results. Continuation adoption does not itself change that policy, safe-fallback
|
||||
eligibility, or durable provenance. Keep raw conversation state in process and
|
||||
out of ordinary manifests/checkpoints; explicit detailed debug remains separate.
|
||||
|
||||
PromptKit should own validation and repair of the private model response.
|
||||
Notarius's serialized schema validator currently receives the codec-produced
|
||||
artifact and its artifact schema, which can differ from the model response.
|
||||
Do not remove that boundary merely because both use JSON Schema. Audit actual
|
||||
duplicate response checks individually; artifact validation and Go/domain
|
||||
interpretation are not automatically redundant. Item 2's filesystem and schema
|
||||
assembly decisions remain open.
|
||||
|
||||
Evidence: [producer attempt loop](../../internal/framework/pipeline/producer_attempts.go),
|
||||
[validation executor](../../internal/framework/pipeline/validation_executor.go),
|
||||
[typed artifact validation](../../internal/framework/pipeline/runner_typed.go),
|
||||
[NPC normalizer](../../internal/modules/dnd/normalize/npcregistry/normalizer.go),
|
||||
[semantic reconciliation](../../internal/framework/semanticreconcile/engine.go),
|
||||
and the native continuation evidence in item 7.
|
||||
|
||||
**Follow-up:** Agree correction scope and repair response contracts; settle budget
|
||||
migration and terminal/fallback behavior; then reconcile the adoption roadmap.
|
||||
Record the changed architecture in a successor to
|
||||
[ADR-0014](../adr/0014-feedback-aware-validation-retries.md) and
|
||||
[ADR-0016](../adr/0016-feedback-aware-module-requested-retries.md).
|
||||
PromptKit v0.23.2's `Coordinate` continues an exchange; it is not a direct
|
||||
replacement for the existing latest-response-only policy or full stage loop.
|
||||
The Audita review in item 7 demonstrates application-owned validation inside
|
||||
native continuation coordination. Evaluate adopting that model on its merits;
|
||||
the existing Notarius history policy is subject to reconsideration under item 1.
|
||||
Migration checks should cover accumulated history, fixed handle mappings,
|
||||
revalidation, final-turn acceptance, native budget exhaustion, safe fallback,
|
||||
validator failure, cancellation, and checkpoint eligibility. Replace tests of
|
||||
the retired message-reconstruction protocol rather than preserving its mechanics.
|
||||
|
||||
## 4. Global Scheduling And Admission
|
||||
|
||||
@@ -373,11 +464,11 @@ Item 2's native underlay direction has a working application precedent. Audita
|
||||
also offers evidence for ordinary embedded filesystems and per-request schema
|
||||
selection, while our filesystem and schema decisions remain open.
|
||||
|
||||
For item 3, native continuation is a practical alternative to the current
|
||||
latest-response-only protocol. Keeping domain validation local does not require
|
||||
keeping conversation assembly or the turn loop local. Decide whether Notarius
|
||||
should adopt accumulated branch history, separate repair prompts, and revised
|
||||
response contracts; none has been selected merely by this comparison.
|
||||
The subsequent item 3 discussion selects native continuation and accumulated
|
||||
conversation history. Keeping domain validation local does not require keeping
|
||||
conversation assembly or the turn loop local. Item 3 owns the proposed Notarius
|
||||
flow and the remaining repair-prompt, response-contract, and policy decisions;
|
||||
the comparison alone did not settle those details.
|
||||
|
||||
Items 4 and 6 can draw on Audita's native admission, observations, and CommandKit
|
||||
integration. Item 5 remains independent upstream/application identity work.
|
||||
|
||||
Reference in New Issue
Block a user