diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md index 1f767ef9..87112128 100644 --- a/docs/roadmap/remediation.md +++ b/docs/roadmap/remediation.md @@ -6,12 +6,18 @@ Review date: 2026-11-22. Reviewed commit: `75444b8ef300ca8efee57d7a2271d21ce258b49c`. +Integration follow-up baseline: `fd134e85e54c6e0f74aab5c77df5bada7c1b1e6a`. +The follow-up baseline adds only the initial review findings; application code +is unchanged between these revisions. + This review followed the 46 stages in [the implementation plan](implementation.md) against the [native migration target](promptkit.md), current owning code, focused tests, maintained examples, and canonical documentation. The migration -is substantially complete. The required follow-up is limited to the two findings -below; neither calls for another compatibility layer or application execution -abstraction. This document records findings, not implemented fixes. +is substantially complete. Findings 1–2 come from the initial review. Findings +3–5 come from a second, stage-ordered pass through the interaction of the native +migration with existing domain, validation, and diagnostic behavior. None calls +for another compatibility layer or application execution abstraction. This +document records findings, not implemented fixes. ## Finding 1: Retain Genuine Native Timeouts During Concurrent Error Selection @@ -127,6 +133,154 @@ retained. prose-snapshot or mandatory-source-validation test is needed for this documentation correction. +## Finding 3: Rebind Cached Scene Identity To The Current Chunk Plan + +Priority: P1. Relevant stages: 11, 23, 27–29, and 37. + +### Evidence And Impact + +The new message-content cache interacts incorrectly with the existing +position-derived scene identity: + +- [`MaterializeChunkPlan`](../../internal/core/source/chunk_plan.go), lines + 169–179, renders only source units into chunk content, while assigning + `chunk-000001`, etc. from the range's position in the plan. +- [Scene extraction](../../internal/modules/dnd/extract/scenedescriptions/extractor.go) + deterministically attaches the current chunk ID and range in `mapResponse`. + The model does not supply this identity, and the ID is not rendered into the + initial scene-description messages. +- [`artifactProducerCache`](../../internal/framework/pipeline/runner_cache.go), + lines 21–37, decodes a hit and checks cited source units, but does not rebind + its scene identity. The hit bypasses extraction interpretation and assessment. + +Changing an earlier scene boundary can therefore renumber an unchanged later +scene without changing that later scene's prompt hash. Its old accepted artifact +is a legitimate content-key hit, but its stored scene ID belongs to the old plan. + +A temporary native-CLI integration probe used real scene planning, extraction, +merging, normalization, default validators, and a filesystem cache. The chunk +binding used `cache: refresh`; scene extraction used ordinary `auto`. The fake +provider returned these two valid plans for the same three-unit transcript: + +1. `[1–2], [3]`, followed by `[1], [2], [3]`: the second run failed normalization + with `scene ID "chunk-000002" has conflicting records`. +2. The reverse change: the second run succeeded but published IDs + `chunk-000001` and `chunk-000003` for a two-chunk plan. The last ID should have + been `chunk-000002`. + +Both cases pass with invocation-wide cache bypass. The unchanged suffix was +reused in the failing cases, as confirmed by provider-call counts. This is not +an invalid generated plan or intentionally skipped semantic revalidation. + +The stale ID also conflicts with the existing [scene registry's exact match](../../internal/modules/dnd/scenedescriptions/registry/registry.go): +combat/enemy consumers require both the current chunk ID and range, so a stale +scene can become unavailable classification and suppress downstream extraction. + +### Required Change + +Add the missing current-operation adaptation at the scene cache-hit boundary. +For a usable single-scene artifact describing the exact current range, +deterministically reattach the current chunk identity while retaining its accepted +kind/title/summary. If a cached artifact cannot be safely adapted, treat it as an +unusable miss before returning it as accepted work. + +Keep domain-specific adaptation with the scene module, using the supplied typed +cache collaboration or a small extension to that boundary if needed. Preserve +the content-only native cache key and no-semantic-reassessment policy. Do not +render an internal scene ID into the prompt, add a partition fingerprint/key +dimension, restore canonical source-only chunking, or ask the model to copy IDs. +Review other current cached artifacts for equivalent operation-derived fields; +do not invent a generic identity framework for families that do not need one. + +### Acceptance + +- Repartitioning earlier units can move an unchanged later scene to a different + chunk ordinal without duplicate or stale scene IDs in the published artifact. +- Current scene IDs/ranges match the chunk map and remain usable by the existing + combat/enemy eligibility resolver. +- The unchanged scene can reuse its accepted semantic content without another + model call or validator assessment when deterministic adaptation is safe. +- Wrong-range, multi-scene, or otherwise unusable entries are not silently + rebound into valid-looking scenes; they miss normally. +- Add assembled two-run expansion/collapse coverage using the existing native + fake-provider/cache setup, and retain ordinary same-plan hit coverage. + +## Finding 4: Bound Framework-Generated Fallback Diagnostic Messages + +Priority: P2. Relevant stages: 9, 11, 24, 29, and 34. + +### Evidence And Impact + +[`FinishProducerFailure`](../../internal/framework/pipeline/producer_outcome.go) +retains the complete failure text in `ProducerOutcome.FallbackReason` after a +supported permissive fallback is successfully constructed. +[`operationDiagnosticGroups`](../../internal/framework/pipeline/runner_operation.go), +lines 51–58, copies that text unchanged into a framework-generated warning sample. +The existing [diagnostic contract](../../internal/framework/contracts/diagnostics.go) +requires valid UTF-8 and a maximum sample-message size of 4096 bytes. + +Native errors and joined assessment errors are not bounded diagnostic samples. +Even individually bounded domain validator messages can exceed the sample limit +after joining several findings and adding validator context. The new fallback +bridge thus creates invalid diagnostics from otherwise valid operation outcomes. + +A temporary offline native-CLI probe ran a minimal spell pipeline in permissive +mode with a fake provider returning a long ordinary error. The valid empty spell +fallback was constructed, but the run then failed with +`producer diagnostic 0: diagnostic sample 0: diagnostic sample message exceeds maximum length`. +The failure occurs even with ordinary visibility hiding optional warnings. + +### Required Change + +Render the framework's fallback warning into a bounded, valid-UTF-8 sample before +promoting it. Reuse or factor the small bounding behavior already present in +[`producerCacheObserver`](../../internal/framework/pipeline/runner_cache.go) +rather than adding a second diagnostic system. + +Bound only the generated diagnostic sample, leaving the original error/outcome +text and requested evidence unchanged. This is operator diagnostic presentation, +not model correction guidance. Preserve validation of module-supplied diagnostics; +do not raise/remove the existing cap or suppress genuine malformed-module +contracts to work around the framework's own unbounded sample. + +### Acceptance + +- A supported permissive fallback still publishes qualified output when its + native or aggregated-assessment cause exceeds the diagnostic sample limit. +- The generated warning is nonblank, bounded, and valid UTF-8, including at a + multibyte truncation boundary. +- Strict handling, caller cancellation, fallback counting/reuse exclusion, + full requested evidence, and module-diagnostic validation remain unchanged. +- Add focused fallback-to-diagnostic integration coverage, including a real + typed producer path with a fake long native error. No model retry is added. + +## Finding 5: Remove The Disconnected Legacy LLM Error Contract + +Priority: P3. Relevant stages: 8, 24, 35, and 44. + +### Evidence And Impact + +[`contracts/errors.go`](../../internal/framework/contracts/errors.go) still +defines the pre-migration `ErrInvalidStructuredOutput`, `ErrLLMCapacityExceeded`, +`ErrLLMGeneration`, `LLMGenerationError`, constructors, and copied failure/attempt +metadata. Current producers and validators retain native PromptKit errors instead. +Graph/code searches find this old taxonomy only in that file and its +[self-test](../../internal/framework/contracts/errors_test.go), not in an active +application consumer. + +This is leftover contract surface from the removed adapter, not a second active +execution path. It nevertheless presents obsolete sentinels and metadata accessors +as available framework conventions, although matching actual native failures +against them no longer works. Stage 35's hard-cutover cleanup is incomplete here. + +### Required Change And Acceptance + +Remove the unused legacy error definitions and their self-test. Keep native +`errors.Is`/`errors.As` behavior and the separate, actively used application +assessment/capture error types. Do not bridge native errors into these retired +sentinels or add aliases for compatibility. Verify no current callers remain and +run contracts, pipeline, CLI, and module integration tests after removal. + ## Review Coverage And Verification Owning boundaries were inspected in implementation order: @@ -158,6 +312,25 @@ Checks completed during review: repository code or tests. Their deliberately failing assertions are review evidence, separate from the passing default suite. +The integration follow-up inspected the stage boundaries above against the +retained source/chunk identity, domain codecs/validators, generated references, +diagnostic limits, physical publication, and subprocess contracts. It reused +the earlier full-suite/vet/build/race results only after confirming that the +application code and dependencies were unchanged. Additional checks were: + +- `GOWORK=off GOPROXY=off go test ./internal/core/config + ./internal/framework/pipeline ./internal/framework/semanticreconcile + ./internal/modules/integration ./internal/cli` passed. +- Real native-CLI/fake-provider Go-overlay probes reproduced both directions of + scene repartitioning in Finding 3. Both corresponding bypass controls passed. +- A native-CLI/fake-provider overlay probe reproduced the long-error permissive + fallback failure in Finding 4. +- Graph/code searches confirmed that Finding 5's legacy taxonomy has no active + callers outside its own definition/self-test. + +These probes live only in a temporary review directory. No production code, +test, configuration, schema, or existing canonical documentation was changed. + No live provider inference was performed. This review does not establish production model quality or performance. Intentional changes such as the hard configuration/CLI cutover, native retry ownership, content-only cache identity,