Files
scriptorium/docs/roadmap/remediation.md

7.4 KiB
Raw Blame History

PromptKit Adoption Review Findings

Review baseline: Scriptorium c5e54edaa5f6621d504cf510b4b5b5c1e4d1cfd4, with the approved PromptKit v0.23.3 pin and ConfigKit v1.2.1. Stage numbers refer to the implementation plan.

R1: Bound the HTTP timeout regression tests (Stage 4)

Complete. The tests use independent parent deadlines and capture the target invocation's error without sharing a channel. All three passed twice under the race detector.

In handler_test.go, TestHandlerDeadlineDuringBuiltInGeneration runs the handler synchronously while the fake provider waits on a channel that is released only after the handler returns. If the operation timeout stops reaching PromptKit, the test can wait for the unrelated provider/transport timeout or the suite watchdog instead of promptly failing. The retry- and capacity-wait tests likewise rely on the behavior being tested to return control. The added observeRunnerFailures helper also sends all failures to a shared, one-slot channel before allowing each runner invocation to return. An unexpected failure of the first request in the capacity test can fill that slot and block the second request from returning the failure the test needs to inspect.

Use a test-only fallback deadline or bounded synchronization independent of the operation budget, and capture the error of the specific observed invocation without a blocking send shared by concurrent requests. Keep the real engine and loopback provider, the assertion on ErrOperationTimeout, and the existing status, observation, and capacity-release checks. This is a test liveness fix, not a new production timeout or release gate.

R2: Describe source-relative message paths without a colocation rule (Stage 7)

Complete. The migration guide describes source-relative resolution and links to PromptKit's path rules. A render with a message in a subdirectory confirmed that no colocation is required.

The upgrade guide instructs users to keep message files beside their owning definitions. PromptKit resolves content_file against the field's declaring source within its rooted filesystem; it does not require the file to be in the same directory. A freshly built Scriptorium binary rendered a valid content_file: ./messages/request.md without moving the file beside its framework YAML. The current wording can lead operators to rearrange valid layouts during migration.

Describe source-relative resolution and link to the pinned PromptKit format contract for containment rules. A colocated file can remain an example, not a requirement. No loader or path-policy change is needed.

The attribution entry for golang.org/x/text v0.44.0 links LICENSE but omits PATENTS. Build dependency inspection confirms Go Text is part of the executable. Its selected module contains the separate patent grant at upstream commit 6070ac32abcdd1670599eeb3ee4d513c8184a513; PromptKit's attribution for that same revision links it. The OSS policy calls for applicable companion texts to be directly reachable.

Add a PATENTS link at the already selected Go Text commit. Retain the verified local BSD text and existing LICENSE link. No separate local patent notice or change to the project license is needed under the adopted attribution format.

R4: Distinguish discovery peers from ordered framework layers (Stages 6–7)

The upgrade guide says a later source selecting the same ID replaces the whole record. That holds for a later framework underlay, not for two checkout entries selected by the same discovery file. Scriptorium turns each resolved promptkit_sources selector into one underlay. The entries discovered by the imports of that selector are composed as peers; their filename, glob, or checkout order is not override precedence.

The built binary rejected two discovered checkout entries defining shared/fast with compose.conflict. Selecting the very same entry files as two ordered promptkit_sources items instead succeeded: the later record's model was selected and max_tokens, absent from that record, did not carry forward from the earlier one. This is the intended distinction, but the migration wording can lead operators to put an intended override in a discovery peer, which prevents the engine from loading.

Qualify whole-record replacement in the upgrade guide and explain in the checkout discovery reference that same-ID conflicts within one discovery underlay fail. For an intentional override, select a separate, later framework source using the existing application selector list. No scanner, de-duplication, new namespace restriction, or change to composition is needed.

R5: Replay framework explanation history once (Stage 3)

Framework configuration inspection calls view.Explain(path) to check existence, discards the complete explanation, then calls view.WriteExplanation. The pinned native writer calls Explain itself. A successful request therefore replays the same captured history and builds the same redacted value snapshots twice. If H is the work to construct that explanation and W the work to render it, the command does roughly 2H + W work instead of H + W. This is a constant-factor cost, not a new quadratic algorithm, but it repeats a history- and value-sized operation and its allocations. The legacy wrapper already used two calls; the native inspection path need not retain them.

Delegate once to view.WriteExplanation, as app configuration inspection already does with its native writer. A direct probe with the selected PromptKit and ConfigKit releases confirmed that this writes a known path, returns an error without output for an unknown path, and preserves writer errors. Keep JSON Pointer parsing, native redaction, and the existing exit and stream behavior; no new formatter or replay algorithm is needed.

R6: Prevent artifact read-limit overflow (Stage 4)

The restricted HTTP artifact reader allows the same nonnegative int64 limit as application configuration and the CLI. It computes maxBytes + 1 for io.LimitReader to detect oversized files. At the valid setting 9223372036854775807 (math.MaxInt64), that addition wraps to a negative number. The limited reader then immediately returns EOF, so a nonempty file is accepted as an empty artifact. A loopback check confirmed HTTP 200 at this setting while the outgoing prompt silently lost the file content. The same synthetic request included the file at zero and at the ordinary positive limit. The defect predates the migration and remains in the reader retained by Stage 4.

Use overflow-safe byte-limit arithmetic: for example, retain the capped limit + 1 read for smaller positive limits and skip the increment at math.MaxInt64, or limit the read first and probe separately for an extra byte. Accepted files should still require linear time and buffer space in the bytes read; normal limits should continue to reject oversized files without a full read. Add a boundary test using a small nonempty file. Do not reduce the accepted configuration range or change containment or the zero-limit semantics.