diff --git a/docs/roadmap/promptkit-architecture-discussion.md b/docs/roadmap/promptkit-architecture-discussion.md index c245a819..9478a1f3 100644 --- a/docs/roadmap/promptkit-architecture-discussion.md +++ b/docs/roadmap/promptkit-architecture-discussion.md @@ -26,9 +26,12 @@ and [LLM Runtime Internals](../internal/llm.md). ## 1. LLM API And Diagnostic Boundary -**Discussion / decision:** Direction selected: remove the Notarius PromptKit -adapter and make the application fully PromptKit-native. Detailed design and -migration consequences remain to be worked through. +**Discussion / decision:** Direction agreed: remove the Notarius PromptKit +adapter and compose the application directly from native PromptKit execution, +typed artifact operations, and pipeline orchestration. A small application +function connects generation to configured validation and acceptance. The design +below incorporates items 2–6 and the Audita review; exact signatures and migration +details remain open. No application changes have been implemented. ### Background And Intent @@ -47,33 +50,103 @@ Missing reusable capabilities may justify upstream feature requests. The discussion is not constrained to what v0.23.2 can replace immediately; any resulting change to the adoption target remains to be selected. -### Design Consequences To Work Through +### Application Structure -- Use native PromptKit APIs and contracts for LLM execution instead of preserving - a parallel provider-neutral completion API or rebuilding the adapter under - another name. -- Review copied request, result, error, inspection, observation, and debug types - for removal. Keep application-owned representations only where they serve a - concrete application need, with compatibility decisions made explicitly. -- Let PromptKit provide reusable execution mechanics. Determine how Notarius - supplies domain inputs, interprets typed outputs, and expresses its acceptance - decisions through the resulting design. -- Review application composition, test seams, diagnostic storage, and disclosure - handling without assuming they require an LLM abstraction layer. -- Evaluate each upstream gap as a requirement to understand and potentially - address in PromptKit, rather than automatically preserving local machinery. +| Part | Responsibility | +| --- | --- | +| Application composition | Load configuration, assemble native prompt sources, and construct one shared PromptKit engine for producers, validators, and continuation runs. | +| Artifact-family modules | Supply prompt inputs and response requirements, interpret responses, and apply domain rules. | +| Pipeline execution | Connect generation to configured validation, decide acceptance or fallback, and manage dependencies and bounded workers. | +| PromptKit | Prepare and execute requests, maintain conversations, enforce native budgets and admission, and report execution facts. | +| Application records and CLI | Store artifacts and checkpoints, attach pipeline meaning to execution facts, and present results through CommandKit. | -Open questions: +Engine construction becomes ordinary application wiring using item 2's embedded +defaults and operator sources. Profile selection is expressed through native +requests. The current `internal/framework/llm` package can shrink to engine +assembly and focused helpers; whether those warrant a separate package remains +an organizational choice, not a reason to retain an adapter. -- What application collaborators and helpers remain after the adapter is removed? -- Which durable output or diagnostic contracts need deliberate migration? -- Which existing behavioral differences should disappear through native adoption, - and which reflect an application need that should be supplied to PromptKit? +Use native requests, results, exchanges, errors, and observers directly in +model-executing application code. Pure domain models, codecs, source-reference +resolution, and deterministic transformations need no PromptKit dependency. +Retain application interfaces for extracting artifacts, validating typed +candidates, encoding output, and loading checkpoints. Deterministic stages +continue to operate without conversation machinery. -**Follow-up:** Apply this direction to items 2–6, compare Audita under item 7, -and reconcile the adoption roadmap. Record the significant boundary change in -a successor ADR and update architecture and implementation documentation when -the change lands. Do not treat the remaining topics as decided by this direction. +### Connect Modules To Pipeline Assessment + +The current [spell extractor](../../internal/modules/dnd/extract/spells/extractor.go) +prepares inputs, invokes the private completion interface, interprets a response, +and returns a copied raw model candidate to the +[producer retry loop](../../internal/framework/pipeline/producer_attempts.go). +Replace that arrangement with direct collaboration between module interpretation, +pipeline assessment, and native execution: + +1. The module prepares source material, references, and any candidate-handle + mappings once, and constructs the native initial request. +2. Application execution calls `Engine.Run`, capturing an exchange when + continuation is enabled. +3. The module interprets the structurally valid response into a typed candidate + or actionable domain findings. +4. The pipeline runs configured candidate validation and decides whether to + accept, request correction, or apply terminal policy. +5. If correction is needed, native `Coordinate` invokes the same interpretation + and assessment for revised responses. The module supplies the repair request; + PromptKit maintains history and enforces the turn budget. + +A small shared application function can connect these responsibilities: the +module supplies native request construction and response interpretation, while +the pipeline supplies configured validation and terminal policy. It uses native +PromptKit types and returns a typed application outcome. It does not introduce +another completion API, conversation protocol, or retry engine. + +Item 3 owns the detailed continuation and assessment rules, including final-turn +assessment, failure handling, complete replacement responses, and safe fallback. +For normalization, a revised reconciliation plan is applied to the original +inputs to produce a fresh candidate. LLM-backed validators can call the same +native engine and return application validation findings; they do not need the +producer's conversation machinery. + +### Removal, Records, And Testing + +Retire `StructuredLLMClient`, the private structured-completion request/response +family, adapter conversions, and opaque correction transport. Items 3, 4, and 6 +specify removal of the outer LLM retry loop, scheduler, debug completion wrapper, +profile-recording plumbing, and duplicate execution accounting. Do not recreate +that wrapper stack around native calls. + +Attach pipeline scope through native callbacks and observers. Use native +`ExecutionGroup` totals and small application projections for manifests and +optional debug artifacts, following item 6. Checkpoint matching and durable +acceptance provenance remain the application responsibilities selected in item 5. + +Audita demonstrates shared engine assembly, native continuation, and CommandKit +integration. Its `RoleClient`, structured-output interface, and opaque exchange +wrapper are not part of this design; item 7 records the comparison's evidence +and limits. + +Test domain interpretation and assessment directly. Exercise model workflows +with a real PromptKit engine and its injectable client or a fake HTTP provider. +Pipeline orchestration can still use fake stage implementations. Testing does +not require preserving a private completion interface. + +### Remaining Implementation Work + +- Select concrete module/pipeline function signatures that connect request + construction, interpretation, configured validation, and terminal disposition. +- Choose the location of engine assembly and focused application helpers without + introducing a replacement LLM service layer. +- Resolve item 2's deferred filesystem and schema choices and the remaining + migration details recorded under items 3–6, including durable compatibility. +- Evaluate any remaining upstream gap against an actual application requirement, + rather than preserving historical machinery by default. + +**Follow-up:** Reconcile the adoption roadmap with this design and items 2–6. +Record the significant boundary change in successor ADRs. Update architecture +and implementation documentation when the change lands, explicitly replacing +the current transport-neutral completion boundary with PromptKit as the execution +dependency of application workflows. Keep domain operations and application +outcomes as the meaningful Notarius-owned boundaries. ## 2. Prompt Assets And Schema Assembly