diff --git a/docs/roadmap/commandkit-migration-discussion.md b/docs/roadmap/commandkit-migration-discussion.md index 65532fa7..c57eae6f 100644 --- a/docs/roadmap/commandkit-migration-discussion.md +++ b/docs/roadmap/commandkit-migration-discussion.md @@ -21,10 +21,11 @@ planning work. Implementation remains pending. The roadmap resolves U10's CommandKit scope, tracks the pending upstream message-content hash in U1, replaces U8's checkpoint design, and narrows U9 to -implementation details within the agreed output/debug contracts. Schema handling, -role-specific correction/fallback policy, worker defaults, and concrete cache -controls/transition remain visible there. Item-level directions below record the -accepted design; the roadmap is the authority for sequencing and current status. +implementation details within the agreed output/debug contracts. U2 now resolves +asset and schema assembly. Role-specific correction/fallback policy, worker +defaults, and concrete cache controls/transition remain visible there. Item-level +directions below record the accepted design; the roadmap is the authority for +sequencing and current status. ## Agreed Proposal @@ -1213,8 +1214,9 @@ separate Phase 7 is retired. Its dependency baseline includes the selected versions and pending upstream hash request. U10 is resolved; U8 now concerns cache operation/transition, and U9 concerns implementation details within the selected output/debug contracts. -Earlier receipt/manifest inventories are retired. Remaining schema, correction, -fallback, worker, and cache-transition choices are explicit in its +Earlier receipt/manifest inventories are retired, and U2 resolves asset/schema +assembly. Remaining correction, fallback, worker, and cache-transition choices +are explicit in its [unresolved-choice table](promptkit.md#unresolved-choices). The master roadmap owns sequencing and implementation status. This notepad and diff --git a/docs/roadmap/promptkit-architecture-discussion.md b/docs/roadmap/promptkit-architecture-discussion.md index 2fa31888..d976d322 100644 --- a/docs/roadmap/promptkit-architecture-discussion.md +++ b/docs/roadmap/promptkit-architecture-discussion.md @@ -144,7 +144,7 @@ not require preserving a private completion interface. construction, interpretation, configured validation, and terminal disposition. - Choose the location of engine assembly and focused application helpers without introducing a replacement LLM service layer. -- Resolve item 2's deferred filesystem and schema choices and the remaining +- Implement item 2's resolved asset/schema design and settle the remaining migration details recorded under items 3–6, including durable compatibility. - Evaluate any remaining upstream gap against an actual application requirement, rather than preserving historical machinery by default. @@ -158,70 +158,40 @@ outcomes as the meaningful Notarius-owned boundaries. ## 2. Prompt Assets And Schema Assembly -**Discussion / decision:** Direction agreed: use native PromptKit declarations -as the sole prompt assembly mechanism and remove the duplicate application -assembly plumbing. Resolution of the custom read-only filesystem and schemas -is explicitly deferred; the next topic may clarify those choices. +**Discussion / decision:** Agreed and incorporated as resolved U2 in the +[master roadmap](promptkit.md#prompt-assets-and-response-schemas-u2-resolved). +The filesystem and schema questions initially deferred here are now settled. +Implementation is pending; the master roadmap owns the detailed design and checks. -**Selected direction:** Keep the embedded content tree and load its prompt and -profile definitions as native PromptKit configuration in the application -collection. Use native filesystem contexts and imports to reference shared -content directly. Application composition assembles the embedded and operator -sources once; modules select prompt IDs, supply domain inputs, and interpret -typed results. +Use ordinary embedded assets, native prompt declarations, and native schema +loading. Keep the content tree and module ownership of prompt meaning and domain +interpretation. Shared files use real native filesystem paths. Remove duplicate +asset registration, per-module manifests, synthetic mounts, flattened schema +namespaces, and the custom read-only filesystem that served that assembly. -The spell extractor illustrates the current duplication: its prompt YAML -declares ordered messages and a schema, while a separate Go asset manifest -lists files, mounts shared fragments, registers filesystems, and hashes them. -The selected direction makes native declarations the sole prompt assembly source. +Prompt definitions select output contracts by default. Initial and correction +prompts can share schema files. Use native per-request replacement only for a +demonstrated runtime-dependent contract, without an application wrapper or schema +registry. Audita demonstrates ordinary embedded schema loading and per-call +contracts; its per-call selection is not required for every Notarius operation. -Candidates for removal: +The default schema source is the embedded asset tree. Operator-selected external +schema directories use native whole-root replacement, not an application overlay. +Prompt underlays do not implicitly overlay schema files. PromptKit owns private +response-schema loading, structural checks, and repair; Notarius keeps typed +interpretation, domain validation, and distinct durable artifact schemas/codecs. -- Prompt and fallback-profile registration plumbing in `AssetRegistry` and - individual modules; schema responsibilities remain under review. -- `ModulePromptFS`, `PromptAssetManifest`, shared-name lookup tables, synthetic - `sharedassets` mounts. -- Manual prompt-file enumeration for checkpoint fingerprints, retired under - item 5's simpler reuse policy. Native prompt hashes remain available for - inspection and diagnostics. +Retire response-schema hashes and copied metadata used only by the adapter, +checkpoints, or reporting. Preserve actual artifact contracts and native diagnostic +snapshots. Neither asset nor schema metadata adds another cache fingerprint. -Keep prompt prose, ordered messages, cache controls, workload profiles, private -response schema documents, and domain interpretation. Durable artifact schemas -remain separate. Content ownership does not require runtime asset registration. +The spell extractor illustrated the duplication: prompt YAML declared ordered +messages and a schema while a Go manifest separately listed, mounted, registered, +and hashed assets. Native declarations now supply the selected assembly model. -Native schema paths resolve against the configured schema root, separately from -message-file source contexts. PromptKit v0.23.2 loads schemas during preparation -and requires self-contained schema documents. A captured prompt hash alone does -not establish schema equality. Existing schema metadata in manifests and -checkpoint inputs must be deliberately replaced or retired; its presence does -not require preserving the complete wrapper system. - -Deferred decisions: - -- Whether the custom read-only filesystem can be removed after reviewing its - remaining responsibilities. -- Schema assembly, root/path selection, wrapper removal, and schema metadata. - Supplying the embedded filesystem as a native schema root remains a candidate, - not a selected design. -- Schema metadata for artifact compatibility and inspection. Item 5 settles that - cache matching does not require exact schema identity; the CommandKit output - decision removes mandatory manifests. - -Implementation questions: - -- Can native sources represent the existing shared fragments and module assets? -- Which registry, remounting, and hashing helpers could disappear? -- What schema metadata remains useful for inspection and artifact loading under - item 5's simpler reuse policy? -- How do we preserve message ordering and cache controls while simplifying - assembly? - -**Follow-up:** Continue with item 3, then revisit the deferred filesystem and -schema decisions with any new clarity. Confirm native source contexts for shared -fragments and operator replacements. Apply item 5's reuse policy and resolve -remaining metadata with item 6. Verify the resulting definitions through native -preparation and fake-provider execution; filesystem remounting tests can retire -with the removed implementation. The agreed direction has not yet been implemented. +**Follow-up:** Implement the master roadmap's Phase 2 and retire tests of removed +mounting/registry machinery. Verify shared paths, operator replacements, native +schema sources, and any necessary complete request-contract replacement. ## 3. Retry And Semantic Correction Ownership @@ -335,8 +305,8 @@ Notarius's serialized schema validator currently receives the codec-produced artifact and its artifact schema, which can differ from the model response. Do not remove that boundary merely because both use JSON Schema. Audit actual duplicate response checks individually; artifact validation and Go/domain -interpretation are not automatically redundant. Item 2's filesystem and schema -assembly decisions remain open. +interpretation are not automatically redundant. Item 2's resolved native asset +and schema design preserves this distinction. Evidence: [producer attempt loop](../../internal/framework/pipeline/producer_attempts.go), [validation executor](../../internal/framework/pipeline/validation_executor.go), @@ -355,9 +325,8 @@ and the native continuation evidence in item 7. candidate may survive a later failed turn. - Failure handling for non-LLM operations and any justified recovery beyond native execution budgets. -- Item 2's deferred filesystem and schema assembly choices, item 5's remaining - cache implementation details, and migration to the selected CommandKit output - contract. +- Item 5's remaining cache implementation details and migration to the selected + CommandKit output contract. **Follow-up:** Resolve the remaining details above, then reconcile the adoption roadmap. Record the changed architecture in a successor to @@ -825,7 +794,8 @@ Evidence: [LLM runtime](../../../audita/docs/internal/llm-runtime.md), Item 2's native underlay direction has a working application precedent. Audita also offers evidence for ordinary embedded filesystems and per-request schema -selection, while our filesystem and schema decisions remain open. +selection. U2 now selects ordinary embedded assets and prompt-declared contracts +by default, reserving per-request replacement for demonstrated runtime needs. The subsequent item 3 discussion selects native continuation and accumulated conversation history. Keeping domain validation local does not require keeping diff --git a/docs/roadmap/promptkit.md b/docs/roadmap/promptkit.md index 97d49c62..2c74b885 100644 --- a/docs/roadmap/promptkit.md +++ b/docs/roadmap/promptkit.md @@ -240,8 +240,8 @@ content tree and module ownership of prompt meaning, response requirements, and domain interpretation. Remove prompt/fallback-profile registration in `AssetRegistry`, per-module `PromptAssetManifest` plumbing, `ModulePromptFS`, shared-name maps, synthetic `sharedassets` mounts, and manual prompt enumeration -for checkpoint fingerprints. Schema-related registry responsibilities remain -subject to the explicit schema decision below. +for checkpoint fingerprints. The resolved schema design below also removes +schema registration and filesystem-flattening machinery. Convert maintained resources, examples, and fixtures to native identified lists, `config:` source controls, and sibling inheritance (`extends: {id: parent}`). @@ -268,19 +268,61 @@ for accumulated conversation history; preserving every old byte or retry message is not a migration requirement. Shared content should use native filesystem contexts and imports rather than reconstructed virtual trees. -PromptKit owns private model-response validation and structural repair. Native -schemas resolve against the configured schema root, separately from message-file -contexts. The reviewed baseline requires self-contained schemas loaded during -preparation. -A request override replaces the complete output contract; it must retain the -intended schema, validation mode, and repair policy. A prompt hash alone does not -establish schema equality. +#### Prompt Assets And Response Schemas (U2 Resolved) -The custom read-only filesystem, schema root/path selection, module-supplied -request contracts versus prompt-declared contracts, wrapper removal, and useful -schema metadata remain open. An ordinary embedded schema filesystem, as used by -Audita, is a candidate rather than a selected requirement. Durable artifact -schemas and their validators remain distinct from private response schemas. +Use the existing root `assets` package's ordinary `embed.FS` directly for native +prompt sources and response schemas, with `fs.Sub` only where useful. Shared +message files use their actual paths within the native filesystem context. +Remove synthetic mounts, file copying, flattened schema namespaces, and module +asset manifests. Retire `readonlyfs` when its prompt-mounting and asset-flattening +consumers disappear; no replacement filesystem abstraction is needed. + +Declare response format, validation mode, schema path, and structural-repair +budget in native prompt definitions by default. Modules select prompt IDs, supply +inputs, and interpret responses. Initial and correction prompts may reference +the same schema file. Execution, inspection, and prepared-request preview then +use the same declared contract without repeating schema selection in Go. +Role-specific prompt contents remain U4, and retry defaults remain U5. + +Use a request-level native `OutputContract` only for a demonstrated runtime- +dependent contract. It replaces the whole configured contract, so supply the +intended format, validation mode, schema, and repair policy together. Do not +introduce a wrapper or per-call registry solely to reproduce Audita's approach. +Operator prompt replacements must still produce responses the selected module +can interpret; that module boundary does not require duplicating each prompt's +schema selection in Go. + +Default to native `WithSchemaFS(assets.FS(), ".")`. Schema paths are relative to +that selected root, for example `dnd/spells/schemas/dnd_spells_llm.v1.json`. +They are separate from message-file source contexts: prompt underlays do not +implicitly overlay schema files. When an operator configures an external schema +directory, select PromptKit's native directory source instead of the embedded +root. It replaces the whole schema source and must supply the schemas used by +the selected prompts. Do not build an application filesystem overlay to mix +embedded and external schemas. Transparent layering would be a separate upstream +capability, not a requirement for this migration. Concrete configuration binding +is an implementation detail within this selected source policy. + +Keep private response schemas as self-contained JSON documents. PromptKit owns +loading, compilation, structural validation, provider structured-output settings, +and structural repair. Remove Notarius's response-schema byte registry, copied +schema wrappers, defensive-copy layer, digest calculation, and duplicate +structural-validation machinery. Native prepared snapshots provide diagnostic +schema details; no schema fingerprint is added to the message-content cache key. + +Keep typed decoding, source/reference checks, reconciliation safety, semantic +validators, artifact codecs, and durable artifact schemas in Notarius. Response +schemas describe model output; artifact schemas describe published and consumed +domain artifacts. Preserve identifiers and versions belonging to actual artifact +contracts. Retire private schema keys, registration records, hashes, and copied +identity/version/name metadata whose only consumers were adapters, checkpoints, +or reporting. No independent schema metadata system replaces them. + +Implementation is pending. Verify native preparation of embedded and operator +prompt definitions, shared-file paths, schema selection from embedded and external +roots, and complete request-contract replacement where actually used. Missing or +invalid schemas should follow native preparation behavior. No new schema check +requiring synthetic requests is added to offline pipeline preflight. ### Native Production, Continuation, And Assessment @@ -669,7 +711,6 @@ reopen the architecture. IDs retain continuity with the earlier roadmap. | ID | Remaining choice or dependency | Resolve for | | --- | --- | --- | | U1 | Supplying PromptKit API/release for the requested message-content hash; verify the final compatible dependency set against the selected baseline. The upstream request is pending. | Track in Phase 1; required to finish cache integration in Phase 5. | -| U2 | Custom read-only filesystem responsibilities; schema root/path and assembly; prompt-declared versus per-request contracts; removable wrappers and useful artifact-compatibility/inspection metadata. Mandatory manifest metadata is retired. | Phase 2. | | U3 | Concrete module/pipeline signatures and helper placement for request construction, interpretation, assessment, and native engine assembly. | Implementation detail in Phase 3, refined in Phase 4. | | U4 | Role-specific initial/repair definitions, response contracts, and profile selection within complete replacement and fixed operation inputs. | Phases 3–4. | | U5 | Producer/validator retry-setting migration to native controls and semantic budgets: field names/defaults, non-LLM failure handling, and any justified recovery beyond native exhaustion. | Phases 3–4 and configuration migration. | @@ -678,6 +719,12 @@ reopen the architecture. IDs retain continuity with the earlier roadmap. | U8 | Cache directory configuration and reuse/bypass/refresh controls (names, defaults, scope); old-checkpoint treatment and unusable-entry diagnostics. Hash inputs, lookup timing, accepted value, and ownership are settled. | Design with Phase 3; finish in Phase 5. | | U9 | Observer/group enrollment and application scope; concrete conditional report fields and debug filenames/serialization within the agreed contracts; publication-layout/consumer migration details preserving intrinsic artifact schemas. | Implementation details in Phases 3–5; no new receipt/manifest inventory or capture-policy decision required. | +**U2 resolved:** Ordinary embedded assets, prompt-declared output contracts, +native schema-source selection, and removal of duplicate schema/filesystem +machinery are selected in [the schema design](#prompt-assets-and-response-schemas-u2-resolved). +Implementation remains in Phase 2; role-specific prompts and retry budgets remain +U4/U5. + **U10 resolved:** CommandKit versions, capabilities, commands, loaders/preflight, output, debug policy, and coordinated cutover sequencing are selected above. The former later CommandKit phase is retired. CLI compatibility is a non-goal; @@ -698,8 +745,8 @@ major boundary changes in successor ADRs without rewriting accepted decisions. Keep the cache dependency explicit without blocking independent work. - Map affected ADRs, current callers, contracts, and removal inventory. Carry forward Audita evidence without adopting its application wrappers. -- Preserve U2–U9 with their dependent phases; do not treat resolved CommandKit, - debug, cache, or output choices as requirements to redesign. +- Preserve U3–U9 with their dependent phases; do not treat resolved assets/schemas, + CommandKit, debug, cache, or output choices as requirements to redesign. **Completion evidence:** reviewed dependency baseline, upstream dependency, scope, and outstanding choices are explicit. This phase does not claim that the hash @@ -707,15 +754,17 @@ already exists or that the migration is implemented. ### Phase 2: Build Native Composition, Sources, And Preflight -- Resolve U2, update available dependencies, introduce configuration version 6, - convert native embedded/companion resources, and establish schema loading. +- Implement the resolved U2 design, update available dependencies, introduce + configuration version 6, convert native embedded/companion resources, and use + native schema loading with embedded/default or operator-selected roots. - Build the small CommandKit application, shared ConfigKit/engine loaders, standard suites, pipeline declarations/native alias, and `internal/preflight`. - Add maintained embedded-only and operator-override examples. Remove duplicated loading and asset assembly as callers migrate. **Completion evidence:** runtime/inspection/preflight share intended definitions; -source contexts, strict peer conflicts, and replacement work offline. Help, +source contexts, shared message paths, prompt-declared output contracts, selected +schema roots, strict peer conflicts, and replacement work offline. Help, version, and completion need no configuration. Preflight checks known dependencies without credentials or synthetic requests. `config diff` handles two selections. @@ -784,7 +833,8 @@ Release preparation/publication is governed separately by release policy. | Retire or replace | Retained application responsibility | | --- | --- | | `StructuredLLMClient`, private request/response types, adapter conversions | Native request construction and typed interpretation. | -| Prompt/profile `AssetRegistry`, `PromptAssetManifest`, `ModulePromptFS`, synthetic mounts, manual fingerprint enumeration | Embedded content and module meaning; schema choices under U2. | +| `AssetRegistry`, `PromptAssetManifest`, `ModulePromptFS`, synthetic mounts, flattened schema namespaces, `readonlyfs`, manual fingerprint enumeration | Ordinary embedded content, native source assembly, and module meaning. | +| Private response-schema registry/wrappers, byte copies, digest and reporting-only identity metadata, duplicate structural validation | Native prompt output contracts and schema loading; application interpretation and durable artifact contracts. | | `single_response_v1`, raw `ModelCandidate`/`SemanticCorrection`, manual replay, correction capability checks | Typed candidates, actionable findings, native exchanges, assessment. | | LLM retry loops in `runProducerAttempts`/`executeValidationChain`, module rerun directives | Ordered validators, domain acceptance/rejection/fallback. | | `Scheduler`, `scheduledClient`, `concurrency.total_llm`, worker coupling | Bounded pipeline work and native backend admission. | @@ -805,7 +855,7 @@ retry tests or freeze complete help text and native debug snapshots. | Area | Required evidence | | --- | --- | -| Configuration/assets | Version 6/new resources work; old formats intentionally rejected; companions/native imports retain main-source context; peer conflicts fail; same-ID replacement and selected schemas work. | +| Configuration/assets | Version 6/new resources work; old formats intentionally rejected; companions/native imports retain main-source context; peer conflicts fail; same-ID replacement, real shared-file paths, prompt-declared contracts, and embedded/external schema roots work; necessary request overrides replace the complete contract. | | Commands/loaders | Required operands, native alias, supplied-value distinctions, references, and run-wide overrides reach intended handlers/roles; help/version/completion need no config; diff selections remain distinct. | | Preflight/inspection | Same definitions across execution and inspection; missing statically selected resources fail with context; partial profiles alone pass; no provider calls, transcript, or credential requirement for offline preflight. | | Domain workflow | Representative extraction and normalization preserve source/reference identity, fixed handles, safe plan application, ordered validators, artifact schemas, and stable output ordering. |