diff --git a/docs/roadmap/audita-request.md b/docs/roadmap/audita-request.md new file mode 100644 index 0000000..d19b044 --- /dev/null +++ b/docs/roadmap/audita-request.md @@ -0,0 +1,81 @@ +# Audita request: main-relative companion resources + +Consumer requirements against ConfigKit v1.1.0. These complement the proposed +[main-relative underlays and overlays](main-relative-inputs.md); that roadmap +owns upstream API design and implementation planning. + +## What Audita is doing + +Audita is a transcript-processing CLI adopting ConfigKit, PromptKit, and +CommandKit's native loading and inspection workflows. ConfigKit should own +discovery, imports, composition, overlays, and configuration resolution. Audita +should receive the resolved configuration and then apply transcript-specific +business logic. + +The main application configuration is either an explicit file or the result of +native `SearchConfig` discovery. Companion resources should follow the directory +containing the selected main file, regardless of the process working directory: + +| Resource | Intended location and use | +| --- | --- | +| Glossaries | Optional `glossary.yml` and recursive `glossary/` imports, mounted at `glossary_collections` in an application underlay. | +| PromptKit definitions | Optional `prompts.yml` and recursive `prompts/` imports, in an operator tier above embedded Audita prompts and PromptKit's bundled catalog. | +| Selected named overlay | Beside the main file, using native `overlays/.yml` lookup. | + +Both recursive imports select `*.yml` and `*.yaml`. Audita supplies their import +declarations, so operators need not repeat discovery boilerplate in every main +file or create a manifest just to use a resource directory. Each pair composes +as peers within its tier; native precedence and diagnostics remain authoritative. +Embedded assets retain their separate filesystem contexts. + +For example, selecting `/srv/campaign/audita.yml` while running from `/tmp` +should use `/srv/campaign/glossary/`, `/srv/campaign/prompts/`, and +`/srv/campaign/overlays/.yml`. + +An aggregate `overlays.yml` was also suggested for symmetry. It is not supported +by today's named-overlay contract and is a separate possible enhancement, not +necessary to solve the directory-binding requirement described here. + +## Missing API support + +In the current [layered API](../config.md#layered-loading), `SearchConfig` +selects the main during loading, while underlay filesystem contexts and +`OverlayFilesystem` must be supplied beforehand. Audita cannot bind these to +the selected directory without duplicating discovery or performing a preliminary +load. Imports in the main file would be peers, not the required lower-priority +underlay. + +1. **Bind an underlay import context to the selected main directory.** Audita + needs this for an application-supplied mapping or bytes containing native + import declarations, including optional file and recursive-directory imports + and their destinations. A constructor for a required main-relative file alone + would not cover this workflow. Resolve the relationship within one layered + load, then let existing import machinery acquire and compose the resources. +2. **Derive the selected overlay context during that same load.** Support an + explicit request for main-relative lookup with either an explicit or discovered + main file. Preserve native overlay naming, precedence, and errors. Audita's + intended lookup base is the main file's directory; distinguish that choice + from a potentially broader filesystem containment root. +3. **Make absent-main behavior explicit.** Audita permits optional discovery to + find no main file and still run with sufficient defaults and CLI settings. + The API should allow an explicit policy for dependent resources in that case, + without manufacturing a main source or guessing from unsuccessful candidates. + A caller-supplied fallback directory is one useful option; whether Audita + chooses the working directory or skips automatic companion imports remains + an application decision. Failures reading a selected main must remain terminal. + +PromptKit's separate engine can be constructed after application loading and use +the selected main's recorded source context. Existing context inspection may +already suffice there; this request does not require combining the two schemas +or adding a second discovery API. + +## Completion from Audita's perspective + +One native application load can discover the main, bind its companion imports +and selected overlay, and return normal source evidence and errors. Changing +only the invocation directory does not relocate companions of the same selected +main. Processing and inspection share this behavior, with no Audita directory +walker, configuration parser, discovery pass, or preliminary full load. + +The API shape is ConfigKit's choice. The important addition is native context +binding for both declared underlay imports and selected overlays. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..95edb53 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,239 @@ +# Implement main-relative companion inputs + +The [feature roadmap](main-relative-inputs.md) owns the target behavior and the +four settled decisions. The [Audita request](audita-request.md) supplies the +consumer workflow. Implement in order as one sprint. Intermediate stages may +leave code or tests incomplete; run focused checks when their dependencies are +in place, and require a coherent, validated tree at the end. Stage boundaries +are handoffs for Luna, not required commit boundaries. + +Start with the [development guide](../development.md) and its applicable +agent, architecture, testing, and documentation policies. Keep the public +facade root-owned. Keep ordinary loading, file source semantics, and existing +explicit layered overlays compatible. No dependency change, new YAML syntax, +second discovery pass, application callback for context selection, or new +release gate is part of this work. + +## Stage 1 — Define the Go API and preflight rules + +In root and engine `layered.go`, add matching descriptions and carry them into +`loadInput` and `loadRequest` in the engine. Use these public names: + +```go +type MainRelativeContext struct { + Fallback *FilesystemContext + RequireBase bool +} + +func MainRelative() *MainRelativeContext +``` + +`MainRelative()` returns a fresh pointer to a zero-valued description. Add +`Input.MainRelative *MainRelativeContext` and +`LayeredOptions[T].OverlayFallback *FilesystemContext`. The latter only needs +a concrete context: a selected overlay already requires a base. Use keyed +literals in new examples. Copy the description and its concrete fallback into +the private request using the existing filesystem-context conversion in +`lifecycle.go`; preserve the caller's native namespace and reader ownership +rules. Store resolved state only in the current load. + +In layered preflight, apply the concrete-context override before checking +binding scope. Reject an active main binding and active bindings on `File`, +`Filesystem.File`, and `SearchConfig` underlays with `CodeOptionsInvalid`. +Keep ordinary source validation, including zero-source rejection. A valid +`NoSource` underlay remains absent; binding does not make it contribute content +or evidence, and `RequireBase` still applies to an active binding. Suppress the +synthetic-source missing-context preflight error only for a valid active +underlay binding. Binding does not enable imports. + +Check the static shape of a fallback that can participate, using the existing +non-file context rules (both base and root, host absolute paths or native +namespace paths). Defer existence, directory, access, and resolved containment +checks until selection. Ignore the shadowed binding entirely when +`Input.Filesystem` is supplied; likewise ignore the overlay fallback when the +overlay is inactive or an explicit `OverlayFilesystem` wins. Keep overlay +name and explicit-context preflight checks. Move the missing-context error for +active layered overlays to Stage 5, where selection can see the main. Do not +change preflight for ordinary `Load`. + +Cover active binding scope, explicit overrides (including invalid explicit +contexts), and fallback shape through public option tests. Reuse existing +option diagnostic codes; avoid promising order among unrelated errors. + +## Stage 2 — Bind companion inputs with a usable context + +In `assembleLayeredInput`, pass the main's `assembledInput.resolvedFilesystem` +to `acquireUnderlayGroupScopes`. Reuse the acquisition handoff in +`acquisition.go` and resolution helpers in `filesystem_context.go`; do not +select or open the main again. Verify the handoff covers a file or selected +search without imports, a native file, explicit non-file contexts, and +`NoSource` or exhausted optional search with explicit context. Preserve the +absent host root-only rule and the native context requirements. A main +acquisition/context error is terminal, never an absent-base signal. + +For each underlay, use its explicit filesystem first; otherwise, for active +bindings, use the resolved main context, then its fallback. For main-derived +contexts copy the resolved base and root and keep the native namespace object +intact. Clear `mainPath` in the copy: the companion is still a synthetic +source, not a second copy of the main. Use the receiving tier's existing host +identity token and visited state; the main's host location and scope carry +over, not its tier bookkeeping. A selected fallback uses the same +resolution/validation rules as a concrete non-file context; pass it through +the normal acquisition path where possible. Validate selected contexts even +when imports are inactive as required by the existing context contract; +do not probe the fallback when main derivation wins. + +Carry a pre-resolved derived context through the private acquisition request +to `acquireImportedInput` and the standalone/context completion path. Use +existing import access, traversal and closure; keep root/base checks, parse +and version processing, budget accounting, and source evidence. Bound +synthetic sources can expose the same passive context as explicitly located +synthetic sources but keep their original kind, label and empty physical path. +The imported files use their own paths and directory bases. + +Make the existing native namespace anchor selection in +`acquireUnderlayGroupScopes` use the **selected** namespace, including +derived or fallback contexts. Determine anchors from selected descriptors +without opening directories or validating later inputs out of turn. Keep the +first input position anchor for each namespace and the existing host and +synthetic anchoring and tier boundaries. Preserve independent sources and +repetition semantics, including two different native namespaces with the +same label and paths. + +Cover successful bindings for host and native contexts, an unrelated working +directory, canonical host symlink target directory, a wider root that permits +contained parent-relative imports, and reuse of one description across loads. +Test selected and unused fallbacks. Defer the no-base path to Stages 3–4. + +## Stage 3 — Share import declaration preparation + +In `import_acquisition.go`, extract the context-independent first pass of +`walkDeclarations` so both normal traversal and the contextless path can +use it. Prepare from the already parsed declaration specs, apply +`resolveImportPolicy` with the load-wide defaults, and retain existing +placement checks and diagnostic sites. For an input placed as a sequence +item, prepare against its local mapping scope and keep the existing diagnostic +path projection. Use the already processed root directives from +`processAcquiredInputVersion`; do not reparse YAML, repeat version checks, +or infer a filesystem path from a source label. + +Keep selection and admission in the existing normal walk. This stage changes +no bound import behavior. Reuse import specification, defaults, placement, +selection, and diagnostic tests for verification. + +## Stage 4 — Finish companion handling when no base exists + +Handle the no-base outcome only for an active underlay binding with no +override, usable main context, or fallback. `RequireBase` fails with +`CodeOptionsInvalid` regardless of import enablement or declaration count. +Otherwise retain normal source parsing, version handling, disabled-import +validation, and contribution assembly. When imports are enabled, use Stage 3 +to validate their declarations even if none will be expanded. + +For required declarations use `CodeImportMissing` at the first named target +site for `path`, or the selector field site for `dir` and `glob`. Say that no +base is available; do not report a file search that did not happen. A required named-path declaration with nonempty +targets fails, as do required directory or glob selections. Empty +`path: []` has no targets even if required. Explicit per-declaration +requiredness continues to override the application default. Skip optional +expansion without calling a filesystem selector. + +Assemble the declaring payload once, with its original source record, +placement, sequence declarations, and version evidence. Share the existing +root contribution assembly and accounting where needed for imports-enabled +payloads, including the pending local scope for sequence-item placement. +Do not turn imports off to bypass validation or introduce a dummy access +object, dummy filesystem, acquired child, resource context, or +`EmptyImportSelections` entry. Charge the declaring payload as usual; charge +no traversal or child work for skipped imports. Normal searches with a +selected context still produce their normal empty-selection evidence. + +Test absent optional discovery and explicit absence, mixed payload with +optional and required declarations, application defaults and explicit +overrides, empty paths under both requiredness values, require-base with and +without enabled imports, malformed declarations and invalid placement, +source version policy, mounted payloads, and absence of synthetic search +evidence. Required discovery and failed selected main processing must not +activate this path. + +## Stage 5 — Derive layered overlay context and lookup base + +In the existing overlay branch of `assembleLayeredInput`, select in order: +explicit `OverlayFilesystem`, usable main context, `OverlayFallback`. +Skip selection entirely for an inactive name. For an active name without a +base, return `CodeOptionsInvalid`; if context validation fails, propagate +its normal diagnostics without trying another context. Keep overlay selection +after source assembly and main version handling, and preserve existing +failures for missing files, invalid overlays, and repeated sources. + +In `selected_overlay.go`, pass a lookup directory separately to the existing +selected overlay loader (for example on `selectedOverlayRequest`). Ordinary +`Load` and explicit layered contexts use the resolved root as before. +Derived and fallback contexts use the resolved base. Open access at the +containment root; join the lookup directory with `overlays/.yml` +using the selected namespace's path operations, then use normal +canonicalization and containment. Keep the namespace, source identities, +version processing, limits, source contexts, and handle ownership on the +existing path. Do not use the lookup directory as a replacement root or +construct a new namespace. + +Cover canonical host symlink targets, wider roots with distinct overlay +files at base and root, native filesystems, context on a non-file or absent +main, inactive overlay with an invalid fallback, explicit override with no +fallback on error, base-relative fallback, and missing selected overlays +without recovery. Retain ordinary and explicit root-relative regression +coverage, including existing namespace and repeated-source cases. + +## Stage 6 — Exercise the combined workflow + +Add a maintained executable example in `examples/consumers` for a single +layered load using `SearchConfig`, a mapping companion with optional named +files and recursive directory imports, an independently located embedded +tier, and one selected overlay. Use actual fixture files and the existing +`*.yml` and `*.yaml` pattern list; show destination placement, transitive +imports, and strict peers. Demonstrate precedence from embedded catalog +through operator underlay, main, and overlay. Include a registered identity +case for whole-record upsert across tiers and a same-tier duplicate failure. +No change to Audita or PromptKit is needed. + +## Stage 7 — Close remaining integration and evidence gaps + +Fill remaining public integration gaps without repeating every engine test: +first and later search candidates, explicit file and mapping/bytes/reader +sources, only the file, only the directory, both, neither and an empty +directory, invalid or inaccessible selected fragments (use a deterministic +filesystem error where permissions are unreliable), and a selected main +failure that cannot resume search or use the fallback. Use focused fixtures +to check layer/index, physical paths, passive resource contexts, normal +source-version evidence, shared acquisition budgets and cleanup on failure. +Check that callbacks execute only in the existing lifecycle, and that a +reused description binds anew. Preserve existing ownership and detached +inspection checks. Do not require new global diagnostic ordering or test +incidental call counts. + +## Stage 8 — Document and verify the completed sprint + +Update the public declaration comments and the discovery, layered loading, +imports, resource context and selected overlay sections of +[the configuration contract](../config.md). Document lookup versus +containment, canonical host symlink behavior, override and fallback +precedence, and absent-base behavior. Link the maintained examples from +consumer guidance. Update the [internal overview](../internal/overview.md) +and [import internals](../internal/imports.md) for the actual handoff, +contextless path, and ownership. + +Record the deferred context decision in the next numbered ADR, extending +[ADR 0019](../adr/0019-explicit-configuration-discovery.md) without rewriting +it. Clarify in the architecture policy that the layered loader binds +application-selected dependent locations after main selection; applications +still own search conventions, and an explicit overlay selection is required. +Preserve existing scope boundaries. Use the documentation policy's canonical +owners rather than restating the full contract in every document. + +Run the focused engine, facade, and consumer tests added or affected above, +then the everyday `./scripts/check.sh`. Run the race suite for the +load-local ownership and description reuse changes. Check changed Markdown +links, public facade boundaries and examples. Use the declared Go toolchain +and existing dependencies. No release or benchmark is part of completion. +Report validation and any remaining limitations; commit the completed sprint +under the agent policy. diff --git a/docs/roadmap/main-relative-inputs.md b/docs/roadmap/main-relative-inputs.md index 3a5eec9..4ce4ae5 100644 --- a/docs/roadmap/main-relative-inputs.md +++ b/docs/roadmap/main-relative-inputs.md @@ -1,7 +1,7 @@ # Main-relative companion imports and overlays -Status: all four design decisions settled; not implemented. Final API spelling -and implementation planning remain. Baseline: `191015e`. +Status: design settled; not implemented. The [implementation plan](implementation.md) +owns the concrete API spelling, execution order, and verification. Baseline: `191015e`. ## Purpose @@ -205,74 +205,6 @@ architectural exclusions. The design follows follow-up for deferred context ownership without rewriting accepted decisions. No release version or additional release gate is selected here. -## Work from the baseline - -1. **Finalize API spelling and examples.** Translate the settled decisions below - into concrete declarations and prepare Go examples for Audita's mapping/bytes - workflow, explicit overrides, non-file main contexts, and independently located - embedded inputs. -2. **Carry binding intent through the public facade and load request.** Update - the relevant input/options declarations and conversions in `layered.go` and - `lifecycle.go`, retaining root-owned public types and normal ownership rules. - Distinguish a deferred relationship from a concrete filesystem context. -3. **Bind underlay import contexts after main context resolution.** Extend - preparation and acquisition in `internal/engine/load_request.go`, `load_input.go`, and - `acquisition.go` as needed. Reuse the main's resolved base, root, and namespace, - apply the settled missing-base policy, and delegate selection to existing import - machinery. Preserve independently supplied contexts. -4. **Derive overlay context and separate lookup from containment.** Remove the - unconditional explicit-context requirement for the new layered default. - Preserve explicit and ordinary lookup while using the main context's resolved - base inside a potentially wider root. Reuse selected-overlay loading and its - failure and resource-ownership behavior. -5. **Verify the complete workflow and document it.** Add maintained consumer - examples, public integration coverage, and focused engine tests. Update the - discovery, layered-loading, import-context, and selected-overlay contracts, - declaration comments, and internal ownership documentation when implemented. - -This is dependency order for a later implementation plan, not a required commit -layout or a requirement to expose intermediate stages to consumers. - -## Validation and completion - -Validate behavior through the public loading boundary, including: - -- Explicit and discovered main files, early and later search candidates, and an - unrelated invocation directory. Companions follow the same selected main. -- Native file mains, bytes/readers/mappings with explicit host or native contexts, - and absent mains with explicit contexts. Cover absent host root-only contexts, - missing and invalid contexts, and context reuse without enabling main imports. -- Mapping, bytes, and reader underlays containing optional named-file and recursive - directory imports with `*.yml`/`*.yaml` filtering and mounted destinations. -- Receiving-input scope: active bindings on main inputs and file-backed underlays - fail with option errors; concrete context overrides retain their precedence. -- File only, directory only, both, neither, empty directories, and invalid or - unreadable selected fragments. Both forms compose as peers when present. -- Operator prompt overrides above embedded catalogs using registered identity; - same-tier duplicate IDs remain conflicts. Main and overlay precedence survive. -- Mixed embedded and companion inputs, multiple underlay tiers, transitive - imports, placement, source-version checks, and load-wide import defaults. -- Automatic derived overlays and explicit overrides, active and inactive names, - broader containment roots, and missing selected overlays. -- Optional discovery exhaustion, required discovery, fallback and skip policies, - mixed payload/required-import cases from OD-2, and terminal selected-main errors. -- Unused fallback paths need not exist or be accessible; selected fallback paths - receive normal validation. Inactive overlays ignore fallback settings. -- Empty `path: []` with either requiredness value when no base exists, including - the independent effect of `RequireBase`, and skipped imports without fabricated - completed-search evidence. -- Symlinks, eligible namespaces, parent-relative paths within the selected scope, - context validity, and namespace identity under OD-3 and OD-4. -- Normal source evidence, diagnostics, passive resource contexts, cleanup, and - budget accounting; no duplicate lifecycle callbacks or preliminary full load. -- Existing ordinary and explicit layered workflows and description reuse. - -Run focused checks during implementation and applicable normal repository checks -before completion. No benchmark campaign is required. Completion means the -application can discover its main, bind companion imports and the selected -overlay, and receive normal resolved configuration and inspection in one load, -without an application directory walker, parser, or extra discovery pass. - ## Settled Decisions ### OD-1: Public representation of deferred context binding — settled