base-layer/configkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Retire completed companion loading roadmaps

Browse Source
This commit is contained in:
Eric Rakestraw
2026-11-10 01:26:41 +00:00
parent 47547bc63e
commit 5f001cf597
5 changed files with 6 additions and 957 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
docs/development.md
View File

@@ -94,6 +94,12 @@ and current facade ownership belongs to the [component overview](internal/overvi
## Baseline validation
The completed main-relative companion roadmap, Audita request, implementation
plan, and remediation report are preserved in Git history at `47547bc` under
`docs/roadmap/`. The [layered loading contract](config.md#layered-loading)
describes current behavior, and [ADR 0022](adr/0022-main-relative-context.md)
records the context binding decision.
The completed field-level sequence resolution roadmap, implementation plan, and
resolved audit findings are preserved in Git history at `7e53863` under
`docs/roadmap/`. For example:

81
docs/roadmap/audita-request.md
View File

@@ -1,81 +0,0 @@
# Audita request: main-relative companion resources
Consumer requirements against ConfigKit v1.1.0. These complement the proposed
[main-relative underlays and overlays](main-relative-inputs.md); that roadmap
owns upstream API design and implementation planning.
## What Audita is doing
Audita is a transcript-processing CLI adopting ConfigKit, PromptKit, and
CommandKit's native loading and inspection workflows. ConfigKit should own
discovery, imports, composition, overlays, and configuration resolution. Audita
should receive the resolved configuration and then apply transcript-specific
business logic.
The main application configuration is either an explicit file or the result of
native `SearchConfig` discovery. Companion resources should follow the directory
containing the selected main file, regardless of the process working directory:
| Resource | Intended location and use |
| --- | --- |
| Glossaries | Optional `glossary.yml` and recursive `glossary/` imports, mounted at `glossary_collections` in an application underlay. |
| PromptKit definitions | Optional `prompts.yml` and recursive `prompts/` imports, in an operator tier above embedded Audita prompts and PromptKit's bundled catalog. |
| Selected named overlay | Beside the main file, using native `overlays/<name>.yml` lookup. |
Both recursive imports select `*.yml` and `*.yaml`. Audita supplies their import
declarations, so operators need not repeat discovery boilerplate in every main
file or create a manifest just to use a resource directory. Each pair composes
as peers within its tier; native precedence and diagnostics remain authoritative.
Embedded assets retain their separate filesystem contexts.
For example, selecting `/srv/campaign/audita.yml` while running from `/tmp`
should use `/srv/campaign/glossary/`, `/srv/campaign/prompts/`, and
`/srv/campaign/overlays/<name>.yml`.
An aggregate `overlays.yml` was also suggested for symmetry. It is not supported
by today's named-overlay contract and is a separate possible enhancement, not
necessary to solve the directory-binding requirement described here.
## Missing API support
In the current [layered API](../config.md#layered-loading), `SearchConfig`
selects the main during loading, while underlay filesystem contexts and
`OverlayFilesystem` must be supplied beforehand. Audita cannot bind these to
the selected directory without duplicating discovery or performing a preliminary
load. Imports in the main file would be peers, not the required lower-priority
underlay.
1. **Bind an underlay import context to the selected main directory.** Audita
needs this for an application-supplied mapping or bytes containing native
import declarations, including optional file and recursive-directory imports
and their destinations. A constructor for a required main-relative file alone
would not cover this workflow. Resolve the relationship within one layered
load, then let existing import machinery acquire and compose the resources.
2. **Derive the selected overlay context during that same load.** Support an
explicit request for main-relative lookup with either an explicit or discovered
main file. Preserve native overlay naming, precedence, and errors. Audita's
intended lookup base is the main file's directory; distinguish that choice
from a potentially broader filesystem containment root.
3. **Make absent-main behavior explicit.** Audita permits optional discovery to
find no main file and still run with sufficient defaults and CLI settings.
The API should allow an explicit policy for dependent resources in that case,
without manufacturing a main source or guessing from unsuccessful candidates.
A caller-supplied fallback directory is one useful option; whether Audita
chooses the working directory or skips automatic companion imports remains
an application decision. Failures reading a selected main must remain terminal.
PromptKit's separate engine can be constructed after application loading and use
the selected main's recorded source context. Existing context inspection may
already suffice there; this request does not require combining the two schemas
or adding a second discovery API.
## Completion from Audita's perspective
One native application load can discover the main, bind its companion imports
and selected overlay, and return normal source evidence and errors. Changing
only the invocation directory does not relocate companions of the same selected
main. Processing and inspection share this behavior, with no Audita directory
walker, configuration parser, discovery pass, or preliminary full load.
The API shape is ConfigKit's choice. The important addition is native context
binding for both declared underlay imports and selected overlays.

255
docs/roadmap/implementation.md
View File

@@ -1,255 +0,0 @@
# Implement main-relative companion inputs
The [feature roadmap](main-relative-inputs.md) owns the target behavior and the
four settled decisions. The [Audita request](audita-request.md) supplies the
consumer workflow. Implement in order as one sprint. Intermediate stages may
leave code or tests incomplete; run focused checks when their dependencies are
in place, and require a coherent, validated tree at the end. Stage boundaries
are handoffs for Luna, not required commit boundaries.
Start with the [development guide](../development.md) and its applicable
agent, architecture, testing, and documentation policies. Keep the public
facade root-owned. Keep ordinary loading, file source semantics, and existing
explicit layered overlays compatible. No dependency change, new YAML syntax,
second discovery pass, application callback for context selection, or new
release gate is part of this work.
## Stage 1 — Define the Go API and preflight rules
**Status:** Complete. Main-relative descriptions and layered preflight are in place.
In root and engine `layered.go`, add matching descriptions and carry them into
`loadInput` and `loadRequest` in the engine. Use these public names:
```go
type MainRelativeContext struct {
Fallback *FilesystemContext
RequireBase bool
}
func MainRelative() *MainRelativeContext
```
`MainRelative()` returns a fresh pointer to a zero-valued description. Add
`Input.MainRelative *MainRelativeContext` and
`LayeredOptions[T].OverlayFallback *FilesystemContext`. The latter only needs
a concrete context: a selected overlay already requires a base. Use keyed
literals in new examples. Copy the description and its concrete fallback into
the private request using the existing filesystem-context conversion in
`lifecycle.go`; preserve the caller's native namespace and reader ownership
rules. Store resolved state only in the current load.
In layered preflight, apply the concrete-context override before checking
binding scope. Reject an active main binding and active bindings on `File`,
`Filesystem.File`, and `SearchConfig` underlays with `CodeOptionsInvalid`.
Keep ordinary source validation, including zero-source rejection. A valid
`NoSource` underlay remains absent; binding does not make it contribute content
or evidence, and `RequireBase` still applies to an active binding. Suppress the
synthetic-source missing-context preflight error only for a valid active
underlay binding. Binding does not enable imports.
Check the static shape of a fallback that can participate, using the existing
non-file context rules (both base and root, host absolute paths or native
namespace paths). Defer existence, directory, access, and resolved containment
checks until selection. Ignore the shadowed binding entirely when
`Input.Filesystem` is supplied; likewise ignore the overlay fallback when the
overlay is inactive or an explicit `OverlayFilesystem` wins. Keep overlay
name and explicit-context preflight checks. Move the missing-context error for
active layered overlays to Stage 5, where selection can see the main. Do not
change preflight for ordinary `Load`.
Cover active binding scope, explicit overrides (including invalid explicit
contexts), and fallback shape through public option tests. Reuse existing
option diagnostic codes; avoid promising order among unrelated errors.
## Stage 2 — Bind companion inputs with a usable context
**Status:** Complete. Focused host, native, symlink, fallback, and grouped-load checks passed.
In `assembleLayeredInput`, pass the main's `assembledInput.resolvedFilesystem`
to `acquireUnderlayGroupScopes`. Reuse the acquisition handoff in
`acquisition.go` and resolution helpers in `filesystem_context.go`; do not
select or open the main again. Verify the handoff covers a file or selected
search without imports, a native file, explicit non-file contexts, and
`NoSource` or exhausted optional search with explicit context. Preserve the
absent host root-only rule and the native context requirements. A main
acquisition/context error is terminal, never an absent-base signal.
For each underlay, use its explicit filesystem first; otherwise, for active
bindings, use the resolved main context, then its fallback. For main-derived
contexts copy the resolved base and root and keep the native namespace object
intact. Clear `mainPath` in the copy: the companion is still a synthetic
source, not a second copy of the main. Use the receiving tier's existing host
identity token and visited state; the main's host location and scope carry
over, not its tier bookkeeping. A selected fallback uses the same
resolution/validation rules as a concrete non-file context; pass it through
the normal acquisition path where possible. Validate selected contexts even
when imports are inactive as required by the existing context contract;
do not probe the fallback when main derivation wins.
Carry a pre-resolved derived context through the private acquisition request
to `acquireImportedInput` and the standalone/context completion path. Use
existing import access, traversal and closure; keep root/base checks, parse
and version processing, budget accounting, and source evidence. Bound
synthetic sources can expose the same passive context as explicitly located
synthetic sources but keep their original kind, label and empty physical path.
The imported files use their own paths and directory bases.
Make the existing native namespace anchor selection in
`acquireUnderlayGroupScopes` use the **selected** namespace, including
derived or fallback contexts. Determine anchors from selected descriptors
without opening directories or validating later inputs out of turn. Keep the
first input position anchor for each namespace and the existing host and
synthetic anchoring and tier boundaries. Preserve independent sources and
repetition semantics, including two different native namespaces with the
same label and paths.
Cover successful bindings for host and native contexts, an unrelated working
directory, canonical host symlink target directory, a wider root that permits
contained parent-relative imports, and reuse of one description across loads.
Test selected and unused fallbacks. Defer the no-base path to Stages 3–4.
## Stage 3 — Share import declaration preparation
**Status:** Complete. Focused import and placement checks passed.
In `import_acquisition.go`, extract the context-independent first pass of
`walkDeclarations` so both normal traversal and the contextless path can
use it. Prepare from the already parsed declaration specs, apply
`resolveImportPolicy` with the load-wide defaults, and retain existing
placement checks and diagnostic sites. For an input placed as a sequence
item, prepare against its local mapping scope and keep the existing diagnostic
path projection. Use the already processed root directives from
`processAcquiredInputVersion`; do not reparse YAML, repeat version checks,
or infer a filesystem path from a source label.
Keep selection and admission in the existing normal walk. This stage changes
no bound import behavior. Reuse import specification, defaults, placement,
selection, and diagnostic tests for verification.
## Stage 4 — Finish companion handling when no base exists
**Status:** Complete. Focused no-base, import, placement, and grouped-load checks passed.
Handle the no-base outcome only for an active underlay binding with no
override, usable main context, or fallback. `RequireBase` fails with
`CodeOptionsInvalid` regardless of import enablement or declaration count.
Otherwise retain normal source parsing, version handling, disabled-import
validation, and contribution assembly. When imports are enabled, use Stage 3
to validate their declarations even if none will be expanded.
For required declarations use `CodeImportMissing` at the first named target
site for `path`, or the selector field site for `dir` and `glob`. Say that no
base is available; do not report a file search that did not happen. A required named-path declaration with nonempty
targets fails, as do required directory or glob selections. Empty
`path: []` has no targets even if required. Explicit per-declaration
requiredness continues to override the application default. Skip optional
expansion without calling a filesystem selector.
Assemble the declaring payload once, with its original source record,
placement, sequence declarations, and version evidence. Share the existing
root contribution assembly and accounting where needed for imports-enabled
payloads, including the pending local scope for sequence-item placement.
Do not turn imports off to bypass validation or introduce a dummy access
object, dummy filesystem, acquired child, resource context, or
`EmptyImportSelections` entry. Charge the declaring payload as usual; charge
no traversal or child work for skipped imports. Normal searches with a
selected context still produce their normal empty-selection evidence.
Test absent optional discovery and explicit absence, mixed payload with
optional and required declarations, application defaults and explicit
overrides, empty paths under both requiredness values, require-base with and
without enabled imports, malformed declarations and invalid placement,
source version policy, mounted payloads, and absence of synthetic search
evidence. Required discovery and failed selected main processing must not
activate this path.
## Stage 5 — Derive layered overlay context and lookup base
**Status:** Complete. Focused host, native, fallback, and existing overlay checks passed.
In the existing overlay branch of `assembleLayeredInput`, select in order:
explicit `OverlayFilesystem`, usable main context, `OverlayFallback`.
Skip selection entirely for an inactive name. For an active name without a
base, return `CodeOptionsInvalid`; if context validation fails, propagate
its normal diagnostics without trying another context. Keep overlay selection
after source assembly and main version handling, and preserve existing
failures for missing files, invalid overlays, and repeated sources.
In `selected_overlay.go`, pass a lookup directory separately to the existing
selected overlay loader (for example on `selectedOverlayRequest`). Ordinary
`Load` and explicit layered contexts use the resolved root as before.
Derived and fallback contexts use the resolved base. Open access at the
containment root; join the lookup directory with `overlays/<name>.yml`
using the selected namespace's path operations, then use normal
canonicalization and containment. Keep the namespace, source identities,
version processing, limits, source contexts, and handle ownership on the
existing path. Do not use the lookup directory as a replacement root or
construct a new namespace.
Cover canonical host symlink targets, wider roots with distinct overlay
files at base and root, native filesystems, context on a non-file or absent
main, inactive overlay with an invalid fallback, explicit override with no
fallback on error, base-relative fallback, and missing selected overlays
without recovery. Retain ordinary and explicit root-relative regression
coverage, including existing namespace and repeated-source cases.
## Stage 6 — Exercise the combined workflow
**Status:** Complete. The executable consumer example and strict-peer regression pass.
Add a maintained executable example in `examples/consumers` for a single
layered load using `SearchConfig`, a mapping companion with optional named
files and recursive directory imports, an independently located embedded
tier, and one selected overlay. Use actual fixture files and the existing
`*.yml` and `*.yaml` pattern list; show destination placement, transitive
imports, and strict peers. Demonstrate precedence from embedded catalog
through operator underlay, main, and overlay. Include a registered identity
case for whole-record upsert across tiers and a same-tier duplicate failure.
No change to Audita or PromptKit is needed.
## Stage 7 — Close remaining integration and evidence gaps
**Status:** Complete. Focused discovery, source evidence, version, optional-resource, budget, and cleanup checks passed.
Fill remaining public integration gaps without repeating every engine test:
first and later search candidates, explicit file and mapping/bytes/reader
sources, only the file, only the directory, both, neither and an empty
directory, invalid or inaccessible selected fragments (use a deterministic
filesystem error where permissions are unreliable), and a selected main
failure that cannot resume search or use the fallback. Use focused fixtures
to check layer/index, physical paths, passive resource contexts, normal
source-version evidence, shared acquisition budgets and cleanup on failure.
Check that callbacks execute only in the existing lifecycle, and that a
reused description binds anew. Preserve existing ownership and detached
inspection checks. Do not require new global diagnostic ordering or test
incidental call counts.
## Stage 8 — Document and verify the completed sprint
**Status:** Complete. Focused integration and consumer checks, the everyday checker, the race suite, and changed local-link checks passed.
Update the public declaration comments and the discovery, layered loading,
imports, resource context and selected overlay sections of
[the configuration contract](../config.md). Document lookup versus
containment, canonical host symlink behavior, override and fallback
precedence, and absent-base behavior. Link the maintained examples from
consumer guidance. Update the [internal overview](../internal/overview.md)
and [import internals](../internal/imports.md) for the actual handoff,
contextless path, and ownership.
Record the deferred context decision in the next numbered ADR, extending
[ADR 0019](../adr/0019-explicit-configuration-discovery.md) without rewriting
it. Clarify in the architecture policy that the layered loader binds
application-selected dependent locations after main selection; applications
still own search conventions, and an explicit overlay selection is required.
Preserve existing scope boundaries. Use the documentation policy's canonical
owners rather than restating the full contract in every document.
Run the focused engine, facade, and consumer tests added or affected above,
then the everyday `./scripts/check.sh`. Run the race suite for the
load-local ownership and description reuse changes. Check changed Markdown
links, public facade boundaries and examples. Use the declared Go toolchain
and existing dependencies. No release or benchmark is part of completion.
Report validation and any remaining limitations; commit the completed sprint
under the agent policy.

379
docs/roadmap/main-relative-inputs.md
View File

@@ -1,379 +0,0 @@
# Main-relative companion imports and overlays
Status: implemented. The [implementation plan](implementation.md)
owns the concrete API spelling, execution order, and verification. Baseline: `191015e`.
## Purpose
Make the common workflow straightforward: discover the main configuration file,
load optional companion fragments beside it, and apply a selected native overlay
in one configuration load. Applications choose their conventions; operators can
drop resource definitions into the declared locations without adding import
boilerplate to the main file or creating a manifest on disk.
The central addition is deferred filesystem-context binding for application-owned
underlay inputs, including `Mapping` and `Bytes` sources containing native import
declarations. Bind their imports to the main input's resolved filesystem context
during loading: for file mains, lookup starts beside the selected main file.
Reuse existing optional-file, directory, recursion, pattern, destination, and
composition behavior. A new file constructor alone would not cover this workflow,
and a second fragment-discovery implementation is unnecessary.
## Prior baseline
The [discovery contract](../config.md#configuration-discovery) resolves
`SearchConfig` inside loading to one host file or absent input. Selected-file
failures do not resume discovery. The constructor returns a source description,
not a selected filename.
Ordinary `Load` already infers overlay context from a file source. Without a
root override, `/srv/campaign/audita.yml` supplies the location
`/srv/campaign/overlays/production.yml` for overlay `production`. Existing overlay
lookup uses the configuration root, which can differ from the main directory.
Optional discovery exhaustion provides no inferred context.
Non-file mains can already carry an explicit filesystem context. Absent inputs,
including optional discovery exhaustion, can also retain a supplied context;
for an absent host input, an explicit root supplies the base when `BaseDir` is
omitted. Reusing these validated contexts requires no filename inference.
The [layered API](../config.md#layered-loading) has the coordination gap:
- An active overlay requires `LayeredOptions.OverlayFilesystem`. Preflight
rejects its absence before discovery runs.
- Underlay sources and contexts are supplied independently. Application-owned
mappings or bytes can already contain import declarations, but cannot obtain
their import context from the main file that discovery will select.
- Native imports already support optional named files and recursive directory
searches with filename filtering and placement. They compose as peers of
their declaring input, so putting declarations in an underlay gives them
underlay priority. Putting the same declarations in the main file does not.
- Main acquisition already precedes underlay acquisition. Existing composition
subsequently applies underlay groups from lowest to highest beneath the main.
This feature extends context selection and binding within that lifecycle.
## Audita workflow
Audita supplies the companion import declarations in application-owned mappings
or bytes. Their filesystem bases follow the selected main file, whether selected
explicitly or through ConfigKit's built-in searches. For a main file at
`/srv/campaign/audita.yml`, running from `/tmp` must not relocate these companions:
| Resource | Application-declared locations | Role |
| --- | --- | --- |
| Glossaries | Optional `glossary.yml` and recursive `glossary/` | Underlay contributions mounted at `glossary_collections` using the appropriate existing destination and placement controls |
| Prompt definitions | Optional `prompts.yml` and recursive `prompts/` | Operator contributions above the bundled PromptKit catalog and Audita's embedded prompts |
| Selected native overlay | `overlays/<name>.yml` | The one named overlay applied after main composition |
Both resource-directory searches select `*.yml` and `*.yaml`. The file and
its corresponding directory contribute together if both exist; neither suppresses
the other. Each pair composes strictly as peers within its tier. Missing optional
resources contribute nothing; unreadable selected files, invalid definitions,
and composition conflicts remain errors.
Prompt overrides rely on existing tier and identity rules. The operator prompt
tier sits above embedded prompt tiers, allowing whole-record upsert by registered
prompt ID under the applicable collection policy. Two operator fragments defining
the same ID in that tier conflict; filename ordering does not choose a winner.
The consuming application selects each fragment's schema and placement.
This does not require combining Audita's and PromptKit's schemas. If PromptKit
uses a separate engine, its catalog load establishes the prompt tiers and can
use the selected main's recorded context after Audita's application load. That
is a separate domain load, not a preliminary load to repeat main discovery.
This feature's acceptance does not require changing another repository.
The overlay directory is not imported wholesale as an underlay: its files are
alternative named configurations. No selection means no overlay. Selecting a
name retains the existing missing-selected-overlay error. An aggregate
`overlays.yml` would need its own named-definition and conflict rules and is
outside this feature; it is not a second spelling for current overlay lookup.
## Targeted end state
### Companion underlays
The application declares a non-file underlay `Input` containing native imports
and selects main-derived filesystem binding for that input. `Mapping`, `Bytes`,
and `Reader` are supported sources for this use. OD-1 settles the public API
structure and receiving-input scope below; no new operator-facing YAML syntax
is needed.
Once the main context is resolved, ConfigKit inherits its base, namespace, and
containment root under OD-3 and OD-4. File mains supply their resolved directory;
non-file or absent mains may supply an explicit context. Existing import selection
then acquires the companion files. Each imported file uses its own directory for
further relative imports. An import of
`extra.yml` from `glossary/terms.yml` resolves to `glossary/extra.yml`.
Applications choose underlay groups and their order. A group composes strictly;
separate groups establish precedence. Existing destinations, sequence placement,
source-version policies, and load-wide import defaults still apply. Binding a
context does not itself enable imports or change collection operations.
The relationship is explicit on companion inputs. Independently located and
embedded inputs keep their own contexts and can coexist in the same load.
Providing a concrete context selects that complete context over derived
binding, without field-by-field fallback or recovery from an invalid explicit
context. OD-1 records the representation and precedence rules.
There is no requirement to add a dedicated main-relative file constructor.
Optional named-file acquisition is already expressible through the underlay's
native imports. Ordinary `File` paths do not silently gain new relative semantics.
### Layered overlays
For an active layered overlay, use an explicit `OverlayFilesystem` when supplied.
Otherwise derive the overlay context automatically from the main input's usable
resolved context; the common workflow requires no `OverlayFromMain` flag. An
invalid explicit context fails rather than falling back. Inactive overlays need no
context. Missing-main and fallback cases are addressed below and in OD-2.
For this newly derived layered context, lookup uses the main context's resolved
base, independently of a wider containment root. For file mains this is the
main file's directory. If the main is `/srv/campaign/audita.yml` and its permitted
root is `/srv`, the selected file
is `/srv/campaign/overlays/production.yml`, not `/srv/overlays/production.yml`.
Widening containment must not relocate the automatically selected companions.
Keep existing explicit-overlay-context lookup behavior and ordinary `Load`
behavior. The implementation must represent the derived lookup base separately
from its containment root where necessary, rather than simply copying today's
root-relative overlay context. Native overlay names, filenames, selection,
precedence, and validation retain their existing semantics.
### Absent main and optional companions
Optional discovery may find no main file while defaults and CLI settings still
permit a successful application load. An explicit context on that absent main
can still supply a usable base under OD-4. When no usable base is available, skip
optional companion import expansion while retaining the declaring input's
ordinary payload and validation. Imports requiring targets or a nonempty selection
still fail; an empty `path: []` retains its existing no-op behavior even when
required. Applications can independently require a base or supply an explicit
fallback context on each derived binding. OD-2 records the settled precedence,
validation timing, skip scope, and fallback behavior.
Do not infer a directory from unsuccessful candidates or silently use the working
directory. An application may explicitly choose the working directory as its
fallback. An active selected overlay still needs a usable context and the selected
file; optional companion imports do not make that overlay optional. Independent
underlays and explicitly located overlays retain their absent-main behavior.
No fallback applies to a failure reading or processing an actually selected main.
### Lifecycle, evidence, and compatibility
Bind dependent contexts after the main information is available and before
traversing their imports or loading the overlay. Preserve main-first acquisition,
underlay precedence, version processing, selected-overlay timing, and the existing
defaults, CLI, resolution, secret, and validation phases. No application context
callback, second discovery API, or preliminary full load is required.
Check static declarations during preflight where possible; defer checks requiring
the selected directory. Resolve and check fallback filesystem locations only when
selected under OD-2. Do not introduce a global ordering guarantee for unrelated
diagnostics or filesystem checks. Selected-file and companion acquisition errors
remain terminal rather than restarting discovery.
Application-supplied mappings or bytes remain ordinary input sources with their
own labels, evidence, and existing validation obligations. Binding a context does
not manufacture a main document, exempt these inputs from source-version policy,
or give them a fabricated physical file path. Imported files retain their actual
source identity, locations, precedence layer, underlay index, and passive resource
context. Preserve namespace identity, repetition scope, shared budgets, detached
inspection, and acquisition-handle cleanup.
Optional imports skipped because no base is available retain their declaring
source's normal evidence. Skipping does not manufacture acquired sources,
resource contexts, or evidence of completed filesystem searches. In particular,
do not add `EmptyImportSelections` records solely because expansion was skipped
for lack of a base. Preserve existing evidence for declarations that actually
complete; no new inspection API is needed for this feature.
Successful existing ordinary and explicitly configured layered loads should retain
their behavior. Automatically deriving a layered overlay context intentionally
relaxes the former missing-context error. Companion directory conventions are
selected by applications; ConfigKit does not universally scan `glossary/` or
`prompts/` without those declarations.
This work adds no general resource-path resolver, remote acquisition, mutation of
the process working directory, filesystem race-protection guarantee, or general
source-provider callback framework. These are scope boundaries, not new permanent
architectural exclusions. The design follows
[ADR 0019](../adr/0019-explicit-configuration-discovery.md). Review ADR and policy
follow-up for deferred context ownership without rewriting accepted decisions.
No release version or additional release gate is selected here.
## Settled Decisions
### OD-1: Public representation of deferred context binding — settled
Add a small, optional main-relative binding description to `Input`, retaining
the concrete `Input.Filesystem` field. A helper supplies the common default
description; explicit settings group the absent-main controls settled in OD-2.
Companion `Mapping` and `Bytes` inputs use native imports for fragment selection.
Preserve the input's source, placement, import enablement, and tier.
The proposed shape is an `Input.MainRelative` field of type
`*MainRelativeContext`, with a `MainRelative()` convenience helper. These names
illustrate the settled structure; final spelling can be finished during API
design. For example, this unimplemented API sketch assumes
`companionDeclarations` contains ordinary native import declarations:
```go
configkit.Input{
Source: configkit.Mapping("companions", companionDeclarations),
Imports: true,
MainRelative: configkit.MainRelative(),
}
```
A nil binding preserves existing independent context behavior. A non-nil binding
requests context derivation during loading. If `Input.Filesystem` is also supplied,
that complete concrete context wins. Do not merge fields or fall back to derivation
when the explicit context is invalid. This permits overriding reusable companion
setup without clearing its binding declaration.
An active binding is one selected in the absence of an overriding
`Input.Filesystem`. It applies to non-file underlay inputs, including `Mapping`,
`Bytes`, and `Reader`. An active binding on the main `Input` is an option error
because it would refer to itself. An active binding on a `File`,
`Filesystem.File`, or `SearchConfig` underlay is also an option error: these
sources determine their own file import bases. The concrete-context override
rule still applies before this active-binding scope check. Existing source
behavior is unchanged; express main-relative file acquisition through native
imports in a non-file underlay.
Layered overlays derive automatically when `OverlayFilesystem` is absent; no
additional opt-in flag is needed. An explicit overlay context wins as a whole.
Support an explicit overlay fallback under OD-2, reusing the small binding
settings where appropriate without requiring a general context-selection API.
Keep the binding on `Input`: `Source` still describes the supplied content and
`FilesystemContext` still describes a concrete location. This does not reinterpret
a file source's own path. Grouping related settings avoids scattering them across
a boolean and sibling fields, while keeping the addition smaller than a general
context selector. No source wrapper, deferred fields on `FilesystemContext`,
callback, or second file-selection API is needed for this design.
### OD-2: Scope and representation of absent-main handling — settled
Put absence behavior on each derived binding. When no usable base exists, the
default skips optional companion import expansion, retaining the declaring
input's ordinary configuration values. Imports requiring targets or a nonempty
selection produce an error; requiredness includes applicable application import
defaults. An empty `path: []` still succeeds even with `required: true`, because
there are no named targets to require. `RequireBase` independently requires a
usable context even for that empty declaration. Continue normal input and
directive validation, so absence does not excuse invalid declarations.
This preserves independent payload and existing required/optional distinctions
without treating the entire input as conditional on main presence.
Support a concrete fallback context and a require-base option within the binding
description. Illustrative fields are `Fallback *FilesystemContext` and
`RequireBase bool`; final spelling remains an API design detail. Select the
context in this order:
1. A supplied `Input.Filesystem` is the complete override.
2. Otherwise, use a usable main-derived context.
3. Otherwise, use the explicitly supplied fallback context, if any.
4. Without any usable context, skip optional imports unless the binding requires
a base. A require-base binding or an import requiring targets or a nonempty
selection produces an error; the empty named-path list remains a no-op.
Validate static binding declaration shape during preflight. Resolve and check
fallback filesystem locations only if the fallback is selected: an unused
fallback need not exist or be accessible. Preserve existing validation of
supplied concrete contexts. A selected invalid fallback produces its normal
error; lazy fallback checks do not provide recovery from invalid selected input.
When no overlay is selected, ignore its fallback settings.
A fallback can satisfy require-base: the option requires a usable context, not
an actual main file. Consumers requiring a main file can make discovery required.
Under OD-4, an explicit context on a non-file or absent main can also satisfy it;
the absence policy applies when no usable base exists, not merely when no main
content exists.
An invalid selected context fails; it does not activate a lower-priority choice.
Fallback applies only when the main cannot provide a base, never after a failure
reading or processing a selected main, or after missing or invalid companion
files beside an existing main. Once bound, normal import selection and failure
rules apply. Do not guess from unsuccessful discovery candidates or implicitly
use the working directory; applications may explicitly choose it as a fallback.
A fallback uses `FilesystemContext.BaseDir` as its lookup directory, with `Root`
and `FS` retaining their containment and namespace roles. This also applies to
the new overlay fallback: look for `overlays/<name>.yml` beneath its base, while
preserving existing explicit `OverlayFilesystem` lookup behavior. An explicit
overlay context wins, followed by main derivation, then the declared overlay
fallback. When no overlay is selected, no overlay context is needed. An active
selection requires a usable context and the named file; never silently skip it.
Companion absence settings do not make a selected overlay optional.
### OD-3: Containment inheritance and symlink convention — settled
Derived companion imports and derived layered overlays inherit the main input's
resolved filesystem namespace and containment root. Lookup uses its resolved
base under OD-4, which for file mains is the file's resolved directory, even when
its configured root is wider. Keep existing explicit overlay context and ordinary
`Load` lookup behavior.
For a host main at `/srv/campaign/audita.yml` with root `/srv`, companion
`glossary.yml` resolves to `/srv/campaign/glossary.yml`, while an import of
`../shared/terms.yml` may resolve to `/srv/shared/terms.yml` within that root.
Overlay `production` resolves to `/srv/campaign/overlays/production.yml`.
Without an explicit wider root, the host main's canonical directory remains
its default containment root. Preserve intentional wider roots rather than
establishing a fresh, narrower companion root. Parent-relative paths remain
permitted when existing containment rules allow them.
For host symlinks, use the canonical target directory. If
`/etc/audita/audita.yml` points to `/srv/releases/v3/audita.yml`, derived
companions and overlays are located under `/srv/releases/v3`. A subsequent load
follows the newly selected bundle if the symlink changes. Document that files
beside the link in `/etc/audita` are not selected by this derived convention.
This matches existing file/import bases; no separate symlink-base selector is
needed for this feature. Applications may supply concrete context overrides
under the normal rules when they choose another location or scope.
Each imported file continues to resolve further imports from its own directory,
within the inherited root. Existing import `follow` controls still govern
discovered symlink entries; they do not select the base of a symlinked main.
Preserve existing containment checks and traversal controls. For native main
contexts under OD-4, retain that filesystem's namespace and link-handling
capabilities without imposing host-filesystem guarantees on every `fs.FS`.
### OD-4: Main-source eligibility beyond discovered host files — settled
Derive from any usable, validated main filesystem context, regardless of the
main source's format. Document `MainRelative` as using the main input's resolved
filesystem context. For the common file workflow, this means lookup beside the
selected main file. Reuse existing context resolution rather than adding another
acquisition mechanism or requiring main imports to be enabled.
- Explicit host files and successful discovery supply the canonical file directory
and resolved containment root.
- `Filesystem.File` mains supply their directory and root within their existing
namespace. Dependent loading stays in that namespace.
- `Bytes`, `Reader`, and `Mapping` mains can supply an explicit context with a
usable base, root, and namespace under their existing validation rules.
- `NoSource` and optional discovery exhaustion can retain an explicit main
context. Reuse its resolved base: existing absent-host rules allow an explicit
root to serve as the base when `BaseDir` is omitted; native contexts retain
their existing requirements.
For example, a bytes main with host `BaseDir: /srv/campaign` and `Root: /srv`
can supply the same companion lookup and containment as a file main in that
directory with the same root. An absent main with a usable explicit context
can likewise supply companion and overlay lookup without creating main content.
Without a usable main context, apply OD-2's fallback, optional-import skipping,
or error behavior. Invalid supplied contexts fail rather than selecting a
fallback. Never infer filesystem locations from diagnostic labels or unsuccessful
search candidates, manufacture a main source, or change source evidence to make
a non-file input appear file-backed.
This preserves one context-reuse rule when applications change how they supply
main content. OD-1's concrete overrides and OD-3's containment, namespace, and
link-handling rules continue to apply. Existing ordinary `Load` and explicit
overlay context lookup semantics remain unchanged.

242
docs/roadmap/remediation.md
View File

@@ -1,242 +0,0 @@
# Main-relative companion implementation review
Reviewed the eight implementation commits from `ae0c94d` through `18af2d2`
against the [implementation plan](implementation.md) and
[feature roadmap](main-relative-inputs.md).
| Stage | Review result |
| --- | --- |
| 1 | The root facade copies binding and fallback descriptions. Preflight checks active binding scope and static context shape. Concrete overrides and inactive overlays take precedence over shadowed settings. |
| 2 | The acquired main's resolved context feeds underlay acquisition. Derived inputs retain the namespace and containment root, clear the main's physical path, and use receiving-tier host identity. Fallback selection and namespace anchors use the chosen context. |
| 3 | Bound traversal and contextless imports share declaration policy and placement preparation, with source-local diagnostic sites. |
| 4 | Without a base, the loader retains payload and version processing, honors requiredness and the empty named-path exception, and emits no fabricated selection evidence. Root assembly has a second implementation. See Finding 2. |
| 5 | Layered overlays select explicit, main-derived, then fallback context. Lookup base and containment root are separate. Native link handling leaves a gap in containment and duplicate detection. See Finding 1. |
| 6 | The executable example covers discovery, embedded defaults, mounted file and recursive directory imports, transitive imports, whole-record upsert, and the selected overlay. A companion test rejects duplicate peers. |
| 7 | Public and engine tests cover discovery, source and version evidence, optional resources, budgets, and cleanup. The exhaustion and lazy fallback branches need focused regression coverage. See Finding 3. |
| 8 | The configuration contract, public comments, architecture policy, and ADR describe deferred binding. Two supporting descriptions misstate ownership. See Finding 4. |
## Finding 1 — Native overlay symlinks bypass containment and duplicate checks
**Status** — Complete. Native overlay lookup resolves supported symlinks before containment and repeat checks. The overlay tests cover escapes, aliases in both directions, and a contained target.
**Priority P2.** The layered branch passes the inherited root and lookup base
to the shared overlay loader. That loader calls `filesystemAccess.canonical`.
For a native filesystem, the function stats the path and returns its lexical
spelling even when the filesystem implements `fs.ReadLinkFS`. The root and
duplicate checks then compare that spelling, while the subsequent read can
follow the link. The same limitation predates this sprint in the shared
loader. Automatic derivation makes it reachable without an explicit overlay
filesystem, and the new contract describes the inherited root as the
containment boundary. [Layered overlay selection](../../internal/engine/load_request.go#L192)
· [Overlay checks](../../internal/engine/selected_overlay.go#L74)
· [Native path lookup](../../internal/engine/filesystem_access.go#L94)
· [Overlay contract](../config.md#selected-overlays)
Reproduced through `LoadLayered` on Go 1.27.3 using
`NewFilesystem("native", os.DirFS(dir))`, which exposes `fs.ReadLinkFS`.
The main is `scope/main.yml` with a context rooted at `scope`. There is no
explicit overlay context.
- `scope/overlays/escape.yml` points to `../../outside.yml`. The load
succeeds and takes `count: 9` from outside the root.
- `scope/overlays/repeat.yml` points to `../main.yml`. The load succeeds
and reports the main content a second time under the overlay alias.
Apply capability-aware link resolution to native overlay targets, using the
existing native link resolver where applicable. Use the resolved path for
containment, repetition, and reading. Check the resolved identity against
already acquired files in the same namespace, including aliases where those
files retain lexical paths. Add focused derived-overlay regressions for the
escape and repeat cases. Preserve root-relative lookup for ordinary and
explicit overlays and document the limit for native filesystems that cannot
expose link targets. This requires no general sandbox guarantee for caller
filesystems. [Native link resolver](../../internal/engine/filesystem_links.go#L227)
## Finding 2 — Contextless imports duplicate root contribution assembly
**Status** — Complete. Bound and contextless imports share root charging, placement, and sequence-item scope assembly. The full test suite passed.
**Priority P3.** Stage 4 calls for sharing the root contribution assembly
and accounting. The new contextless path has its own source charging,
mapping placement, wrapper charging, pending sequence-item scope
construction, and placement reservation. Bound imports perform those same
steps independently. Both paths pass today’s tests, but changes to placement
or accounting now need to be made twice for the same declaring payload.
The initial limit diagnostics already differ in specificity.
[Contextless path](../../internal/engine/contextless_imports.go#L51)
· [Bound import assembly](../../internal/engine/import_acquisition.go#L115)
Share the preparation and finalization of the declaring root contribution
where the paths have the same semantics. Keep the contextless requiredness
decision and bound traversal separate. Preserve their existing evidence and
the contextless path's lack of filesystem access. Use existing mapping,
sequence-item, and budget checks to verify the extraction.
## Finding 3 — Exhaustion and lazy fallback lack focused regression coverage
**Status** — Complete. Public tests cover search exhaustion, root-only main context, and missing fallbacks when they win or are shadowed.
**Priority P3.** The context-selection test uses an existing fallback when
checking that main derivation wins. It would still pass if the loader
prematurely opened that fallback. The no-base tests use `NoSource()`, and
the discovery integration covers a successful selection or a selected
parse failure. These cases leave the optional-search exhaustion handoff to
both companion and overlay selection unprotected.
[Context selection tests](../../facade_main_relative_test.go#L39)
· [Discovery integration tests](../../facade_main_relative_integration_test.go#L12)
A disposable public API probe confirmed the current behavior. Exhausted
optional search without context retained the declaring payload with no empty
selection evidence. The same search with a root-only main context loaded a
companion and overlay despite a well-shaped but nonexistent fallback. With
the main context removed, the selected missing fallback failed. Add a small
public regression table for those outcomes. Include an overlay-only failure
case if the shared fixture would otherwise stop at the companion error
before reaching overlay selection.
## Finding 4 — Supporting documentation misstates binding ownership
**Status** — Complete. The consumer guide identifies Input.MainRelative, and the internal guide describes the receiving tier’s identity bookkeeping.
**Priority P3.** The consumer guide locates main-relative bindings in
`LayeredOptions`, but a companion binds through `Input.MainRelative`.
The import internals guide says passing the main context replaces tier-local
identity state. The engine preserves the receiving tier's bookkeeping and
sets the derived host token to that tier's token. Correct these descriptions
and leave the observable rules with the configuration contract.
[Consumer guide](../consumers/README.md#choosing-a-loading-entry-point)
· [Import internals](../internal/imports.md#discovery-and-acquisition)
· [Receiving tier](../../internal/engine/load_request.go#L276)
## Verification
The everyday checker and the full race suite passed on Linux with Go 1.27.3.
Disposable public API probes reproduced Finding 1 and checked the exhaustion
and fallback behavior in Finding 3. The review found no additional contract
strengthening beyond the native containment wording discussed in Finding 1.
## Integration follow-up
The follow-up reviewed the same implementation through its existing callers
and downstream consumers. Findings 1–4 remain open.
| Stage | Interaction reviewed |
| --- | --- |
| 1 | The public facade copies the new descriptions into each request. Existing concrete context checks, disabled import handling, and overlay name validation still run on their established paths. |
| 2 | Derived imports preserve the receiving tier's repeat detection and the native namespace token. A probe confirmed that a derived import and explicit file repeat within the same namespace, while separately constructed namespaces with the same label and path compose independently. Standalone host validation diverges from explicit context handling. See Finding 5. |
| 3 | Both traversal paths prepare declarations from the existing parsed specs. Placement diagnostics still use the local mapping scope for sequence items. |
| 4 | With optional imports and no base, the declaring source enters the established scope composition and version evidence paths. A probe with a mounted sequence item and required source version produced the same record and version evidence with or without an explicit base. Only the bound path reported a completed empty selection. |
| 5 | The selected overlay uses the existing loader, version protection, aggregate limits, and source context construction after main and underlay assembly. The inherited native link limitation is recorded in Finding 1. |
| 6 | The executable example exercises the handoff from discovery through embedded and operator tiers, identified record upsert, and overlay application using existing composition operations. |
| 7 | Focused regressions passed for main-relative loading, ordinary and layered overlays, source contexts, import defaults, source versions, and sequence-item placement. The host access mismatch calls for the regression in Finding 5. |
| 8 | The public contract requires a usable base for `RequireBase` and describes the established context checks. Finding 5 concerns code behavior under that contract. The ownership corrections from Finding 4 still apply. |
## Finding 5 — Derived host roots skip standalone access validation
**Status** — Complete. Derived and explicit standalone contexts share the host root open and close checks. The permission regression passed for both paths.
**Priority P2.** When a companion binds to the main context with imports
disabled, `acquireStandaloneInput` checks the resolved directories with
`Stat`. An equivalent explicit context goes through
`validateFilesystemContext`, which also opens and closes the host root.
The derived path can therefore accept a directory that the existing
validation rejects. It publishes a passive `SourceContext` whose host
filesystem immediately fails to open that same root.
[Derived and explicit validation branches](../../internal/engine/acquisition.go#L139)
· [Existing context validator](../../internal/engine/filesystem.go#L10)
· [Passive host filesystem](../../internal/engine/resource_context.go#L77)
A public API probe reproduced the difference as an unprivileged user on
Linux with Go 1.27.3. Put a readable `main.yml` and `resource.txt` in a
directory with owner search permission and no read permission (mode
`0100`). The main file can be read by name and its directory can be
statted, but `os.OpenRoot` returns permission denied. An imports-disabled
companion with `MainRelativeContext{RequireBase: true}` loads and returns
a source context, then `fs.ReadFile` through that context fails. The
same companion with an explicit context at that directory fails during
loading with `import.permission`. The derived case also fails with
`import.permission` when imports are enabled.
Share the existing resolved-context validation between derived and
explicit standalone acquisition, including the host root open and close
checks. Keep the native directory checks and existing error handling.
An access failure in the selected context should remain terminal.
Add a regression at the derived versus explicit boundary, using a
deterministic root-open failure or a permission fixture that verifies
the platform actually denies the open. The check concerns root access
during loading. Later resource operations retain their existing filesystem
semantics. [Layered loading contract](../config.md#layered-loading)
## Algorithm audit
The audit covers the implementation from `ae0c94d` through `18af2d2`.
It analyzes the work and storage implied by the code. The planning decisions
on precedence, selection, import traversal, and context ownership remain the
basis for the analysis.
| Stage | Work and storage reviewed |
| --- | --- |
| 1 | Facade conversion and binding preflight visit each input once, with work proportional to the option data copied or validated. The existing diagnostic sort orders failures after collection. |
| 2 | Binding and namespace anchor selection use a fixed number of linear passes. The temporary context table and per-input resolved copies are avoidable. See Finding 7. |
| 3 | Declaration preparation visits each declaration once, but resolving a shared pattern default copies its entries for each inheriting directory declaration. See Finding 6. |
| 4 | The no-base path prepares declarations and scans requiredness before assembling one payload. The pattern copies contribute a multiplicative cost even though no traversal runs. Root assembly duplication is already tracked in Finding 2. |
| 5 | Context selection considers at most three choices, and the shared overlay loader makes one linear pass over acquired sources for duplicate detection. The native path correctness issue remains in Finding 1. |
| 6 | The example uses one layered load and the existing declaration, import, and tier assembly paths. It adds no separate scan for companion files. |
| 7 | Source budgeting and evidence reuse the existing load state. The focused suite passed for the new loading paths and their import, overlay, and sequence-item dependencies. |
| 8 | The documented single discovery and deferred selection match the call path reviewed above. The identified improvements concern private storage and work, with no change to the selected algorithms or public behavior. |
## Finding 6 — Skipped directory imports copy shared patterns per declaration
**Status** — Complete. Inherited directory policies borrow the read-only load snapshot. Import selection and no-base tests passed.
**Priority P3.** With no base, `acquireContextlessInput` calls
`prepareImportDeclarations`, which holds one full resolved policy per
declaration. Each directory declaration that inherits the load-wide
`ImportDefaults.Pattern` allocates a separate copy of the pattern slice.
The no-base path reads the requiredness from those policies and never uses
the copied patterns.
[No-base preparation](../../internal/engine/contextless_imports.go#L22)
· [Policy slice](../../internal/engine/import_acquisition.go#L245)
· [Default pattern copy](../../internal/engine/import_spec.go#L122)
For **D** directory declarations that inherit **P** patterns, this adds
Θ(D × P) string-entry copies in time and peak temporary storage for a
single input. It becomes quadratic when both counts grow together. The
pattern text itself is shared. The work is separate from validating the
default patterns once and from matching filenames on the bound path.
Keep the load-owned default pattern snapshot and borrow its slice read-only
when resolving an inherited pattern. The selectors already read these
entries without modifying them. This reduces pattern storage and copying
to Θ(D + P) for the policy headers and shared entries. A separate option
is to share the context-independent validation and have the no-base path
retain only the policy fields it actually uses. Either approach should
preserve explicit empty pattern overrides, validation of skipped
declarations, and existing diagnostic sites.
[Default snapshot](../../internal/engine/import_spec.go#L72)
· [Filename filtering](../../internal/engine/import_selection.go#L230)
## Finding 7 — Each derived companion keeps a copy of the same tier context
**Status** — Complete. Each tier creates at most one shared derived context and has no per-input context table. The development check and full race suite passed.
**Priority P3.** `acquireUnderlayGroupScopes` allocates a context pointer
slot for every input, even in tiers with no bindings. Each derived input
then gets an addressable copy of the same main context with the same
cleared main path and receiving-tier host token. The separate copies
carry no input-specific values.
[Context preparation](../../internal/engine/load_request.go#L276)
For a tier with **U** inputs and **C** derived bindings, this adds Θ(U)
temporary pointer slots and Θ(C) context values. The passes themselves
are linear. Build at most one read-only derived context per tier and
select it during the existing ordered anchor and acquisition passes.
Use the private input's selected concrete context for fallbacks. This
reduces the extra context bookkeeping to O(1), excluding the existing
input list and native namespace map. Preserve per-input validation and
handle ownership, as well as first-input namespace anchors. Share the
context description without caching filesystem observations.
Complexity was derived from the loops and copies above. Focused Go 1.27.3
tests passed for the companion workflow, layered loading, overlays,
import policy and selection, and sequence-item placement.