configkit/docs/roadmap/main-relative-inputs.md

Main-relative companion imports and overlays

Status: implemented. The implementation plan owns the concrete API spelling, execution order, and verification. Baseline: 191015e.

Purpose

Make the common workflow straightforward: discover the main configuration file, load optional companion fragments beside it, and apply a selected native overlay in one configuration load. Applications choose their conventions; operators can drop resource definitions into the declared locations without adding import boilerplate to the main file or creating a manifest on disk.

The central addition is deferred filesystem-context binding for application-owned underlay inputs, including Mapping and Bytes sources containing native import declarations. Bind their imports to the main input's resolved filesystem context during loading: for file mains, lookup starts beside the selected main file. Reuse existing optional-file, directory, recursion, pattern, destination, and composition behavior. A new file constructor alone would not cover this workflow, and a second fragment-discovery implementation is unnecessary.

Prior baseline

The discovery contract resolves SearchConfig inside loading to one host file or absent input. Selected-file failures do not resume discovery. The constructor returns a source description, not a selected filename.

Ordinary Load already infers overlay context from a file source. Without a root override, /srv/campaign/audita.yml supplies the location /srv/campaign/overlays/production.yml for overlay production. Existing overlay lookup uses the configuration root, which can differ from the main directory. Optional discovery exhaustion provides no inferred context.

Non-file mains can already carry an explicit filesystem context. Absent inputs, including optional discovery exhaustion, can also retain a supplied context; for an absent host input, an explicit root supplies the base when BaseDir is omitted. Reusing these validated contexts requires no filename inference.

The layered API has the coordination gap:

• An active overlay requires LayeredOptions.OverlayFilesystem. Preflight rejects its absence before discovery runs.
• Underlay sources and contexts are supplied independently. Application-owned mappings or bytes can already contain import declarations, but cannot obtain their import context from the main file that discovery will select.
• Native imports already support optional named files and recursive directory searches with filename filtering and placement. They compose as peers of their declaring input, so putting declarations in an underlay gives them underlay priority. Putting the same declarations in the main file does not.
• Main acquisition already precedes underlay acquisition. Existing composition subsequently applies underlay groups from lowest to highest beneath the main.

This feature extends context selection and binding within that lifecycle.

Audita workflow

Audita supplies the companion import declarations in application-owned mappings or bytes. Their filesystem bases follow the selected main file, whether selected explicitly or through ConfigKit's built-in searches. For a main file at /srv/campaign/audita.yml, running from /tmp must not relocate these companions:

Resource	Application-declared locations	Role
Glossaries	Optional glossary.yml and recursive glossary/	Underlay contributions mounted at glossary_collections using the appropriate existing destination and placement controls
Prompt definitions	Optional prompts.yml and recursive prompts/	Operator contributions above the bundled PromptKit catalog and Audita's embedded prompts
Selected native overlay	overlays/<name>.yml	The one named overlay applied after main composition

Both resource-directory searches select *.yml and *.yaml. The file and its corresponding directory contribute together if both exist; neither suppresses the other. Each pair composes strictly as peers within its tier. Missing optional resources contribute nothing; unreadable selected files, invalid definitions, and composition conflicts remain errors.

Prompt overrides rely on existing tier and identity rules. The operator prompt tier sits above embedded prompt tiers, allowing whole-record upsert by registered prompt ID under the applicable collection policy. Two operator fragments defining the same ID in that tier conflict; filename ordering does not choose a winner. The consuming application selects each fragment's schema and placement.

This does not require combining Audita's and PromptKit's schemas. If PromptKit uses a separate engine, its catalog load establishes the prompt tiers and can use the selected main's recorded context after Audita's application load. That is a separate domain load, not a preliminary load to repeat main discovery. This feature's acceptance does not require changing another repository.

The overlay directory is not imported wholesale as an underlay: its files are alternative named configurations. No selection means no overlay. Selecting a name retains the existing missing-selected-overlay error. An aggregate overlays.yml would need its own named-definition and conflict rules and is outside this feature; it is not a second spelling for current overlay lookup.

Targeted end state

Companion underlays

The application declares a non-file underlay Input containing native imports and selects main-derived filesystem binding for that input. Mapping, Bytes, and Reader are supported sources for this use. OD-1 settles the public API structure and receiving-input scope below; no new operator-facing YAML syntax is needed.

Once the main context is resolved, ConfigKit inherits its base, namespace, and containment root under OD-3 and OD-4. File mains supply their resolved directory; non-file or absent mains may supply an explicit context. Existing import selection then acquires the companion files. Each imported file uses its own directory for further relative imports. An import of extra.yml from glossary/terms.yml resolves to glossary/extra.yml.

Applications choose underlay groups and their order. A group composes strictly; separate groups establish precedence. Existing destinations, sequence placement, source-version policies, and load-wide import defaults still apply. Binding a context does not itself enable imports or change collection operations.

The relationship is explicit on companion inputs. Independently located and embedded inputs keep their own contexts and can coexist in the same load. Providing a concrete context selects that complete context over derived binding, without field-by-field fallback or recovery from an invalid explicit context. OD-1 records the representation and precedence rules.

There is no requirement to add a dedicated main-relative file constructor. Optional named-file acquisition is already expressible through the underlay's native imports. Ordinary File paths do not silently gain new relative semantics.

Layered overlays

For an active layered overlay, use an explicit OverlayFilesystem when supplied. Otherwise derive the overlay context automatically from the main input's usable resolved context; the common workflow requires no OverlayFromMain flag. An invalid explicit context fails rather than falling back. Inactive overlays need no context. Missing-main and fallback cases are addressed below and in OD-2.

For this newly derived layered context, lookup uses the main context's resolved base, independently of a wider containment root. For file mains this is the main file's directory. If the main is /srv/campaign/audita.yml and its permitted root is /srv, the selected file is /srv/campaign/overlays/production.yml, not /srv/overlays/production.yml. Widening containment must not relocate the automatically selected companions.

Keep existing explicit-overlay-context lookup behavior and ordinary Load behavior. The implementation must represent the derived lookup base separately from its containment root where necessary, rather than simply copying today's root-relative overlay context. Native overlay names, filenames, selection, precedence, and validation retain their existing semantics.

Absent main and optional companions

Optional discovery may find no main file while defaults and CLI settings still permit a successful application load. An explicit context on that absent main can still supply a usable base under OD-4. When no usable base is available, skip optional companion import expansion while retaining the declaring input's ordinary payload and validation. Imports requiring targets or a nonempty selection still fail; an empty path: [] retains its existing no-op behavior even when required. Applications can independently require a base or supply an explicit fallback context on each derived binding. OD-2 records the settled precedence, validation timing, skip scope, and fallback behavior.

Do not infer a directory from unsuccessful candidates or silently use the working directory. An application may explicitly choose the working directory as its fallback. An active selected overlay still needs a usable context and the selected file; optional companion imports do not make that overlay optional. Independent underlays and explicitly located overlays retain their absent-main behavior. No fallback applies to a failure reading or processing an actually selected main.

Lifecycle, evidence, and compatibility

Bind dependent contexts after the main information is available and before traversing their imports or loading the overlay. Preserve main-first acquisition, underlay precedence, version processing, selected-overlay timing, and the existing defaults, CLI, resolution, secret, and validation phases. No application context callback, second discovery API, or preliminary full load is required.

Check static declarations during preflight where possible; defer checks requiring the selected directory. Resolve and check fallback filesystem locations only when selected under OD-2. Do not introduce a global ordering guarantee for unrelated diagnostics or filesystem checks. Selected-file and companion acquisition errors remain terminal rather than restarting discovery.

Application-supplied mappings or bytes remain ordinary input sources with their own labels, evidence, and existing validation obligations. Binding a context does not manufacture a main document, exempt these inputs from source-version policy, or give them a fabricated physical file path. Imported files retain their actual source identity, locations, precedence layer, underlay index, and passive resource context. Preserve namespace identity, repetition scope, shared budgets, detached inspection, and acquisition-handle cleanup.

Optional imports skipped because no base is available retain their declaring source's normal evidence. Skipping does not manufacture acquired sources, resource contexts, or evidence of completed filesystem searches. In particular, do not add EmptyImportSelections records solely because expansion was skipped for lack of a base. Preserve existing evidence for declarations that actually complete; no new inspection API is needed for this feature.

Successful existing ordinary and explicitly configured layered loads should retain their behavior. Automatically deriving a layered overlay context intentionally relaxes the former missing-context error. Companion directory conventions are selected by applications; ConfigKit does not universally scan glossary/ or prompts/ without those declarations.

This work adds no general resource-path resolver, remote acquisition, mutation of the process working directory, filesystem race-protection guarantee, or general source-provider callback framework. These are scope boundaries, not new permanent architectural exclusions. The design follows ADR 0019. Review ADR and policy follow-up for deferred context ownership without rewriting accepted decisions. No release version or additional release gate is selected here.

Settled Decisions

OD-1: Public representation of deferred context binding — settled

Add a small, optional main-relative binding description to Input, retaining the concrete Input.Filesystem field. A helper supplies the common default description; explicit settings group the absent-main controls settled in OD-2. Companion Mapping and Bytes inputs use native imports for fragment selection. Preserve the input's source, placement, import enablement, and tier.

The proposed shape is an Input.MainRelative field of type *MainRelativeContext, with a MainRelative() convenience helper. These names illustrate the settled structure; final spelling can be finished during API design. For example, this unimplemented API sketch assumes companionDeclarations contains ordinary native import declarations:

configkit.Input{
    Source:       configkit.Mapping("companions", companionDeclarations),
    Imports:      true,
    MainRelative: configkit.MainRelative(),
}

A nil binding preserves existing independent context behavior. A non-nil binding requests context derivation during loading. If Input.Filesystem is also supplied, that complete concrete context wins. Do not merge fields or fall back to derivation when the explicit context is invalid. This permits overriding reusable companion setup without clearing its binding declaration.

An active binding is one selected in the absence of an overriding Input.Filesystem. It applies to non-file underlay inputs, including Mapping, Bytes, and Reader. An active binding on the main Input is an option error because it would refer to itself. An active binding on a File, Filesystem.File, or SearchConfig underlay is also an option error: these sources determine their own file import bases. The concrete-context override rule still applies before this active-binding scope check. Existing source behavior is unchanged; express main-relative file acquisition through native imports in a non-file underlay.

Layered overlays derive automatically when OverlayFilesystem is absent; no additional opt-in flag is needed. An explicit overlay context wins as a whole. Support an explicit overlay fallback under OD-2, reusing the small binding settings where appropriate without requiring a general context-selection API.

Keep the binding on Input: Source still describes the supplied content and FilesystemContext still describes a concrete location. This does not reinterpret a file source's own path. Grouping related settings avoids scattering them across a boolean and sibling fields, while keeping the addition smaller than a general context selector. No source wrapper, deferred fields on FilesystemContext, callback, or second file-selection API is needed for this design.

OD-2: Scope and representation of absent-main handling — settled

Put absence behavior on each derived binding. When no usable base exists, the default skips optional companion import expansion, retaining the declaring input's ordinary configuration values. Imports requiring targets or a nonempty selection produce an error; requiredness includes applicable application import defaults. An empty path: [] still succeeds even with required: true, because there are no named targets to require. RequireBase independently requires a usable context even for that empty declaration. Continue normal input and directive validation, so absence does not excuse invalid declarations. This preserves independent payload and existing required/optional distinctions without treating the entire input as conditional on main presence.

Support a concrete fallback context and a require-base option within the binding description. Illustrative fields are Fallback *FilesystemContext and RequireBase bool; final spelling remains an API design detail. Select the context in this order:

1. A supplied Input.Filesystem is the complete override.
2. Otherwise, use a usable main-derived context.
3. Otherwise, use the explicitly supplied fallback context, if any.
4. Without any usable context, skip optional imports unless the binding requires a base. A require-base binding or an import requiring targets or a nonempty selection produces an error; the empty named-path list remains a no-op.

Validate static binding declaration shape during preflight. Resolve and check fallback filesystem locations only if the fallback is selected: an unused fallback need not exist or be accessible. Preserve existing validation of supplied concrete contexts. A selected invalid fallback produces its normal error; lazy fallback checks do not provide recovery from invalid selected input. When no overlay is selected, ignore its fallback settings.

A fallback can satisfy require-base: the option requires a usable context, not an actual main file. Consumers requiring a main file can make discovery required. Under OD-4, an explicit context on a non-file or absent main can also satisfy it; the absence policy applies when no usable base exists, not merely when no main content exists. An invalid selected context fails; it does not activate a lower-priority choice. Fallback applies only when the main cannot provide a base, never after a failure reading or processing a selected main, or after missing or invalid companion files beside an existing main. Once bound, normal import selection and failure rules apply. Do not guess from unsuccessful discovery candidates or implicitly use the working directory; applications may explicitly choose it as a fallback.

A fallback uses FilesystemContext.BaseDir as its lookup directory, with Root and FS retaining their containment and namespace roles. This also applies to the new overlay fallback: look for overlays/<name>.yml beneath its base, while preserving existing explicit OverlayFilesystem lookup behavior. An explicit overlay context wins, followed by main derivation, then the declared overlay fallback. When no overlay is selected, no overlay context is needed. An active selection requires a usable context and the named file; never silently skip it. Companion absence settings do not make a selected overlay optional.

OD-3: Containment inheritance and symlink convention — settled

Derived companion imports and derived layered overlays inherit the main input's resolved filesystem namespace and containment root. Lookup uses its resolved base under OD-4, which for file mains is the file's resolved directory, even when its configured root is wider. Keep existing explicit overlay context and ordinary Load lookup behavior.

For a host main at /srv/campaign/audita.yml with root /srv, companion glossary.yml resolves to /srv/campaign/glossary.yml, while an import of ../shared/terms.yml may resolve to /srv/shared/terms.yml within that root. Overlay production resolves to /srv/campaign/overlays/production.yml. Without an explicit wider root, the host main's canonical directory remains its default containment root. Preserve intentional wider roots rather than establishing a fresh, narrower companion root. Parent-relative paths remain permitted when existing containment rules allow them.

For host symlinks, use the canonical target directory. If /etc/audita/audita.yml points to /srv/releases/v3/audita.yml, derived companions and overlays are located under /srv/releases/v3. A subsequent load follows the newly selected bundle if the symlink changes. Document that files beside the link in /etc/audita are not selected by this derived convention. This matches existing file/import bases; no separate symlink-base selector is needed for this feature. Applications may supply concrete context overrides under the normal rules when they choose another location or scope.

Each imported file continues to resolve further imports from its own directory, within the inherited root. Existing import follow controls still govern discovered symlink entries; they do not select the base of a symlinked main. Preserve existing containment checks and traversal controls. For native main contexts under OD-4, retain that filesystem's namespace and link-handling capabilities without imposing host-filesystem guarantees on every fs.FS.

OD-4: Main-source eligibility beyond discovered host files — settled

Derive from any usable, validated main filesystem context, regardless of the main source's format. Document MainRelative as using the main input's resolved filesystem context. For the common file workflow, this means lookup beside the selected main file. Reuse existing context resolution rather than adding another acquisition mechanism or requiring main imports to be enabled.

• Explicit host files and successful discovery supply the canonical file directory and resolved containment root.
• Filesystem.File mains supply their directory and root within their existing namespace. Dependent loading stays in that namespace.
• Bytes, Reader, and Mapping mains can supply an explicit context with a usable base, root, and namespace under their existing validation rules.
• NoSource and optional discovery exhaustion can retain an explicit main context. Reuse its resolved base: existing absent-host rules allow an explicit root to serve as the base when BaseDir is omitted; native contexts retain their existing requirements.

For example, a bytes main with host BaseDir: /srv/campaign and Root: /srv can supply the same companion lookup and containment as a file main in that directory with the same root. An absent main with a usable explicit context can likewise supply companion and overlay lookup without creating main content.

Without a usable main context, apply OD-2's fallback, optional-import skipping, or error behavior. Invalid supplied contexts fail rather than selecting a fallback. Never infer filesystem locations from diagnostic labels or unsuccessful search candidates, manufacture a main source, or change source evidence to make a non-file input appear file-backed.

This preserves one context-reuse rule when applications change how they supply main content. OD-1's concrete overrides and OD-3's containment, namespace, and link-handling rules continue to apply. Existing ordinary Load and explicit overlay context lookup semantics remain unchanged.