configkit/docs/roadmap/main-relative-inputs.md

# Main-relative companion imports and overlays

Status: design settled; not implemented. The [implementation plan](implementation.md)
owns the concrete API spelling, execution order, and verification. Baseline: `191015e`.

## Purpose

Make the common workflow straightforward: discover the main configuration file,
load optional companion fragments beside it, and apply a selected native overlay
in one configuration load. Applications choose their conventions; operators can
drop resource definitions into the declared locations without adding import
boilerplate to the main file or creating a manifest on disk.

The central addition is deferred filesystem-context binding for application-owned
underlay inputs, including `Mapping` and `Bytes` sources containing native import
declarations. Bind their imports to the main input's resolved filesystem context
during loading: for file mains, lookup starts beside the selected main file.
Reuse existing optional-file, directory, recursion, pattern, destination, and
composition behavior. A new file constructor alone would not cover this workflow,
and a second fragment-discovery implementation is unnecessary.

## Current baseline

The [discovery contract](../config.md#configuration-discovery) resolves
`SearchConfig` inside loading to one host file or absent input. Selected-file
failures do not resume discovery. The constructor returns a source description,
not a selected filename.

Ordinary `Load` already infers overlay context from a file source. Without a
root override, `/srv/campaign/audita.yml` supplies the location
`/srv/campaign/overlays/production.yml` for overlay `production`. Existing overlay
lookup uses the configuration root, which can differ from the main directory.
Optional discovery exhaustion provides no inferred context.

Non-file mains can already carry an explicit filesystem context. Absent inputs,
including optional discovery exhaustion, can also retain a supplied context;
for an absent host input, an explicit root supplies the base when `BaseDir` is
omitted. Reusing these validated contexts requires no filename inference.

The [layered API](../config.md#layered-loading) has the coordination gap:

- An active overlay requires `LayeredOptions.OverlayFilesystem`. Preflight
  rejects its absence before discovery runs.
- Underlay sources and contexts are supplied independently. Application-owned
  mappings or bytes can already contain import declarations, but cannot obtain
  their import context from the main file that discovery will select.
- Native imports already support optional named files and recursive directory
  searches with filename filtering and placement. They compose as peers of
  their declaring input, so putting declarations in an underlay gives them
  underlay priority. Putting the same declarations in the main file does not.
- Main acquisition already precedes underlay acquisition. Existing composition
  subsequently applies underlay groups from lowest to highest beneath the main.

This feature extends context selection and binding within that lifecycle.

## Audita workflow

Audita supplies the companion import declarations in application-owned mappings
or bytes. Their filesystem bases follow the selected main file, whether selected
explicitly or through ConfigKit's built-in searches. For a main file at
`/srv/campaign/audita.yml`, running from `/tmp` must not relocate these companions:

| Resource | Application-declared locations | Role |
| --- | --- | --- |
| Glossaries | Optional `glossary.yml` and recursive `glossary/` | Underlay contributions mounted at `glossary_collections` using the appropriate existing destination and placement controls |
| Prompt definitions | Optional `prompts.yml` and recursive `prompts/` | Operator contributions above the bundled PromptKit catalog and Audita's embedded prompts |
| Selected native overlay | `overlays/<name>.yml` | The one named overlay applied after main composition |

Both resource-directory searches select `*.yml` and `*.yaml`. The file and
its corresponding directory contribute together if both exist; neither suppresses
the other. Each pair composes strictly as peers within its tier. Missing optional
resources contribute nothing; unreadable selected files, invalid definitions,
and composition conflicts remain errors.

Prompt overrides rely on existing tier and identity rules. The operator prompt
tier sits above embedded prompt tiers, allowing whole-record upsert by registered
prompt ID under the applicable collection policy. Two operator fragments defining
the same ID in that tier conflict; filename ordering does not choose a winner.
The consuming application selects each fragment's schema and placement.

This does not require combining Audita's and PromptKit's schemas. If PromptKit
uses a separate engine, its catalog load establishes the prompt tiers and can
use the selected main's recorded context after Audita's application load. That
is a separate domain load, not a preliminary load to repeat main discovery.
This feature's acceptance does not require changing another repository.

The overlay directory is not imported wholesale as an underlay: its files are
alternative named configurations. No selection means no overlay. Selecting a
name retains the existing missing-selected-overlay error. An aggregate
`overlays.yml` would need its own named-definition and conflict rules and is
outside this feature; it is not a second spelling for current overlay lookup.

## Targeted end state

### Companion underlays

The application declares a non-file underlay `Input` containing native imports
and selects main-derived filesystem binding for that input. `Mapping`, `Bytes`,
and `Reader` are supported sources for this use. OD-1 settles the public API
structure and receiving-input scope below; no new operator-facing YAML syntax
is needed.

Once the main context is resolved, ConfigKit inherits its base, namespace, and
containment root under OD-3 and OD-4. File mains supply their resolved directory;
non-file or absent mains may supply an explicit context. Existing import selection
then acquires the companion files. Each imported file uses its own directory for
further relative imports. An import of
`extra.yml` from `glossary/terms.yml` resolves to `glossary/extra.yml`.

Applications choose underlay groups and their order. A group composes strictly;
separate groups establish precedence. Existing destinations, sequence placement,
source-version policies, and load-wide import defaults still apply. Binding a
context does not itself enable imports or change collection operations.

The relationship is explicit on companion inputs. Independently located and
embedded inputs keep their own contexts and can coexist in the same load.
Providing a concrete context selects that complete context over derived
binding, without field-by-field fallback or recovery from an invalid explicit
context. OD-1 records the representation and precedence rules.

There is no requirement to add a dedicated main-relative file constructor.
Optional named-file acquisition is already expressible through the underlay's
native imports. Ordinary `File` paths do not silently gain new relative semantics.

### Layered overlays

For an active layered overlay, use an explicit `OverlayFilesystem` when supplied.
Otherwise derive the overlay context automatically from the main input's usable
resolved context; the common workflow requires no `OverlayFromMain` flag. An
invalid explicit context fails rather than falling back. Inactive overlays need no
context. Missing-main and fallback cases are addressed below and in OD-2.

For this newly derived layered context, lookup uses the main context's resolved
base, independently of a wider containment root. For file mains this is the
main file's directory. If the main is `/srv/campaign/audita.yml` and its permitted
root is `/srv`, the selected file
is `/srv/campaign/overlays/production.yml`, not `/srv/overlays/production.yml`.
Widening containment must not relocate the automatically selected companions.

Keep existing explicit-overlay-context lookup behavior and ordinary `Load`
behavior. The implementation must represent the derived lookup base separately
from its containment root where necessary, rather than simply copying today's
root-relative overlay context. Native overlay names, filenames, selection,
precedence, and validation retain their existing semantics.

### Absent main and optional companions

Optional discovery may find no main file while defaults and CLI settings still
permit a successful application load. An explicit context on that absent main
can still supply a usable base under OD-4. When no usable base is available, skip
optional companion import expansion while retaining the declaring input's
ordinary payload and validation. Imports requiring targets or a nonempty selection
still fail; an empty `path: []` retains its existing no-op behavior even when
required. Applications can independently require a base or supply an explicit
fallback context on each derived binding. OD-2 records the settled precedence,
validation timing, skip scope, and fallback behavior.

Do not infer a directory from unsuccessful candidates or silently use the working
directory. An application may explicitly choose the working directory as its
fallback. An active selected overlay still needs a usable context and the selected
file; optional companion imports do not make that overlay optional. Independent
underlays and explicitly located overlays retain their absent-main behavior.
No fallback applies to a failure reading or processing an actually selected main.

### Lifecycle, evidence, and compatibility

Bind dependent contexts after the main information is available and before
traversing their imports or loading the overlay. Preserve main-first acquisition,
underlay precedence, version processing, selected-overlay timing, and the existing
defaults, CLI, resolution, secret, and validation phases. No application context
callback, second discovery API, or preliminary full load is required.

Check static declarations during preflight where possible; defer checks requiring
the selected directory. Resolve and check fallback filesystem locations only when
selected under OD-2. Do not introduce a global ordering guarantee for unrelated
diagnostics or filesystem checks. Selected-file and companion acquisition errors
remain terminal rather than restarting discovery.

Application-supplied mappings or bytes remain ordinary input sources with their
own labels, evidence, and existing validation obligations. Binding a context does
not manufacture a main document, exempt these inputs from source-version policy,
or give them a fabricated physical file path. Imported files retain their actual
source identity, locations, precedence layer, underlay index, and passive resource
context. Preserve namespace identity, repetition scope, shared budgets, detached
inspection, and acquisition-handle cleanup.

Optional imports skipped because no base is available retain their declaring
source's normal evidence. Skipping does not manufacture acquired sources,
resource contexts, or evidence of completed filesystem searches. In particular,
do not add `EmptyImportSelections` records solely because expansion was skipped
for lack of a base. Preserve existing evidence for declarations that actually
complete; no new inspection API is needed for this feature.

Successful existing ordinary and explicitly configured layered loads should retain
their behavior. Automatically deriving a layered overlay context intentionally
relaxes the former missing-context error. Companion directory conventions are
selected by applications; ConfigKit does not universally scan `glossary/` or
`prompts/` without those declarations.

This work adds no general resource-path resolver, remote acquisition, mutation of
the process working directory, filesystem race-protection guarantee, or general
source-provider callback framework. These are scope boundaries, not new permanent
architectural exclusions. The design follows
[ADR 0019](../adr/0019-explicit-configuration-discovery.md). Review ADR and policy
follow-up for deferred context ownership without rewriting accepted decisions.
No release version or additional release gate is selected here.

## Settled Decisions

### OD-1: Public representation of deferred context binding — settled

Add a small, optional main-relative binding description to `Input`, retaining
the concrete `Input.Filesystem` field. A helper supplies the common default
description; explicit settings group the absent-main controls settled in OD-2.
Companion `Mapping` and `Bytes` inputs use native imports for fragment selection.
Preserve the input's source, placement, import enablement, and tier.

The proposed shape is an `Input.MainRelative` field of type
`*MainRelativeContext`, with a `MainRelative()` convenience helper. These names
illustrate the settled structure; final spelling can be finished during API
design. For example, this unimplemented API sketch assumes
`companionDeclarations` contains ordinary native import declarations:

```go
configkit.Input{
    Source:       configkit.Mapping("companions", companionDeclarations),
    Imports:      true,
    MainRelative: configkit.MainRelative(),
}
```

A nil binding preserves existing independent context behavior. A non-nil binding
requests context derivation during loading. If `Input.Filesystem` is also supplied,
that complete concrete context wins. Do not merge fields or fall back to derivation
when the explicit context is invalid. This permits overriding reusable companion
setup without clearing its binding declaration.

An active binding is one selected in the absence of an overriding
`Input.Filesystem`. It applies to non-file underlay inputs, including `Mapping`,
`Bytes`, and `Reader`. An active binding on the main `Input` is an option error
because it would refer to itself. An active binding on a `File`,
`Filesystem.File`, or `SearchConfig` underlay is also an option error: these
sources determine their own file import bases. The concrete-context override
rule still applies before this active-binding scope check. Existing source
behavior is unchanged; express main-relative file acquisition through native
imports in a non-file underlay.

Layered overlays derive automatically when `OverlayFilesystem` is absent; no
additional opt-in flag is needed. An explicit overlay context wins as a whole.
Support an explicit overlay fallback under OD-2, reusing the small binding
settings where appropriate without requiring a general context-selection API.

Keep the binding on `Input`: `Source` still describes the supplied content and
`FilesystemContext` still describes a concrete location. This does not reinterpret
a file source's own path. Grouping related settings avoids scattering them across
a boolean and sibling fields, while keeping the addition smaller than a general
context selector. No source wrapper, deferred fields on `FilesystemContext`,
callback, or second file-selection API is needed for this design.

### OD-2: Scope and representation of absent-main handling — settled

Put absence behavior on each derived binding. When no usable base exists, the
default skips optional companion import expansion, retaining the declaring
input's ordinary configuration values. Imports requiring targets or a nonempty
selection produce an error; requiredness includes applicable application import
defaults. An empty `path: []` still succeeds even with `required: true`, because
there are no named targets to require. `RequireBase` independently requires a
usable context even for that empty declaration. Continue normal input and
directive validation, so absence does not excuse invalid declarations.
This preserves independent payload and existing required/optional distinctions
without treating the entire input as conditional on main presence.

Support a concrete fallback context and a require-base option within the binding
description. Illustrative fields are `Fallback *FilesystemContext` and
`RequireBase bool`; final spelling remains an API design detail. Select the
context in this order:

1. A supplied `Input.Filesystem` is the complete override.
2. Otherwise, use a usable main-derived context.
3. Otherwise, use the explicitly supplied fallback context, if any.
4. Without any usable context, skip optional imports unless the binding requires
   a base. A require-base binding or an import requiring targets or a nonempty
   selection produces an error; the empty named-path list remains a no-op.

Validate static binding declaration shape during preflight. Resolve and check
fallback filesystem locations only if the fallback is selected: an unused
fallback need not exist or be accessible. Preserve existing validation of
supplied concrete contexts. A selected invalid fallback produces its normal
error; lazy fallback checks do not provide recovery from invalid selected input.
When no overlay is selected, ignore its fallback settings.

A fallback can satisfy require-base: the option requires a usable context, not
an actual main file. Consumers requiring a main file can make discovery required.
Under OD-4, an explicit context on a non-file or absent main can also satisfy it;
the absence policy applies when no usable base exists, not merely when no main
content exists.
An invalid selected context fails; it does not activate a lower-priority choice.
Fallback applies only when the main cannot provide a base, never after a failure
reading or processing a selected main, or after missing or invalid companion
files beside an existing main. Once bound, normal import selection and failure
rules apply. Do not guess from unsuccessful discovery candidates or implicitly
use the working directory; applications may explicitly choose it as a fallback.

A fallback uses `FilesystemContext.BaseDir` as its lookup directory, with `Root`
and `FS` retaining their containment and namespace roles. This also applies to
the new overlay fallback: look for `overlays/<name>.yml` beneath its base, while
preserving existing explicit `OverlayFilesystem` lookup behavior. An explicit
overlay context wins, followed by main derivation, then the declared overlay
fallback. When no overlay is selected, no overlay context is needed. An active
selection requires a usable context and the named file; never silently skip it.
Companion absence settings do not make a selected overlay optional.

### OD-3: Containment inheritance and symlink convention — settled

Derived companion imports and derived layered overlays inherit the main input's
resolved filesystem namespace and containment root. Lookup uses its resolved
base under OD-4, which for file mains is the file's resolved directory, even when
its configured root is wider. Keep existing explicit overlay context and ordinary
`Load` lookup behavior.

For a host main at `/srv/campaign/audita.yml` with root `/srv`, companion
`glossary.yml` resolves to `/srv/campaign/glossary.yml`, while an import of
`../shared/terms.yml` may resolve to `/srv/shared/terms.yml` within that root.
Overlay `production` resolves to `/srv/campaign/overlays/production.yml`.
Without an explicit wider root, the host main's canonical directory remains
its default containment root. Preserve intentional wider roots rather than
establishing a fresh, narrower companion root. Parent-relative paths remain
permitted when existing containment rules allow them.

For host symlinks, use the canonical target directory. If
`/etc/audita/audita.yml` points to `/srv/releases/v3/audita.yml`, derived
companions and overlays are located under `/srv/releases/v3`. A subsequent load
follows the newly selected bundle if the symlink changes. Document that files
beside the link in `/etc/audita` are not selected by this derived convention.
This matches existing file/import bases; no separate symlink-base selector is
needed for this feature. Applications may supply concrete context overrides
under the normal rules when they choose another location or scope.

Each imported file continues to resolve further imports from its own directory,
within the inherited root. Existing import `follow` controls still govern
discovered symlink entries; they do not select the base of a symlinked main.
Preserve existing containment checks and traversal controls. For native main
contexts under OD-4, retain that filesystem's namespace and link-handling
capabilities without imposing host-filesystem guarantees on every `fs.FS`.

### OD-4: Main-source eligibility beyond discovered host files — settled

Derive from any usable, validated main filesystem context, regardless of the
main source's format. Document `MainRelative` as using the main input's resolved
filesystem context. For the common file workflow, this means lookup beside the
selected main file. Reuse existing context resolution rather than adding another
acquisition mechanism or requiring main imports to be enabled.

- Explicit host files and successful discovery supply the canonical file directory
  and resolved containment root.
- `Filesystem.File` mains supply their directory and root within their existing
  namespace. Dependent loading stays in that namespace.
- `Bytes`, `Reader`, and `Mapping` mains can supply an explicit context with a
  usable base, root, and namespace under their existing validation rules.
- `NoSource` and optional discovery exhaustion can retain an explicit main
  context. Reuse its resolved base: existing absent-host rules allow an explicit
  root to serve as the base when `BaseDir` is omitted; native contexts retain
  their existing requirements.

For example, a bytes main with host `BaseDir: /srv/campaign` and `Root: /srv`
can supply the same companion lookup and containment as a file main in that
directory with the same root. An absent main with a usable explicit context
can likewise supply companion and overlay lookup without creating main content.

Without a usable main context, apply OD-2's fallback, optional-import skipping,
or error behavior. Invalid supplied contexts fail rather than selecting a
fallback. Never infer filesystem locations from diagnostic labels or unsuccessful
search candidates, manufacture a main source, or change source evidence to make
a non-file input appear file-backed.

This preserves one context-reuse rule when applications change how they supply
main content. OD-1's concrete overrides and OD-3's containment, namespace, and
link-handling rules continue to apply. Existing ordinary `Load` and explicit
overlay context lookup semantics remain unchanged.