configkit/docs/roadmap/main-relative-inputs.md

Main-relative companion imports and overlays

Status: accepted direction, public API structure, and absent-main behavior; containment and main-source eligibility remain open. Not implemented. Baseline: 191015e.

Purpose

Make the common workflow straightforward: discover the main configuration file, load optional companion fragments beside it, and apply a selected native overlay in one configuration load. Applications choose their conventions; operators can drop resource definitions into the declared locations without adding import boilerplate to the main file or creating a manifest on disk.

The central addition is deferred filesystem-context binding for application-owned underlay inputs, including Mapping and Bytes sources containing native import declarations. Bind their imports to the selected main directory during loading. Reuse existing optional-file, directory, recursion, pattern, destination, and composition behavior. A new file constructor alone would not cover this workflow, and a second fragment-discovery implementation is unnecessary.

Current baseline

The discovery contract resolves SearchConfig inside loading to one host file or absent input. Selected-file failures do not resume discovery. The constructor returns a source description, not a selected filename.

Ordinary Load already infers overlay context from a file source. Without a root override, /srv/campaign/audita.yml supplies the location /srv/campaign/overlays/production.yml for overlay production. Existing overlay lookup uses the configuration root, which can differ from the main directory. Optional discovery exhaustion provides no inferred context.

The layered API has the coordination gap:

• An active overlay requires LayeredOptions.OverlayFilesystem. Preflight rejects its absence before discovery runs.
• Underlay sources and contexts are supplied independently. Application-owned mappings or bytes can already contain import declarations, but cannot obtain their import context from the main file that discovery will select.
• Native imports already support optional named files and recursive directory searches with filename filtering and placement. They compose as peers of their declaring input, so putting declarations in an underlay gives them underlay priority. Putting the same declarations in the main file does not.
• Main acquisition already precedes underlay acquisition. Existing composition subsequently applies underlay groups from lowest to highest beneath the main.

This feature extends context selection and binding within that lifecycle.

Audita workflow

Audita supplies the companion import declarations in application-owned mappings or bytes. Their filesystem bases follow the selected main file, whether selected explicitly or through ConfigKit's built-in searches. For a main file at /srv/campaign/audita.yml, running from /tmp must not relocate these companions:

Resource	Application-declared locations	Role
Glossaries	Optional glossary.yml and recursive glossary/	Underlay contributions mounted at glossary_collections using the appropriate existing destination and placement controls
Prompt definitions	Optional prompts.yml and recursive prompts/	Operator contributions above the bundled PromptKit catalog and Audita's embedded prompts
Selected native overlay	overlays/<name>.yml	The one named overlay applied after main composition

Both resource-directory searches select *.yml and *.yaml. The file and its corresponding directory contribute together if both exist; neither suppresses the other. Each pair composes strictly as peers within its tier. Missing optional resources contribute nothing; unreadable selected files, invalid definitions, and composition conflicts remain errors.

Prompt overrides rely on existing tier and identity rules. The operator prompt tier sits above embedded prompt tiers, allowing whole-record upsert by registered prompt ID under the applicable collection policy. Two operator fragments defining the same ID in that tier conflict; filename ordering does not choose a winner. The consuming application selects each fragment's schema and placement.

This does not require combining Audita's and PromptKit's schemas. If PromptKit uses a separate engine, its catalog load establishes the prompt tiers and can use the selected main's recorded context after Audita's application load. That is a separate domain load, not a preliminary load to repeat main discovery. This feature's acceptance does not require changing another repository.

The overlay directory is not imported wholesale as an underlay: its files are alternative named configurations. No selection means no overlay. Selecting a name retains the existing missing-selected-overlay error. An aggregate overlays.yml would need its own named-definition and conflict rules and is outside this feature; it is not a second spelling for current overlay lookup.

Targeted end state

Companion underlays

The application declares an underlay Input containing native imports and selects main-derived filesystem binding for that input. Mapping and Bytes are first-class supported sources for this use. OD-1 settles the public API structure below; no new operator-facing YAML syntax is needed.

Once the main is selected, ConfigKit supplies the underlay import base from that file's directory and binds the namespace and containment policy. Existing import selection then acquires the companion files. Every imported file uses its own directory for its further relative imports. For example, an import of extra.yml from glossary/terms.yml resolves to glossary/extra.yml.

Applications choose underlay groups and their order. A group composes strictly; separate groups establish precedence. Existing destinations, sequence placement, source-version policies, and load-wide import defaults still apply. Binding a context does not itself enable imports or change collection operations.

The relationship is explicit on companion inputs. Independently located and embedded inputs keep their own contexts and can coexist in the same load. Providing a concrete context selects that complete context over derived binding, without field-by-field fallback or recovery from an invalid explicit context. OD-1 records the representation and precedence rules.

There is no requirement to add a dedicated main-relative file constructor. Optional named-file acquisition is already expressible through the underlay's native imports. Ordinary File paths do not silently gain new relative semantics.

Layered overlays

For an active layered overlay, use an explicit OverlayFilesystem when supplied. Otherwise derive the overlay context automatically from the available main-file information; the common workflow requires no OverlayFromMain flag. An invalid explicit context fails rather than falling back. Inactive overlays need no context. Missing-main and fallback cases are addressed below and in OD-2.

For this newly derived layered context, the lookup location is the main file's directory, independently of a wider containment root. If the main is /srv/campaign/audita.yml and its permitted root is /srv, the selected file is /srv/campaign/overlays/production.yml, not /srv/overlays/production.yml. Widening containment must not relocate the automatically selected companions.

Keep existing explicit-overlay-context lookup behavior and ordinary Load behavior. The implementation must represent the derived lookup base separately from its containment root where necessary, rather than simply copying today's root-relative overlay context. Native overlay names, filenames, selection, precedence, and validation retain their existing semantics.

Absent main and optional companions

Optional discovery may find no main file while defaults and CLI settings still permit a successful application load. When no usable base is available, skip optional companion import expansion while retaining the declaring input's ordinary payload and validation. Required imports still fail. Applications can require a base or supply an explicit fallback context on each derived binding. OD-2 records the settled precedence, skip scope, and fallback behavior.

Do not infer a directory from unsuccessful candidates or silently use the working directory. An application may explicitly choose the working directory as its fallback. An active selected overlay still needs a usable context and the selected file; optional companion imports do not make that overlay optional. Independent underlays and explicitly located overlays retain their absent-main behavior. No fallback applies to a failure reading or processing an actually selected main.

Lifecycle, evidence, and compatibility

Bind dependent contexts after the main information is available and before traversing their imports or loading the overlay. Preserve main-first acquisition, underlay precedence, version processing, selected-overlay timing, and the existing defaults, CLI, resolution, secret, and validation phases. No application context callback, second discovery API, or preliminary full load is required.

Check static declarations during preflight where possible; defer checks requiring the selected directory. Do not introduce a global ordering guarantee for unrelated diagnostics or filesystem checks. Selected-file and companion acquisition errors remain terminal rather than restarting discovery.

Application-supplied mappings or bytes remain ordinary input sources with their own labels, evidence, and existing validation obligations. Binding a context does not manufacture a main document, exempt these inputs from source-version policy, or give them a fabricated physical file path. Imported files retain their actual source identity, locations, precedence layer, underlay index, and passive resource context. Preserve namespace identity, repetition scope, shared budgets, detached inspection, and acquisition-handle cleanup.

Successful existing ordinary and explicitly configured layered loads should retain their behavior. Automatically deriving a layered overlay context intentionally relaxes the former missing-context error. Companion directory conventions are selected by applications; ConfigKit does not universally scan glossary/ or prompts/ without those declarations.

This work adds no general resource-path resolver, remote acquisition, mutation of the process working directory, filesystem race-protection guarantee, or general source-provider callback framework. These are scope boundaries, not new permanent architectural exclusions. The design follows ADR 0019. Review ADR and policy follow-up for deferred context ownership without rewriting accepted decisions. No release version or additional release gate is selected here.

Work from the baseline

1. Finalize context declarations and absence policy. Resolve the remaining choices below and prepare Go examples for Audita's mapping/bytes workflow, explicit context overrides, and independently located embedded inputs.
2. Carry binding intent through the public facade and load request. Update the relevant input/options declarations and conversions in layered.go and lifecycle.go, retaining root-owned public types and normal ownership rules. Distinguish a deferred relationship from a concrete filesystem context.
3. Bind underlay import contexts after main selection. Extend preparation and acquisition in internal/engine/load_request.go, load_input.go, and acquisition.go as needed. Reuse the acquired main's directory and namespace, apply the chosen missing-base policy, and delegate selection to existing import machinery. Preserve independently supplied contexts.
4. Derive overlay context and separate lookup from containment. Remove the unconditional explicit-context requirement for the new layered default. Preserve explicit and ordinary lookup while enabling a main-directory lookup base inside a potentially wider root. Reuse selected-overlay loading and its failure and resource-ownership behavior.
5. Verify the complete workflow and document it. Add maintained consumer examples, public integration coverage, and focused engine tests. Update the discovery, layered-loading, import-context, and selected-overlay contracts, declaration comments, and internal ownership documentation when implemented.

This is dependency order for a later implementation plan, not a required commit layout or a requirement to expose intermediate stages to consumers.

Validation and completion

Validate behavior through the public loading boundary, including:

• Explicit and discovered main files, early and later search candidates, and an unrelated invocation directory. Companions follow the same selected main.
• Mapping and bytes underlays containing optional named-file and recursive directory imports with *.yml/*.yaml filtering and mounted destinations.
• File only, directory only, both, neither, empty directories, and invalid or unreadable selected fragments. Both forms compose as peers when present.
• Operator prompt overrides above embedded catalogs using registered identity; same-tier duplicate IDs remain conflicts. Main and overlay precedence survive.
• Mixed embedded and companion inputs, multiple underlay tiers, transitive imports, placement, source-version checks, and load-wide import defaults.
• Automatic derived overlays and explicit overrides, active and inactive names, broader containment roots, and missing selected overlays.
• Optional discovery exhaustion, required discovery, fallback and skip policies, mixed payload/required-import cases from OD-2, and terminal selected-main errors.
• Symlinks, eligible namespaces, parent-relative paths within the selected scope, context validity, and namespace identity under OD-3 and OD-4.
• Normal source evidence, diagnostics, passive resource contexts, cleanup, and budget accounting; no duplicate lifecycle callbacks or preliminary full load.
• Existing ordinary and explicit layered workflows and description reuse.

Run focused checks during implementation and applicable normal repository checks before completion. No benchmark campaign is required. Completion means the application can discover its main, bind companion imports and the selected overlay, and receive normal resolved configuration and inspection in one load, without an application directory walker, parser, or extra discovery pass.

Settled Decisions

OD-1: Public representation of deferred context binding — settled

Add a small, optional main-relative binding description to Input, retaining the concrete Input.Filesystem field. A helper supplies the common default description; explicit settings group the absent-main controls settled in OD-2. Companion Mapping and Bytes inputs use native imports for fragment selection. Preserve the input's source, placement, import enablement, and tier.

The proposed shape is an Input.MainRelative field of type *MainRelativeContext, with a MainRelative() convenience helper. These names illustrate the settled structure; final spelling can be finished during API design. For example, this unimplemented API sketch assumes companionDeclarations contains ordinary native import declarations:

configkit.Input{
    Source:       configkit.Mapping("companions", companionDeclarations),
    Imports:      true,
    MainRelative: configkit.MainRelative(),
}

A nil binding preserves existing independent context behavior. A non-nil binding requests context derivation during loading. If Input.Filesystem is also supplied, that complete concrete context wins. Do not merge fields or fall back to derivation when the explicit context is invalid. This permits overriding reusable companion setup without clearing its binding declaration.

Layered overlays derive automatically when OverlayFilesystem is absent; no additional opt-in flag is needed. An explicit overlay context wins as a whole. Support an explicit overlay fallback under OD-2, reusing the small binding settings where appropriate without requiring a general context-selection API.

Keep the binding on Input: Source still describes the supplied content and FilesystemContext still describes a concrete location. This does not reinterpret a file source's own path. Grouping related settings avoids scattering them across a boolean and sibling fields, while keeping the addition smaller than a general context selector. No source wrapper, deferred fields on FilesystemContext, callback, or second file-selection API is needed for this design.

OD-2: Scope and representation of absent-main handling — settled

Put absence behavior on each derived binding. When no usable base exists, the default skips optional companion import expansion, retaining the declaring input's ordinary configuration values. Required imports produce an error; requiredness includes applicable application import defaults. Continue normal input and directive validation, so absence does not excuse invalid declarations. This preserves independent payload and existing required/optional distinctions without treating the entire input as conditional on main presence.

Support a concrete fallback context and a require-base option within the binding description. Illustrative fields are Fallback *FilesystemContext and RequireBase bool; final spelling remains an API design detail. Select the context in this order:

1. A supplied Input.Filesystem is the complete override.
2. Otherwise, use a usable main-derived context.
3. Otherwise, use the explicitly supplied fallback context, if any.
4. Without any usable context, skip optional imports unless the binding requires a base. A require-base binding or required import produces an error.

A fallback can satisfy require-base: the option requires a usable context, not an actual main file. Consumers requiring a main file can make discovery required. An invalid selected context fails; it does not activate a lower-priority choice. Fallback applies only when the main cannot provide a base, never after a failure reading or processing a selected main, or after missing or invalid companion files beside an existing main. Once bound, normal import selection and failure rules apply. Do not guess from unsuccessful discovery candidates or implicitly use the working directory; applications may explicitly choose it as a fallback.

A fallback uses FilesystemContext.BaseDir as its lookup directory, with Root and FS retaining their containment and namespace roles. This also applies to the new overlay fallback: look for overlays/<name>.yml beneath its base, while preserving existing explicit OverlayFilesystem lookup behavior. An explicit overlay context wins, followed by main derivation, then the declared overlay fallback. When no overlay is selected, no overlay context is needed. An active selection requires a usable context and the named file; never silently skip it. Companion absence settings do not make a selected overlay optional.

Open Decisions

OD-3: Containment inheritance and symlink convention

Settled: derived companion imports and derived layered overlays use the main file's directory for lookup, even with a wider configured containment root. Explicit overlay contexts retain their current lookup semantics.

Recommendation: use the selected main's canonical directory, and inherit its resolved namespace and containment root for derived bindings. Existing import rules then provide file-local bases for transitive imports. Permit parent-relative paths when existing containment rules allow them. This reuses current canonical file behavior and preserves intentional wider roots without relocating companions. A concrete underlay context override can select a different scope under its normal rules. Document the symlink-target-directory convention explicitly.

Alternative: create a fresh containment root at the selected main directory. This keeps companions locally scoped but discards intentional wider roots and prevents some sibling-directory layouts. Anchoring to the lexical symlink directory is also viable for deployment bundles; it differs from current file/import behavior and would need distinct lookup and containment handling.

OD-4: Main-source eligibility beyond discovered host files

Settled: both explicitly selected and discovered host mains support the workflow. Non-file companion inputs, especially mappings and bytes, are in scope; this decision concerns the main input that supplies their base.

Recommendation: also support native Filesystem.File mains using their namespace's path and ownership rules. For an initial file-based scope, non-file mains use independently supplied or fallback contexts rather than implicitly providing a main-file directory. This covers the requested workflow and avoids making discovery a prerequisite for equivalent file inputs.

Alternative: derive from any usable main context, including BaseDir on a byte, reader, or mapping main. This is more uniform with context-based loading and avoids excluding a known base, but requires explaining main-relative as main-context-relative and deciding how root-only or absent-main contexts behave. A host-only initial scope is smaller but leaves native-file consumers with the same coordination gap.