From 47547bc63eb5a5019b80f1b71fe966b23e235b6f Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 9 Nov 2026 21:34:17 +0000 Subject: [PATCH] Plan single-file overlays with strict peer composition --- docs/roadmap/overlays.md | 275 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 275 insertions(+) create mode 100644 docs/roadmap/overlays.md diff --git a/docs/roadmap/overlays.md b/docs/roadmap/overlays.md new file mode 100644 index 0000000..13ba0c0 --- /dev/null +++ b/docs/roadmap/overlays.md @@ -0,0 +1,275 @@ +# Single-file named overlays + +Status: proposal; strict composition between the two forms is accepted. Remaining +decisions are listed below. Not implemented. Implementation baseline: `ce6005c`. + +## Purpose + +Support a single `overlays.yml` containing named overlay definitions alongside +the existing `overlays/.yml` convention. Small applications can keep their +profiles together, while applications with larger configurations can split them +into individual files or contribute through both forms. + +Applications continue to select one overlay name. ConfigKit selects the matching +contributions, strictly composes them, and applies the resulting overlay once. +This follows [ADR 0001](../adr/0001-separate-composition-and-overlay.md): peer +composition and intentional precedence remain separate operations. + +## Current baseline + +The [selected-overlay contract](../config.md#selected-overlays) currently loads +one YAML mapping from `overlays/.yml`. Overlay names are not paths. Empty +selection and the exact lowercase name `default` select no overlay. A selected +missing file fails; there is no extension fallback or overlay stack. + +Ordinary loads and explicit layered overlay contexts locate that file beneath +the configured root. Layered main-derived and fallback contexts use their +resolved base for lookup, with a separate containment root. The implemented +[main-relative context decision](../adr/0022-main-relative-context.md) supplies +the context needed by this feature; it does not implement aggregate overlays. + +The existing loader in `internal/engine/selected_overlay.go` acquires and parses +the file, checks overlay directives and source-version policy, and applies its +payload with the effective collection policy. Overlays apply after base +composition and main-version checking, before inheritance, defaults, decoding, +CLI overrides, application resolution, and final validation. Source records, +locations, resource contexts, history, repetition checks, limits, and handle +cleanup already cover the individual-file form. + +## Targeted end state + +### File layout and selection + +The proposed catalog is a mapping from overlay names directly to overlay bodies: + +```yaml +# overlays.yml — proposed format +production: + service: + workers: 8 +development: + service: + workers: 1 + log_level: debug +``` + +Selecting `production` takes only that entry as the catalog's contribution. +The same selection may also acquire this existing-format file: + +```yaml +# overlays/production.yml +service: + log_level: warning +``` + +Strict composition produces this one overlay payload: + +```yaml +service: + workers: 8 + log_level: warning +``` + +If the individual file instead also defines `service.workers: 12`, ordinary +strict-composition conflict rules apply. File order grants no precedence. +The other catalog entries do not contribute to the selected configuration. +The exact catalog structure and its validation boundary remain in OD-1 through +OD-3; this example illustrates the recommended structure. + +Use the same lookup directory for the catalog and the existing `overlays/` +directory. Preserve ordinary versus layered lookup semantics, explicit context +precedence, main-relative and fallback behavior, namespace identity, symlink +handling, and containment. The new filename variants and activation behavior +remain in OD-4 and OD-5. + +No selected overlay means neither form is read. With a selection, either form +may supply the named overlay. An absent catalog entry is not an error when the +individual file supplies the selection, and a missing individual file is not an +error when the catalog supplies it. If neither contributes, report the existing +missing-overlay condition. A selected empty mapping is a present contribution, +not a missing overlay. Acquisition or parsing failures are not a reason to +silently prefer the other form. + +### Strict composition, then one overlay application + +**Settled:** contributions for the selected name from both forms are peers. +Strictly compose them before applying their combined result to the lower +configuration. Do not apply them sequentially as precedence layers, suppress +one form, or reject a shared name merely because both forms contribute. + +Reuse existing composition and collection-policy rules. Effective lower-tier +identity and applicable selected-overlay declarations must be available when +composing the peers. Disjoint identified records can compose; conflicting +identities among peers fail. The combined collection then uses its applicable +`replace`, `append`, or `upsert` operation against the lower value once. +Unidentified sequences retain existing strict-composition and overlay behavior. + +Resolve collection declarations across the contributions using the existing +same-tier rules; do not independently apply each contribution's mode to the +lower configuration. Keep the existing rules for version-path protection and +forbidding overlay imports. This feature adds neither overlay inheritance nor +selection of several overlay names. + +### Processing and source evidence + +Acquire selected contributions within the existing load, after base processing +reaches the overlay phase. Preserve subsequent lifecycle order and normal +failure behavior. Reuse parser and composition representations rather than +decoding the catalog into application structs and reconstructing a document. + +The selected catalog body's application paths are relative to that body, while +its locations still identify the real catalog file and authored line/column. +Diagnostics must distinguish application coordinates from the catalog wrapper; +never fabricate a physical file such as `overlays.yml/production`. Relative +resource contexts follow the physical declaring file, so resources beside +`overlays.yml` and beside `overlays/production.yml` have different bases. +Preserve both contributors when mappings combine, and normal field origins and +history when the completed overlay is applied. + +Keep real acquisition accounting: parse the acquired catalog under existing +YAML limits and charge its complete bytes and parsed nodes, including unselected +bodies. Each acquired physical file contributes a source record; unused names +are not separately acquired sources. Preserve repetition/containment checks, +detached inspection, and cleanup on every exit. Do not add discovery-attempt +history or a new inspection API solely for this feature. + +Use deterministic acquisition order for source IDs and sequence assembly. The +recommended order is catalog contribution, then individual-file contribution; +this orders evidence and composed records, not precedence. Reuse existing +diagnostic classifications where they fit. Source-version/control scope is an +open decision in OD-2 and must be reflected consistently in inspection. + +## Work from the baseline + +1. **Settle the remaining contract choices.** Resolve the decisions below, + including activation compatibility, and prepare examples for each source + form, combined contributions, source-version modes, and collection controls. +2. **Separate overlay acquisition from application.** Adapt the selected-overlay + loader to acquire optional contributions and distinguish missing selection + from acquisition failure. Retain existing context selection and access rules. +3. **Select catalog bodies without losing source evidence.** Parse the physical + catalog, select the requested body, and retain its actual locations. Apply + the chosen version/directive scope and account for the complete acquired file. +4. **Compose peers and apply once.** Reuse strict composition with the lower + collection policy and selected declarations, then feed the composed result + into the existing overlay operation. Preserve contributor origins, history, + sensitivity, version protection, and resource contexts. +5. **Verify and document.** Add public integration coverage and maintained + consumer examples; update selected-overlay contracts, public declaration + comments as needed, and internal ownership documentation when implemented. + +This is a dependency order for a later implementation plan, not a prescribed +commit layout. Do not change existing ADR decisions merely to document a new +storage form. Review any durable ownership change separately if one emerges. + +## Validation and completion + +Cover the public loading boundary with: + +- Catalog only, individual file only, both, neither, a catalog without the + selected name, selected empty mappings, and inactive/default selections. +- Complementary mappings, conflicting fields, identified-sequence composition, + duplicate identities, mode declarations, and one final collection operation. +- Ordinary and layered loading, discovered/explicit/non-file/absent mains, + explicit overlay contexts, main-derived contexts, fallback contexts, wider + roots, host and native filesystems, and existing link-handling capabilities. +- Invalid YAML, catalog shape, selected/unselected body validation, source-version + modes, protected version paths, and forbidden directives under the settled rules. +- Actual source paths and locations, origins and contributor history, sensitive + values, distinct resource bases, source IDs, and detached snapshots. +- Missing versus unreadable sources, repetition, complete catalog accounting, + limits, and cleanup on failure. No duplicate load lifecycle callbacks. +- Existing individual-file workflows and the activation compatibility cases + settled in OD-5. + +Use focused tests and normal repository checks during implementation. No new +benchmark campaign or release gate is required. Completion means applications +can select one named overlay from either or both forms with the same composition, +precedence, and inspection model. + +## Open Decisions + +### OD-1: Catalog structure + +**Recommendation:** use the direct name-to-body mapping illustrated above. +It adds only one enclosing mapping and keeps each body recognizable as an +ordinary overlay. The catalog wrapper is a selection container, not an application +document; overlay names retain existing spelling and selection rules. The special +`default` selection continues to mean no overlay. + +**Alternative:** wrap entries under an `overlays` key, leaving space for catalog +metadata. This gives file-level headers an explicit home, but adds boilerplate +and another reserved structure before a need for catalog metadata is established. + +### OD-2: Source-version declarations and document controls + +**Recommendation:** treat each selected body as the overlay's logical document +root for existing source-version and overlay-control processing. A version header +or `config.sequences` declaration appears inside that body, just as it appears +at the root of an individual overlay file. Process each participating contribution +before strict composition, including its own required version check when enabled. +Preserve the physical file's authored locations and report version evidence for +the selected logical document. Imports remain forbidden under existing overlay +rules; nested application data is not reinterpreted as root controls. + +This permits moving a body between forms without rewriting its payload or +changing its policy meaning. It requires explicitly distinguishing the catalog's +physical document from the selected logical overlay in version processing and +inspection; it must not silently change source-version semantics for other inputs. + +**Alternative:** declare the version and shared controls once at catalog level, +with entries inheriting them. This reduces repeated headers but needs the wrapper +from OD-1 or another metadata convention, plus rules for shared versus per-entry +controls. It also makes a standalone overlay body differ from its catalog form. + +### OD-3: Validation of unselected bodies + +**Recommendation:** require a valid YAML mapping catalog and look up the selected +name. Preserve parser-level validation of the whole file, including duplicate +mapping-key rules and limits. Require the selected body to satisfy normal overlay +mapping, directive, and version rules, but do not run those semantic checks or +application callbacks for unselected bodies. Final application validation assesses +the resulting effective configuration, not each profile in isolation. + +This keeps a load focused on the requested configuration while acknowledging +that malformed YAML anywhere prevents reliable parsing. An empty mapping catalog +contains no definitions; selected null/scalar/sequence bodies follow the existing +mapping-root rules rather than becoming shorthand for absence. + +**Alternative:** validate every body's shape, name, directives, and version when +reading the catalog. This catches unused-profile defects earlier, but makes one +profile's load depend on unrelated definitions and expands checker invocation. +A whole-catalog lint facility could serve that purpose separately if requested. + +### OD-4: Filename variants + +**Recommendation:** initially recognize exactly `overlays.yml` and the existing +`overlays/.yml`. This matches current overlay spelling without adding a +separate extension-precedence decision. Both recognized forms compose as peers. + +**Alternative:** also recognize `.yaml`. This accommodates common YAML filenames, +but requires deciding whether multiple extensions are additional peers or fallback +alternatives. If adopted, prefer strict peer composition over hidden precedence; +document stable acquisition order and collisions consistently for both forms. + +### OD-5: Activation and compatibility + +**Recommendation:** use the existing selected overlay name to activate lookup of +both forms, without another option in the common workflow. The storage layout +should not require consumers to implement different selection APIs. Document +that an acquired catalog participates in parsing even if its selected name is +absent, and that conflicting selected contributions fail strict composition. + +Before committing to release compatibility, assess existing deployments that +have an `overlays.yml` previously ignored by ConfigKit: automatic participation +can expose invalid content or conflicts alongside an otherwise successful +individual-file load. The [release policy](../policy/release.md#versioning-and-compatibility) +preserves successful valid configurations; merely accepting additional layouts +does not resolve this interaction. No release version is selected here. + +**Alternative:** add explicit catalog enablement, preserving existing directory-only +lookup until selected by the application. This provides a clear compatibility +boundary and lets consumers adopt the feature deliberately, but introduces an +extra setting for the common layout. Once enabled, contributions from both forms +must still compose strictly; this alternative does not reopen the settled +coexistence decision.