diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index f314432..3f6c683 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -102,3 +102,29 @@ string equality must not be presented as semantic equality. Keeping `Path` solely as an opaque query/building value remains an alternative. No current contract changes or implementation commitment follow from this entry; a comparison operation can be added compatibly later. + +## 5. Single-file named overlays + +Status: deferred. Consider supporting a single `overlays.yml` containing named +overlay bodies alongside the existing +[selected-overlay directory convention](../config.md#selected-overlays). +This would let small applications keep their overlays in one file. Planning was +tabled because the added semantics outweighed the immediate convenience; the +draft is preserved at `3e50388:docs/roadmap/overlays.md`. + +The draft selected a direct name-to-body mapping, strict peer composition when +both storage forms contribute to the selected name, and file-level version +checks using the application's existing rule and mode. These provide a starting +point for reconsideration, not an implementation commitment. + +### Questions to resolve before implementation + +- Where should collection controls live: within each named body or at catalog + level, and how should that interact with document-root control ownership? +- How should configured version metadata paths coexist with overlay names, + including nested paths and modes that do not check overlay versions? +- Which semantic checks should apply to unselected bodies? +- Should `.yaml` also be recognized, and how should multiple matching files + interact? +- How should catalog lookup be enabled without disrupting successful loads that + previously ignored an adjacent `overlays.yml`? diff --git a/docs/roadmap/overlays.md b/docs/roadmap/overlays.md deleted file mode 100644 index ec31699..0000000 --- a/docs/roadmap/overlays.md +++ /dev/null @@ -1,357 +0,0 @@ -# Single-file named overlays - -Status: strict composition, direct catalog structure, and file-level version -checking are settled. Remaining decisions are listed below. Not implemented. -Implementation baseline: `ce6005c`. - -## Purpose - -Support a single `overlays.yml` containing named overlay definitions alongside -the existing `overlays/.yml` convention. Small applications can keep their -profiles together, while applications with larger configurations can split them -into individual files or contribute through both forms. - -Applications continue to select one overlay name. ConfigKit selects the matching -contributions, strictly composes them, and applies the resulting overlay once. -This follows [ADR 0001](../adr/0001-separate-composition-and-overlay.md): peer -composition and intentional precedence remain separate operations. - -This roadmap extends file-based selected overlays. The selected contributions -compose as peers at the overlay stage; overlay definitions do not acquire -underlay or main-input precedence tiers. A configuration-wide catalog populated -through inline `config.overlays`, ordinary imports, or application underlays is -outside this feature's scope. No new application or source-control key is reserved. - -## Current baseline - -The [selected-overlay contract](../config.md#selected-overlays) currently loads -one YAML mapping from `overlays/.yml`. Overlay names are not paths. Empty -selection and the exact lowercase name `default` select no overlay. A selected -missing file fails; there is no extension fallback or overlay stack. - -Ordinary loads and explicit layered overlay contexts locate that file beneath -the configured root. Layered main-derived and fallback contexts use their -resolved base for lookup, with a separate containment root. The implemented -[main-relative context decision](../adr/0022-main-relative-context.md) supplies -the context needed by this feature; it does not implement aggregate overlays. - -The existing loader in `internal/engine/selected_overlay.go` acquires and parses -the file, checks overlay directives and source-version policy, and applies its -payload with the effective collection policy. Overlays apply after base -composition and main-version checking, before inheritance, defaults, decoding, -CLI overrides, application resolution, and final validation. Source records, -locations, resource contexts, history, repetition checks, limits, and handle -cleanup already cover the individual-file form. - -## Targeted end state - -### File layout and selection - -The catalog maps overlay names directly to overlay bodies, without an enclosing -`overlays` key. It may also carry file-level version metadata under the application's -configured version path. The following example omits optional version metadata: - -```yaml -# overlays.yml — planned format -production: - service: - workers: 8 -development: - service: - workers: 1 - log_level: debug -``` - -Selecting `production` takes only that entry as the catalog's contribution. -The same selection may also acquire this existing-format file: - -```yaml -# overlays/production.yml -service: - log_level: warning -``` - -Strict composition produces this one overlay payload: - -```yaml -service: - workers: 8 - log_level: warning -``` - -If the individual file instead also defines `service.workers: 12`, ordinary -strict-composition conflict rules apply. File order grants no precedence. -The other catalog entries do not contribute to the selected configuration. -OD-1 settles the direct catalog structure. Version checks are file-scoped as -described below; metadata-name collisions, collection-control placement, and -the validation boundary remain in OD-2 and OD-3. - -Use the same lookup directory for the catalog and the existing `overlays/` -directory. Preserve ordinary versus layered lookup semantics, explicit context -precedence, main-relative and fallback behavior, namespace identity, symlink -handling, and containment. The new filename variants and activation behavior -remain in OD-4 and OD-5. - -No selected overlay means neither form is read. With a selection, either form -may supply the named overlay. An absent catalog entry is not an error when the -individual file supplies the selection, and a missing individual file is not an -error when the catalog supplies it. If neither contributes, report the existing -missing-overlay condition. A selected empty mapping is a present contribution, -not a missing overlay. Acquisition or parsing failures are not a reason to -silently prefer the other form. - -### File-level version checks - -**Settled:** a version declaration checks whether a physical configuration file -declares an acceptable specification version. It does not independently version -each overlay body. Use the existing application-configured `Definition.Version` -path and source-version mode; do not hard-code the spelling `version`. - -For an application using the root `version` key, a catalog can contain: - -```yaml -version: 2 -production: - service: - workers: 8 -development: - service: - workers: 1 -``` - -Apply the configured source-version policy once to each acquired physical file, -before its contribution enters strict composition. The catalog check is independent -of which name is selected and also applies when the acquired catalog lacks that -name. No selected overlay still means neither form is read or checked. In enabled -modes, evaluate the catalog declaration at its physical document root, not inside -the selected body. Preserve the existing distinction between check-when-present -and required modes and the application's authority to accept the declared value. - -An acquired individual overlay file has its own declaration and existing version -processing. When both files contribute, each is checked independently; one file -cannot satisfy another's required declaration. Do not check or require separate -version declarations in the catalog's named bodies. Keep catalog version metadata -out of the applied overlay, and keep its evidence attached to the actual catalog -source and authored location. Body values remain subject to normal protection of -the application's version path when applying the overlay. - -This decision concerns stale-file detection. Collection controls describe overlay -operations and need not share the version declaration's scope. OD-2 separates the -remaining control-placement and catalog-metadata recognition decisions. - -### Strict composition, then one overlay application - -**Settled:** contributions for the selected name from both forms are peers. -Strictly compose them before applying their combined result to the lower -configuration. Do not apply them sequentially as precedence layers, suppress -one form, or reject a shared name merely because both forms contribute. - -Reuse existing composition and collection-policy rules. Effective lower-tier -identity and applicable selected-overlay declarations must be available when -composing the peers. Disjoint identified records can compose; conflicting -identities among peers fail. The combined collection then uses its applicable -`replace`, `append`, or `upsert` operation against the lower value once. -Unidentified sequences retain existing strict-composition and overlay behavior. - -Resolve collection declarations across the contributions using the existing -same-tier rules; do not independently apply each contribution's mode to the -lower configuration. Keep the existing rules for version-path protection and -forbidding overlay imports. This feature adds neither overlay inheritance nor -selection of several overlay names. - -### Processing and source evidence - -Acquire selected contributions within the existing load, after base processing -reaches the overlay phase. Preserve subsequent lifecycle order and normal -failure behavior. Reuse parser and composition representations rather than -decoding the catalog into application structs and reconstructing a document. - -The selected catalog body's application paths are relative to that body, while -its locations still identify the real catalog file and authored line/column. -Diagnostics must distinguish application coordinates from the catalog wrapper; -never fabricate a physical file such as `overlays.yml/production`. Relative -resource contexts follow the physical declaring file, so resources beside -`overlays.yml` and beside `overlays/production.yml` have different bases. -Preserve both contributors when mappings combine, and normal field origins and -history when the completed overlay is applied. - -Keep real acquisition accounting: parse the acquired catalog under existing -YAML limits and charge its complete bytes and parsed nodes, including unselected -bodies. Each acquired physical file contributes a source record; unused names -are not separately acquired sources. Preserve repetition/containment checks, -detached inspection, and cleanup on every exit. Do not add discovery-attempt -history or a new inspection API solely for this feature. - -Use deterministic acquisition order for source IDs and sequence assembly. The -recommended order is catalog contribution, then individual-file contribution; -this orders evidence and composed records, not precedence. Reuse existing -diagnostic classifications where they fit. Record file-level version evidence -once per acquired source; collection-control scope remains open in OD-2. - -## Work from the baseline - -1. **Settle the remaining contract choices.** Resolve the decisions below, - including activation compatibility, and prepare examples for each source - form, combined contributions, source-version modes, and collection controls. -2. **Separate overlay acquisition from application.** Adapt the selected-overlay - loader to acquire optional contributions and distinguish missing selection - from acquisition failure. Retain existing context selection and access rules. -3. **Check files and select catalog bodies without losing source evidence.** Parse - and apply file-level source-version policy to each acquired file. Separate - catalog metadata from named definitions, select the requested body, and retain - actual locations. Apply the chosen collection-control scope and account for - the complete acquired file. -4. **Compose peers and apply once.** Reuse strict composition with the lower - collection policy and selected declarations, then feed the composed result - into the existing overlay operation. Preserve contributor origins, history, - sensitivity, version protection, and resource contexts. -5. **Verify and document.** Add public integration coverage and maintained - consumer examples; update selected-overlay contracts, public declaration - comments as needed, and internal ownership documentation when implemented. - -This is a dependency order for a later implementation plan, not a prescribed -commit layout. Do not change existing ADR decisions merely to document a new -storage form. Review any durable ownership change separately if one emerges. - -## Validation and completion - -Cover the public loading boundary with: - -- Catalog only, individual file only, both, neither, a catalog without the - selected name, selected empty mappings, and inactive/default selections. -- Complementary mappings, conflicting fields, identified-sequence composition, - duplicate identities, mode declarations, and one final collection operation. -- Ordinary and layered loading, discovered/explicit/non-file/absent mains, - explicit overlay contexts, main-derived contexts, fallback contexts, wider - roots, host and native filesystems, and existing link-handling capabilities. -- Invalid YAML, catalog shape, selected/unselected body validation, source-version - modes, protected version paths, and forbidden directives under the settled rules. -- One file-level version evaluation per acquired catalog, independent of the - selected name, including a catalog without that name. Cover both source forms - under required and check-when-present modes, configured metadata paths and name - collisions, metadata exclusion from payloads, and actual declaration locations. -- Actual source paths and locations, origins and contributor history, sensitive - values, distinct resource bases, source IDs, and detached snapshots. -- Missing versus unreadable sources, repetition, complete catalog accounting, - limits, and cleanup on failure. No duplicate load lifecycle callbacks. -- Existing individual-file workflows and the activation compatibility cases - settled in OD-5. - -Use focused tests and normal repository checks during implementation. No new -benchmark campaign or release gate is required. Completion means applications -can select one named overlay from either or both forms with the same composition, -precedence, and inspection model. - -## Settled Decisions - -### OD-1: Catalog structure — settled - -Use the direct name-to-body mapping illustrated above, with no enclosing -`overlays` mapping. The filename identifies the document's role; each named body -remains recognizable as an individual overlay file's contents. The top-level -mapping is a selection container and is not passed to application decoding. -Overlay names retain existing spelling and selection rules, including empty -selection and exact lowercase `default` meaning no overlay. - -This settles the storage structure within the narrower file-based feature. -File-level version metadata coexists with the named entries. Its recognition and -name-collision rules, and collection-control placement, remain subject to OD-2. - -## Open Decisions - -### OD-2: File-level versions and overlay controls — partially settled - -**Settled:** source-version checking operates once per acquired physical file, -using the existing application rule and mode. The catalog's declaration belongs -at its configured file-level path, not inside every named body. Collection-control -placement is a separate decision. The targeted end state above records the -version behavior; the remaining questions follow. - -#### Collection-control placement - -**Recommendation:** interpret `config.sequences` inside the selected body, using -application-relative destinations such as `/prompts`. Process each selected -contribution's controls before strict composition under existing same-tier rules. -This keeps an overlay's collection operations beside its payload and preserves -their meaning when moving that payload between forms. The file-level version -declaration stays separate. Imports remain forbidden under existing overlay rules; -nested application data does not activate controls. - -**Alternative:** place shared collection controls at catalog level and apply them -to every selected body. This reduces repetition where profiles use identical -operations, but needs metadata-name rules, makes operations less local to their -payloads, and differs from the structure of individual overlay files. - -#### Version metadata recognition and name collisions - -**Recommendation:** when the application configures a version path, give that -path a defined metadata role in the catalog and reserve its enclosing top-level -entry from overlay-name selection. For a root `version` path this reserves only -the catalog name `version`; for a nested path it makes the enclosing entry a -metadata container. This avoids guessing whether a value is a header or a profile. -Do not reserve a hard-coded version name when no version rule is configured. - -Specify how recognition works in main-only mode separately from whether a checker -runs, keeping metadata out of overlay payloads. Retain support for existing version -path forms; concrete examples must cover nested paths and retained empty ancestors -after header consumption. These rules are still open, not implicit additions to -the settled file-level check. - -**Alternative:** reserve only the exact version path and allow other fields in -its enclosing entry to form an overlay body. This permits more names and avoids -reserving a whole metadata container, but gives one mapping both metadata and -profile roles. Selection, header removal, and empty-body behavior become harder -to explain and need explicit rules. - -### OD-3: Validation of unselected bodies - -**Recommendation:** require a valid YAML mapping catalog and look up the selected -name. Preserve parser-level validation of the whole file, including duplicate -mapping-key rules and limits. Require the selected body to satisfy normal overlay -mapping and directive rules, but do not run those semantic checks or application -callbacks for unselected bodies. File-level version policy still applies to the -acquired catalog independently of selection. Final application validation assesses -the resulting effective configuration, not each profile in isolation. - -This keeps a load focused on the requested configuration while acknowledging -that malformed YAML anywhere prevents reliable parsing. An empty mapping catalog -contains no definitions; selected null/scalar/sequence bodies follow the existing -mapping-root rules rather than becoming shorthand for absence. - -**Alternative:** validate every body's shape, name, and directives when -reading the catalog. This catches unused-profile defects earlier, but makes one -profile's load depend on unrelated definitions and expands semantic validation. -A whole-catalog lint facility could serve that purpose separately if requested. - -### OD-4: Filename variants - -**Recommendation:** initially recognize exactly `overlays.yml` and the existing -`overlays/.yml`. This matches current overlay spelling without adding a -separate extension-precedence decision. Both recognized forms compose as peers. - -**Alternative:** also recognize `.yaml`. This accommodates common YAML filenames, -but requires deciding whether multiple extensions are additional peers or fallback -alternatives. If adopted, prefer strict peer composition over hidden precedence; -document stable acquisition order and collisions consistently for both forms. - -### OD-5: Activation and compatibility - -**Recommendation:** use the existing selected overlay name to activate lookup of -both forms, without another option in the common workflow. The storage layout -should not require consumers to implement different selection APIs. Document -that an acquired catalog participates in parsing even if its selected name is -absent, and that conflicting selected contributions fail strict composition. - -Before committing to release compatibility, assess existing deployments that -have an `overlays.yml` previously ignored by ConfigKit: automatic participation -can expose invalid content or conflicts alongside an otherwise successful -individual-file load. The [release policy](../policy/release.md#versioning-and-compatibility) -preserves successful valid configurations; merely accepting additional layouts -does not resolve this interaction. No release version is selected here. - -**Alternative:** add explicit catalog enablement, preserving existing directory-only -lookup until selected by the application. This provides a clear compatibility -boundary and lets consumers adopt the feature deliberately, but introduces an -extra setting for the common layout. Once enabled, contributions from both forms -must still compose strictly; this alternative does not reopen the settled -coexistence decision.