From e1336f74e537d16ffa31269bb307b85b4cdb7436 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 17 Nov 2026 22:28:16 +0000 Subject: [PATCH] Revise secrets roadmap around explicit configuration bootstrap --- docs/roadmap/secrets-directory.md | 222 +++++++++++++++++------------- 1 file changed, 125 insertions(+), 97 deletions(-) diff --git a/docs/roadmap/secrets-directory.md b/docs/roadmap/secrets-directory.md index b72c864..b18ea97 100644 --- a/docs/roadmap/secrets-directory.md +++ b/docs/roadmap/secrets-directory.md @@ -9,34 +9,49 @@ argument uses the process working directory. Using a directory beside the discovered file therefore requires the application to take over main-file selection or perform a preliminary load. [Configuration discovery](../config.md#configuration-discovery) · [Secrets bootstrap](../config.md#explicit-secrets-directory-bootstrap) -ConfigKit already carries the selected main's filesystem context into layered -loading. It resolves effective typed `Secret` references after application -resolution and before final validation. Those two points provide the base for -the directory and the set of credential names to read. [Main-relative contexts](../adr/0022-main-relative-context.md) · [Typed secret references](../config.md#typed-secret-references) +ConfigKit already retains filesystem context for acquired main files and binds +companion inputs to that context. Typed `Secret` references read the environment +after application resolution and before final validation. A companion bootstrap +helper can use the main's context before loading while preserving this +processing order. [Main-relative contexts](../adr/0022-main-relative-context.md) · [Typed secret references](../config.md#typed-secret-references) ## Target behavior ### Declaration and path resolution -An application opts in through Go when calling `Load` or `LoadLayered`. The -selected main document supplies one relative directory through its root control +An application explicitly calls a companion to +`LoadSecretsIntoEnvironment` before `Load` or `LoadLayered`. The selected main +document supplies the directory and overwrite policy through its root control mapping. For example, the following declaration in `/etc/demo/config.yml` -selects `/etc/demo/secrets` when discovery selects that main file. +selects `/etc/demo/secrets` and authorizes replacement of existing variables +when the companion helper is called. ```yaml config: secrets_dir: secrets + secrets_overwrite: true api_key: DEMO_API_KEY ``` -Interpret the declaration at the main document's root before placement and -composition. Diagnose a declaration used without opt-in or in an unsupported -source-control context. Nested application data follows the existing -`config` rules. The location comes from the main document even when an import, -overlay, CLI override, default, or application resolution supplies the final -credential name. Environment lookup alone applies when there is no declaration, -including when an optional search finds no main file. +Interpret these fields at the main document's root before placement and +composition. They are bootstrap settings consumed as document controls. +`Load` and `LoadLayered` recognize the controls and perform their usual +environment lookup without invoking bootstrap. The companion helper call is +the explicit authorization for environment writes. + +Only the main document selects the bootstrap directory and YAML overwrite +policy. Diagnose these controls in unsupported source-control contexts. +Nested application data follows the existing `config` rules. Imports, +underlays, overlays, CLI overrides, and application resolution retain their +ordinary effects on application configuration. They do not redirect this +bootstrap operation. + +`secrets_overwrite` is an optional Boolean with a default of `false`. +An explicitly supplied Go overwrite choice takes precedence over YAML, +including when that choice is `false`. If the main supplies no directory, +bootstrap is a no-op and the prepared main can still be loaded. An optional +search that finds no main preserves the existing absent-main lifecycle. Resolve the directory from the selected main file's canonical base within its existing filesystem root. For a discovered symlink, the base is the selected @@ -45,102 +60,115 @@ containment rule as other main-relative inputs. The declaration must be a nonempty relative path and requires a usable main-file context. Keep the declaration and any diagnostics in source-local coordinates. [Document-root controls](../adr/0020-document-root-controls.md) · [Layered loading](../config.md#layered-loading) -### Lookup within a load +### Companion helper and prepared main source -The usual opt-in mode checks the environment first. A present variable wins, -including an empty one, which produces the existing empty-secret failure. For -an unset variable, read the file whose name matches the effective typed -`Secret` reference. The directory is accessed only if a reference needs a file. +The companion accepts the main `Source`, including a `SearchConfig` source, +and returns a secrets report and a prepared main source. The public name and +option type below are illustrative. -An application can also select file-first lookup for a load. An absent file -allows environment lookup. A file that is present but unreadable or invalid -produces a safe diagnostic, and an empty file follows the existing rule that a -resolved credential must be nonempty. If neither source supplies a credential, -report the failure at the affected configuration field. +```go +main := configkit.SearchConfig("demo", search) +report, preparedMain, err := configkit.LoadSecretsIntoEnvironmentFromConfig(main) +if err != nil { + // Handle the bootstrap failure and inspect report. + return err +} +result, err := configkit.Load[AppConfig](preparedMain) +``` -Resolve surviving references at the existing secret-resolution step and share -the lookup result for repeated names within that load. Use the existing -secret-file checks, content rules, and per-file and cumulative byte limits for -the files read. Exact-name lookup leaves unrelated directory entries alone. -Close library-owned handles during the load. Final validation sees the -resolved credentials, with byte access through `Secret.Value()`. Inspection -can distinguish the environment and a file as the credential provider while -keeping credential bytes outside diagnostics, history, and effective YAML. +For layered loading, supply the prepared source as the main `Input.Source`. +The helper resolves discovery once, reads the selected main document, extracts +the bootstrap controls, and performs bootstrap. It returns the captured main +contents with the selected file's identity and filesystem context so the later +load consumes that same main document. Keep its remaining controls, including +imports and collection declarations, available to normal loading. -### Explicit startup export +The helper prepares only the main document. Composition, import acquisition, +inheritance, typed decoding, application resolution, secret lookup, and +validation run through the subsequent ordinary or layered load. Main +preparation must retain the applicable source limits and root constraints. +Source records, diagnostics, placements, imports, and main-relative companions +use the selected main's identity and context. -Applications that start SDKs or subprocesses which read the environment can -explicitly request file-first loading and environment export. This is a -startup operation distinct from the ordinary load entry points. Go code -authorizes writes and overwrites. The main-file declaration selects the -directory. +Release library-owned file handles before the helper returns, including on +error. Prepared sources retain captured document data and passive filesystem +context and follow the source reuse contract without sharing mutable load +state. Later imports and resources observe the filesystem when accessed. A +later load reads the environment at its normal secret-resolution step. +[Complete-on-return loading](../adr/0008-complete-on-return-loading.md) · [Source resource contexts](../config.md#source-resource-contexts) -Complete configuration loading and validation before writing to the process -environment. Export the values of the file-backed effective references used -by that successful load, once per name. Each successful write sets the -variable to the same value used by the returned configuration. If a file is -absent and lookup uses an existing environment value, leave that variable alone. -Invoke this operation during startup before starting concurrent work. It runs -synchronously and reports successful writes and partial failures without -credential bytes, following the existing bootstrap reporting model. ConfigKit -does not roll back earlier writes when a later write fails. +### Shared bootstrap and overwrite behavior -Ordinary `Load` and `LoadLayered` resolve credentials for their own results -without writing to the environment. The existing -`LoadSecretsIntoEnvironment(directory)` remains the explicit startup helper -for callers supplying a directory independently. [Complete-on-return loading](../adr/0008-complete-on-return-loading.md) · [Secret references and explicit bootstrap](../adr/0004-secret-references-and-explicit-bootstrap.md) +Extend the existing direct-directory helper with an optional named Boolean +option, keeping existing calls valid. The option type name in this example is +provisional. -## Work needed +```go +report, err := configkit.LoadSecretsIntoEnvironment("/etc/demo/secrets") +report, err = configkit.LoadSecretsIntoEnvironment( + "/etc/demo/secrets", + configkit.SecretsOptions{Overwrite: true}, +) +``` -- Extend main-document control parsing and the public load options to capture - the opted-in declaration and bind it to the main context already obtained - during acquisition. Carry the binding into secret resolution without another - search or parse. -- Reuse the secret-file reading and validation rules for exact-name lookup. - Extend the operation-local lookup to report value-free filesystem failures - and retain the provider for inspection separately from configuration value - provenance. -- Add the explicit startup operation around the shared load lifecycle. Use the - resolved file values for export and reuse the bootstrap reporting semantics - where they fit. -- Verify discovery from a different working directory, canonical main paths, - declaration scope, both lookup orders, and effective references after - composition and application resolution. Cover unused and missing files, - empty values, read errors, safe inspection, and startup export after - validation, including partial writes and absence of writes on a failed load. - Use disposable credential values and deterministic failure seams in line - with the [testing policy](../policy/testing.md). -- Document the resulting contract and a consumer startup example. Record the - architectural decision to add file-backed resolution and explicit startup - export alongside ADR 0004, and align the current architecture and component - documentation when the feature lands. +Both helpers use the same directory-bootstrap implementation. With overwrite +disabled, existing variables win, including empty ones, with presence checked +again immediately before application as today. With overwrite enabled, +eligible files are read even for existing variables and their values replace +those variables. Variables without a corresponding file are left alone. +An applied empty file supplies an empty value in either mode. -## Open Decisions +Bootstrap processes the directory's eligible entries, including credentials +used by SDKs or subprocesses outside typed `Secret` fields. Reuse the existing +dot-entry handling, file checks, content rules, limits, and filename ordering. +Validate metadata and read all content needed by the chosen overwrite policy +before beginning writes. +Directory and file failures produce the existing safe reporting behavior. +The secrets report identifies writes, skips, and partial failures without +retaining credential bytes. -### Startup API and export failure reporting +Call bootstrap during startup before starting concurrent work. Successful +bootstrap precedes configuration loading and validation. A later load failure +leaves the completed environment writes in place. A bootstrap write failure +stops further writes, leaves earlier writes in place, and reports remaining +entries as `not_applied`. The existing lack of rollback and concurrent +environment transaction guarantees continues to apply. -**Recommendation.** Provide explicit startup entry points corresponding to -ordinary and layered loading, backed by the same load implementation. Return -the configuration outcome and the export report so callers can distinguish a -failed load from a successful load followed by an environment write failure. -The successful load's private file values can be handed directly to export -within the call, which keeps the application flow short and the lifetime of -those values local to the startup operation. - -**Alternative.** Export from a successfully returned result through a separate -explicit call. This gives applications control over when to perform the writes -and avoids a parallel pair of load entry points. It requires a result contract -for retaining the original file-backed assignments or defining how later -application edits to the result affect export. +Keep bootstrap reporting separate from the subsequent configuration's secret +lookup evidence. Each operation describes what it observed and performed. +[Secrets bootstrap](../config.md#explicit-secrets-directory-bootstrap) · [Secret references and explicit bootstrap](../adr/0004-secret-references-and-explicit-bootstrap.md) ### Filesystem scope -**Recommendation.** Support discovered and explicitly selected host files -first, including their existing main-root options. These cover the requested -deployment pattern and allow reuse of the existing native secret-file checks. +Support discovered and explicitly selected host files, including their existing +main-root options. This covers the requested deployment pattern and reuses the +native secret-file checks. Caller-supplied filesystem support is a possible +later extension requiring decisions about file-type and symlink capabilities. -**Alternative.** Support caller-supplied filesystems with a usable main-file -context in the same change. This aligns with companion inputs and supports -virtual filesystems. It requires an explicit decision about file-type and -symlink handling where a caller filesystem exposes different metadata or -access capabilities. +## Work needed + +- Add `config.secrets_dir` and `config.secrets_overwrite` to the main-document + control grammar and consume them during ordinary loading without triggering + bootstrap. Retain source-local positions for diagnostics. +- Factor main-file selection and capture for use by the companion helper and + normal acquisition. Add a prepared-source handoff with retained main contents, + identity, context, and ownership. Select an API for supplying applicable + main-root constraints and representing an absent or explicit Go overwrite + choice, including `false`. +- Add the optional overwrite setting to direct-directory bootstrap and reuse + that operation from the companion helper, with matching reports and failures. +- Verify discovery from a different working directory, canonical main paths, + optional search exhaustion, declaration scope, default and explicit + overwrite choices, Go-over-YAML precedence, and both helper interfaces. + Cover empty values, dot entries, read failures before writes, and partial + write failures. Verify bootstrap includes entries unused by typed fields and + that later load failures preserve bootstrap effects. +- Verify prepared-source reuse, handle cleanup, normal loading without + environment writes, and agreement with selected-file provenance, placement, + relative imports, and main-relative companions. Use disposable credential + values and deterministic failure seams in line with the + [testing policy](../policy/testing.md). +- Document the startup workflow and add a consumer example. Record the prepared + main-source handoff and optional overwrite policy in an ADR referencing + ADR 0004, and update the canonical configuration contract and affected + architecture and component documentation when implemented.