22 KiB
Configured secrets bootstrap implementation plan
Scope and execution
Implement the secrets-directory roadmap and accepted
ADR 0023. The starting
revision is fb0560d. Read development, the
agent policy, and the applicable architecture, testing,
and documentation policies before implementing a stage.
The stages run sequentially in one sprint. Intermediate stages may leave callers temporarily uncompilable while an internal handoff changes. Finish the shared integration before final validation. Keep tests with their behavior's owning stage and reuse existing fixtures and failure seams. Each stage is intended to fit one Luna implementation prompt.
The completed feature provides explicit bootstrap before ordinary configuration loading. Discovery and the captured main document are shared between those two calls. Ordinary secret resolution continues to read the environment at its existing lifecycle point. Filesystem access and bootstrap finish synchronously.
Decisions for implementation
Public API
Keep the existing function and its exact function type. Add these root-owned public declarations, following the existing facade-to-engine conversion pattern.
func LoadSecretsIntoEnvironment(directory string) (*SecretsReport, error)
type SecretsOptions struct {
Overwrite bool
}
func LoadSecretsIntoEnvironmentWithOptions(
directory string, options SecretsOptions,
) (*SecretsReport, error)
type SecretsFromConfigOptions struct {
Filesystem *FilesystemContext
Overwrite *bool
}
func LoadSecretsIntoEnvironmentFromConfig(
main Source, supplied ...SecretsFromConfigOptions,
) (*SecretsReport, Source, error)
The companion accepts zero or one options value, matching the load entry points.
Multiple values produce LoadError with CodeOptionsInvalid, an empty non-nil
report, and zero Source before acquisition or writes. Snapshot the supplied
context and Boolean value at call entry using existing conversion conventions.
Nil overwrite uses YAML, then false. A supplied false or true wins over YAML.
The companion accepts host File and SearchConfig sources and NoSource.
A prepared host source can also be supplied to an explicit companion call using
its captured main. Reject other source kinds and native filesystem contexts as
LoadError option failures before consuming them. Preserve constructor and search
validation diagnostics through the existing helpers. NoSource and optional search
exhaustion succeed with an empty report and NoSource(). Explicit context on an
absent input follows existing absent-host-context validation; subsequent absent
loading uses the ordinary caller-supplied context rules.
Every failure returns a non-nil report and zero Source. Discovery, acquisition,
parse, control, and preparation-context failures use LoadError. Directory
resolution/inspection, credential reads, and environment writes use
SecretsError. Keep both error wrappers and their existing safe rendering and
detached records. Successful preparation with no directory returns the prepared
main and an empty report.
Controls and source roles
Recognize root config.secrets_dir as a nonempty string containing a relative
host path and root config.secrets_overwrite as a Boolean. Accept ordinary YAML
Boolean spellings through the parser's existing type handling. Nulls, wrong
types, empty directory strings, absolute paths, and host volume-qualified paths
are invalid. Treat path text literally using host path operations. Parent-relative
paths are usable when their canonical targets stay inside the selected root.
Only the main input's document root owns these controls. Imports, underlays,
and selected overlays containing them produce a source-local control diagnostic.
Nested application config mappings and root application data introduced by
placement, defaults, CLI, or resolution retain their existing meaning under
ADR 0020.
Normal loading checks and consumes the main controls, with imports enabled or disabled. Filesystem checks for the declared secrets directory happen in the companion helper. Both ordinary loading entry points perform their usual secret lookup. Preparation validates the root control mapping and the bootstrap fields, while other control grammars and import enablement are checked by the later load. Retain the original main tree so imports and collection declarations are still available to that load. A valid overwrite declaration without a directory is a no-op. Invalid bootstrap fields fail even if a Go overwrite choice is supplied.
Use existing diagnostic codes. Use CodeSecretsInvalidPath for invalid directory
syntax, CodeSecretsInvalidValue for a non-Boolean overwrite setting, and the
existing directive-context codes for misplaced controls (CodeImportInvalid or
CodeOverlayDirective). Root mapping errors keep their current classification.
Bootstrap-field diagnostics carry /config/secrets_dir or
/config/secrets_overwrite and source-local locations, inside a LoadError.
Avoid raw input, parser errors, or credential contents in messages.
Captured-source ownership and context
Represent preparation privately as a captured host-file payload attached to an
engine sourceFile. Keep public Source opaque and report SourceKindFile.
Capture the original parsed document, original byte count, canonical main path,
canonical base, and canonical containment root. Retain the unconsumed controls.
Keep captured data read-only and give each load an owned document clone with
fresh source IDs and accounting. Namespace tokens and acquisition state belong
to each load. Retain passive path facts with the source.
Normal acquisition consumes the captured main without discovery, file reading, or bootstrap. Apply the usual byte, document, placement, and aggregate accounting to the captured input in every load. A preparation-time count never substitutes for load-wide admission. Imports and resources continue to be acquired when their normal operations run. Main deletion or modification does not change captured contents; later filesystem operations may fail under their existing contracts.
Use the captured context when later loading supplies none. Resolve an explicit
later host context against the captured canonical main path. It is compatible
when its canonical root, base, and main path match the captured facts. Compare
path facts using the existing host path rules, allowing equivalent spellings
and root symlink aliases. Native contexts, a supplied file BaseDir, and genuinely
different roots fail with CodeOptionsInvalid in LoadError. Avoid re-resolving
the original main filename to compare contexts, since it may now point elsewhere.
Use the same check for ordinary options and layered main input contexts before
import or overlay handling can substitute a context.
Directory bootstrap semantics
Reuse loadSecretsDir and its environment/access seams for all three public
helpers. Preservation mode retains both existing-variable checks, including
empty values and a variable created between preflight and application.
Overwrite mode reads existing names' eligible files and applies them regardless
of environment presence. Unrelated environment variables retain their values.
Use the existing loaded/skipped/failed/not-applied statuses. ExistingEmpty
continues to describe skipped existing values.
Configured directory resolution uses the captured base and root. Canonicalize
the selected directory with the existing filesystem helpers and check containment
of that canonical target before reads or writes. A directory symlink can resolve
to a contained target, following the existing bootstrap directory convention.
Return SecretsError with CodeSecretsInvalidPath for an outside-root target;
use existing filesystem classifications for missing or unreadable targets.
The direct-directory helpers keep their existing path resolution behavior.
Bootstrap includes all eligible directory entries, independently of the later typed schema. Reuse dot-entry exclusion, regular-file checks, limits, sorted order, newline handling, empty-file handling, and NUL rejection. Complete metadata checks and required reads before writes. A write failure retains prior writes and reports remaining entries as not applied. Later load failures leave bootstrap effects in place. Reports and prepared sources retain no credential bytes from secret files. Applications own startup ordering and concurrent process-environment mutation under the existing bootstrap contract.
Stage 1 — Extend shared directory bootstrap with overwrite
Work in internal/engine/secrets.go and internal/engine/secrets_test.go.
- Add a small engine options value for overwrite and keep
LoadSecretsDirdelegating with preservation enabled. Add its options-bearing counterpart. Extend the shared private worker and existing test seams accordingly. - Make overwrite affect only staging and applying values. Preserve sorted enumeration, all metadata validation before reads, bounded reads before writes, skipped-file behavior in preservation mode, and failure reporting.
- Add focused behavioral cases for overwriting nonempty and empty existing variables, overwriting with empty file content, and leaving names without files alone. Extend the existing staged-presence-race and partial-apply-failure cases to cover overwrite. Reuse current read-limit, dot-file, NUL, and safe error coverage. Process-mutating tests restore state and run serially.
Stage completion means both policies work through one bootstrap worker and
existing preservation behavior remains covered. Run the relevant existing and
new LoadSecretsDir tests once callers compile.
Stage 2 — Add main-scoped bootstrap control parsing
Work primarily in internal/engine/directives.go, acquisition.go, and the
directive/version parsing handoffs in source_version.go.
- Introduce a small private bootstrap-settings value with declaration presence, directory text, overwrite value, and source-local diagnostic locations. Share its field validation between preparation and normal control extraction.
- Give extraction enough source-role information to allow these controls only
on the actual main document.
rootDirectiveContextSourceandrootDirectiveContextLayeredalone cannot identify that role because those contexts also occur on contributions. Carry the role through the relevant acquisition/version and import traversal call sites. - Keep bootstrap-only extraction usable before normal imports/sequence grammar validation. The companion can prepare a main that declares imports without enabling or acquiring them. Normal extraction checks the complete control block and removes it before placement and application decoding.
- Extend directive-scope and facade-control tests with main declarations, disabled imports, wrong types, explicit false, misplaced controls, and nested ordinary data. Verify source-version control-overlap checks recognize the new fields and that version acceptance itself stays in the load.
Stage completion means the normal load accepts and consumes valid main settings while leaving environment state untouched. Other source-root controls retain their existing grammar, enablement, and error precedence.
Stage 3 — Share host-main selection and capture
Work in internal/engine/source.go, parse.go, discovery.go, and the initial
selection/read portion of acquisition.go. A focused prepared_source.go may
hold the new private payload and preparation helpers.
- Factor the existing selection and bounded main parsing handoff for reuse by
normal acquisition and preparation. Reuse
resolveSearchInput, search option checks,readSource, and YAML adapter diagnostics. Keep candidate ordering, optional exhaustion, selected-file failure, and regular-file semantics. - Add the captured host payload specified above. Preserve original source line
locations and all controls. Construct independent documents for subsequent
loads with
document.CloneandsetDocumentSourceID. Count captured bytes under the current read limits before returning a document to acquisition. - For capture, resolve and validate the host base/root through existing
filesystemOptionDiagnostics,resolveFilesystemContext, and directory validation. Store path facts without access handles or a preparation token. - Add internal cases proving captured data stays fixed after the original file changes, becomes absent, or another search candidate appears. Retain ordinary File/SearchConfig reread behavior. Cover parser/read failures and limits with existing narrow test fixtures. Keep sources and payloads private until the public wrapper stage.
Stage completion means the acquisition front end can consume an ordinary host file or its captured payload through the same parsed-source handoff.
Stage 4 — Integrate captured contexts with ordinary loading
Work in options.go, filesystem_context.go, filesystem.go, acquisition.go,
load_input.go, selected_overlay.go, and source_context.go as needed.
- Make context resolution select captured path facts and enforce the explicit context equivalence rule at the shared acquisition boundary. Ensure both imported and standalone branches receive the selected resolved context.
- Carry the captured base/root into standalone main acquisition, ordinary overlay resolution, and passive source-context completion. Ordinary overlays keep their established root-relative lookup convention.
- Preserve file-kind records, canonical paths, source-local diagnostics, and placement projection. Reuse normal source registration and accounting.
- Add focused internal cases for omitted/equivalent/incompatible later contexts, explicit wider roots, a symlinked selected main, main removal, destinations, and ordinary overlay/resource-context behavior. Compare relevant records and values with a normal load of the same unmodified file.
Stage completion means an ordinary load uses captured contents and context through all its normal paths. Real filesystem failures are handled at the existing access boundaries without retaining an open handle in the source.
Stage 5 — Integrate imports and layered main-relative consumers
Work in import_acquisition.go, load_request.go, and layered/context owners
identified by the shared acquisition handoff.
- Feed the captured main document and resolved context into imported acquisition. Charge its raw bytes and nodes through existing accounting exactly once per load. Keep placement wrapper accounting and per-tier repetition tracking.
- Treat the captured main as the same canonical host file for repeated-file and cycle checks. A fresh load gets its own host namespace and source IDs.
- Ensure layered acquisition propagates the captured main context to declared MainRelative underlays and selected overlays, with existing explicit context and fallback precedence. Preserve main-first acquisition and tier precedence.
- Verify live imports after main capture, repeat/cycle detection involving the captured main, source/version evidence with imports, an explicitly wider root permitting contained parent-relative imports, and independent repeated loads. Include one integrated layered case with a main-relative underlay and overlay. Use existing import/placement/version suites for unchanged semantics.
Stage completion means prepared main sources behave as host main inputs across ordinary and layered composition, metadata, and main-relative bindings.
Stage 6 — Implement configured bootstrap orchestration
Add a focused engine companion in secrets_from_config.go, using the capture
and bootstrap helpers from earlier stages.
- Validate and snapshot options, resolve the main, capture its document and context, and extract bootstrap settings. Return the no-directory and absent cases under the public return contract. A valid Go overwrite override does not bypass validation of malformed YAML bootstrap fields.
- Join the declared directory to the captured base. Add an optional internal containment-root constraint to shared directory preparation so canonical resolution and containment validation occur before the worker uses that canonical directory. Direct-directory calls supply no such constraint.
- Select overwrite using Go, YAML, then false, and invoke the shared worker. Return the prepared source only after successful bootstrap. All branches return the non-nil report and appropriate existing error family.
- Test no declaration, overwrite without a directory, optional/required missing main, unsupported source/context, malformed controls, missing or outside-root directory, and directory symlink containment. Assert empty/partial reports and zero prepared sources on errors. Use the existing environment seam for deterministic apply failures and verify safe error/report output.
Stage completion means configured bootstrap can finish independently of schema, imports, version hooks, resolution, and validation, returning a reusable main only when its own operation succeeds.
Stage 7 — Expose root-owned APIs and convert both error families
Work in root secrets.go, source.go, and existing facade conversion owners.
- Add the two options types and new functions with the signatures above. Keep the existing function type intact and route its default behavior through the shared bootstrap implementation.
- Convert the companion's report, prepared engine Source, and errors to the
existing public wrappers. Preserve
LoadErrorsource records as well as diagnostics. PreserveSecretsErrorwithout exposing underlying filesystem or environment errors. Add only the small conversion logic this boundary needs, reusing existing converters. - Document signatures, options precedence, reports, error types, and source ownership on their declarations. Explain captured reuse versus File/SearchConfig reuse in Source's documentation. Keep implementation types private.
- Add facade tests for both new functions and assignment of the old function to
func(string) (*SecretsReport, error). Exercise YAML true with explicit Go false, YAML false with Go true, omitted choices, and multiple options. Verify publicerrors.Asbehavior for preparation and bootstrap failures.
Stage completion means applications can use the complete two-call startup flow through ConfigKit's public package with the documented ownership and errors.
Stage 8 — Verify the complete startup handoff
Extend public integration coverage and add one maintained consumer startup example with ordinary and layered use explained from the same helper.
- Start in a different working directory, discover a main with relative secrets, bootstrap all eligible entries including an SDK-only name, and load the prepared source. Verify configured overwrite and existing empty-variable behavior with disposable environment names.
- Change the main, discovery candidates, an import, and an environment value after preparation. Verify captured main values, live import values, fresh environment lookup, and independent metadata/results across subsequent loads. Include selected-main provenance and source resource context assertions.
- Exercise the layered handoff with explicit root preparation and companion inputs. Confirm compatible later context succeeds and incompatible context fails safely after the already-completed bootstrap.
- Show a later version/decoding or validation failure preserves the bootstrap report and environment effects. Separately verify ordinary Load/LoadLayered on a main containing the controls perform no environment writes, even with YAML overwrite true. Existing secret-resolution tests continue to own credential lookup and redaction behavior.
- Verify captured sources retain no handles through successful and failed preparation. Use existing resource-lifetime techniques and narrow seams, avoiding a new filesystem framework. Exercise independent captured-source loads under the race detector without concurrent process mutation.
Stage completion means the feature works through normal entry points and its integration risks are covered without duplicating every internal test at the facade boundary.
Stage 9 — Update canonical documentation and validate the sprint
- Update
docs/config.mdwith the two main controls, preparation and bootstrap boundaries, lookup timing, overwrite precedence, reports/errors, host scope, and prepared-source reuse/context behavior. Keep source-local controls distinct from application paths and preserve existing loading semantics. - Update
docs/consumers/README.mdand its test map to link the new maintained example. Updatedocs/development.mdonly where navigation/test ownership changes. Explain internal capture and acquisition handoffs indocs/internal/overview.mdand relevant import documentation. - Update the implemented-behavior paragraphs in architecture policy and retire its roadmap implementation-status link. Keep ADR 0023's accepted decision text intact. Preserve the existing bootstrap limits and concurrency/rollback contract while documenting explicit overwrite as the selected extension.
- Format changed Go files and run the focused tests for changed owners. Run
./scripts/check.shandgo test -race ./...on the completed implementation. Verify edited Markdown links and headings, exported declaration comments, and the public example. Reuse successful checks for unchanged content. - Review the final diff for captured-state aliasing, handles, extra parsing or discovery, secret disclosure, omitted accounting, and accidental changes to current configuration processing or helper signatures. Resolve failures and finish the documented target before marking the sprint complete.
The final result is the implemented feature with current documentation and appropriate behavioral coverage. Release preparation follows its separate procedure when requested.