27 KiB
ConfigKit v1.2.1 implementation plan
Scope and authority
Implement the approved ConfigKit roadmap. It owns the target, native-semantics intent, and four approved product decisions. This plan replaces the completed v0.5.0 plan. Its audience is a Luna coding agent implementing one end-to-end sprint in the numbered order below.
Select go.maximumdirect.net/configkit v1.2.1, upstream commit
51ef3144158db1a09fabdbd1c7ae59908c84923b. The pinned
consumer guide
and configuration contract
own upstream semantics. Read the sections identified by each stage and public
declarations in the released module as needed. The sibling checkout is optional
reference material, not a build or test prerequisite.
Working instructions
- Start with
docs/development.mdanddocs/policy/agents.md. Architecture, testing, documentation, OSS, and release policies apply; reuse unchanged policy text already read. Prefer graph tools for code discovery. - Implement the full native configuration workflow. Discovery, destinations, collection controls, resource bases, secondary imports, materialization, and native inspection are part of this sprint.
- Stages are work boundaries, not deployment releases. Intermediate stages may fail to compile or have dependent tests awaiting later work. Record that state accurately. Avoid temporary adapters or extra abstractions merely to make every intermediate commit green. Final validation covers the assembled system.
- Commit completed nonempty units with short plain-English messages. Add focused behavioral tests with their owner; run meaningful available checks. The final stage runs repository-wide validation against the finished source.
- Use ConfigKit's public loading, presence, provenance, recording, rendering, and comparison APIs. Narratio supplies domain meaning and runtime use. Retire obsolete source-authoring checks and tests rather than reproducing them.
- Leave import defaults unspecified so ConfigKit supplies its defaults. Explicit false/zero/empty/null and collection semantics follow the native contract and actual Narratio domain rules.
- Use temporary inputs, real ConfigKit, and controlled synthetic credentials. The default tests remain offline and independent of installed subprocesses.
- Write documentation in concise, affirmative, factual language. The roadmap owns intent; this file owns sequencing and implementation choices.
Preserve stage ordering, artifact dependency meaning, the canonical party schema, exact-name credential forwarding, publish commit semantics, and the existing Linux amd64/arm64 plus best-effort macOS arm64 targets. This sprint does not implement pending Audita/workspace features, migrate old sessions, rewrite downstream configurations, add fragment version mandates or new path checks, adopt ReleaseKit for Narratio, or publish a release.
Concrete integration choices
- Consumer Go minimum: 1.27.0. Development/release validation and build toolchain: official Go 1.27.4, an available patch on that line. Selecting it is not a new publication gate.
- Pipeline source when no explicit file is supplied:
SearchConfig("narratio", Searches(WorkingDirectory(), UserConfigDirectory(), UnixSystemDirectories()), SearchOptions{Required: true}). An explicit--configusesFileexclusively. - Bootstrap: optional
/usr/local/etc/narratio/secrets, or explicit--secrets-dir, throughLoadSecretsIntoEnvironmentbefore typed loading. Existing environment values win. The implicit sibling directory retires. - Local campaign/session/party loading enables imports. Only the pipeline gets its selected overlay. Party version checking remains main-only; fragments need no new version header. Remote session bytes are contextless snapshots.
- Existing campaign registry selection and local session search locations remain
domain conventions. Use native optional
SearchConfigwithSearchPathsfor local session candidates. Only actual absence permits remote fallback. - Final-domain inspection uses a safe DTO and a defaults-backed
Load(NoSource(), Definition.Defaults)snapshot after domain resolution. Credential fields contain reference names, not typed secrets that would resolve again. Original results supply actual file provenance. Stages 8–9 explain this boundary.
Stage 1: Establish the dependency and toolchain baseline [x]
Read go.mod, go.sum, .woodpecker/*.yml, release scripts/runbook, and
attribution. Read the upstream v1.0.0 migration table and v1.2.1 release note.
- Select Go 1.27.0 in
go.modand ConfigKit v1.2.1 under its vanity path. Migrate all production/test imports together and remove the old requirement. Use the released module rather than a filesystem replacement. - Replace APIs actually used here:
LoadSecretsDir→LoadSecretsIntoEnvironment,LayerBase→LayerMain, and genericResolution.Recordcalls withRecordDerivedDefaultorRecordNormalization. Audit other removed public names and compared layer or diagnostic strings. Use keyed expanded options/definition literals. - Rename genuine source-root
configkit:controls toconfig:throughout fixtures and examples. Keep the package name,configkit:"secret"Go tag, and ordinary data keys distinct from source controls. - Align Go CI images and local/release instructions with Go 1.27.4. Preserve platform targets and publication mechanics. Update incorporated Go notices.
- Refresh ConfigKit and stable go-yaml v4.0.0 attribution from selected source notices. Review actual module graph churn and tool-only dependencies. Keep Narratio's separate YAML v3 use unless another scoped change needs it.
Check module identity and attribution. Attempt focused config/app tests; record failures assigned to later stages. Complete when the dependency/API baseline is established, without preserving old behavioral expectations as compatibility.
Stage 2: Integrate native pipeline discovery and independent bootstrap [x]
Read internal/app/pipeline_config_path.go, secrets_bootstrap.go,
command_dependencies.go, loadPipelineConfigWithDependencies and callers,
inspection loaders, and internal/config/pipeline_configkit.go.
Read upstream discovery and secrets-bootstrap contracts.
- Have the configuration owner accept an explicit file or the native required
discovery source above. Retain explicit-file package entry points where useful.
Commands load through one composition path instead of first probing with
os.Statto obtain a default pathname. - Bootstrap once before typed loading or adapter creation from the independent directory. Preserve explicit-path error behavior, safe partial reports, and existing-empty environment precedence.
- Obtain the actual selected root and import/overlay evidence from ConfigKit. Carry that root into campaign selection, provenance, and command summaries. Remove dependencies/search helpers used only for old pipeline preselection.
- Apply this composition to run/range/alias commands, session init/plan/status/ restore, inspection, and remote-session workflows. A selected invalid source fails rather than falling through to another pipeline candidate.
- For overlay diff, load the left through native selection and the right from that selected root with its own overlay. Share invocation bootstrap; neither side performs a preliminary base-only load.
Test precedence and exclusive explicit selection with controlled working/user directories and temporary native search descriptions. Test selected invalid input, no source, bootstrap override/default behavior, and diff bootstrap once. Keep tests isolated from the operator's system configuration and secrets.
Stage 3: Resolve pipeline resources from native origin context [x]
Read resolvePipelineConfig, resolveNotariusPaths,
finalizeLoadedPipeline, effective_digest.go, and the upstream source-aware
path example. Resolve paths before normalization erases entry origins.
-
Introduce one small config-owned host-resource base helper using
Resolution.EntryInfo,Resolution.Source, andResolution.SourceContext. Use the value's effective origin rather than its child's inheritance directive. A distinct usable source context supplies its canonical base; missing or composite context falls back to the main source. Explicit CLI-origin file references use the captured invocation directory. If the main input is contextless too, use that invocation directory rather than interpreting a diagnostic label as a filesystem base. This covers remote byte snapshots without turning their temporary filenames into resource roots. -
Normalize nonempty relative references with recorded helpers. Absolute values stay absolute. Derive Notarius working directory from its already resolved config path and record that dependency. Preserve domain validation of relevant empty/disabled values.
-
Apply this classification:
Fields Interpretation Audita/Scriptorium/Notarius config_path; Notariusworking_directoryFile/directory resource based on effective origin. Relative workspace.root,campaigns.root,spool.root,cache.rootHost directory reference based on effective origin; absolute defaults stay unchanged. Campaign template/stable-input paths; session stable-input/audio paths Same resource policy, completed in Stage 4. Artifact output paths, family output patterns, publish destinations, source IDs, prompt/profile IDs, schema identifiers, binary commands, Notarius reference bindings and chunk-cache modes Existing domain interpretation, not file-resource rebasing. -
Remove duplicate pipeline-root rebasing and logical-Notarius path caches once callers use resolved domain values. Show/digest describe those domain values, rather than reconstructing an alternative authoring-path view.
-
Keep resource access at existing consumer boundaries; base selection adds no target-existence check, root restriction, or history acceptance gate.
Test imported, inherited, overlay, absent-default, absolute, and ambiguous-origin cases. Equal resolved resources have equal effective semantics across layouts; relative text resolving to a different resource is a meaningful change.
Stage 4: Load campaign/session imports with correct contexts
Read document_configkit.go, load.go, pathresolve.go,
party_resolution.go, app session discovery/remote-session composition, and
session init. Read native import and resource-context contracts.
- Use
Filewith imports enabled for local campaign/session inputs. Share definitions with byte loading and retain successful results/source contexts. ConfigKit owns per-source and aggregate acquisition limits. - Replace local session candidate probing with native optional discovery over the existing candidate paths. Permit absent input solely for remote fallback; distinguish absence through native source evidence. Structure callbacks to allow that discovery-only no-source result. Acquired invalid files still receive ordinary validation and fail without remote fallback.
- Downloaded session bytes receive no context inferred from their temporary filename. Retain object provenance and current explicit session/previous-ID checks. Use native contextless-import behavior; no remote import fetcher or custom structural rejection layer is added.
- Resolve campaign
session_template_file, stable-input paths, and session overrides/audio references with Stage 3's helper. Resolve eachaudio_fileselement at its entry index before ordering. Keep campaign/ session stable-input precedence and campaign ownership of party input. Provenance identifies the effective supplying configuration source. - Check unresolved template tokens in effective session string values, including imported values. Session init still reads an authored template for intentional substitution; runtime loading stays a configuration operation.
- Retain detached campaign and session document values after their own loading and resource resolution. Capture the session document before campaign-inherited fields are copied into runtime session state. Stage 6 serializes these values.
Test local imports, resource bases, effective placeholder rejection, identity mismatches, imported session-init template paths, absence versus selected-source failure, and remote snapshots independent of temporary directories. Add no secondary overlay flags.
Stage 5: Resolve imported party definitions and canonical roster output
Read party.go, party_resolution.go, artifact_families.go, the party
integration contract, and focused party/family tests.
- Add a file-backed party entry point with imports enabled for campaign
resolution. Keep
ParseParty(bytes)contextless for snapshots and tests. Reuse the definition, main-only version rule, and domain validator. - Preserve
narratio.party.v1, existing character/name/class/alias validation, and campaign ownership. Native destinations allow split roster contributions; no fragment version mandate or abstract/template record type is added. - Use native metadata for roster order and evidence, preserving authored order within a source. Extend the current ordering helper using source acquisition order, then line/column and ID as deterministic ties across sources. Prefer a child's own declaration/inheritance location over its parent's location. This is a presentation convention, not a new source acceptance rule.
- Render canonical party YAML from the effective roster: schema version, keyed
characters, singular external
alias, classes and optional levels. Preserve meaningful roster/list order. A small typed DTO/YAML-node writer is suitable for this Narratio-owned external format. - Keep players derivation and family expansion based on that same roster.
Replace output/checksum uses of
CanonicalParty.Rawwith canonical rendering; retain raw data only where an actual diagnostic caller needs it.
Test split/inherited rosters, aliases/classes, effective duplicate-name rejection, order, reloadable standalone canonical output, players, and family identities. Use native evidence rather than a second source parse to reconstruct order.
Stage 6: Materialize effective inputs and align prepare resume checks
Read internal/stage/prepare.go, prepare_party.go, prepare_resume.go,
prepared-input descriptors, and prepare semantic evidence/tests.
- Add configuration-owned deterministic campaign/session renderers from Stage 4's detached document values. Serialize public schemas and resolved file references, excluding native directives and runtime/provenance state.
- Preserve ownership: campaign-inherited stable inputs must not become explicit
session overrides. A published session that omitted
speakers_filemust still inherit the receiving campaign's value. Explicit session overrides remain explicit. Include actual session identity values where application resolution supplies them, without freezing campaign defaults into the session. - Prepare writes these documents and Stage 5's party bytes through existing artifact-store helpers. Keep filenames, input kinds, provenance fields, and write/checksum behavior. Resume uses the same renderers instead of comparing prepared effective output with a raw imported root.
- Ensure imported effective changes reach existing stage-owned semantic/input evidence. Equivalent domain values remain equivalent. Full pipeline digests, source mtimes, or every authoring file are not new blanket invalidation keys.
- Preserve ordinary stable-file copying/audio handling. Already resolved paths must not be rebased again against the session root. Retire raw campaign/session file requirements used solely for copying the old authoring documents.
Test standalone prepared documents, imported changes, equivalent layouts, and remote-origin metadata. Canonical-byte format changes can cause a normal prepare rerun; no session migration or new manifest schema is needed for serialization.
Stage 7: Close publication, restore, and remote-session round trips
Read publish input selection, remote-session loading, restore contracts, and existing in-memory storage integration tests.
- Trace the actual bytes published as remote
session.yml. Publish the canonical prepared session rather than the original root or command-scoped download. Preserve immutable object/commit-pointer ordering and lock behavior. - Confirm campaign/party prepared products flow through existing selections with their identities intact and without unresolved companion imports.
- Verify a published canonical session loads through the existing contextless path after authoring fragments and temporary files disappear. Restore uses recorded contents and existing safety/locking. No remote import graph or producer-host filesystem search is introduced.
- Verify omitted session stable fields inherit the receiving campaign's values. Keep remote object identity evidence separate from checksums of locally materialized canonical bytes.
Use fake/in-memory storage for prepare → publish selection → remote load and relevant restore coverage. Fix actual producers/consumers rather than adding a parallel publication path. Existing atomic commit tests remain authoritative.
Stage 8: Build native final-domain inspection snapshots
Read effective_projection.go, effective_digest.go,
effective_pipeline.go, pipeline finalization/family expansion, and native
completion, defaults, rendering, and typed-secret contracts.
- Keep one safe schema-specific final-domain DTO containing concrete expanded
artifacts and generated publish outputs. Omit unexpanded family declarations
and bookkeeping. Audit all credentials, including S3 fields and Audita/
Notarius/Scriptorium lists: use exact reference names, with no resolved
Secretfields in the inspection DTO. - Build a native DTO result with
Load(NoSource(), Options{Definition: Definition{Defaults: ...}}), supplying a fresh safe DTO and no external credential lookup or application callback rerun. ConfigKit copies and freezes these already resolved values. Using defaults avoids interpreting finalconfig/extendsdata as fresh directives or converting precise numbers through a generic mapping-source reconstruction. - Build/refresh the snapshot at existing domain-finalization boundaries, after applicable path/dependency/family resolution. Keep original authoring results separately. Synthetic default origins describe this derived snapshot, not original file provenance. No general snapshot service is needed.
- Route effective show and
pipeline.resolved.ymlrendering through nativeWriteEffective. Use the same DTO's deterministic JSON for Narratio's semantic digest. Preserve exact integers, domain null/empty values, and credential identities. Update the digest discriminator if its format changes. - Remove flattening/serialization machinery used only to duplicate native show/diff. Keep narrow path enumeration and family/party context genuinely needed by catalog/source presentation and stage-owned reuse evidence.
Test final family/publish expansion, inherited/expanded equivalence, large integer variables, null/empty values, safe rendering, credential-reference changes versus rotation, and no credential resolution during snapshot creation.
Stage 9: Adopt native diff and truthful operator source inspection
Read internal/app/config_diff.go, config command loaders/tests,
artifact_catalog.go, and native Diff/source/history contracts.
- Fully resolve each overlay against the same selected campaign/party before
comparing Stage 8 snapshots with
configkit.Diff. RemovediffEffectivePipelineValues, its general map comparison, and consistency checks predicated on the old flattened-leaf change model. - Present canonical JSON Pointer paths in native order. Derive added/removed/
changed from snapshot
Present, not nil or redaction text. Use compact JSON for safe values and exposeHiddenChangedif present. Equal values produceno differences. Document pointer notation and native changed-region/list granularity instead of emulating the old engine. - Source/catalog evidence comes from authoring results through public source
lookup and metadata history/origins. Add clearly labeled empty optional-import
selection evidence to
config sources, without guessed candidates or reasons for absence. - Preserve artifact/family/character IDs and generated party context. Distinguish
normalization origins from retained definition evidence. Use native
mainwhere an upstream layer is reported; root/import presentation roles may remain where they express actual application context.
Test changed family members and publish rules, equivalent layouts, composite contributors, optional empty imports, atomic lists, and pointer escaping in allowed dynamic mapping keys. Inspection remains secret-safe and read-only, without source reparsing or provenance-based acceptance gates.
Stage 10: Exercise native authoring and maintain ergonomic examples
Read the production/testing bundle, artifact/family tests, native destinations and collection rules, and maintained example tests.
- Convert artifact/family files to destination-based definitions. IDs remain explicit destination keys or keyed payloads; filenames do not imply IDs.
- Demonstrate a declared directory import with
pattern: "*.yml", essentialrequired: trueinput, an optional selection, inheritance, and a useful testing-overlay list customization throughconfig.sequences. - Cover append/upsert/replace on an existing list using an appropriate authored
identity, such as publish output
source. Whole-record upsert and empty-list behavior follow ConfigKit. No application identity registration is required merely to enable authored collection controls. - Demonstrate a split roster and local secondary imports using the sample campaign, with companion resource paths following source-origin policy. Explain that full-root and disjoint definitions remain accepted too.
- Use ordinary
Loadfor this authored-import workflow. Layered loading is useful for actual lower-priority sources, not required for defaults alone. If such a source is genuinely needed, useLoadLayeredandMainRelativedirectly with focused coverage instead of another merge/search mechanism.
Load examples through production definitions. Check dependent logical IDs, concrete family members, testing differences, and standalone prepared inputs. Keep examples synthetic and credential-free.
Stage 11: Align canonical documentation and project-specific policy
Read documentation/OSS/release policies, affected references, and the approved roadmap. Finish current-behavior documentation for the assembled implementation.
- Update config/CLI references, operations/bootstrap guidance, troubleshooting, configuration internals, prepare internals, and party integration docs for the new discovery, import, resource-base, materialization, and inspection behavior.
- Replace obsolete import restrictions with an accurate native summary and
pinned upstream links. Explain optional imports,
required, filters, destinations, source-local root controls, list modes, and reserved-key scope. Keep effective domain validation Narratio-owned. - Align project-specific architecture configuration/startup statements and the current raw-party-copy description in its actual owner. Preserve accepted ADR 0003 and generic adopted policies. Routine integration details do not require new ADRs.
- Write manual upgrade guidance for control names, native discovery filenames/ precedence, explicit custom filenames, secrets override, relative path changes, required imports where desired, and list operations. Explain effective prepared documents versus authored roots; downstream migrations remain separate.
- Finish Go prerequisites, CI/runbook toolchain alignment, and attribution from Stage 1. Record impact for eventual major-release preparation without inventing a release version or changing published notes. Leave pending Audita/workspace roadmaps and Proposed ADRs intact.
Run doccheck and maintained example validation. Each volatile contract has one canonical owner. Retire completed roadmap files only when separately requested.
Stage 12: Verify the assembled migration and close the sprint
- Extend a command-level temporary bundle with native discovery, explicit secrets override, imported campaign/party/session resources, destination-based artifacts, family expansion, and a testing overlay. Reuse preceding focused coverage instead of repeating ConfigKit's structural matrix.
- Exercise validate/show/sources/artifacts/diff through real composition. Check selected root, resources, generated values, optional-import evidence, safe credentials, and absence of runtime workspace creation by inspection. Exercise plan or prepare with fake external seams to establish execution parity.
- Trace remaining callers of old search, rebasing, raw-copy, and diff helpers. Remove dead code/test seams. Review intentional test expectation changes against the roadmap and preserve existing consequential domain/safety tests.
- With Go 1.27.4 and dependencies prepared, run
go test ./...,go test -race ./...,go vet ./...,go build ./...,go mod tidy -diff, andgit diff --check. Use the existing release asset builder with a temporary output directory and its existingv0.0.0-testlabel to check the three targets; this invokes no publication or new release note. - Inspect
go list -mfor the exact release, absence of the old module path and filesystem replacements, and expected notices. Review every roadmap capability and all four approved decisions; native structural behavior must not be narrowed by new Narratio acceptance gates. - Commit the finished unit and report checks, compatibility changes, revision, and any genuine unfinished work. Leave a clean workspace. Actual release publication and the next feature require separate authorization.
Complete when the native workflow works end to end, prepared inputs survive loss of authoring companions, operator inspection covers final domain values, and final validation passes.
Open Questions
None required for this sprint. The four product choices are approved. This plan selects concrete native snapshot, source-context, materialization, and toolchain approaches without adding compatibility shims, structural restrictions, or stronger runtime guarantees.