Files
narratio/docs/roadmap/implementation.md

432 lines
27 KiB
Markdown
Raw Normal View History

2026-11-17 20:41:34 +00:00
# ConfigKit v1.2.1 implementation plan
## Scope and authority
2026-11-17 20:41:34 +00:00
Implement [the approved ConfigKit roadmap](configkit.md). It owns the target,
native-semantics intent, and four approved product decisions. This plan replaces
the completed v0.5.0 plan. Its audience is a Luna coding agent implementing one
end-to-end sprint in the numbered order below.
Select `go.maximumdirect.net/configkit v1.2.1`, upstream commit
`51ef3144158db1a09fabdbd1c7ae59908c84923b`. The pinned
[consumer guide](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/consumers/README.md)
and [configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md)
own upstream semantics. Read the sections identified by each stage and public
declarations in the released module as needed. The sibling checkout is optional
reference material, not a build or test prerequisite.
## Working instructions
2026-11-17 20:41:34 +00:00
- Start with `docs/development.md` and `docs/policy/agents.md`. Architecture,
testing, documentation, OSS, and release policies apply; reuse unchanged
policy text already read. Prefer graph tools for code discovery.
- Implement the full native configuration workflow. Discovery, destinations,
collection controls, resource bases, secondary imports, materialization, and
native inspection are part of this sprint.
- Stages are work boundaries, not deployment releases. Intermediate stages may
fail to compile or have dependent tests awaiting later work. Record that state
accurately. Avoid temporary adapters or extra abstractions merely to make
every intermediate commit green. Final validation covers the assembled system.
- Commit completed nonempty units with short plain-English messages. Add focused
behavioral tests with their owner; run meaningful available checks. The final
stage runs repository-wide validation against the finished source.
- Use ConfigKit's public loading, presence, provenance, recording, rendering,
and comparison APIs. Narratio supplies domain meaning and runtime use.
Retire obsolete source-authoring checks and tests rather than reproducing them.
- Leave import defaults unspecified so ConfigKit supplies its defaults.
Explicit false/zero/empty/null and collection semantics follow the native
contract and actual Narratio domain rules.
- Use temporary inputs, real ConfigKit, and controlled synthetic credentials.
The default tests remain offline and independent of installed subprocesses.
- Write documentation in concise, affirmative, factual language. The roadmap
owns intent; this file owns sequencing and implementation choices.
Preserve stage ordering, artifact dependency meaning, the canonical party schema,
exact-name credential forwarding, publish commit semantics, and the existing
Linux amd64/arm64 plus best-effort macOS arm64 targets. This sprint does not
implement pending Audita/workspace features, migrate old sessions, rewrite
downstream configurations, add fragment version mandates or new path checks,
adopt ReleaseKit for Narratio, or publish a release.
## Concrete integration choices
- Consumer Go minimum: **1.27.0**. Development/release validation and build
toolchain: official **Go 1.27.4**, an available patch on that line. Selecting
it is not a new publication gate.
- Pipeline source when no explicit file is supplied:
`SearchConfig("narratio", Searches(WorkingDirectory(), UserConfigDirectory(),
UnixSystemDirectories()), SearchOptions{Required: true})`.
An explicit `--config` uses `File` exclusively.
- Bootstrap: optional `/usr/local/etc/narratio/secrets`, or explicit
`--secrets-dir`, through `LoadSecretsIntoEnvironment` before typed loading.
Existing environment values win. The implicit sibling directory retires.
- Local campaign/session/party loading enables imports. Only the pipeline gets
its selected overlay. Party version checking remains main-only; fragments
need no new version header. Remote session bytes are contextless snapshots.
- Existing campaign registry selection and local session search locations remain
domain conventions. Use native optional `SearchConfig` with `SearchPaths`
for local session candidates. Only actual absence permits remote fallback.
- Final-domain inspection uses a safe DTO and a defaults-backed
`Load(NoSource(), Definition.Defaults)` snapshot after domain resolution.
Credential fields contain reference names, not typed secrets that would
resolve again. Original results supply actual file provenance. Stages 8–9
explain this boundary.
## Stage 1: Establish the dependency and toolchain baseline [x]
2026-11-17 20:41:34 +00:00
Read `go.mod`, `go.sum`, `.woodpecker/*.yml`, release scripts/runbook, and
attribution. Read the upstream v1.0.0 migration table and v1.2.1 release note.
1. Select Go 1.27.0 in `go.mod` and ConfigKit v1.2.1 under its vanity path.
Migrate all production/test imports together and remove the old requirement.
Use the released module rather than a filesystem replacement.
2. Replace APIs actually used here: `LoadSecretsDir` →
`LoadSecretsIntoEnvironment`, `LayerBase` → `LayerMain`, and generic
`Resolution.Record` calls with `RecordDerivedDefault` or
`RecordNormalization`. Audit other removed public names and compared layer
or diagnostic strings. Use keyed expanded options/definition literals.
3. Rename genuine source-root `configkit:` controls to `config:` throughout
fixtures and examples. Keep the package name, `configkit:"secret"` Go tag,
and ordinary data keys distinct from source controls.
4. Align Go CI images and local/release instructions with Go 1.27.4. Preserve
platform targets and publication mechanics. Update incorporated Go notices.
5. Refresh ConfigKit and stable go-yaml v4.0.0 attribution from selected source
notices. Review actual module graph churn and tool-only dependencies. Keep
Narratio's separate YAML v3 use unless another scoped change needs it.
Check module identity and attribution. Attempt focused config/app tests; record
failures assigned to later stages. Complete when the dependency/API baseline is
established, without preserving old behavioral expectations as compatibility.
## Stage 2: Integrate native pipeline discovery and independent bootstrap [x]
2026-11-17 20:41:34 +00:00
Read `internal/app/pipeline_config_path.go`, `secrets_bootstrap.go`,
`command_dependencies.go`, `loadPipelineConfigWithDependencies` and callers,
inspection loaders, and `internal/config/pipeline_configkit.go`.
Read upstream discovery and secrets-bootstrap contracts.
1. Have the configuration owner accept an explicit file or the native required
discovery source above. Retain explicit-file package entry points where useful.
Commands load through one composition path instead of first probing with
`os.Stat` to obtain a default pathname.
2. Bootstrap once before typed loading or adapter creation from the independent
directory. Preserve explicit-path error behavior, safe partial reports, and
existing-empty environment precedence.
3. Obtain the actual selected root and import/overlay evidence from ConfigKit.
Carry that root into campaign selection, provenance, and command summaries.
Remove dependencies/search helpers used only for old pipeline preselection.
4. Apply this composition to run/range/alias commands, session init/plan/status/
restore, inspection, and remote-session workflows. A selected invalid source
fails rather than falling through to another pipeline candidate.
5. For overlay diff, load the left through native selection and the right from
that selected root with its own overlay. Share invocation bootstrap; neither
side performs a preliminary base-only load.
Test precedence and exclusive explicit selection with controlled working/user
directories and temporary native search descriptions. Test selected invalid
input, no source, bootstrap override/default behavior, and diff bootstrap once.
Keep tests isolated from the operator's system configuration and secrets.
## Stage 3: Resolve pipeline resources from native origin context [x]
2026-11-17 20:41:34 +00:00
Read `resolvePipelineConfig`, `resolveNotariusPaths`,
`finalizeLoadedPipeline`, `effective_digest.go`, and the upstream source-aware
path example. Resolve paths before normalization erases entry origins.
1. Introduce one small config-owned host-resource base helper using
`Resolution.EntryInfo`, `Resolution.Source`, and
`Resolution.SourceContext`. Use the value's effective origin rather than
its child's inheritance directive. A distinct usable source context supplies
its canonical base; missing or composite context falls back to the main source.
Explicit CLI-origin file references use the captured invocation directory.
If the main input is contextless too, use that invocation directory rather
than interpreting a diagnostic label as a filesystem base. This covers remote
byte snapshots without turning their temporary filenames into resource roots.
2. Normalize nonempty relative references with recorded helpers. Absolute values
stay absolute. Derive Notarius working directory from its already resolved
config path and record that dependency. Preserve domain validation of relevant
empty/disabled values.
3. Apply this classification:
| Fields | Interpretation |
| --- | --- |
| Audita/Scriptorium/Notarius `config_path`; Notarius `working_directory` | File/directory resource based on effective origin. |
| Relative `workspace.root`, `campaigns.root`, `spool.root`, `cache.root` | Host directory reference based on effective origin; absolute defaults stay unchanged. |
| Campaign template/stable-input paths; session stable-input/audio paths | Same resource policy, completed in Stage 4. |
| Artifact output paths, family output patterns, publish destinations, source IDs, prompt/profile IDs, schema identifiers, binary commands, Notarius reference bindings and chunk-cache modes | Existing domain interpretation, not file-resource rebasing. |
4. Remove duplicate pipeline-root rebasing and logical-Notarius path caches once
callers use resolved domain values. Show/digest describe those domain values,
rather than reconstructing an alternative authoring-path view.
5. Keep resource access at existing consumer boundaries; base selection adds no
target-existence check, root restriction, or history acceptance gate.
Test imported, inherited, overlay, absent-default, absolute, and ambiguous-origin
cases. Equal resolved resources have equal effective semantics across layouts;
relative text resolving to a different resource is a meaningful change.
## Stage 4: Load campaign/session imports with correct contexts [x]
2026-11-17 20:41:34 +00:00
Read `document_configkit.go`, `load.go`, `pathresolve.go`,
`party_resolution.go`, app session discovery/remote-session composition, and
session init. Read native import and resource-context contracts.
1. Use `File` with imports enabled for local campaign/session inputs. Share
definitions with byte loading and retain successful results/source contexts.
ConfigKit owns per-source and aggregate acquisition limits.
2. Replace local session candidate probing with native optional discovery over
the existing candidate paths. Permit absent input solely for remote fallback;
distinguish absence through native source evidence. Structure callbacks to
allow that discovery-only no-source result. Acquired invalid files still
receive ordinary validation and fail without remote fallback.
3. Downloaded session bytes receive no context inferred from their temporary
filename. Retain object provenance and current explicit session/previous-ID
checks. Use native contextless-import behavior; no remote import fetcher or
custom structural rejection layer is added.
4. Resolve campaign `session_template_file`, stable-input paths, and session
overrides/audio references with Stage 3's helper. Resolve each
`audio_files` element at its entry index before ordering. Keep campaign/
session stable-input precedence and campaign ownership of party input.
Provenance identifies the effective supplying configuration source.
5. Check unresolved template tokens in effective session string values, including
imported values. Session init still reads an authored template for intentional
substitution; runtime loading stays a configuration operation.
6. Retain detached campaign and session document values after their own loading
and resource resolution. Capture the session document before campaign-inherited
fields are copied into runtime session state. Stage 6 serializes these values.
Test local imports, resource bases, effective placeholder rejection, identity
mismatches, imported session-init template paths, absence versus selected-source
failure, and remote snapshots independent of temporary directories. Add no
secondary overlay flags.
## Stage 5: Resolve imported party definitions and canonical roster output [x]
2026-11-17 20:41:34 +00:00
Read `party.go`, `party_resolution.go`, `artifact_families.go`, the party
integration contract, and focused party/family tests.
1. Add a file-backed party entry point with imports enabled for campaign
resolution. Keep `ParseParty(bytes)` contextless for snapshots and tests.
Reuse the definition, main-only version rule, and domain validator.
2. Preserve `narratio.party.v1`, existing character/name/class/alias validation,
and campaign ownership. Native destinations allow split roster contributions;
no fragment version mandate or abstract/template record type is added.
3. Use native metadata for roster order and evidence, preserving authored order
within a source. Extend the current ordering helper using source acquisition
order, then line/column and ID as deterministic ties across sources. Prefer a
child's own declaration/inheritance location over its parent's location.
This is a presentation convention, not a new source acceptance rule.
4. Render canonical party YAML from the effective roster: schema version, keyed
characters, singular external `alias`, classes and optional levels. Preserve
meaningful roster/list order. A small typed DTO/YAML-node writer is suitable
for this Narratio-owned external format.
5. Keep players derivation and family expansion based on that same roster.
Replace output/checksum uses of `CanonicalParty.Raw` with canonical rendering;
retain raw data only where an actual diagnostic caller needs it.
Test split/inherited rosters, aliases/classes, effective duplicate-name rejection,
order, reloadable standalone canonical output, players, and family identities.
Use native evidence rather than a second source parse to reconstruct order.
## Stage 6: Materialize effective inputs and align prepare resume checks [x]
2026-11-17 20:41:34 +00:00
Read `internal/stage/prepare.go`, `prepare_party.go`, `prepare_resume.go`,
prepared-input descriptors, and prepare semantic evidence/tests.
1. Add configuration-owned deterministic campaign/session renderers from Stage
4's detached document values. Serialize public schemas and resolved file
references, excluding native directives and runtime/provenance state.
2. Preserve ownership: campaign-inherited stable inputs must not become explicit
session overrides. A published session that omitted `speakers_file` must
still inherit the receiving campaign's value. Explicit session overrides
remain explicit. Include actual session identity values where application
resolution supplies them, without freezing campaign defaults into the session.
3. Prepare writes these documents and Stage 5's party bytes through existing
artifact-store helpers. Keep filenames, input kinds, provenance fields, and
write/checksum behavior. Resume uses the same renderers instead of comparing
prepared effective output with a raw imported root.
4. Ensure imported effective changes reach existing stage-owned semantic/input
evidence. Equivalent domain values remain equivalent. Full pipeline digests,
source mtimes, or every authoring file are not new blanket invalidation keys.
5. Preserve ordinary stable-file copying/audio handling. Already resolved paths
must not be rebased again against the session root. Retire raw campaign/session
file requirements used solely for copying the old authoring documents.
Test standalone prepared documents, imported changes, equivalent layouts, and
remote-origin metadata. Canonical-byte format changes can cause a normal prepare
rerun; no session migration or new manifest schema is needed for serialization.
## Stage 7: Close publication, restore, and remote-session round trips [x]
2026-11-17 20:41:34 +00:00
Read publish input selection, remote-session loading, restore contracts, and
existing in-memory storage integration tests.
1. Trace the actual bytes published as remote `session.yml`. Publish the canonical
prepared session rather than the original root or command-scoped download.
Preserve immutable object/commit-pointer ordering and lock behavior.
2. Confirm campaign/party prepared products flow through existing selections
with their identities intact and without unresolved companion imports.
3. Verify a published canonical session loads through the existing contextless
path after authoring fragments and temporary files disappear. Restore uses
recorded contents and existing safety/locking. No remote import graph or
producer-host filesystem search is introduced.
4. Verify omitted session stable fields inherit the receiving campaign's values.
Keep remote object identity evidence separate from checksums of locally
materialized canonical bytes.
Use fake/in-memory storage for prepare → publish selection → remote load and
relevant restore coverage. Fix actual producers/consumers rather than adding a
parallel publication path. Existing atomic commit tests remain authoritative.
## Stage 8: Build native final-domain inspection snapshots [x]
2026-11-17 20:41:34 +00:00
Read `effective_projection.go`, `effective_digest.go`,
`effective_pipeline.go`, pipeline finalization/family expansion, and native
completion, defaults, rendering, and typed-secret contracts.
1. Keep one safe schema-specific final-domain DTO containing concrete expanded
artifacts and generated publish outputs. Omit unexpanded family declarations
and bookkeeping. Audit all credentials, including S3 fields and Audita/
Notarius/Scriptorium lists: use exact reference names, with no resolved
`Secret` fields in the inspection DTO.
2. Build a native DTO result with `Load(NoSource(), Options{Definition:
Definition{Defaults: ...}})`, supplying a fresh safe DTO and no external
credential lookup or application callback rerun. ConfigKit copies and freezes
these already resolved values. Using defaults avoids interpreting final
`config`/`extends` data as fresh directives or converting precise numbers
through a generic mapping-source reconstruction.
3. Build/refresh the snapshot at existing domain-finalization boundaries, after
applicable path/dependency/family resolution. Keep original authoring results
separately. Synthetic default origins describe this derived snapshot, not
original file provenance. No general snapshot service is needed.
4. Route effective show and `pipeline.resolved.yml` rendering through native
`WriteEffective`. Use the same DTO's deterministic JSON for Narratio's
semantic digest. Preserve exact integers, domain null/empty values, and
credential identities. Update the digest discriminator if its format changes.
5. Remove flattening/serialization machinery used only to duplicate native
show/diff. Keep narrow path enumeration and family/party context genuinely
needed by catalog/source presentation and stage-owned reuse evidence.
Test final family/publish expansion, inherited/expanded equivalence, large
integer variables, null/empty values, safe rendering, credential-reference
changes versus rotation, and no credential resolution during snapshot creation.
## Stage 9: Adopt native diff and truthful operator source inspection [x]
2026-11-17 20:41:34 +00:00
Read `internal/app/config_diff.go`, config command loaders/tests,
`artifact_catalog.go`, and native Diff/source/history contracts.
1. Fully resolve each overlay against the same selected campaign/party before
comparing Stage 8 snapshots with `configkit.Diff`. Remove
`diffEffectivePipelineValues`, its general map comparison, and consistency
checks predicated on the old flattened-leaf change model.
2. Present canonical JSON Pointer paths in native order. Derive added/removed/
changed from snapshot `Present`, not nil or redaction text. Use compact JSON
for safe values and expose `HiddenChanged` if present. Equal values produce
`no differences`. Document pointer notation and native changed-region/list
granularity instead of emulating the old engine.
3. Source/catalog evidence comes from authoring results through public source
lookup and metadata history/origins. Add clearly labeled empty optional-import
selection evidence to `config sources`, without guessed candidates or
reasons for absence.
4. Preserve artifact/family/character IDs and generated party context. Distinguish
normalization origins from retained definition evidence. Use native `main`
where an upstream layer is reported; root/import presentation roles may remain
where they express actual application context.
Test changed family members and publish rules, equivalent layouts, composite
contributors, optional empty imports, atomic lists, and pointer escaping in
allowed dynamic mapping keys. Inspection remains secret-safe and read-only,
without source reparsing or provenance-based acceptance gates.
## Stage 10: Exercise native authoring and maintain ergonomic examples
Read the production/testing bundle, artifact/family tests, native destinations
and collection rules, and maintained example tests.
1. Convert artifact/family files to destination-based definitions. IDs remain
explicit destination keys or keyed payloads; filenames do not imply IDs.
2. Demonstrate a declared directory import with `pattern: "*.yml"`, essential
`required: true` input, an optional selection, inheritance, and a useful
testing-overlay list customization through `config.sequences`.
3. Cover append/upsert/replace on an existing list using an appropriate authored
identity, such as publish output `source`. Whole-record upsert and empty-list
behavior follow ConfigKit. No application identity registration is required
merely to enable authored collection controls.
4. Demonstrate a split roster and local secondary imports using the sample
campaign, with companion resource paths following source-origin policy.
Explain that full-root and disjoint definitions remain accepted too.
5. Use ordinary `Load` for this authored-import workflow. Layered loading is
useful for actual lower-priority sources, not required for defaults alone.
If such a source is genuinely needed, use `LoadLayered` and `MainRelative`
directly with focused coverage instead of another merge/search mechanism.
Load examples through production definitions. Check dependent logical IDs,
concrete family members, testing differences, and standalone prepared inputs.
Keep examples synthetic and credential-free.
## Stage 11: Align canonical documentation and project-specific policy
Read documentation/OSS/release policies, affected references, and the approved
roadmap. Finish current-behavior documentation for the assembled implementation.
1. Update config/CLI references, operations/bootstrap guidance, troubleshooting,
configuration internals, prepare internals, and party integration docs for the
new discovery, import, resource-base, materialization, and inspection behavior.
2. Replace obsolete import restrictions with an accurate native summary and
pinned upstream links. Explain optional imports, `required`, filters,
destinations, source-local root controls, list modes, and reserved-key scope.
Keep effective domain validation Narratio-owned.
3. Align project-specific architecture configuration/startup statements and the
current raw-party-copy description in its actual owner. Preserve accepted
ADR 0003 and generic adopted policies. Routine integration details do not
require new ADRs.
4. Write manual upgrade guidance for control names, native discovery filenames/
precedence, explicit custom filenames, secrets override, relative path changes,
required imports where desired, and list operations. Explain effective prepared
documents versus authored roots; downstream migrations remain separate.
5. Finish Go prerequisites, CI/runbook toolchain alignment, and attribution from
Stage 1. Record impact for eventual major-release preparation without inventing
a release version or changing published notes. Leave pending Audita/workspace
roadmaps and Proposed ADRs intact.
Run doccheck and maintained example validation. Each volatile contract has one
canonical owner. Retire completed roadmap files only when separately requested.
## Stage 12: Verify the assembled migration and close the sprint
1. Extend a command-level temporary bundle with native discovery, explicit
secrets override, imported campaign/party/session resources, destination-based
artifacts, family expansion, and a testing overlay. Reuse preceding focused
coverage instead of repeating ConfigKit's structural matrix.
2. Exercise validate/show/sources/artifacts/diff through real composition. Check
selected root, resources, generated values, optional-import evidence, safe
credentials, and absence of runtime workspace creation by inspection. Exercise
plan or prepare with fake external seams to establish execution parity.
3. Trace remaining callers of old search, rebasing, raw-copy, and diff helpers.
Remove dead code/test seams. Review intentional test expectation changes
against the roadmap and preserve existing consequential domain/safety tests.
4. With Go 1.27.4 and dependencies prepared, run `go test ./...`,
`go test -race ./...`, `go vet ./...`, `go build ./...`,
`go mod tidy -diff`, and `git diff --check`. Use the existing release asset
builder with a temporary output directory and its existing `v0.0.0-test` label
to check the three targets; this invokes no publication or new release note.
5. Inspect `go list -m` for the exact release, absence of the old module path
and filesystem replacements, and expected notices. Review every roadmap
capability and all four approved decisions; native structural behavior must
not be narrowed by new Narratio acceptance gates.
6. Commit the finished unit and report checks, compatibility changes, revision,
and any genuine unfinished work. Leave a clean workspace. Actual release
publication and the next feature require separate authorization.
Complete when the native workflow works end to end, prepared inputs survive
loss of authoring companions, operator inspection covers final domain values,
and final validation passes.
## Open Questions
2026-11-17 20:41:34 +00:00
None required for this sprint. The four product choices are approved. This plan
selects concrete native snapshot, source-context, materialization, and toolchain
approaches without adding compatibility shims, structural restrictions, or
stronger runtime guarantees.