Files
narratio/docs/roadmap/configkit.md

360 lines
20 KiB
Markdown

# Adopt ConfigKit v1.2.1 and native configuration ergonomics
## Purpose
Adopt current ConfigKit as Narratio's configuration owner and make its native
authoring and inspection features useful to operators. Align configuration
semantics with other ConfigKit consumers. Existing Narratio assembly conventions
are migration inputs rather than compatibility requirements for this work.
This roadmap defines the target and required changes. A subsequent implementation
plan will define staged work after the open decisions are resolved. This pass
adds documentation only. The earlier [v0.5.0 roadmap](config.md) and
[implementation plan](implementation.md) describe the completed baseline.
The ownership boundary in [ADR 0003](../adr/0003-assign-configuration-ownership.md)
continues to apply: ConfigKit resolves configuration before its final handoff;
Narratio owns domain validation, application derivation, and runtime use.
Supported callbacks and native metadata provide the integration boundary.
## Reviewed baseline and releases
Narratio currently selects `gitea.maximumdirect.net/eric/ConfigKit v0.5.0`,
commit `dc09959cca4e105f49ec316c6d4ba585bd2340bd`, with Go 1.26.0 as its
consumer minimum. Release builds currently select Go 1.26.7.
The adoption target is `go.maximumdirect.net/configkit v1.2.1`, commit
`51ef3144158db1a09fabdbd1c7ae59908c84923b`. The published tag and the sibling
checkout agree on that revision. ConfigKit requires Go 1.27.0 and uses
`go.yaml.in/yaml/v4 v4.0.0`. Its code license remains LGPL-3.0-or-later.
The pinned [configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md)
and [consumer guide](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/consumers/README.md)
own upstream semantics. The intervening releases supply the following relevant
changes:
| Release | Adoption relevance |
| --- | --- |
| v0.6.0 | Loading and metadata optimizations; Go minimum moves to 1.26.0. |
| v0.7.0 | Native safe effective-configuration `Diff`. |
| v0.7.1 | Maintainer-tool update; runtime behavior unchanged. |
| v0.8.0 | Input/import destinations, sequence-item placement, shared source-version checks, and initial identified-sequence composition. |
| v0.9.0 | Directory and expanded import selection; optional-by-default imports; identified-list replace/append/upsert and inheritance; CLI assignments move after decoding and inheritance. Its final collection APIs replace the initial v0.8 registrations. |
| v0.10.0 | Native ordered configuration discovery through `SearchConfig`. |
| v0.10.1 | Loading consolidation and import bookkeeping improvements. |
| v0.10.2 | Correctness fixes for decoding, callback/resource lifetime, discovery, inspection budgets, and collection operations. |
| v1.0.0 | Vanity module path, source-root `config` controls, public API cleanup, source lookup, provenance and sensitivity fixes, and Go 1.27.0 minimum. |
| v1.1.0 | Field-level list resolution records and identity-aware inspection across list movement. |
| v1.2.0 | Main-relative companion inputs and overlay context for layered loading; overlay filesystem fixes. |
| v1.2.1 | Stable go-yaml v4.0.0 dependency; maintainer release-tool update. |
The corresponding `docs/releases/<version>.md` files at the pinned revision
provide the migration details. Select v1.2.1 directly; intermediate releases
provide context rather than required deployment stops.
## Current Narratio integration
The [configuration reference](../config.md) and
[configuration internals](../internal/configuration.md) describe current behavior.
- Commands select a pipeline path through Narratio's search loop, bootstrap
credentials from an explicit directory or a sibling `secrets/` directory,
and then call ConfigKit.
- Pipeline loading enables explicit path/glob imports and one selected overlay.
Campaign, session, and canonical-party documents use separate loads. Session
loading also supports command-scoped downloaded input snapshots.
- Typed defaults, recorded normalization, domain validation, and native
provenance are integrated. Additional path/dependency resolution and
campaign-owned family expansion occur after the pipeline load.
- Artifact files currently repeat enclosing `scriptorium.artifacts` mappings.
Logical IDs come from map keys. Families expand against canonical party data.
- Show, diff, and digest use a safe finalized-domain projection. Narratio owns
the current mapping-diff algorithm. Native load metadata supplies source and
artifact-catalog evidence.
Some public import descriptions still state restrictions from older ConfigKit
behavior. Replace these with current native semantics during adoption rather
than recreating them as Narratio checks.
## Targeted end state
### Native assembly and operator choice
ConfigKit owns source discovery, acquisition, composition, inheritance, presence,
typed secrets, provenance, and safe native inspection. Narratio supplies its
schema, domain defaults and validation, supported application resolution, and
operator-selected inputs.
Use ordinary `Load` for an ordinary configuration. Use public native facilities
where they serve the operator workflow. Layered loading is appropriate when
actual lower-priority inputs are needed; typed defaults remain appropriate for
simple defaults. Main-relative companion inputs allow such underlays to follow
a discovered main without Narratio searching for it again.
Expose the library's authored `config.imports` and `config.sequences` semantics
where applicable. Imports use ConfigKit defaults, including optional absence,
unless the operator explicitly requests other supported behavior. An invalid
selected source still fails. Operators mark essential imports `required: true`.
Directory imports can select YAML with an explicit `pattern`; unrestricted
filenames remain the library default.
Native destinations, nested imports, discovery filters, source-local relative
import paths, collection operations, and inheritance work under ConfigKit's
rules. Narratio validates the final effective domain entries. It adds no
fragment-count rule, mandatory fragment version, import-mode allowlist, or
parallel composition engine.
### Ergonomic artifact files and list customization
Support an artifact definition without repeated enclosing mappings through an
explicit import destination. For example:
```yaml
config:
imports:
- path: artifacts/session-recap.yml
destination: [scriptorium, artifacts, session_recap]
required: true
```
The imported file contains the artifact fields such as `prompt_id`, `inputs`,
and `output_path`. The literal destination defines the reusable key
`session_recap`, with logical source `narratio.artifact.session_recap`.
The filename is an organizational choice. Keyed files mounted at
`[scriptorium, artifacts]` support declared directory or glob imports when many
definitions should be discovered together. Apply the same pattern to families.
Keep keyed artifact/family mappings and campaign-owned character data. They
already fit native inheritance and stable identity. Full-root and disjoint
definitions remain valid native inputs; a one-artifact-per-file organization is
guidance rather than a validity requirement.
Allow authored native collection declarations for existing list fields where
their final shapes satisfy Narratio's schema. Operators can choose replacement,
append, or upsert and identified-record inheritance. Explain that upsert
replaces whole matching records and that an empty append/upsert contribution
does not clear a list. Register application-owned sequence identities only
where Narratio has a real domain identity requirement; generic library
capabilities do not establish new uniqueness requirements for every list.
### Discovery, credentials, and resource paths
Replace Narratio-owned local candidate probing with ConfigKit search descriptions
and `SearchConfig`. Explicit `--config` selection remains an exclusive `File`
input. Report the source actually selected by the library. The open decisions
below select search conventions and the independent bootstrap directory needed
before a discovered load resolves credentials.
Bootstrap uses `LoadSecretsIntoEnvironment` once per invocation before typed
credential resolution and concurrent work. Existing environment variables win,
including explicitly empty variables. Keep explicit `--secrets-dir`, safe
partial-report handling, and exact-name forwarding to subprocesses. The selected
bootstrap convention must allow native discovery without a second preliminary
configuration load or a duplicate file-selection loop.
Use native source/resource context for application path interpretation. The
resource-base decision below applies to file references rather than artifact
destinations, command names, or remote object keys. ConfigKit's consumer examples
illustrate application policy; they do not impose it on Narratio.
### Final-domain inspection through native APIs
Keep validate, show, sources, artifact catalog, and diff useful for the operator's
fully resolved pipeline, including campaign-driven members and publish rules.
Use native effective rendering and `Diff` where they own the operation, with
Narratio formatting and generated-domain context at the presentation boundary.
A raw pipeline `Result` is a frozen load snapshot. Mutating its `Config` later
does not update rendering or diff. Integrate this distinction explicitly:
record domain resolution within a supported resolver when its inputs are
available there; where later campaign resolution remains appropriate, create
a completed native inspection result from the finalized safe domain projection
using the public mapping-source API. That result describes application-derived
values; retained authoring results supply actual file provenance.
Use native comparison rather than maintaining a second general mapping-diff
engine. Preserve final-domain coverage and credential identities. Narratio's
digest and stage-owned reuse evidence remain application responsibilities.
Source-layout changes alone do not change equivalent domain values. Source
inspection can report composite origins, empty optional-import selections, and
application derivation without inventing a unique defining-file contract.
### Scope and compatibility
This is a clean native cutover. Update source-root `configkit:` controls to
`config:` and remove old-module/API references in maintained code and fixtures.
Nested `config` data remains ordinary data under ConfigKit's source-root rules.
The `configkit` package name and `configkit:"secret"` Go tag retain their
upstream spelling. Mapping and identified-list `extends` follow current native
rules.
This roadmap deliberately permits revised configuration semantics. Update
tests and documentation for selected changes rather than preserving historical
acceptance rules through compatibility loaders or keyword aliases.
Application identity, artifact dependency meaning, runtime output destinations,
campaign-owned party authority, remote session retrieval, and stage reuse remain
Narratio domain concerns. This scope does not redesign the runtime workspace,
change subprocess adapters, adopt ReleaseKit for Narratio, or add external
provider probes. Audita and workspace work retain their separate roadmaps.
## Required changes
### Dependency, toolchain, and notices
- Select the released `go.maximumdirect.net/configkit v1.2.1` module and migrate
all imports and exchanged named types together. Remove the old requirement;
use the released dependency rather than a filesystem replacement.
- Raise `go.mod` to Go 1.27.0 and select an available official Go 1.27 patch for
development/release validation and release builds. Align CI, runbook setup,
and incorporated Go attribution with that selection.
- Update ConfigKit source/license links to the reviewed commit and YAML v4
attribution to the selected stable release. Check the actual resulting module
graph. Maintainer tool dependencies are not automatically Narratio runtime
components. Narratio's separate direct YAML use need not be rewritten merely
because ConfigKit changed its internal parser dependency.
### Integration and simplification
- Apply the v1 API replacements actually used here, including
`LoadSecretsIntoEnvironment`, `LayerMain`, `RecordDerivedDefault`, and
`RecordNormalization`. Review other public names against the migration guide.
- Use keyed expanded options and definition literals. Keep metadata projections
compatible with current source evidence, native layer names, and safe errors.
- Replace broad list-record workarounds with supported field-level recording
when the application changes only fields. Record whole-list changes where
order or membership changes warrant them.
- Adopt native discovery, destination-based artifact authoring, authored
collection operations, and native final-domain rendering/comparison. Remove
replaced search, comparison, and redundant metadata machinery.
- Follow the decisions below for resource bases and secondary-document loading.
Share definitions and loading context between execution and inspection so an
operator validates the configuration the pipeline will actually consume.
- If secondary-document imports are adopted, review input materialization and
publication as well as parsing. A copied root file can retain imports whose
companions are absent at its new location. Materialize the effective domain
document where a subprocess or restored session consumes it, retaining native
authoring evidence separately. Review placeholder checks against the effective
session input and preserve required external party/session formats.
### Documentation and maintained examples
Update the configuration reference, CLI reference, operations/bootstrap guidance,
configuration internals, contributor prerequisites, and release procedure for
the implemented semantics. Keep canonical facts in their respective owners.
Update project-specific architecture wording where selected path/bootstrap
decisions change it; preserve ADR 0003's ownership boundary.
Migrate the production/testing bundle to `config:`. Demonstrate destination-based
artifact and family files, a native directory import restricted to YAML, an
essential required import, native inheritance, and a useful selected-overlay
list customization. Keep illustrative optional choices labeled as choices.
Show how the declared artifact key is referenced by dependent artifacts.
Provide concise manual upgrade instructions for existing Narratio configurations:
control-key/API changes, optional import defaults, collection behavior, source
discovery, and selected path/bootstrap changes. Separately installed applications
own their configuration versions; Narratio's migration does not automatically
rewrite their files.
The eventual Narratio release requires a major version because existing valid
configuration control blocks change. The Go minimum increase alone is permitted
in a minor release under the adopted release policy. This roadmap does not
authorize tagging, publication, or retrospective release-note changes.
### Verification
Use temporary inputs and controlled synthetic credentials. Exercise native
discovery and the chosen bootstrap convention together, explicit file selection,
selected-source errors, optional and required imports, import destinations,
inherited artifact identities, native list customization, and final domain
validation. Add coverage at Narratio boundaries rather than duplicating
ConfigKit's complete structural test matrix.
Verify inherited presence/defaulting, party order, final campaign/family
expansion, resource paths, source attribution, and secret-safe inspection.
Diff/show must include derived members and publish rules; compare equal effective
configurations across different source arrangements. Credential rotation behind
the same reference name must remain absent from semantic diffs and digests.
Update tests that encode intentionally retired syntax or policies. Run focused
configuration/command/example/documentation checks and the established full
test, race, vet, build, and module checks on the adopted Go line. Preserve the
current supported build targets. No new live-service or CI-completion gate is
introduced by this migration.
## Open Decisions
### 1. Default pipeline discovery conventions
**Recommendation:** use the native `WorkingDirectory`, `UserConfigDirectory`,
and `UnixSystemDirectories` presets in that order with application name
`narratio`. Use their documented filenames and retain `--config` for exclusive
selection. This makes development-local configuration convenient and matches
ConfigKit's standard naming. Existing `pipeline.yml` files remain explicitly
selectable; migration guidance explains the new discovery names and precedence.
**Alternative:** use ConfigKit `SearchPaths` for the current two system
`pipeline.yml` paths, or add those paths to a broader ordered native search.
This reduces deployment disruption while still removing Narratio's probing
loop, but preserves project-specific naming and gives local development less
convenience. Adding fallback paths also creates more precedence to explain.
### 2. Secrets bootstrap before native discovery
**Recommendation:** use an invocation-owned conventional directory
`/usr/local/etc/narratio/secrets`, with `--secrets-dir` selecting another
directory and existing environment variables taking priority. Missing
conventional secrets remain optional; an explicitly selected directory follows
the existing explicit-selection behavior. This separates bootstrap from main
file discovery and lets ConfigKit perform the first and only configuration
selection. Service environments can provide credentials directly; development
invocations can choose a project-local secrets directory explicitly.
**Alternative:** preserve implicit `secrets/` beside the selected main file.
This is convenient for portable configuration bundles, but ConfigKit's discovery
source selects its file during loading. Preserving pre-load bootstrap would
require a coordinated upstream discovery/bootstrap facility or retaining
Narratio-owned preselection. The latter reduces native consolidation; moving
environment mutation into a hidden resolver would change the existing startup
boundary. An upstream request is viable if sibling-directory convenience is
more important than an immediately self-contained migration.
### 3. Relative resource paths in imported definitions
**Recommendation:** resolve relative file-resource references from their
effective defining source's ConfigKit resource context. Inherited values follow
their effective origin. CLI-relative paths use the invocation directory;
application-derived paths record their dependencies. Use the main source as a
documented fallback for origins without a distinct usable resource base.
This makes imported artifact/integration definitions portable across bundles.
Review actual path-bearing fields explicitly; session-relative products and
remote keys retain their domain bases. Introduce no new target-existence or
root-containment checks merely to adopt this policy.
**Alternative:** retain current per-field pipeline-root bases while using native
metadata elsewhere. This minimizes path edits and remains a valid ConfigKit
consumer policy, but moving reusable definitions still requires knowing the
destination bundle's root. Absolute resource paths are another viable operator
choice, with clear meaning but reduced portability.
### 4. Imports in campaign, party, and session documents
**Recommendation:** enable native explicit imports for locally sourced campaign,
party, and session documents as well as pipelines. This supports split rosters
and shared metadata with consistent library semantics. Keep campaign selection,
party schema validation, and session domain rules application-owned. Remote
session snapshots retain their explicit source context: remote companion-file
retrieval needs its own supported input contract, rather than resolving relative
imports against an incidental download directory. Pipeline overlay selection
remains associated with the pipeline; it does not implicitly select overlays
for every document. Adoption includes effective-document materialization where
raw root-file copying would leave unresolved imports in downstream inputs or
published session data.
**Alternative:** enable the richer facilities only for pipelines and retain
single-source secondary documents. This bounds the migration and simplifies
remote/local parity, but leaves an intentional project-specific difference in
authoring ergonomics. If selected, document that scope explicitly rather than
presenting it as a ConfigKit limitation.