360 lines
20 KiB
Markdown
360 lines
20 KiB
Markdown
|
|
# Adopt ConfigKit v1.2.1 and native configuration ergonomics
|
||
|
|
|
||
|
|
## Purpose
|
||
|
|
|
||
|
|
Adopt current ConfigKit as Narratio's configuration owner and make its native
|
||
|
|
authoring and inspection features useful to operators. Align configuration
|
||
|
|
semantics with other ConfigKit consumers. Existing Narratio assembly conventions
|
||
|
|
are migration inputs rather than compatibility requirements for this work.
|
||
|
|
|
||
|
|
This roadmap defines the target and required changes. A subsequent implementation
|
||
|
|
plan will define staged work after the open decisions are resolved. This pass
|
||
|
|
adds documentation only. The earlier [v0.5.0 roadmap](config.md) and
|
||
|
|
[implementation plan](implementation.md) describe the completed baseline.
|
||
|
|
|
||
|
|
The ownership boundary in [ADR 0003](../adr/0003-assign-configuration-ownership.md)
|
||
|
|
continues to apply: ConfigKit resolves configuration before its final handoff;
|
||
|
|
Narratio owns domain validation, application derivation, and runtime use.
|
||
|
|
Supported callbacks and native metadata provide the integration boundary.
|
||
|
|
|
||
|
|
## Reviewed baseline and releases
|
||
|
|
|
||
|
|
Narratio currently selects `gitea.maximumdirect.net/eric/ConfigKit v0.5.0`,
|
||
|
|
commit `dc09959cca4e105f49ec316c6d4ba585bd2340bd`, with Go 1.26.0 as its
|
||
|
|
consumer minimum. Release builds currently select Go 1.26.7.
|
||
|
|
|
||
|
|
The adoption target is `go.maximumdirect.net/configkit v1.2.1`, commit
|
||
|
|
`51ef3144158db1a09fabdbd1c7ae59908c84923b`. The published tag and the sibling
|
||
|
|
checkout agree on that revision. ConfigKit requires Go 1.27.0 and uses
|
||
|
|
`go.yaml.in/yaml/v4 v4.0.0`. Its code license remains LGPL-3.0-or-later.
|
||
|
|
|
||
|
|
The pinned [configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md)
|
||
|
|
and [consumer guide](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/consumers/README.md)
|
||
|
|
own upstream semantics. The intervening releases supply the following relevant
|
||
|
|
changes:
|
||
|
|
|
||
|
|
| Release | Adoption relevance |
|
||
|
|
| --- | --- |
|
||
|
|
| v0.6.0 | Loading and metadata optimizations; Go minimum moves to 1.26.0. |
|
||
|
|
| v0.7.0 | Native safe effective-configuration `Diff`. |
|
||
|
|
| v0.7.1 | Maintainer-tool update; runtime behavior unchanged. |
|
||
|
|
| v0.8.0 | Input/import destinations, sequence-item placement, shared source-version checks, and initial identified-sequence composition. |
|
||
|
|
| v0.9.0 | Directory and expanded import selection; optional-by-default imports; identified-list replace/append/upsert and inheritance; CLI assignments move after decoding and inheritance. Its final collection APIs replace the initial v0.8 registrations. |
|
||
|
|
| v0.10.0 | Native ordered configuration discovery through `SearchConfig`. |
|
||
|
|
| v0.10.1 | Loading consolidation and import bookkeeping improvements. |
|
||
|
|
| v0.10.2 | Correctness fixes for decoding, callback/resource lifetime, discovery, inspection budgets, and collection operations. |
|
||
|
|
| v1.0.0 | Vanity module path, source-root `config` controls, public API cleanup, source lookup, provenance and sensitivity fixes, and Go 1.27.0 minimum. |
|
||
|
|
| v1.1.0 | Field-level list resolution records and identity-aware inspection across list movement. |
|
||
|
|
| v1.2.0 | Main-relative companion inputs and overlay context for layered loading; overlay filesystem fixes. |
|
||
|
|
| v1.2.1 | Stable go-yaml v4.0.0 dependency; maintainer release-tool update. |
|
||
|
|
|
||
|
|
The corresponding `docs/releases/<version>.md` files at the pinned revision
|
||
|
|
provide the migration details. Select v1.2.1 directly; intermediate releases
|
||
|
|
provide context rather than required deployment stops.
|
||
|
|
|
||
|
|
## Current Narratio integration
|
||
|
|
|
||
|
|
The [configuration reference](../config.md) and
|
||
|
|
[configuration internals](../internal/configuration.md) describe current behavior.
|
||
|
|
|
||
|
|
- Commands select a pipeline path through Narratio's search loop, bootstrap
|
||
|
|
credentials from an explicit directory or a sibling `secrets/` directory,
|
||
|
|
and then call ConfigKit.
|
||
|
|
- Pipeline loading enables explicit path/glob imports and one selected overlay.
|
||
|
|
Campaign, session, and canonical-party documents use separate loads. Session
|
||
|
|
loading also supports command-scoped downloaded input snapshots.
|
||
|
|
- Typed defaults, recorded normalization, domain validation, and native
|
||
|
|
provenance are integrated. Additional path/dependency resolution and
|
||
|
|
campaign-owned family expansion occur after the pipeline load.
|
||
|
|
- Artifact files currently repeat enclosing `scriptorium.artifacts` mappings.
|
||
|
|
Logical IDs come from map keys. Families expand against canonical party data.
|
||
|
|
- Show, diff, and digest use a safe finalized-domain projection. Narratio owns
|
||
|
|
the current mapping-diff algorithm. Native load metadata supplies source and
|
||
|
|
artifact-catalog evidence.
|
||
|
|
|
||
|
|
Some public import descriptions still state restrictions from older ConfigKit
|
||
|
|
behavior. Replace these with current native semantics during adoption rather
|
||
|
|
than recreating them as Narratio checks.
|
||
|
|
|
||
|
|
## Targeted end state
|
||
|
|
|
||
|
|
### Native assembly and operator choice
|
||
|
|
|
||
|
|
ConfigKit owns source discovery, acquisition, composition, inheritance, presence,
|
||
|
|
typed secrets, provenance, and safe native inspection. Narratio supplies its
|
||
|
|
schema, domain defaults and validation, supported application resolution, and
|
||
|
|
operator-selected inputs.
|
||
|
|
|
||
|
|
Use ordinary `Load` for an ordinary configuration. Use public native facilities
|
||
|
|
where they serve the operator workflow. Layered loading is appropriate when
|
||
|
|
actual lower-priority inputs are needed; typed defaults remain appropriate for
|
||
|
|
simple defaults. Main-relative companion inputs allow such underlays to follow
|
||
|
|
a discovered main without Narratio searching for it again.
|
||
|
|
|
||
|
|
Expose the library's authored `config.imports` and `config.sequences` semantics
|
||
|
|
where applicable. Imports use ConfigKit defaults, including optional absence,
|
||
|
|
unless the operator explicitly requests other supported behavior. An invalid
|
||
|
|
selected source still fails. Operators mark essential imports `required: true`.
|
||
|
|
Directory imports can select YAML with an explicit `pattern`; unrestricted
|
||
|
|
filenames remain the library default.
|
||
|
|
|
||
|
|
Native destinations, nested imports, discovery filters, source-local relative
|
||
|
|
import paths, collection operations, and inheritance work under ConfigKit's
|
||
|
|
rules. Narratio validates the final effective domain entries. It adds no
|
||
|
|
fragment-count rule, mandatory fragment version, import-mode allowlist, or
|
||
|
|
parallel composition engine.
|
||
|
|
|
||
|
|
### Ergonomic artifact files and list customization
|
||
|
|
|
||
|
|
Support an artifact definition without repeated enclosing mappings through an
|
||
|
|
explicit import destination. For example:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
config:
|
||
|
|
imports:
|
||
|
|
- path: artifacts/session-recap.yml
|
||
|
|
destination: [scriptorium, artifacts, session_recap]
|
||
|
|
required: true
|
||
|
|
```
|
||
|
|
|
||
|
|
The imported file contains the artifact fields such as `prompt_id`, `inputs`,
|
||
|
|
and `output_path`. The literal destination defines the reusable key
|
||
|
|
`session_recap`, with logical source `narratio.artifact.session_recap`.
|
||
|
|
The filename is an organizational choice. Keyed files mounted at
|
||
|
|
`[scriptorium, artifacts]` support declared directory or glob imports when many
|
||
|
|
definitions should be discovered together. Apply the same pattern to families.
|
||
|
|
|
||
|
|
Keep keyed artifact/family mappings and campaign-owned character data. They
|
||
|
|
already fit native inheritance and stable identity. Full-root and disjoint
|
||
|
|
definitions remain valid native inputs; a one-artifact-per-file organization is
|
||
|
|
guidance rather than a validity requirement.
|
||
|
|
|
||
|
|
Allow authored native collection declarations for existing list fields where
|
||
|
|
their final shapes satisfy Narratio's schema. Operators can choose replacement,
|
||
|
|
append, or upsert and identified-record inheritance. Explain that upsert
|
||
|
|
replaces whole matching records and that an empty append/upsert contribution
|
||
|
|
does not clear a list. Register application-owned sequence identities only
|
||
|
|
where Narratio has a real domain identity requirement; generic library
|
||
|
|
capabilities do not establish new uniqueness requirements for every list.
|
||
|
|
|
||
|
|
### Discovery, credentials, and resource paths
|
||
|
|
|
||
|
|
Replace Narratio-owned local candidate probing with ConfigKit search descriptions
|
||
|
|
and `SearchConfig`. Explicit `--config` selection remains an exclusive `File`
|
||
|
|
input. Report the source actually selected by the library. The open decisions
|
||
|
|
below select search conventions and the independent bootstrap directory needed
|
||
|
|
before a discovered load resolves credentials.
|
||
|
|
|
||
|
|
Bootstrap uses `LoadSecretsIntoEnvironment` once per invocation before typed
|
||
|
|
credential resolution and concurrent work. Existing environment variables win,
|
||
|
|
including explicitly empty variables. Keep explicit `--secrets-dir`, safe
|
||
|
|
partial-report handling, and exact-name forwarding to subprocesses. The selected
|
||
|
|
bootstrap convention must allow native discovery without a second preliminary
|
||
|
|
configuration load or a duplicate file-selection loop.
|
||
|
|
|
||
|
|
Use native source/resource context for application path interpretation. The
|
||
|
|
resource-base decision below applies to file references rather than artifact
|
||
|
|
destinations, command names, or remote object keys. ConfigKit's consumer examples
|
||
|
|
illustrate application policy; they do not impose it on Narratio.
|
||
|
|
|
||
|
|
### Final-domain inspection through native APIs
|
||
|
|
|
||
|
|
Keep validate, show, sources, artifact catalog, and diff useful for the operator's
|
||
|
|
fully resolved pipeline, including campaign-driven members and publish rules.
|
||
|
|
Use native effective rendering and `Diff` where they own the operation, with
|
||
|
|
Narratio formatting and generated-domain context at the presentation boundary.
|
||
|
|
|
||
|
|
A raw pipeline `Result` is a frozen load snapshot. Mutating its `Config` later
|
||
|
|
does not update rendering or diff. Integrate this distinction explicitly:
|
||
|
|
record domain resolution within a supported resolver when its inputs are
|
||
|
|
available there; where later campaign resolution remains appropriate, create
|
||
|
|
a completed native inspection result from the finalized safe domain projection
|
||
|
|
using the public mapping-source API. That result describes application-derived
|
||
|
|
values; retained authoring results supply actual file provenance.
|
||
|
|
|
||
|
|
Use native comparison rather than maintaining a second general mapping-diff
|
||
|
|
engine. Preserve final-domain coverage and credential identities. Narratio's
|
||
|
|
digest and stage-owned reuse evidence remain application responsibilities.
|
||
|
|
Source-layout changes alone do not change equivalent domain values. Source
|
||
|
|
inspection can report composite origins, empty optional-import selections, and
|
||
|
|
application derivation without inventing a unique defining-file contract.
|
||
|
|
|
||
|
|
### Scope and compatibility
|
||
|
|
|
||
|
|
This is a clean native cutover. Update source-root `configkit:` controls to
|
||
|
|
`config:` and remove old-module/API references in maintained code and fixtures.
|
||
|
|
Nested `config` data remains ordinary data under ConfigKit's source-root rules.
|
||
|
|
The `configkit` package name and `configkit:"secret"` Go tag retain their
|
||
|
|
upstream spelling. Mapping and identified-list `extends` follow current native
|
||
|
|
rules.
|
||
|
|
|
||
|
|
This roadmap deliberately permits revised configuration semantics. Update
|
||
|
|
tests and documentation for selected changes rather than preserving historical
|
||
|
|
acceptance rules through compatibility loaders or keyword aliases.
|
||
|
|
|
||
|
|
Application identity, artifact dependency meaning, runtime output destinations,
|
||
|
|
campaign-owned party authority, remote session retrieval, and stage reuse remain
|
||
|
|
Narratio domain concerns. This scope does not redesign the runtime workspace,
|
||
|
|
change subprocess adapters, adopt ReleaseKit for Narratio, or add external
|
||
|
|
provider probes. Audita and workspace work retain their separate roadmaps.
|
||
|
|
|
||
|
|
## Required changes
|
||
|
|
|
||
|
|
### Dependency, toolchain, and notices
|
||
|
|
|
||
|
|
- Select the released `go.maximumdirect.net/configkit v1.2.1` module and migrate
|
||
|
|
all imports and exchanged named types together. Remove the old requirement;
|
||
|
|
use the released dependency rather than a filesystem replacement.
|
||
|
|
- Raise `go.mod` to Go 1.27.0 and select an available official Go 1.27 patch for
|
||
|
|
development/release validation and release builds. Align CI, runbook setup,
|
||
|
|
and incorporated Go attribution with that selection.
|
||
|
|
- Update ConfigKit source/license links to the reviewed commit and YAML v4
|
||
|
|
attribution to the selected stable release. Check the actual resulting module
|
||
|
|
graph. Maintainer tool dependencies are not automatically Narratio runtime
|
||
|
|
components. Narratio's separate direct YAML use need not be rewritten merely
|
||
|
|
because ConfigKit changed its internal parser dependency.
|
||
|
|
|
||
|
|
### Integration and simplification
|
||
|
|
|
||
|
|
- Apply the v1 API replacements actually used here, including
|
||
|
|
`LoadSecretsIntoEnvironment`, `LayerMain`, `RecordDerivedDefault`, and
|
||
|
|
`RecordNormalization`. Review other public names against the migration guide.
|
||
|
|
- Use keyed expanded options and definition literals. Keep metadata projections
|
||
|
|
compatible with current source evidence, native layer names, and safe errors.
|
||
|
|
- Replace broad list-record workarounds with supported field-level recording
|
||
|
|
when the application changes only fields. Record whole-list changes where
|
||
|
|
order or membership changes warrant them.
|
||
|
|
- Adopt native discovery, destination-based artifact authoring, authored
|
||
|
|
collection operations, and native final-domain rendering/comparison. Remove
|
||
|
|
replaced search, comparison, and redundant metadata machinery.
|
||
|
|
- Follow the decisions below for resource bases and secondary-document loading.
|
||
|
|
Share definitions and loading context between execution and inspection so an
|
||
|
|
operator validates the configuration the pipeline will actually consume.
|
||
|
|
- If secondary-document imports are adopted, review input materialization and
|
||
|
|
publication as well as parsing. A copied root file can retain imports whose
|
||
|
|
companions are absent at its new location. Materialize the effective domain
|
||
|
|
document where a subprocess or restored session consumes it, retaining native
|
||
|
|
authoring evidence separately. Review placeholder checks against the effective
|
||
|
|
session input and preserve required external party/session formats.
|
||
|
|
|
||
|
|
### Documentation and maintained examples
|
||
|
|
|
||
|
|
Update the configuration reference, CLI reference, operations/bootstrap guidance,
|
||
|
|
configuration internals, contributor prerequisites, and release procedure for
|
||
|
|
the implemented semantics. Keep canonical facts in their respective owners.
|
||
|
|
Update project-specific architecture wording where selected path/bootstrap
|
||
|
|
decisions change it; preserve ADR 0003's ownership boundary.
|
||
|
|
|
||
|
|
Migrate the production/testing bundle to `config:`. Demonstrate destination-based
|
||
|
|
artifact and family files, a native directory import restricted to YAML, an
|
||
|
|
essential required import, native inheritance, and a useful selected-overlay
|
||
|
|
list customization. Keep illustrative optional choices labeled as choices.
|
||
|
|
Show how the declared artifact key is referenced by dependent artifacts.
|
||
|
|
|
||
|
|
Provide concise manual upgrade instructions for existing Narratio configurations:
|
||
|
|
control-key/API changes, optional import defaults, collection behavior, source
|
||
|
|
discovery, and selected path/bootstrap changes. Separately installed applications
|
||
|
|
own their configuration versions; Narratio's migration does not automatically
|
||
|
|
rewrite their files.
|
||
|
|
|
||
|
|
The eventual Narratio release requires a major version because existing valid
|
||
|
|
configuration control blocks change. The Go minimum increase alone is permitted
|
||
|
|
in a minor release under the adopted release policy. This roadmap does not
|
||
|
|
authorize tagging, publication, or retrospective release-note changes.
|
||
|
|
|
||
|
|
### Verification
|
||
|
|
|
||
|
|
Use temporary inputs and controlled synthetic credentials. Exercise native
|
||
|
|
discovery and the chosen bootstrap convention together, explicit file selection,
|
||
|
|
selected-source errors, optional and required imports, import destinations,
|
||
|
|
inherited artifact identities, native list customization, and final domain
|
||
|
|
validation. Add coverage at Narratio boundaries rather than duplicating
|
||
|
|
ConfigKit's complete structural test matrix.
|
||
|
|
|
||
|
|
Verify inherited presence/defaulting, party order, final campaign/family
|
||
|
|
expansion, resource paths, source attribution, and secret-safe inspection.
|
||
|
|
Diff/show must include derived members and publish rules; compare equal effective
|
||
|
|
configurations across different source arrangements. Credential rotation behind
|
||
|
|
the same reference name must remain absent from semantic diffs and digests.
|
||
|
|
|
||
|
|
Update tests that encode intentionally retired syntax or policies. Run focused
|
||
|
|
configuration/command/example/documentation checks and the established full
|
||
|
|
test, race, vet, build, and module checks on the adopted Go line. Preserve the
|
||
|
|
current supported build targets. No new live-service or CI-completion gate is
|
||
|
|
introduced by this migration.
|
||
|
|
|
||
|
|
## Open Decisions
|
||
|
|
|
||
|
|
### 1. Default pipeline discovery conventions
|
||
|
|
|
||
|
|
**Recommendation:** use the native `WorkingDirectory`, `UserConfigDirectory`,
|
||
|
|
and `UnixSystemDirectories` presets in that order with application name
|
||
|
|
`narratio`. Use their documented filenames and retain `--config` for exclusive
|
||
|
|
selection. This makes development-local configuration convenient and matches
|
||
|
|
ConfigKit's standard naming. Existing `pipeline.yml` files remain explicitly
|
||
|
|
selectable; migration guidance explains the new discovery names and precedence.
|
||
|
|
|
||
|
|
**Alternative:** use ConfigKit `SearchPaths` for the current two system
|
||
|
|
`pipeline.yml` paths, or add those paths to a broader ordered native search.
|
||
|
|
This reduces deployment disruption while still removing Narratio's probing
|
||
|
|
loop, but preserves project-specific naming and gives local development less
|
||
|
|
convenience. Adding fallback paths also creates more precedence to explain.
|
||
|
|
|
||
|
|
### 2. Secrets bootstrap before native discovery
|
||
|
|
|
||
|
|
**Recommendation:** use an invocation-owned conventional directory
|
||
|
|
`/usr/local/etc/narratio/secrets`, with `--secrets-dir` selecting another
|
||
|
|
directory and existing environment variables taking priority. Missing
|
||
|
|
conventional secrets remain optional; an explicitly selected directory follows
|
||
|
|
the existing explicit-selection behavior. This separates bootstrap from main
|
||
|
|
file discovery and lets ConfigKit perform the first and only configuration
|
||
|
|
selection. Service environments can provide credentials directly; development
|
||
|
|
invocations can choose a project-local secrets directory explicitly.
|
||
|
|
|
||
|
|
**Alternative:** preserve implicit `secrets/` beside the selected main file.
|
||
|
|
This is convenient for portable configuration bundles, but ConfigKit's discovery
|
||
|
|
source selects its file during loading. Preserving pre-load bootstrap would
|
||
|
|
require a coordinated upstream discovery/bootstrap facility or retaining
|
||
|
|
Narratio-owned preselection. The latter reduces native consolidation; moving
|
||
|
|
environment mutation into a hidden resolver would change the existing startup
|
||
|
|
boundary. An upstream request is viable if sibling-directory convenience is
|
||
|
|
more important than an immediately self-contained migration.
|
||
|
|
|
||
|
|
### 3. Relative resource paths in imported definitions
|
||
|
|
|
||
|
|
**Recommendation:** resolve relative file-resource references from their
|
||
|
|
effective defining source's ConfigKit resource context. Inherited values follow
|
||
|
|
their effective origin. CLI-relative paths use the invocation directory;
|
||
|
|
application-derived paths record their dependencies. Use the main source as a
|
||
|
|
documented fallback for origins without a distinct usable resource base.
|
||
|
|
This makes imported artifact/integration definitions portable across bundles.
|
||
|
|
Review actual path-bearing fields explicitly; session-relative products and
|
||
|
|
remote keys retain their domain bases. Introduce no new target-existence or
|
||
|
|
root-containment checks merely to adopt this policy.
|
||
|
|
|
||
|
|
**Alternative:** retain current per-field pipeline-root bases while using native
|
||
|
|
metadata elsewhere. This minimizes path edits and remains a valid ConfigKit
|
||
|
|
consumer policy, but moving reusable definitions still requires knowing the
|
||
|
|
destination bundle's root. Absolute resource paths are another viable operator
|
||
|
|
choice, with clear meaning but reduced portability.
|
||
|
|
|
||
|
|
### 4. Imports in campaign, party, and session documents
|
||
|
|
|
||
|
|
**Recommendation:** enable native explicit imports for locally sourced campaign,
|
||
|
|
party, and session documents as well as pipelines. This supports split rosters
|
||
|
|
and shared metadata with consistent library semantics. Keep campaign selection,
|
||
|
|
party schema validation, and session domain rules application-owned. Remote
|
||
|
|
session snapshots retain their explicit source context: remote companion-file
|
||
|
|
retrieval needs its own supported input contract, rather than resolving relative
|
||
|
|
imports against an incidental download directory. Pipeline overlay selection
|
||
|
|
remains associated with the pipeline; it does not implicitly select overlays
|
||
|
|
for every document. Adoption includes effective-document materialization where
|
||
|
|
raw root-file copying would leave unresolved imports in downstream inputs or
|
||
|
|
published session data.
|
||
|
|
|
||
|
|
**Alternative:** enable the richer facilities only for pipelines and retain
|
||
|
|
single-source secondary documents. This bounds the migration and simplifies
|
||
|
|
remote/local parity, but leaves an intentional project-specific difference in
|
||
|
|
authoring ergonomics. If selected, document that scope explicitly rather than
|
||
|
|
presenting it as a ConfigKit limitation.
|