diff --git a/docs/roadmap/config.md b/docs/roadmap/config.md deleted file mode 100644 index 2185e1a..0000000 --- a/docs/roadmap/config.md +++ /dev/null @@ -1,243 +0,0 @@ -# Adopt ConfigKit v0.5.0 - -## Purpose - -Upgrade Narratio from ConfigKit v0.4.2 to v0.5.0 and integrate its native -sibling inheritance with Narratio's configuration validation and inspection. -Keep the upgrade focused on the existing loading model and the adjustments -needed to consume the new release correctly. Rely on ConfigKit's assembly and -metadata features; validate Narratio's effective domain configuration without -adding acceptance rules about how that configuration was authored. - -This adoption precedes [Audita adoption](audita.md) and the -[workspace reorganization](workspace.md). This document defines the target -and required changes; [the implementation plan](implementation.md) defines -the steps for reaching that target. - -## Reviewed Upstream Release - -ConfigKit v0.5.0 identifies commit -`dc09959cca4e105f49ec316c6d4ba585bd2340bd`. The -[release notes](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.5.0/docs/releases/v0.5.0.md), -[configuration contract](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.5.0/docs/config.md), -and [consumer guide](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.5.0/docs/consumers/README.md) -own the upstream behavior. - -The release introduces native `extends` relationships between sibling mappings, -layered loading with ordered underlays, caller-owned filesystem and mapping -sources, and passive source-resource contexts. Inheritance preserves explicit -presence, provenance, and secret classification. - -The existing loading API remains available. ConfigKit's minimum Go line, YAML -dependency, and LGPL-3.0-or-later license are unchanged. Narratio can retain its -current Go toolchain and ordinary loading calls. - -## Current Baseline - -The [configuration reference](../config.md) owns Narratio's current public -semantics; [configuration internals](../internal/configuration.md) describe -loading and inspection. - -- Pipeline loading uses ConfigKit v0.5.0 with explicit additive imports and one - selected overlay, typed defaults, recorded resolution, and domain validation. -- Campaign, session, and canonical-party inputs use separate single-source - loads. Secret-directory bootstrap remains an explicit invocation operation. -- Fields with root-relative pipeline path semantics retain that base when their - declarations move into imports. For example, enabled Notarius configuration - and working-directory paths resolve from the root pipeline directory. -- Distinct fields of one artifact or artifact-family entry can compose across - base files under ConfigKit's structural rules. Narratio validates the resulting - effective entry, and the selected overlay may replace or remove those values. -- Source inspection and artifact catalogs consume retained ConfigKit metadata. - Effective show, diff, and digest projections describe the resolved domain - configuration rather than source layout. - -The released v0.5.0 module is now selected directly, and its attribution points -to the reviewed commit. Cross-file artifact and family inheritance and disjoint -entry composition now pass through Narratio's effective configuration checks. -The remaining integration work verifies native provenance, explicit presence, -safe projections, and the assembled inspection workflow described below. - -## Targeted End State - -### Native inheritance - -Use ConfigKit's built-in sibling inheritance wherever its input contract and -Narratio's domain schema permit it. ConfigKit resolves `extends` after source -composition and overlay precedence, before typed decoding and application -resolution. Narratio receives the resulting typed configuration. - -Inheritance can reduce repetition between artifacts and artifact families, -including definitions in separate imported files. Distinct child keys retain -their identities. Each resulting entry undergoes ordinary Narratio validation, -family expansion, dependency resolution, and execution selection. - -Parent entries remain ordinary domain entries. They must satisfy applicable -validation and participate in the effective configuration. This adoption does -not introduce abstract artifacts or a separate template registry. Operators -must override inherited fields where needed to keep the resulting configuration -valid, such as output destinations. - -Inherited false, zero, empty, and null input retains ConfigKit's explicit -presence semantics. Narratio's absence-based defaults must not refill such -values. Normal domain validation still determines whether a resulting value -is acceptable. - -### Effective configuration and native provenance - -Remove Narratio's one-base-file-per-artifact/family restriction. If ConfigKit -accepts composition of one entry from several files, Narratio accepts that -entry when its effective domain values are valid. ConfigKit continues to reject -structural conflicts under its own contract. Narratio does not reconstruct -declaration history to add another conflict rule or reject domain values that -were removed or replaced before the final configuration handoff. - -Domain validation concerns the effective configuration, including application -resolution and family expansion. Family publication requires an applicable -effective publish configuration, not separate evidence that the section was -explicitly authored rather than defaulted. Existing checks on destinations, -dependencies, identities, and effective publish rules remain. - -Retain `config sources` and useful provenance in `config artifacts`. Consume -ConfigKit's public source, origin, presence, history, and explanation APIs as -intended. A thin formatting/projection layer may add Narratio-specific family, -character, and party context. It must not duplicate upstream source tracking, -inheritance analysis, or derivation bookkeeping. - -An entry can have several contributors. Report the evidence ConfigKit provides -without inventing a unique defining-file guarantee or choosing an arbitrary -contributor as the owner. Keep logical artifact IDs independent of source files. -Small representation changes needed for truthful composite provenance belong -to this adoption; retiring inspection commands does not. - -Effective configuration, diffs, and digests continue to reflect resolved -domain values. Equivalent inherited and explicitly authored configurations -should have equivalent effective projections, regardless of their provenance. -Stage reuse continues to use existing semantic evidence. - -### Loading and ownership boundaries - -Retain ordinary ConfigKit loading, centralized typed defaults, imports, selected -overlays, domain resolution, credential bootstrap, and exact-name credential -forwarding. Preserve current path bases and read-only inspection workflows. - -ConfigKit owns parsing, source composition, structural inheritance, presence, -provenance, and secret resolution. Narratio owns its schema, domain defaults, -semantic normalization and validation, campaign/family resolution, and runtime -use. Supported defaults, resolver, and validator callbacks remain appropriate; -there is no requirement to move them after `Load` merely to establish this -ownership boundary. ConfigKit presence APIs remain suitable for absence-based -defaults, and native metadata remains suitable for operator inspection and -existing party ordering. - -Layered loading, embedded configuration sources, programmatic mapping sources, -and source-relative resource rebasing are available upstream but are outside -this focused adoption. Do not relax empty/null default semantics or change -party ordering just to avoid using useful native APIs. - -## Required Changes - -### Dependency and attribution - -Update `go.mod` and `go.sum` to ConfigKit v0.5.0. Update its entry in -[attribution](../oss/attribution.md) to the selected version and the reviewed -commit's source and notice links. Existing applicable license texts can remain. - -### Configuration integration - -Delete the duplicate-definition check and its obsolete declaration-accounting -tests. Remove the family-publish explicit-declaration gate and metadata retained -only for that gate. Use existing final-value domain checks instead. - -Simplify source and artifact-definition projections around ConfigKit's public -metadata. Remove duplicated tracking or inferred ownership where native evidence -already provides the needed information. Retain application context and -formatting that actually serve operators. Use the completed load's evidence -rather than reparsing sources or performing a second base-only load that could -resolve superseded credential references. - -Keep effective rendering and digest construction grounded in the existing safe -typed projection, with credential identities rather than resolved values. - -Use ConfigKit's native operation directly. No application-owned inheritance -engine, collection registration, alternative keyword, or compatibility loader -is needed. - -### Policy alignment - -Revise the Configuration section of the -[architecture policy](../policy/architecture.md) to record this responsibility -boundary: native loading and metadata belong to ConfigKit; Narratio owns domain -meaning and fail-fast effective validation. Explicitly permit supported -callbacks and native provenance inspection. Remove wording inconsistent with -the established explicit-presence/defaulting semantics rather than introducing -a new empty-value policy. General agent, testing, documentation, and OSS -policies need no broad revisions for this feature. - -### Compatibility and operator guidance - -Document that `extends` is now reserved structural input throughout ConfigKit -loads, including dynamic mappings such as artifact IDs and prompt variables. -Operators using that key as ordinary data must rename it. Inheritance is active -without an opt-in switch; imports and overlay-file selection remain separately -controlled as before. - -Explain native sibling lookup, inherited explicit presence, and the requirement -that parent and child entries satisfy Narratio's domain rules. Link to the -pinned upstream contract for detailed inheritance semantics. Clarify that a -selected overlay may contribute sibling inheritance directives; this does not -add inheritance or stacking between overlay files. - -Replace documentation of the one-base-file restriction with ConfigKit's -composition contract and Narratio's effective-value validation boundary. -Describe any inspection representation changes without promising a unique -defining file for a composite entry. - -Update the configuration reference, internal configuration guide, and a small -maintained example when implementation lands. Preserve each field's current -path-resolution rule when its value is inherited. Record the reserved-key -compatibility impact in the eventual release note; automatic configuration -conversion is outside this feature. - -### Verification - -Add focused Narratio coverage for cross-file artifact/family inheritance and -valid disjoint contributions to one entry. Verify that effective invalid -configuration still fails before execution and that structural conflicts still -surface as ConfigKit errors. Old source-history rejection tests are retired or -rewritten for the new contract, including overlay replacement/removal cases. -Verify inherited explicit values against Narratio's defaulting rules, credential -safety, native contributor reporting, and equivalent effective projections. - -Concentrate tests on Narratio's integration behavior. ConfigKit's own tests own -its generic inheritance algorithm and source lifecycle. Narratio's default -tests remain offline, use temporary configuration bundles and controlled secret -values, and do not depend on the sibling ConfigKit checkout. Run the existing -suite against the selected released module before completing the upgrade. - -## Completion Criteria - -Narratio selects the released ConfigKit v0.5.0 module without a local checkout -replacement. Cross-file artifact and family inheritance works through the -ordinary loader; valid composition is not rejected by Narratio's declaration -history checks. Effective domain errors fail before execution. Native provenance -remains inspectable without unique-file assumptions; effective projections, -explicit-presence defaulting, and credential handling retain their semantics. - -Maintained guidance and examples explain native inheritance and the reserved -`extends` key, and the architecture policy reflects the ConfigKit/Narratio -boundary. Attribution identifies the selected upstream revision. Focused -regression coverage and the existing suite pass without implementing Audita or -workspace changes as part of this upgrade. - -## Relationship To Subsequent Work - -ConfigKit adoption provides a verified configuration baseline for the later -Audita and workspace features. It makes no changes to subprocess destinations, -session layout, or cleanup behavior. - -Audita adoption then migrates downstream profiles and adds adapter support for -explicit report/debug destinations. Audita owns its separate ConfigKit and -PromptKit dependencies. Shared latest-attempt links, final workspace paths, and -retention protection remain part of the workspace reorganization, so those -facilities are implemented once at their shared owner. diff --git a/docs/roadmap/configkit.md b/docs/roadmap/configkit.md deleted file mode 100644 index 59e9093..0000000 --- a/docs/roadmap/configkit.md +++ /dev/null @@ -1,315 +0,0 @@ -# Adopt ConfigKit v1.2.1 and native configuration ergonomics - -## Purpose - -Adopt current ConfigKit as Narratio's configuration owner and make its native -authoring and inspection features useful to operators. Align configuration -semantics with other ConfigKit consumers. Existing Narratio assembly conventions -are migration inputs rather than compatibility requirements for this work. - -This roadmap defines the target, user intent, and required changes. The -[implementation plan](implementation.md) defines the work needed to reach that -target. The earlier [v0.5.0 roadmap](config.md) records the completed adoption -that forms the current baseline. - -The ownership boundary in [ADR 0003](../adr/0003-assign-configuration-ownership.md) -continues to apply: ConfigKit resolves configuration before its final handoff; -Narratio owns domain validation, application derivation, and runtime use. -Supported callbacks and native metadata provide the integration boundary. - -## Reviewed baseline and releases - -Narratio currently selects `gitea.maximumdirect.net/eric/ConfigKit v0.5.0`, -commit `dc09959cca4e105f49ec316c6d4ba585bd2340bd`, with Go 1.26.0 as its -consumer minimum. Release builds currently select Go 1.26.7. - -The adoption target is `go.maximumdirect.net/configkit v1.2.1`, commit -`51ef3144158db1a09fabdbd1c7ae59908c84923b`. The published tag and the sibling -checkout agree on that revision. ConfigKit requires Go 1.27.0 and uses -`go.yaml.in/yaml/v4 v4.0.0`. Its code license remains LGPL-3.0-or-later. - -The pinned [configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md) -and [consumer guide](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/consumers/README.md) -own upstream semantics. The intervening releases supply the following relevant -changes: - -| Release | Adoption relevance | -| --- | --- | -| v0.6.0 | Loading and metadata optimizations; Go minimum moves to 1.26.0. | -| v0.7.0 | Native safe effective-configuration `Diff`. | -| v0.7.1 | Maintainer-tool update; runtime behavior unchanged. | -| v0.8.0 | Input/import destinations, sequence-item placement, shared source-version checks, and initial identified-sequence composition. | -| v0.9.0 | Directory and expanded import selection; optional-by-default imports; identified-list replace/append/upsert and inheritance; CLI assignments move after decoding and inheritance. Its final collection APIs replace the initial v0.8 registrations. | -| v0.10.0 | Native ordered configuration discovery through `SearchConfig`. | -| v0.10.1 | Loading consolidation and import bookkeeping improvements. | -| v0.10.2 | Correctness fixes for decoding, callback/resource lifetime, discovery, inspection budgets, and collection operations. | -| v1.0.0 | Vanity module path, source-root `config` controls, public API cleanup, source lookup, provenance and sensitivity fixes, and Go 1.27.0 minimum. | -| v1.1.0 | Field-level list resolution records and identity-aware inspection across list movement. | -| v1.2.0 | Main-relative companion inputs and overlay context for layered loading; overlay filesystem fixes. | -| v1.2.1 | Stable go-yaml v4.0.0 dependency; maintainer release-tool update. | - -The corresponding `docs/releases/.md` files at the pinned revision -provide the migration details. Select v1.2.1 directly; intermediate releases -provide context rather than required deployment stops. - -## Current Narratio integration - -The [configuration reference](../config.md) and -[configuration internals](../internal/configuration.md) describe current behavior. - -- Commands select a pipeline path through Narratio's search loop, bootstrap - credentials from an explicit directory or a sibling `secrets/` directory, - and then call ConfigKit. -- Pipeline loading enables explicit path/glob imports and one selected overlay. - Campaign, session, and canonical-party documents use separate loads. Session - loading also supports command-scoped downloaded input snapshots. -- Typed defaults, recorded normalization, domain validation, and native - provenance are integrated. Additional path/dependency resolution and - campaign-owned family expansion occur after the pipeline load. -- Artifact files currently repeat enclosing `scriptorium.artifacts` mappings. - Logical IDs come from map keys. Families expand against canonical party data. -- Show, diff, and digest use a safe finalized-domain projection. Narratio owns - the current mapping-diff algorithm. Native load metadata supplies source and - artifact-catalog evidence. - -Some public import descriptions still state restrictions from older ConfigKit -behavior. Replace these with current native semantics during adoption rather -than recreating them as Narratio checks. - -## Targeted end state - -### Native assembly and operator choice - -ConfigKit owns source discovery, acquisition, composition, inheritance, presence, -typed secrets, provenance, and safe native inspection. Narratio supplies its -schema, domain defaults and validation, supported application resolution, and -operator-selected inputs. - -Use ordinary `Load` for an ordinary configuration. Use public native facilities -where they serve the operator workflow. Layered loading is appropriate when -actual lower-priority inputs are needed; typed defaults remain appropriate for -simple defaults. Main-relative companion inputs allow such underlays to follow -a discovered main without Narratio searching for it again. - -Expose the library's authored `config.imports` and `config.sequences` semantics -where applicable. Imports use ConfigKit defaults, including optional absence, -unless the operator explicitly requests other supported behavior. An invalid -selected source still fails. Operators mark essential imports `required: true`. -Directory imports can select YAML with an explicit `pattern`; unrestricted -filenames remain the library default. - -Native destinations, nested imports, discovery filters, source-local relative -import paths, collection operations, and inheritance work under ConfigKit's -rules. Narratio validates the final effective domain entries. It adds no -fragment-count rule, mandatory fragment version, import-mode allowlist, or -parallel composition engine. - -### Ergonomic artifact files and list customization - -Support an artifact definition without repeated enclosing mappings through an -explicit import destination. For example: - -```yaml -config: - imports: - - path: artifacts/session-recap.yml - destination: [scriptorium, artifacts, session_recap] - required: true -``` - -The imported file contains the artifact fields such as `prompt_id`, `inputs`, -and `output_path`. The literal destination defines the reusable key -`session_recap`, with logical source `narratio.artifact.session_recap`. -The filename is an organizational choice. Keyed files mounted at -`[scriptorium, artifacts]` support declared directory or glob imports when many -definitions should be discovered together. Apply the same pattern to families. - -Keep keyed artifact/family mappings and campaign-owned character data. They -already fit native inheritance and stable identity. Full-root and disjoint -definitions remain valid native inputs; a one-artifact-per-file organization is -guidance rather than a validity requirement. - -Allow authored native collection declarations for existing list fields where -their final shapes satisfy Narratio's schema. Operators can choose replacement, -append, or upsert and identified-record inheritance. Explain that upsert -replaces whole matching records and that an empty append/upsert contribution -does not clear a list. Register application-owned sequence identities only -where Narratio has a real domain identity requirement; generic library -capabilities do not establish new uniqueness requirements for every list. - -### Discovery, credentials, and resource paths - -Replace Narratio-owned local candidate probing with ConfigKit search descriptions -and `SearchConfig`. Use the native `WorkingDirectory`, `UserConfigDirectory`, -and `UnixSystemDirectories` presets in that order with application name -`narratio`. Use their documented filenames. Explicit `--config` selection -remains an exclusive `File` input. Report the source actually selected by the -library. This provides convenient development-local configuration and standard -ConfigKit naming. Existing `pipeline.yml` files remain explicitly selectable; -migration guidance explains the new discovery names and precedence. - -Bootstrap uses `LoadSecretsIntoEnvironment` once per invocation before typed -credential resolution and concurrent work. Existing environment variables win, -including explicitly empty variables. Use the invocation-owned conventional -directory `/usr/local/etc/narratio/secrets`, with `--secrets-dir` selecting -another directory. A missing conventional directory remains optional; an -explicit selection follows the existing explicit-selection behavior. Retain -safe partial-report handling and exact-name forwarding to subprocesses. - -The independent directory lets bootstrap precede ConfigKit's first and only -configuration selection. Service environments can provide credentials directly; -development invocations can choose a project-local secrets directory explicitly. -The implicit sibling `secrets/` convention retires with this cutover. - -Resolve relative file-resource references from their effective defining source's -ConfigKit resource context. Inherited values follow their effective origin; -CLI-relative paths use the invocation directory. Application-derived paths -record their dependencies. Use the main source as a documented fallback for -origins without a distinct usable resource base. This makes imported definitions -portable across bundles. - -Review actual path-bearing fields explicitly. Artifact destinations, command -names, session-relative products, and remote object keys retain their domain -meanings. This policy introduces no new target-existence or root-containment -checks. ConfigKit's consumer examples illustrate application policy; Narratio -owns the selected interpretation. - -### Imports in secondary documents - -Enable native explicit imports for locally sourced campaign, party, and session -documents as well as pipelines. This supports split rosters and shared metadata -with consistent library semantics. Campaign selection, party schema validation, -and session domain rules remain application-owned. Pipeline overlay selection -remains associated with the pipeline. - -Materialize effective documents where raw root-file copying would leave -unresolved imports in downstream inputs or published session data. Remote -session snapshots retain their explicit source context; resolving imports -against an incidental download directory would give them an incorrect base. -Remote companion-file retrieval remains separate work. - -### Final-domain inspection through native APIs - -Keep validate, show, sources, artifact catalog, and diff useful for the operator's -fully resolved pipeline, including campaign-driven members and publish rules. -Use native effective rendering and `Diff` where they own the operation, with -Narratio formatting and generated-domain context at the presentation boundary. - -A raw pipeline `Result` is a frozen load snapshot. Mutating its `Config` later -does not update rendering or diff. Integrate this distinction explicitly: -record domain resolution within a supported resolver when its inputs are -available there; where later campaign resolution remains appropriate, create -a completed native inspection result from the finalized safe domain projection -using ConfigKit's public loading APIs. That result describes application-derived -values; retained authoring results supply actual file provenance. - -Use native comparison rather than maintaining a second general mapping-diff -engine. Preserve final-domain coverage and credential identities. Narratio's -digest and stage-owned reuse evidence remain application responsibilities. -Source-layout changes alone do not change equivalent domain values. Source -inspection can report composite origins, empty optional-import selections, and -application derivation without inventing a unique defining-file contract. - -### Scope and compatibility - -This is a clean native cutover. Update source-root `configkit:` controls to -`config:` and remove old-module/API references in maintained code and fixtures. -Nested `config` data remains ordinary data under ConfigKit's source-root rules. -The `configkit` package name and `configkit:"secret"` Go tag retain their -upstream spelling. Mapping and identified-list `extends` follow current native -rules. - -This roadmap deliberately permits revised configuration semantics. Update -tests and documentation for selected changes rather than preserving historical -acceptance rules through compatibility loaders or keyword aliases. - -Application identity, artifact dependency meaning, runtime output destinations, -campaign-owned party authority, remote session retrieval, and stage reuse remain -Narratio domain concerns. This scope does not redesign the runtime workspace, -change subprocess adapters, adopt ReleaseKit for Narratio, or add external -provider probes. Audita and workspace work retain their separate roadmaps. - -## Required changes - -### Dependency, toolchain, and notices - -- Select the released `go.maximumdirect.net/configkit v1.2.1` module and migrate - all imports and exchanged named types together. Remove the old requirement; - use the released dependency rather than a filesystem replacement. -- Raise `go.mod` to Go 1.27.0 and select an available official Go 1.27 patch for - development/release validation and release builds. Align CI, runbook setup, - and incorporated Go attribution with that selection. -- Update ConfigKit source/license links to the reviewed commit and YAML v4 - attribution to the selected stable release. Check the actual resulting module - graph. Maintainer tool dependencies are not automatically Narratio runtime - components. Narratio's separate direct YAML use need not be rewritten merely - because ConfigKit changed its internal parser dependency. - -### Integration and simplification - -- Apply the v1 API replacements actually used here, including - `LoadSecretsIntoEnvironment`, `LayerMain`, `RecordDerivedDefault`, and - `RecordNormalization`. Review other public names against the migration guide. -- Use keyed expanded options and definition literals. Keep metadata projections - compatible with current source evidence, native layer names, and safe errors. -- Replace broad list-record workarounds with supported field-level recording - when the application changes only fields. Record whole-list changes where - order or membership changes warrant them. -- Adopt native discovery, destination-based artifact authoring, authored - collection operations, and native final-domain rendering/comparison. Remove - replaced search, comparison, and redundant metadata machinery. -- Apply the agreed source-relative resource bases and secondary-document imports. - Share definitions and loading context between execution and inspection so an - operator validates the configuration the pipeline will actually consume. -- Review secondary-document input materialization and publication as well as - parsing. A copied root file can retain imports whose - companions are absent at its new location. Materialize the effective domain - document where a subprocess or restored session consumes it, retaining native - authoring evidence separately. Review placeholder checks against the effective - session input and preserve required external party/session formats. - -### Documentation and maintained examples - -Update the configuration reference, CLI reference, operations/bootstrap guidance, -configuration internals, contributor prerequisites, and release procedure for -the implemented semantics. Keep canonical facts in their respective owners. -Update project-specific architecture wording where selected path/bootstrap -decisions change it; preserve ADR 0003's ownership boundary. - -Migrate the production/testing bundle to `config:`. Demonstrate destination-based -artifact and family files, a native directory import restricted to YAML, an -essential required import, native inheritance, and a useful selected-overlay -list customization. Keep illustrative optional choices labeled as choices. -Show how the declared artifact key is referenced by dependent artifacts. - -Provide concise manual upgrade instructions for existing Narratio configurations: -control-key/API changes, optional import defaults, collection behavior, source -discovery, and selected path/bootstrap changes. Separately installed applications -own their configuration versions; Narratio's migration does not automatically -rewrite their files. - -The eventual Narratio release requires a major version because existing valid -configuration control blocks change. The Go minimum increase alone is permitted -in a minor release under the adopted release policy. This roadmap does not -authorize tagging, publication, or retrospective release-note changes. - -### Verification - -Use temporary inputs and controlled synthetic credentials. Exercise native -discovery and the chosen bootstrap convention together, explicit file selection, -selected-source errors, optional and required imports, import destinations, -inherited artifact identities, native list customization, and final domain -validation. Add coverage at Narratio boundaries rather than duplicating -ConfigKit's complete structural test matrix. - -Verify inherited presence/defaulting, party order, final campaign/family -expansion, resource paths, source attribution, and secret-safe inspection. -Diff/show must include derived members and publish rules; compare equal effective -configurations across different source arrangements. Credential rotation behind -the same reference name must remain absent from semantic diffs and digests. - -Update tests that encode intentionally retired syntax or policies. Run focused -configuration/command/example/documentation checks and the established full -test, race, vet, build, and module checks on the adopted Go line. Preserve the -current supported build targets. No new live-service or CI-completion gate is -introduced by this migration. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 896aaeb..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,431 +0,0 @@ -# ConfigKit v1.2.1 implementation plan - -## Scope and authority - -Implement [the approved ConfigKit roadmap](configkit.md). It owns the target, -native-semantics intent, and four approved product decisions. This plan replaces -the completed v0.5.0 plan. Its audience is a Luna coding agent implementing one -end-to-end sprint in the numbered order below. - -Select `go.maximumdirect.net/configkit v1.2.1`, upstream commit -`51ef3144158db1a09fabdbd1c7ae59908c84923b`. The pinned -[consumer guide](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/consumers/README.md) -and [configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md) -own upstream semantics. Read the sections identified by each stage and public -declarations in the released module as needed. The sibling checkout is optional -reference material, not a build or test prerequisite. - -## Working instructions - -- Start with `docs/development.md` and `docs/policy/agents.md`. Architecture, - testing, documentation, OSS, and release policies apply; reuse unchanged - policy text already read. Prefer graph tools for code discovery. -- Implement the full native configuration workflow. Discovery, destinations, - collection controls, resource bases, secondary imports, materialization, and - native inspection are part of this sprint. -- Stages are work boundaries, not deployment releases. Intermediate stages may - fail to compile or have dependent tests awaiting later work. Record that state - accurately. Avoid temporary adapters or extra abstractions merely to make - every intermediate commit green. Final validation covers the assembled system. -- Commit completed nonempty units with short plain-English messages. Add focused - behavioral tests with their owner; run meaningful available checks. The final - stage runs repository-wide validation against the finished source. -- Use ConfigKit's public loading, presence, provenance, recording, rendering, - and comparison APIs. Narratio supplies domain meaning and runtime use. - Retire obsolete source-authoring checks and tests rather than reproducing them. -- Leave import defaults unspecified so ConfigKit supplies its defaults. - Explicit false/zero/empty/null and collection semantics follow the native - contract and actual Narratio domain rules. -- Use temporary inputs, real ConfigKit, and controlled synthetic credentials. - The default tests remain offline and independent of installed subprocesses. -- Write documentation in concise, affirmative, factual language. The roadmap - owns intent; this file owns sequencing and implementation choices. - -Preserve stage ordering, artifact dependency meaning, the canonical party schema, -exact-name credential forwarding, publish commit semantics, and the existing -Linux amd64/arm64 plus best-effort macOS arm64 targets. This sprint does not -implement pending Audita/workspace features, migrate old sessions, rewrite -downstream configurations, add fragment version mandates or new path checks, -adopt ReleaseKit for Narratio, or publish a release. - -## Concrete integration choices - -- Consumer Go minimum: **1.27.0**. Development/release validation and build - toolchain: official **Go 1.27.4**, an available patch on that line. Selecting - it is not a new publication gate. -- Pipeline source when no explicit file is supplied: - `SearchConfig("narratio", Searches(WorkingDirectory(), UserConfigDirectory(), - UnixSystemDirectories()), SearchOptions{Required: true})`. - An explicit `--config` uses `File` exclusively. -- Bootstrap: optional `/usr/local/etc/narratio/secrets`, or explicit - `--secrets-dir`, through `LoadSecretsIntoEnvironment` before typed loading. - Existing environment values win. The implicit sibling directory retires. -- Local campaign/session/party loading enables imports. Only the pipeline gets - its selected overlay. Party version checking remains main-only; fragments - need no new version header. Remote session bytes are contextless snapshots. -- Existing campaign registry selection and local session search locations remain - domain conventions. Use native optional `SearchConfig` with `SearchPaths` - for local session candidates. Only actual absence permits remote fallback. -- Final-domain inspection uses a safe DTO and a defaults-backed - `Load(NoSource(), Definition.Defaults)` snapshot after domain resolution. - Credential fields contain reference names, not typed secrets that would - resolve again. Original results supply actual file provenance. Stages 8–9 - explain this boundary. - -## Stage 1: Establish the dependency and toolchain baseline [x] - -Read `go.mod`, `go.sum`, `.woodpecker/*.yml`, release scripts/runbook, and -attribution. Read the upstream v1.0.0 migration table and v1.2.1 release note. - -1. Select Go 1.27.0 in `go.mod` and ConfigKit v1.2.1 under its vanity path. - Migrate all production/test imports together and remove the old requirement. - Use the released module rather than a filesystem replacement. -2. Replace APIs actually used here: `LoadSecretsDir` → - `LoadSecretsIntoEnvironment`, `LayerBase` → `LayerMain`, and generic - `Resolution.Record` calls with `RecordDerivedDefault` or - `RecordNormalization`. Audit other removed public names and compared layer - or diagnostic strings. Use keyed expanded options/definition literals. -3. Rename genuine source-root `configkit:` controls to `config:` throughout - fixtures and examples. Keep the package name, `configkit:"secret"` Go tag, - and ordinary data keys distinct from source controls. -4. Align Go CI images and local/release instructions with Go 1.27.4. Preserve - platform targets and publication mechanics. Update incorporated Go notices. -5. Refresh ConfigKit and stable go-yaml v4.0.0 attribution from selected source - notices. Review actual module graph churn and tool-only dependencies. Keep - Narratio's separate YAML v3 use unless another scoped change needs it. - -Check module identity and attribution. Attempt focused config/app tests; record -failures assigned to later stages. Complete when the dependency/API baseline is -established, without preserving old behavioral expectations as compatibility. - -## Stage 2: Integrate native pipeline discovery and independent bootstrap [x] - -Read `internal/app/pipeline_config_path.go`, `secrets_bootstrap.go`, -`command_dependencies.go`, `loadPipelineConfigWithDependencies` and callers, -inspection loaders, and `internal/config/pipeline_configkit.go`. -Read upstream discovery and secrets-bootstrap contracts. - -1. Have the configuration owner accept an explicit file or the native required - discovery source above. Retain explicit-file package entry points where useful. - Commands load through one composition path instead of first probing with - `os.Stat` to obtain a default pathname. -2. Bootstrap once before typed loading or adapter creation from the independent - directory. Preserve explicit-path error behavior, safe partial reports, and - existing-empty environment precedence. -3. Obtain the actual selected root and import/overlay evidence from ConfigKit. - Carry that root into campaign selection, provenance, and command summaries. - Remove dependencies/search helpers used only for old pipeline preselection. -4. Apply this composition to run/range/alias commands, session init/plan/status/ - restore, inspection, and remote-session workflows. A selected invalid source - fails rather than falling through to another pipeline candidate. -5. For overlay diff, load the left through native selection and the right from - that selected root with its own overlay. Share invocation bootstrap; neither - side performs a preliminary base-only load. - -Test precedence and exclusive explicit selection with controlled working/user -directories and temporary native search descriptions. Test selected invalid -input, no source, bootstrap override/default behavior, and diff bootstrap once. -Keep tests isolated from the operator's system configuration and secrets. - -## Stage 3: Resolve pipeline resources from native origin context [x] - -Read `resolvePipelineConfig`, `resolveNotariusPaths`, -`finalizeLoadedPipeline`, `effective_digest.go`, and the upstream source-aware -path example. Resolve paths before normalization erases entry origins. - -1. Introduce one small config-owned host-resource base helper using - `Resolution.EntryInfo`, `Resolution.Source`, and - `Resolution.SourceContext`. Use the value's effective origin rather than - its child's inheritance directive. A distinct usable source context supplies - its canonical base; missing or composite context falls back to the main source. - Explicit CLI-origin file references use the captured invocation directory. - If the main input is contextless too, use that invocation directory rather - than interpreting a diagnostic label as a filesystem base. This covers remote - byte snapshots without turning their temporary filenames into resource roots. -2. Normalize nonempty relative references with recorded helpers. Absolute values - stay absolute. Derive Notarius working directory from its already resolved - config path and record that dependency. Preserve domain validation of relevant - empty/disabled values. -3. Apply this classification: - - | Fields | Interpretation | - | --- | --- | - | Audita/Scriptorium/Notarius `config_path`; Notarius `working_directory` | File/directory resource based on effective origin. | - | Relative `workspace.root`, `campaigns.root`, `spool.root`, `cache.root` | Host directory reference based on effective origin; absolute defaults stay unchanged. | - | Campaign template/stable-input paths; session stable-input/audio paths | Same resource policy, completed in Stage 4. | - | Artifact output paths, family output patterns, publish destinations, source IDs, prompt/profile IDs, schema identifiers, binary commands, Notarius reference bindings and chunk-cache modes | Existing domain interpretation, not file-resource rebasing. | - -4. Remove duplicate pipeline-root rebasing and logical-Notarius path caches once - callers use resolved domain values. Show/digest describe those domain values, - rather than reconstructing an alternative authoring-path view. -5. Keep resource access at existing consumer boundaries; base selection adds no - target-existence check, root restriction, or history acceptance gate. - -Test imported, inherited, overlay, absent-default, absolute, and ambiguous-origin -cases. Equal resolved resources have equal effective semantics across layouts; -relative text resolving to a different resource is a meaningful change. - -## Stage 4: Load campaign/session imports with correct contexts [x] - -Read `document_configkit.go`, `load.go`, `pathresolve.go`, -`party_resolution.go`, app session discovery/remote-session composition, and -session init. Read native import and resource-context contracts. - -1. Use `File` with imports enabled for local campaign/session inputs. Share - definitions with byte loading and retain successful results/source contexts. - ConfigKit owns per-source and aggregate acquisition limits. -2. Replace local session candidate probing with native optional discovery over - the existing candidate paths. Permit absent input solely for remote fallback; - distinguish absence through native source evidence. Structure callbacks to - allow that discovery-only no-source result. Acquired invalid files still - receive ordinary validation and fail without remote fallback. -3. Downloaded session bytes receive no context inferred from their temporary - filename. Retain object provenance and current explicit session/previous-ID - checks. Use native contextless-import behavior; no remote import fetcher or - custom structural rejection layer is added. -4. Resolve campaign `session_template_file`, stable-input paths, and session - overrides/audio references with Stage 3's helper. Resolve each - `audio_files` element at its entry index before ordering. Keep campaign/ - session stable-input precedence and campaign ownership of party input. - Provenance identifies the effective supplying configuration source. -5. Check unresolved template tokens in effective session string values, including - imported values. Session init still reads an authored template for intentional - substitution; runtime loading stays a configuration operation. -6. Retain detached campaign and session document values after their own loading - and resource resolution. Capture the session document before campaign-inherited - fields are copied into runtime session state. Stage 6 serializes these values. - -Test local imports, resource bases, effective placeholder rejection, identity -mismatches, imported session-init template paths, absence versus selected-source -failure, and remote snapshots independent of temporary directories. Add no -secondary overlay flags. - -## Stage 5: Resolve imported party definitions and canonical roster output [x] - -Read `party.go`, `party_resolution.go`, `artifact_families.go`, the party -integration contract, and focused party/family tests. - -1. Add a file-backed party entry point with imports enabled for campaign - resolution. Keep `ParseParty(bytes)` contextless for snapshots and tests. - Reuse the definition, main-only version rule, and domain validator. -2. Preserve `narratio.party.v1`, existing character/name/class/alias validation, - and campaign ownership. Native destinations allow split roster contributions; - no fragment version mandate or abstract/template record type is added. -3. Use native metadata for roster order and evidence, preserving authored order - within a source. Extend the current ordering helper using source acquisition - order, then line/column and ID as deterministic ties across sources. Prefer a - child's own declaration/inheritance location over its parent's location. - This is a presentation convention, not a new source acceptance rule. -4. Render canonical party YAML from the effective roster: schema version, keyed - characters, singular external `alias`, classes and optional levels. Preserve - meaningful roster/list order. A small typed DTO/YAML-node writer is suitable - for this Narratio-owned external format. -5. Keep players derivation and family expansion based on that same roster. - Replace output/checksum uses of `CanonicalParty.Raw` with canonical rendering; - retain raw data only where an actual diagnostic caller needs it. - -Test split/inherited rosters, aliases/classes, effective duplicate-name rejection, -order, reloadable standalone canonical output, players, and family identities. -Use native evidence rather than a second source parse to reconstruct order. - -## Stage 6: Materialize effective inputs and align prepare resume checks [x] - -Read `internal/stage/prepare.go`, `prepare_party.go`, `prepare_resume.go`, -prepared-input descriptors, and prepare semantic evidence/tests. - -1. Add configuration-owned deterministic campaign/session renderers from Stage - 4's detached document values. Serialize public schemas and resolved file - references, excluding native directives and runtime/provenance state. -2. Preserve ownership: campaign-inherited stable inputs must not become explicit - session overrides. A published session that omitted `speakers_file` must - still inherit the receiving campaign's value. Explicit session overrides - remain explicit. Include actual session identity values where application - resolution supplies them, without freezing campaign defaults into the session. -3. Prepare writes these documents and Stage 5's party bytes through existing - artifact-store helpers. Keep filenames, input kinds, provenance fields, and - write/checksum behavior. Resume uses the same renderers instead of comparing - prepared effective output with a raw imported root. -4. Ensure imported effective changes reach existing stage-owned semantic/input - evidence. Equivalent domain values remain equivalent. Full pipeline digests, - source mtimes, or every authoring file are not new blanket invalidation keys. -5. Preserve ordinary stable-file copying/audio handling. Already resolved paths - must not be rebased again against the session root. Retire raw campaign/session - file requirements used solely for copying the old authoring documents. - -Test standalone prepared documents, imported changes, equivalent layouts, and -remote-origin metadata. Canonical-byte format changes can cause a normal prepare -rerun; no session migration or new manifest schema is needed for serialization. - -## Stage 7: Close publication, restore, and remote-session round trips [x] - -Read publish input selection, remote-session loading, restore contracts, and -existing in-memory storage integration tests. - -1. Trace the actual bytes published as remote `session.yml`. Publish the canonical - prepared session rather than the original root or command-scoped download. - Preserve immutable object/commit-pointer ordering and lock behavior. -2. Confirm campaign/party prepared products flow through existing selections - with their identities intact and without unresolved companion imports. -3. Verify a published canonical session loads through the existing contextless - path after authoring fragments and temporary files disappear. Restore uses - recorded contents and existing safety/locking. No remote import graph or - producer-host filesystem search is introduced. -4. Verify omitted session stable fields inherit the receiving campaign's values. - Keep remote object identity evidence separate from checksums of locally - materialized canonical bytes. - -Use fake/in-memory storage for prepare → publish selection → remote load and -relevant restore coverage. Fix actual producers/consumers rather than adding a -parallel publication path. Existing atomic commit tests remain authoritative. - -## Stage 8: Build native final-domain inspection snapshots [x] - -Read `effective_projection.go`, `effective_digest.go`, -`effective_pipeline.go`, pipeline finalization/family expansion, and native -completion, defaults, rendering, and typed-secret contracts. - -1. Keep one safe schema-specific final-domain DTO containing concrete expanded - artifacts and generated publish outputs. Omit unexpanded family declarations - and bookkeeping. Audit all credentials, including S3 fields and Audita/ - Notarius/Scriptorium lists: use exact reference names, with no resolved - `Secret` fields in the inspection DTO. -2. Build a native DTO result with `Load(NoSource(), Options{Definition: - Definition{Defaults: ...}})`, supplying a fresh safe DTO and no external - credential lookup or application callback rerun. ConfigKit copies and freezes - these already resolved values. Using defaults avoids interpreting final - `config`/`extends` data as fresh directives or converting precise numbers - through a generic mapping-source reconstruction. -3. Build/refresh the snapshot at existing domain-finalization boundaries, after - applicable path/dependency/family resolution. Keep original authoring results - separately. Synthetic default origins describe this derived snapshot, not - original file provenance. No general snapshot service is needed. -4. Route effective show and `pipeline.resolved.yml` rendering through native - `WriteEffective`. Use the same DTO's deterministic JSON for Narratio's - semantic digest. Preserve exact integers, domain null/empty values, and - credential identities. Update the digest discriminator if its format changes. -5. Remove flattening/serialization machinery used only to duplicate native - show/diff. Keep narrow path enumeration and family/party context genuinely - needed by catalog/source presentation and stage-owned reuse evidence. - -Test final family/publish expansion, inherited/expanded equivalence, large -integer variables, null/empty values, safe rendering, credential-reference -changes versus rotation, and no credential resolution during snapshot creation. - -## Stage 9: Adopt native diff and truthful operator source inspection [x] - -Read `internal/app/config_diff.go`, config command loaders/tests, -`artifact_catalog.go`, and native Diff/source/history contracts. - -1. Fully resolve each overlay against the same selected campaign/party before - comparing Stage 8 snapshots with `configkit.Diff`. Remove - `diffEffectivePipelineValues`, its general map comparison, and consistency - checks predicated on the old flattened-leaf change model. -2. Present canonical JSON Pointer paths in native order. Derive added/removed/ - changed from snapshot `Present`, not nil or redaction text. Use compact JSON - for safe values and expose `HiddenChanged` if present. Equal values produce - `no differences`. Document pointer notation and native changed-region/list - granularity instead of emulating the old engine. -3. Source/catalog evidence comes from authoring results through public source - lookup and metadata history/origins. Add clearly labeled empty optional-import - selection evidence to `config sources`, without guessed candidates or - reasons for absence. -4. Preserve artifact/family/character IDs and generated party context. Distinguish - normalization origins from retained definition evidence. Use native `main` - where an upstream layer is reported; root/import presentation roles may remain - where they express actual application context. - -Test changed family members and publish rules, equivalent layouts, composite -contributors, optional empty imports, atomic lists, and pointer escaping in -allowed dynamic mapping keys. Inspection remains secret-safe and read-only, -without source reparsing or provenance-based acceptance gates. - -## Stage 10: Exercise native authoring and maintain ergonomic examples [x] - -Read the production/testing bundle, artifact/family tests, native destinations -and collection rules, and maintained example tests. - -1. Convert artifact/family files to destination-based definitions. IDs remain - explicit destination keys or keyed payloads; filenames do not imply IDs. -2. Demonstrate a declared directory import with `pattern: "*.yml"`, essential - `required: true` input, an optional selection, inheritance, and a useful - testing-overlay list customization through `config.sequences`. -3. Cover append/upsert/replace on an existing list using an appropriate authored - identity, such as publish output `source`. Whole-record upsert and empty-list - behavior follow ConfigKit. No application identity registration is required - merely to enable authored collection controls. -4. Demonstrate a split roster and local secondary imports using the sample - campaign, with companion resource paths following source-origin policy. - Explain that full-root and disjoint definitions remain accepted too. -5. Use ordinary `Load` for this authored-import workflow. Layered loading is - useful for actual lower-priority sources, not required for defaults alone. - If such a source is genuinely needed, use `LoadLayered` and `MainRelative` - directly with focused coverage instead of another merge/search mechanism. - -Load examples through production definitions. Check dependent logical IDs, -concrete family members, testing differences, and standalone prepared inputs. -Keep examples synthetic and credential-free. - -## Stage 11: Align canonical documentation and project-specific policy [x] - -Read documentation/OSS/release policies, affected references, and the approved -roadmap. Finish current-behavior documentation for the assembled implementation. - -1. Update config/CLI references, operations/bootstrap guidance, troubleshooting, - configuration internals, prepare internals, and party integration docs for the - new discovery, import, resource-base, materialization, and inspection behavior. -2. Replace obsolete import restrictions with an accurate native summary and - pinned upstream links. Explain optional imports, `required`, filters, - destinations, source-local root controls, list modes, and reserved-key scope. - Keep effective domain validation Narratio-owned. -3. Align project-specific architecture configuration/startup statements and the - current raw-party-copy description in its actual owner. Preserve accepted - ADR 0003 and generic adopted policies. Routine integration details do not - require new ADRs. -4. Write manual upgrade guidance for control names, native discovery filenames/ - precedence, explicit custom filenames, secrets override, relative path changes, - required imports where desired, and list operations. Explain effective prepared - documents versus authored roots; downstream migrations remain separate. -5. Finish Go prerequisites, CI/runbook toolchain alignment, and attribution from - Stage 1. Record impact for eventual major-release preparation without inventing - a release version or changing published notes. Leave pending Audita/workspace - roadmaps and Proposed ADRs intact. - -Run doccheck and maintained example validation. Each volatile contract has one -canonical owner. Retire completed roadmap files only when separately requested. - -## Stage 12: Verify the assembled migration and close the sprint [x] - -1. Extend a command-level temporary bundle with native discovery, explicit - secrets override, imported campaign/party/session resources, destination-based - artifacts, family expansion, and a testing overlay. Reuse preceding focused - coverage instead of repeating ConfigKit's structural matrix. -2. Exercise validate/show/sources/artifacts/diff through real composition. Check - selected root, resources, generated values, optional-import evidence, safe - credentials, and absence of runtime workspace creation by inspection. Exercise - plan or prepare with fake external seams to establish execution parity. -3. Trace remaining callers of old search, rebasing, raw-copy, and diff helpers. - Remove dead code/test seams. Review intentional test expectation changes - against the roadmap and preserve existing consequential domain/safety tests. -4. With Go 1.27.4 and dependencies prepared, run `go test ./...`, - `go test -race ./...`, `go vet ./...`, `go build ./...`, - `go mod tidy -diff`, and `git diff --check`. Use the existing release asset - builder with a temporary output directory and its existing `v0.0.0-test` label - to check the three targets; this invokes no publication or new release note. -5. Inspect `go list -m` for the exact release, absence of the old module path - and filesystem replacements, and expected notices. Review every roadmap - capability and all four approved decisions; native structural behavior must - not be narrowed by new Narratio acceptance gates. -6. Commit the finished unit and report checks, compatibility changes, revision, - and any genuine unfinished work. Leave a clean workspace. Actual release - publication and the next feature require separate authorization. - -Complete when the native workflow works end to end, prepared inputs survive -loss of authoring companions, operator inspection covers final domain values, -and final validation passes. - -## Open Questions - -None required for this sprint. The four product choices are approved. This plan -selects concrete native snapshot, source-context, materialization, and toolchain -approaches without adding compatibility shims, structural restrictions, or -stronger runtime guarantees. diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md deleted file mode 100644 index e7176ce..0000000 --- a/docs/roadmap/remediation.md +++ /dev/null @@ -1,459 +0,0 @@ -# ConfigKit adoption review remediation - -## Review scope - -Reviewed the twelve completed stages in [implementation.md](implementation.md) -against [the approved target](configkit.md), the implementation diff from -`65d0e51` through `39a958675d68e940799f4cb8c8e6748830d1483a`, current code, -documentation, focused diagnostic reproductions, and repository checks. - -An integration-focused follow-up reviewed the same implementation at -`bccf03a99f45c2b9b342412b85f70e356663df61`, tracing each stage into existing -command, preparation, publication, restore, and inspection workflows. Items -5–6 were additional findings from that pass; items 1–4 were outstanding then. - -An algorithm-focused pass reviewed all twelve stages at -`583c5bdf1bc8f0af8ec2e257bcb03ecccbf283b1`. Items 7–10 are refactoring -opportunities for consideration. Complexity follows the code and the selected -ConfigKit v1.2.1 implementation; no operational timings or benchmarks were -collected. The pass preserves algorithms expressly chosen by the plan. - -All ten items below are now complete. The original findings are retained with -completion annotations; final verification is recorded at the end of this file. - -## 1. P2: Distinguish effective resource origins from superseded history - -Status: Complete. Native pre-resolution Previous evidence supplies current -contributors; regression coverage includes overlay replacement and inheritance. - -Stage: 9, with documentation coverage in Stage 11. - -Location: `internal/config/effective_pipeline.go:274–280`, -`effectivePipelineOwners`. - -For a resolution-origin field, the projection adds every main/overlay origin -from native history to the same current source-role records. This conflates -effective ownership with historical definitions. For example, a root containing -`audita: {config_path: original.yml}` and an overlay containing -`audita: {config_path: replacement.yml}` produce `audita.config_path` records -for the root, overlay, and resolution. The root's superseded value did not -supply the effective resource; its record is indistinguishable from an effective -contributor. The error is visible to operators using `config sources` to locate -the definition that controls an integration path. - -Required change: use native effective/pre-normalization evidence for effective -input attribution. If historical definitions are also displayed, label them -separately. Preserve historical contributor information in the artifact catalog -where it is explicitly definition evidence. Keep the projection thin; another -provenance reconstruction service is unnecessary. - -Verification: cover a normalized resource with a superseded root/import value, -an overlay replacement, and inheritance. The effective overlay and resolution -remain visible, while superseded evidence is either absent from current-owner -records or explicitly identified as history. Update the command reference for -the resulting roles. - -## 2. P2: Remove the restore test's incidental startup deadline - -Status: Complete. Cancellation is synchronized after discovery with the session -lock held. Ten focused race repetitions and the full repository race suite pass. - -Stage: 12; the fragile timing assumption predates this migration. - -Location: `internal/app/restore_execution_test.go:383–387`, -`TestExecuteRestoreLockConflictFailsAndWritesNothing`. - -The test starts a 100 ms context deadline before calling the complete restore -command, then requires an error from session-lock acquisition. With the current -configuration workflow under the full race suite, the deadline can expire -during earlier work. The reproduced failure was: - -```text -restore: remote check current commit pointer "dnd/campaigns/sample-campaign/sessions/2026-05-03/current/commit-pointer.json": context deadline exceeded, want lock failure -``` - -An isolated application-suite run passed, while a subsequent full race run -failed at this assertion. A green isolated run therefore does not establish -Stage 12's repeatable final validation. The test implicitly requires unrelated -startup work to finish within 100 ms, which is not an application requirement. -This is not evidence that production session locking is incorrect. - -Required change: make the contention/cancellation test target the intended -lock boundary without timing all preceding configuration and remote discovery. -Use synchronization at an appropriate existing test boundary, or separate -lock-service contention coverage from command-level no-write coverage. A larger -arbitrary startup timeout alone leaves the underlying timing dependence. -Preserve the assertions that restore cannot write protected files while another -invocation owns the session lock. - -Verification: repeat the focused test and rerun the complete race suite under -normal package concurrency. Report actual outcomes rather than treating an -isolated pass as full-suite success. - -## 3. P3: Finish the planned field-level publish resolution cleanup - -Status: Complete. Field-only defaults and destinations retain precise native -history. Publish and authored append/upsert/replace regression checks pass. - -Stages: 1 and 3; the roadmap's integration scope requires field-level recording -for field-only changes. - -Location: `internal/config/pipeline_configkit.go:202–215,232–235`, -`resolvePipelineConfig`. - -Publish requiredness and derived destinations already use field-level native -helpers. The resolver additionally tracks `defaultedOutput`, records the entire -`publish.outputs` list for those field edits, and retains a comment claiming -that child records cannot cover the changes. ConfigKit v1.2.1 supports exactly -this field-level coverage. The ancestor record adds redundant history: querying -an untouched output source also returns a whole-list normalization operation -with publish-default/destination reasons. - -Required change: remove the obsolete list-coverage tracking, redundant ancestor -records, and inaccurate comment for these field-only edits. Keep field helpers -and any whole-list records actually needed for structural membership/order -changes. This is completion of the selected simplification, not a new ban on -container-level records. - -Verification: a temporary diagnostic overlay removing only the obsolete -workarounds successfully loaded defaulted/derived publish outputs and passed -the append/upsert/replace collection tests. Add suitable behavioral coverage -for effective values and precise field history, without asserting private -helper invocation counts. - -## 4. P3: Pin Go attribution links to the selected source commit - -Status: Complete. Go license, patent, and source links identify the verified -Go 1.27.4 source commit; documentation checks pass. - -Stages: 1 and 11. - -Location: `docs/oss/attribution.md:125–128`. - -The updated Go entry uses `go1.27.4` in the LICENSE, PATENTS, and Source URLs. -The adopted OSS policy requires commit-specific upstream source and notice -links. The preceding entry used a source hash; this update replaced it with -tag-based references. - -Required change: use the verified Go 1.27.4 source commit -`7b2881024c95a11ddb86663a616340aa3a88b67f` in those three URLs. The hash was -resolved through the official Go repository's release tag. Keep the existing -license texts, factual notice wording, and Go version identification. - -Verification: confirm the links identify that source revision and run the -documentation checks. This requires no licensing change or new release gate. - -## 5. P2: Give session initialization the correct authoring context - -Status: Complete. Rendered templates load with native local context and imports; -local and remote products are standalone effective documents. Relative CLI audio, -import removal, placeholder/identity, and initialization race checks pass. - -Stage: 4, interacting with materialization and command validation in Stages -6, 11, and 12. - -Locations: `internal/app/operator_session_init.go:78–102` and -`internal/config/load.go`, `LoadSessionBytesWithOptions`. - -The existing initializer validates its rendered document through -`LoadSessionBytesWithOptions`. That loader now deliberately describes a -contextless remote snapshot with imports disabled. A generated local session -is an authored file, so this caller no longer uses the same semantics as the -ordinary file loader. - -Two focused reproductions show the mismatch: - -- A local template with a required `shared.yml` import containing constant - metadata fails with `import.disabled`, even when the import exists beside - the generated destination and the resulting local session is otherwise valid. -- With `--audio-dir ./audio` and an output in another directory, initialization - validates the invocation-directory audio path, then writes the original - relative text. Loading that generated file selects the output-directory - audio path. The persisted resource differs from the validated resource and - the documented invocation-relative CLI interpretation. - -Required change: distinguish authored initialization from contextless snapshot -loading. Use ConfigKit's supported source/context facilities for local -composition, and make persisted resource references reproduce the validated -configuration. Materialize effective values where moving or uploading the -document would leave authoring imports unresolved. Preserve the narrow template -substitution contract and the existing contextless remote-loading boundary. - -Verification: initialize a local imported template, load its output through the -ordinary loader, and check the imported values. Cover a destination outside the -invocation directory and relative CLI audio input. Check that remote generation -emits a consumable standalone snapshot and that ordinary placeholder and identity -validation still applies. These cases exercise initialization through the real -configuration owner without adding remote companion retrieval. - -## 6. P2: Reconcile remote session updates with committed-session selection - -Status: Complete. Authored remote input takes precedence for later commands; -committed snapshots remain immutable fallback input and restore authority. -Publication/update regression coverage checks changed title and audio selection. - -Stage: 7, interacting with the existing initializer and shared command loader -from Stages 2 and 4. - -Locations: `internal/app/config_loader.go:194–223` and -`internal/app/operator_session_init.go:113–138`. - -The new remote loader selects a committed `run:prepare/inputs/session.yml` -whenever the current commit declares one. The existing `session init --remote` -writer continues to update the session-root `session.yml`. After publication, -the writer and reader therefore use different authoritative objects. - -A prepare → publish → `session init --remote --force --title "After publication"` -reproduction succeeds and writes the requested title to the root object. A -subsequent ordinary command load still returns the old title from the committed -prepared session. The same selection would ignore changed audio or previous-session -settings. The documented overwrite workflow silently stops affecting later runs -once the session has been published. - -Required change: reconcile remote input authoring and selection so successful -operator updates can be used by subsequent runs. Preserve immutable published -snapshots and committed restore authority. Keep an effective remote editing or -override workflow; silently ignoring updates or introducing a blanket ban on -editing published sessions would leave the integration unresolved. Update the -canonical operator guidance for the resulting selection behavior. - -Verification: publish a prepared session, update it through the supported remote -workflow, then load it through the ordinary run/plan configuration path and check -the updated values. Include audio or previous-session selection as well as title. -Verify that restore still uses its selected immutable commit and that published -objects are unchanged by the authoring update. - -## 7. P3: Reuse enumerated source records and one inspection source index - -Status: Complete. Source enumeration uses complete native records directly; -catalog projection shares one index and retained family sources. Nested empty -imports and existing composite/family inspection checks pass. - -Stage: 9. - -Locations: `internal/config/effective_pipeline.go:57–75`, -`internal/config/pipeline_configkit.go:354–371`, and -`internal/config/artifact_catalog.go`, `declarationDefinitionSources`. - -`emptyImportSelections` enumerates `Result.Sources()` and then calls -`Result.Source(id)` for every enumerated record. `pipelineSourcePaths` does the -same lookup for every retained source. In ConfigKit v1.2.1, completed-result -source lookup scans the source slice. Enumerating S sources this way performs -Θ(S²) source-ID comparisons, although the enumerated records already contain -the required information, including empty-import evidence. - -The artifact catalog additionally rebuilds this source map for each declaration. -For A entries this introduces O(A S²) source-index work; with one imported file -per artifact, that component can become cubic in A. This is separate from the -native history queries and necessary output construction. - -Potential refactoring: consume the complete enumerated records directly. Build -the source-ID/path map once per inspection operation and share it with declaration -projection. This reduces empty-import enumeration to O(S + E), where E is the -returned selection evidence, and source-index work across the catalog to -expected O(S + A). Expanded members can also reuse the family-source lists -already retained in the family catalog. Keep native history and origin queries -as the provenance authority. - -Verification: temporary functional checks confirmed that enumerated records and -individual lookups have equal contents, and that direct enumeration preserves -source paths and nested empty-import selections. Preserve operator source/catalog -output, including composite and inherited contributors, when refactoring. - -## 8. P3: Reuse party metadata and per-character provenance within inspection - -Status: Complete. Each source/catalog projection shares one native metadata -snapshot, one source index, and a per-character cache. Imported-party, composite -provenance, family, and command agreement tests pass under the race detector. - -Stage: 9, consuming Stage 5's imported party result. - -Locations: `internal/config/artifact_catalog.go:67–105` and -`internal/config/effective_pipeline.go`, `appendGeneratedArtifactSources`. - -`partyCharacterSources` calls `result.Metadata()` separately for Lookup and -History. ConfigKit v1.2.1 returns detached metadata by copying the complete -effective-path and field maps, including their origin records. This helper is -called for each generated artifact in the catalog and for each generated -artifact field in source inspection. The same character's provenance is -therefore repeatedly recomputed across fields and families. - -Let M be the copied party-metadata size and P the number of provenance requests. -The avoidable metadata-copy work and cumulative allocation are Θ(P M). Even -with one family and bounded-size character entries, M and P both grow with the -character count C, producing quadratic work. More families and fields multiply -the requests. This describes the provenance pass, not the complexity of the -whole command or unchanged party validation. - -Potential refactoring: obtain one native Metadata snapshot and source index per -inspection operation. Resolve each requested character's source list once and -reuse it for that operation. The full-metadata copy term becomes O(M), with -native per-character query costs and required output emission remaining. Keep -the cache local to the projection; a new persistent provenance service is -unnecessary. Apply the same snapshot reuse to campaign provenance helpers where -convenient, although their fixed field count is not a quadratic case. - -Verification: a temporary imported/inherited-roster check confirmed equivalent -source lists using shared native metadata and an enumerated source index. -Retain contributor accuracy, ordering, and secret-free inspection output. - -## 9. P3: Read the selected remote session without enumerating run history - -Status: Complete. Remote loading opens the selected object once, verifies its -returned identity, and streams into a command-owned temporary file. Remote and -publication round trips pass under race; a rejecting-list fake checks enumeration -is unnecessary, and cleanup remains idempotent. - -Stage: 7. - -Location: `internal/app/config_loader.go:194–228`, `findRemoteSessionConfig`. - -After selecting the committed prepared-session key, this function lists the -entire session prefix and scans it for that exact key. If H objects are retained -across historical runs, this adds Θ(H) listing/scan work and Θ(H) object-record -memory to every remote session load. The production S3 adapter paginates and -accumulates the complete listing. The scan predates the migration and was -retained in the new committed-object selection path. - -Potential refactoring: retrieve the exact selected key through the existing -`ObjectStore.Read` boundary, which returns metadata with the opened body. Use -that metadata for the existing size/generation checks and stream the body into -the command-owned temporary file while retaining checksum verification. The -object lookup/transfer then takes a constant number of requests independent of -H, plus the necessary O(B) work for B session bytes. Current-commit acquisition -and validation costs remain. Apply this after reconciling the selection policy -in item 6; no new backend interface is needed. - -Verification: the production S3 Read implementation performs one exact-key -GetObject and returns its size and generation. Preserve fake-store coverage for -missing objects, selected identity, checksum/size/generation mismatches, and -temporary-file cleanup. Confirm that unrelated historical keys are not enumerated. - -## 10. P3: Stream prepared-input archival copies - -Status: Complete. Verified descriptors stream through the existing atomic-reader -copy helper, with unchanged-destination reuse. Archive bytes, checksums, modes, -and tamper rejection are covered; full ordinary and race suites pass. - -Stage: 7. - -Location: `internal/stage/publish_sources.go:29–64`, -`collectPublishPreparedInputs`. - -After opening and verifying each prepared document, the new archival path reads -the whole file with `io.ReadAll` and passes the byte slice to -`writeBytesIfChanged`. For a document of B bytes this requires Θ(B) auxiliary -transfer memory. Expanded imported party documents can be larger than any one -authoring fragment. A full in-memory document is unnecessary for this file copy. - -Potential refactoring: preserve the current confined source opening and checksum -checks, then stream the opened descriptor into the run archive. The existing -`fileops.WriteReaderAtomicWithChecksum` already supplies reader-based atomic -replacement and hashing through `io.Copy`. Preserve unchanged-destination reuse -using the known checksum. Transfer time remains Θ(B), while auxiliary copy -memory becomes O(1) with a fixed-size buffer. This does not alter the plan's -separate canonical YAML rendering algorithm or total in-memory configuration size. - -Verification: preserve archived bytes/checksums, workspace file modes, source -validation, failure handling, and immutable publication behavior. Existing -reader-copy and prepared-session publication tests provide suitable behavioral -coverage; allocation or wall-clock thresholds are unnecessary. - -## Stage-by-stage disposition at the initial reviews - -| Stage | Review result | -| --- | --- | -| 1: Dependency/toolchain | ConfigKit v1.2.1 and Go 1.27.0/1.27.4 selected; remaining notice pinning in item 4. | -| 2: Discovery/bootstrap | Native discovery, exclusive explicit selection, independent bootstrap, and shared diff bootstrap are present. | -| 3: Resource bases | Native origin-based resolution and fallbacks are present; field-record cleanup remains in item 3. | -| 4: Campaign/session imports | File contexts, contextless remote snapshots, native optional discovery, and detached document values are present; initialization still uses the snapshot path in item 5. | -| 5: Party imports | Imported rosters, source-aware ordering, canonical rendering, players, and family resolution are present; the planned roster sort is retained. | -| 6: Prepared inputs/resume | Canonical document materialization and matching resume evidence are present. | -| 7: Publish/restore | Prepared documents flow through immutable committed run objects and restore; item 6 is a correctness gap and items 9–10 are scalability opportunities. | -| 8: Final-domain snapshots | Defaults-backed native snapshots, safe credential identities, rendering, and digest integration are present. | -| 9: Diff/provenance | Native diff and pointer paths are present; item 1 concerns attribution correctness and items 7–8 concern avoidable provenance work. | -| 10: Examples | Destination-based definitions, directory imports, list operations, split roster, and secondary imports are demonstrated. | -| 11: Documentation/policy | Current references cover the migration; finish attribution, source-role, and initialization/selection corrections above. | -| 12: Final verification | Existing integration checks pass; add coverage for items 5–6. Full race validation remains unreliable because of item 2. | - -The approved discovery, bootstrap, relative-resource, and secondary-import -changes are deliberate contract changes. Items 5–6 describe implementation -inconsistencies in existing workflows. Item 2 concerns an incidental test-time -performance assumption, not a requested startup guarantee. - -## Initial review validation - -Using Go 1.27.4: - -- `go test ./...` passed. -- `go test -race ./...` failed at item 2. The isolated application race suite - passed, as did five subsequent repetitions of the focused restore test. -- `go vet ./...`, `go build ./...`, and `go mod tidy -diff` passed. -- Release-asset cross-builds passed for Linux amd64, Linux arm64, and macOS - arm64 with the existing `v0.0.0-test` label and temporary output directory. -- Focused diagnostic reproductions confirmed items 1 and 3. Temporary Go - overlays kept those diagnostics and proposed-code experiments outside the - repository. -- Documentation checks and `git diff --check` passed. - -Review execution made no application changes and invoked no live subprocess -services or release publication. - -## Integration follow-up validation - -Using Go 1.27.4 at the follow-up revision: - -- `go test ./...` passed with local test-server sockets permitted. The sandboxed - attempt was blocked by `httptest` socket restrictions. -- Focused migration checks across config, stage, and app packages passed both - ordinarily and under `-race`. These exercised native discovery/inspection, - imported inputs, canonical materialization, published-session round trips, - initialization, and command/execution agreement. -- Temporary overlay diagnostics reproduced both aspects of item 5 and the - publication/update conflict in item 6, ordinarily and under `-race`. -- The full race suite was not repeated in this pass; item 2 retains the initial - review's observed failure. Application code is unchanged. - -Only this remediation document changed. Diagnostic files remained outside the -repository, and object storage used an in-memory fake. - -## Algorithm-focused review coverage and validation - -The stage-by-stage pass covered dependency/API substitutions (1), native -discovery/bootstrap (2), resource-base queries (3), secondary loading and -materialization data flow (4), roster ordering/rendering (5), prepare/resume -serialization (6), archive/remote/restore paths (7), finalized snapshots (8), -diff/provenance projection (9), examples (10), documentation (11), and assembled -verification (12). - -The chosen roster ordering, defaults-backed snapshot construction, native Diff, -and required content validation remain accepted. Canonical rendering is linear -in emitted data; the planned roster comparison sort is O(C log C) after deriving -ordering keys. Unchanged algorithms such as pairwise party-name validation and -prepare-input matching are not new findings from this migration-focused pass. - -Cost analysis followed the released module's public API wrappers through native -source lookup and metadata-copy implementations. Temporary functional Go-overlay -checks validated source-enumeration and shared-metadata equivalence, ordinarily -and under `-race`. Focused configuration, file-reader copy, S3 opened-object -metadata, native inspection, and prepared-session publication checks passed on -Go 1.27.4. No performance data, timing assertions, benchmark gates, or application -changes were introduced. - -## Completed remediation validation - -Each issue was addressed in order and committed with its completion marker. -Using Go 1.27.4 on the completed implementation: - -- `go test ./...` passed. -- `go test -race ./...` passed, including the corrected restore contention test. -- `go vet ./...`, `go build ./...`, and `go mod tidy -diff` passed. -- Focused regressions cover effective provenance, field-level publish history, - local/imported initialization, standalone remote input, remote authoring after - publication, enumeration-free loading, and verified streaming archives. -- Documentation checks and `git diff --check` passed. - -Remote session authoring updates affect future command inputs. Committed -publication objects remain immutable and restore continues to validate its -pointer-selected remote state. Inspection caches are scoped to one operation; -ConfigKit remains the authority for configuration and provenance.