diff --git a/docs/cli.md b/docs/cli.md index bd7ce60..ed9f190 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -59,8 +59,8 @@ Rules: base configuration, while an unknown named overlay fails. The same selection applies to all common-flag commands, including `regenerate-artifacts`. - `--secrets-dir` selects ConfigKit credential-file bootstrap explicitly. A - relative value uses caller CWD. Without the flag, a present `secrets/` - directory beside the resolved root pipeline is used; an absent one is skipped. + relative value uses caller CWD. Without the flag, Narratio uses the optional + `/usr/local/etc/narratio/secrets` directory. - `clean --all` cannot be combined with campaign/session selectors. - notification delivery is currently limited to the configured `noop` mode; see the [configuration reference](./config.md#notifications). diff --git a/docs/config.md b/docs/config.md index 301d829..c61162a 100644 --- a/docs/config.md +++ b/docs/config.md @@ -12,10 +12,20 @@ Narratio resolves three YAML documents: ### `pipeline.yml` -When `--config` is omitted, search order is: +An explicit `--config` selects exactly that file, including a custom filename. +Otherwise ConfigKit searches these locations in order: -1. `/usr/local/etc/narratio/pipeline.yml` -2. `/etc/narratio/pipeline.yml` +1. The working directory, for `narratio.yml` then `narratio.yaml`. +2. The Go user-config directory's `narratio/` subdirectory, for those named + files followed by `config.yml`, `config.yaml`, `main.yml`, and `main.yaml`. +3. The Go user-config directory itself, for the named files. +4. `/usr/local/etc/narratio/` and then `/etc/narratio/`, each for named files + followed by the generic filenames above. +5. `/etc/`, for the named files. + +On Linux, the user-config directory normally follows `XDG_CONFIG_HOME`, or +`~/.config` when it is unset. A selected invalid file fails loading; discovery +does not continue to another candidate. ### `campaign.yml` @@ -32,15 +42,12 @@ When `--session` is omitted, local search order is: 1. `/usr/local/etc/narratio/session.yml` 2. `/etc/narratio/session.yml` -If local session discovery fails and a `session_id` is known, Narratio attempts remote session loading from: - -- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/session.yml` - -using configured object storage. - -The downloaded remote session file is command-scoped: Narratio removes it after -the command finishes and records only the remote object provenance alongside -the durable copied session input. +If no local session exists and a `session_id` is known, Narratio loads the +session from configured object storage. It selects the prepared session in the +current committed run when available; if no current pointer exists or an older +commit has no prepared session, it falls back to the session-init `session.yml` +object. Downloaded session bytes are contextless snapshots, with remote object +provenance retained independently of locally materialized checksums. ### Read-only effective pipeline inspection @@ -72,9 +79,10 @@ several files for one family member. The [CLI reference](cli.md#config-validate- owns command syntax and output conventions. `config diff` compares normalized field values rather than YAML text or source -ownership. It emits sorted `added`, `removed`, and `changed` records, uses -compact deterministic JSON values, treats lists atomically, and reports `no -differences` when the complete effective configurations are equal. Concrete +ownership. It emits native ordered `added`, `removed`, and `changed` records with +JSON Pointer paths and compact JSON values. Lists are compared atomically, and +changed regions may contain whole values. It reports `no differences` when the +complete effective configurations are equal. Concrete family members and generated publish rules participate after expansion; moving an equal value between eligible root/import sources does not create a difference. @@ -105,23 +113,14 @@ remote state with an unsafe legacy identity must be migrated before use. - ConfigKit performs strict typed YAML decoding and accepts exactly one document; unknown fields, duplicate keys, and trailing documents fail load. -- A pipeline file may explicitly import additive YAML fragments through the - root-only `configkit.imports` list. Imported files contribute fields to one - logical pipeline document; they do not override fields supplied by the root - or another import. -- An omitted overlay, an empty overlay option, and the exact name `default` - select the base configuration. Any other name selects - `overlays/.yml` relative to the pipeline root. Move previously selected - `profiles/.yml` files there; the old directory is not searched, and - `--profile` is no longer accepted for Narratio pipeline selection. Explicit - imports from a directory named `profiles` remain valid. -- Campaign, session, and canonical-party documents are strict single-source - ConfigKit schemas. They support sibling inheritance but do not enable imports, - selected overlays, or `configkit` directives. +- Local pipeline, campaign, session, and party documents may declare native + imports and sibling inheritance. Only the pipeline selects an overlay. +- An omitted overlay, empty overlay option, and exact `default` select the base; + another name selects `overlays/.yml` relative to the pipeline root. - Configured timeout and retry-delay durations must be positive. An omitted artifact timeout continues to inherit its configured Scriptorium timeout. - Session files must be concrete; unresolved `{{ ... }}` placeholders fail load. -- Session and party inputs inspected before typed loading are limited to 8 MiB. +- ConfigKit applies its acquisition limits to configuration sources and imports. - Pipeline defaults are applied before validation. - Defaults apply only to absent fields. Explicit empty strings, zero values, `false`, and `null` retain ConfigKit's typed meaning and are then checked by @@ -143,82 +142,74 @@ remote state with an unsafe legacy identity must be migrated before use. ### Pipeline imports and overlays -Large pipeline configurations may be split into explicitly named fragments and -may use fixed-path selected overlays: +Local pipeline, campaign, session, and party files can use source-root +`config.imports` and `config.sequences`. Every acquired document has its own +source-root controls, including imported fragments. Nested `config` keys are +ordinary data. `extends` is reserved for native sibling inheritance; parents +remain ordinary entries subject to Narratio's effective-domain validation. ```yaml config: imports: - - path: config/storage.yml - - path: config/integrations.yaml - -campaigns: - root: /usr/local/share/narratio/campaigns + - path: artifacts/session-recap.yml + destination: [scriptorium, artifacts, session_recap] + required: true + - dir: artifacts/families + pattern: "*.yml" ``` -The maintained [production/testing bundle](../examples/production-testing/pipeline.yml) -is a complete copyable example of this structure, including canonical-party -artifact families. +Imports support `path`, `dir`, and `glob` selectors, applicable filters and +relative destinations. Paths are relative to the declaring source. Missing or +empty selections are optional by default; use `required: true` when a selection +is essential. Selected invalid input fails. Destinations place content under +literal mapping keys, independent of filenames; nested imports can declare +relative destinations. Strict peer contributions must satisfy ConfigKit's +composition rules, and Narratio validates the resulting effective entries. -Imports are resolved relative to the directory containing the root pipeline -file and are loaded in declaration order. Narratio does not scan directories or -infer fragments. Each import must be a confined regular `.yml` or `.yaml` file: -absolute paths, traversal, symlinks, directories, duplicate files, and an -import of the root pipeline itself are rejected. Imported files cannot declare -another `configkit` control section. +A selected pipeline overlay is applied after base composition. It may declare +root `config.sequences` but cannot import other files. Ordinary overlay lists +replace the lower list; declared collections can use native `replace`, `append`, +or `upsert`. Upsert replaces a whole matching record; an empty append or upsert +does not clear the lower list. Mapping and identified-list inheritance follow +native sibling rules. See the pinned [ConfigKit configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md) +and [consumer guide](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/consumers/README.md) +for full selection, composition, collection, and reserved-key semantics. +The [maintained bundle](../examples/production-testing/pipeline.yml) illustrates +these controls, family expansion, and a split roster. -Base merging is additive. A map may be extended by multiple files when every -leaf is distinct, but a scalar, list, or map/list/scalar kind cannot be claimed -more than once, even when the repeated values are identical. Conflict errors -name the full field path and every source that claimed it. These rules apply at -every map path, including artifact and artifact-family entries, so distinct -fields of one entry can compose across imports. Narratio validates the resulting -effective entry using its ordinary domain rules. +Relative host-resource paths, including integration configuration files, +workspace/campaign/spool/cache roots, campaign template and stable inputs, and +session stable inputs and local audio references, resolve from the effective +value's defining source. Inherited values follow their origin. When no distinct +usable source context exists, the main source is the fallback; if it too is +contextless, the invocation directory is used. Explicit CLI-relative file paths +use the invocation directory. Artifact outputs, publish destinations, logical +IDs, and binary names retain their domain meanings. -After base sources are composed and the selected overlay is applied, ConfigKit -resolves native sibling inheritance before typed decoding. Any mapping may use -`extends: ` to inherit from an exact key in the same enclosing map; -lookup does not search parent maps or unrelated scopes. The child keeps its own -key and overlays its explicit fields on the resolved parent. Mappings combine -recursively, while explicit scalar, list, false, zero, empty, and null values -remain the child's values. Parent entries remain ordinary configuration entries -and must satisfy Narratio's normal effective-domain validation; inheritance does -not create templates exempt from domain rules. Inherited explicit input counts -as present for absence-based defaults. -Use [`config sources`](./cli.md#config-validate-config-show-config-sources-config-artifacts-and-config-diff) -to inspect native effective contributors and inheritance origins; composite -artifact definitions may report multiple source paths. +The effective digest excludes source layout, overlay identity, and secret values; +credential reference names and resolved resource paths participate. Prepared +campaign, session, and party documents contain effective standalone values rather +than authored import directives. Session fields omitted by the author remain +eligible to inherit the receiving campaign's values. -The key `extends` is reserved structural input wherever ConfigKit is used, -including pipeline roots, imported files, selected overlay values, dynamic -artifact IDs, prompt-variable mappings, campaigns, sessions, and canonical-party -inputs. Rename any existing ordinary-data key named `extends`; -there is no opt-in or automatic conversion. See the pinned [ConfigKit v0.5.0 -configuration contract](https://forge.maximumdirect.net/base-layer/configkit/src/commit/51ef3144158db1a09fabdbd1c7ae59908c84923b/docs/config.md#sibling-inheritance) -for exact lookup, composition, presence, and provenance semantics. +### Upgrade from earlier configuration -Overlay names are case-sensitive and use ConfigKit's safe fixed-path naming -rules. Only the selected overlay is opened; a malformed unselected overlay does -not affect a base or different-overlay load. Maps merge recursively, overlay -scalars replace base scalars, and overlay lists replace base lists completely. -Explicit `false`, zero, empty strings, empty containers, and null values remain -explicit inputs and are not refilled by defaults. Overlay files cannot inherit -from or stack on other overlay files, import files, or declare another ConfigKit -control section. Values in the selected overlay may use native sibling -`extends` directives; those relationships resolve after overlay application. +- Rename source-root `configkit:` controls to `config:`. Nested `config` data + and the Go `configkit` package and secret tags have different meanings. +- Rename or move pipeline files to the discovery names above, or retain a custom + filename such as `pipeline.yml` and pass it with `--config`. +- Move conventional secrets to `/usr/local/etc/narratio/secrets`, supply them + through the environment, or pass the previous directory with `--secrets-dir`. +- Review relative resource references moved into fragments: their defining + source now supplies the base. Mark necessary imports `required: true`. +- Review lists and use `config.sequences` where append or whole-record upsert is + intended. Verify the effective result with the inspection commands. +- Prepared documents contain effective values; retain authored roots and their + companions separately for editing. Separately installed downstream tools own + their own configuration migrations. -During ConfigKit loading, Narratio applies fresh typed defaults, recorded -entry-level defaults and normalization, read-only file-local validation, and -ordinary path resolution before computing a deterministic effective -configuration digest. The digest represents the normalized, secret-free -runtime pipeline mapping; it excludes ConfigKit directives, source -provenance, overlay identity, and raw environment secret values. Equivalent -effective mappings therefore have the same digest regardless of how fields are -split among the root and imports. - -An imported field has the same meaning it would have in a monolithic root -pipeline. In particular, ordinary relative pipeline paths continue to resolve -from the root pipeline directory, not from the importing fragment's directory. +These control-key changes require a major version for the eventual Narratio +release. This migration does not itself publish a release. ## Minimal Working Configuration @@ -263,8 +254,8 @@ inputs: stage range, or eligible for resume. Omit the field or set it to `null` when that deployment should not require the credential. - Every command that loads a pipeline accepts `--secrets-dir `. When the - flag is omitted, Narratio uses a present `secrets/` directory beside the - resolved root `pipeline.yml`; an absent conventional directory is ignored. + flag is omitted, Narratio uses `/usr/local/etc/narratio/secrets` when present; + an absent conventional directory is ignored. Explicit relative paths are resolved from the caller's working directory, and a missing or empty explicit selection is an error. - Each immediate regular file whose name is a valid environment-variable name @@ -320,7 +311,7 @@ Rules: | Field | Type | Required | Default / Rule | | --- | --- | --- | --- | -| `configkit.imports[].path` | string | No | explicit additive pipeline fragment relative to the root pipeline directory; `.yml` or `.yaml` regular files only | +| `config.imports` | list | No | native source-root import selections; see [imports and overlays](#pipeline-imports-and-overlays) | | `pipeline.workspace.root` | string | No | `/var/lib/narratio` | | `pipeline.workspace.cleanup_after_publish` | bool | No | `false`; selects the published run's workspace target for final cleanup and requires effective `pipeline.publish.enabled=true` and `pipeline.publish.upload_run=true` | | `pipeline.campaigns.root` | string | No | `/usr/local/share/narratio/campaigns` | @@ -385,10 +376,10 @@ Rules: | `pipeline.notarius.binary` | string | No | `notarius` | | `pipeline.notarius.credentials[]` | list[secret reference] | No | empty; exact environment-variable names forwarded to Notarius | | `pipeline.notarius.config_overlay` | string | No | unset; exact Notarius ConfigKit overlay selector | -| `pipeline.notarius.config_path` | string | Conditional | required when enabled; relative paths resolve from the pipeline file directory | +| `pipeline.notarius.config_path` | string | Conditional | required when enabled; relative paths resolve from their defining source | | `pipeline.notarius.pipeline_id` | string | Conditional | required when enabled | | `pipeline.notarius.timeout` | duration | No | `3h`; must be positive | -| `pipeline.notarius.working_directory` | string | No | directory containing resolved `config_path`; relative paths resolve from the pipeline file directory | +| `pipeline.notarius.working_directory` | string | No | directory containing resolved `config_path`; relative paths resolve from their defining source | | `pipeline.notarius.resume` | bool | No | `false`; request compatible Notarius checkpoint reuse during ordinary executed extraction | | `pipeline.notarius.chunk_cache` | string | No | empty; when set, exactly `auto`, `bypass`, or `refresh` | | `pipeline.notarius.references` | map[string]string | No | empty; maps normalized Notarius selectors to supported prepared Narratio source IDs; maximum 256 entries | @@ -697,7 +688,7 @@ integration. | `inputs.speakers_file` | string | Yes | stable input default | | `inputs.autocorrect_file` | string | Yes | stable input default | | `inputs.glossary_file` | string | Yes | stable input default | -| `inputs.party_file` | string | Yes | stable campaign party source; relative paths resolve from `campaign.yml` | +| `inputs.party_file` | string | Yes | stable campaign party source; relative paths resolve from their defining source | | `inputs.spell_catalog_file` | string | No | optional spell-catalog overlay default; required when a Notarius reference selects `narratio.input.spell_catalog` | ### Session diff --git a/docs/integrations/party.md b/docs/integrations/party.md index ef4a507..d15149b 100644 --- a/docs/integrations/party.md +++ b/docs/integrations/party.md @@ -1,8 +1,8 @@ # Canonical Party Input -`party.yml` is a campaign-owned roster input. It must be a single, versioned -`narratio.party.v1` document when Narratio resolves a pipeline, campaign, and -session together. +`party.yml` is a campaign-owned roster input. Its main document must declare +`narratio.party.v1` when Narratio resolves the roster. Local party files may +import fragments; fragments need no version header. ```yaml schema_version: narratio.party.v1 @@ -48,8 +48,8 @@ The configuration rules and source-relative path behavior are defined in the ## Derived players document -During `prepare`, Narratio copies the canonical party source bytes unchanged -to `inputs/party.yml` and writes this deterministic players-only projection to +During `prepare`, Narratio renders the effective roster as standalone canonical +`inputs/party.yml` and writes this deterministic players-only projection to `inputs/players.yml`: ```yaml diff --git a/docs/internal/command-restore.md b/docs/internal/command-restore.md index 546a158..331f092 100644 --- a/docs/internal/command-restore.md +++ b/docs/internal/command-restore.md @@ -37,6 +37,8 @@ Restore planner action kinds: Planner behavior: +- prepared campaign, session, and party inputs are restored from their declared + immutable run objects to their canonical local input paths; - a new-protocol restore uses only the selected commit's declared artifact set; each action carries that artifact's immutable key, checksum, size, and generation; diff --git a/docs/internal/configuration.md b/docs/internal/configuration.md index b5d08a4..46a8f14 100644 --- a/docs/internal/configuration.md +++ b/docs/internal/configuration.md @@ -1,210 +1,81 @@ # Configuration Internals -User-visible fields, defaults, and selection behavior belong in the -[Configuration Reference](../config.md). This document describes the internal -pipeline-loading boundary implemented by `internal/config`. +User-visible fields, defaults, discovery, and selection belong in the +[Configuration Reference](../config.md). This document describes the +`internal/config` loading and resolution boundary. ## Pipeline Loading -`LoadPipeline` assembles and validates a pipeline in this order: +Commands bootstrap secrets once before loading, then select an explicit file or +native required discovery. ConfigKit loads typed defaults, source-root imports, +and the selected overlay, resolves inheritance, and strictly decodes the +result. Narratio applies recorded defaults, normalization, resource resolution, +and effective-domain validation. The selected root, source contexts, metadata, +and authoring result are retained for provenance. Family expansion and checks +requiring the campaign occur after campaign and party resolution. -1. Give ConfigKit a fresh typed `PipelineConfig` default value, the root file, - enabled imports, and the caller's overlay selection. Omitted, empty, and - exact `default` selections use the base; named overlays use the fixed - `overlays/.yml` convention. -2. ConfigKit parses the root and declared imports, additively merges base - sources, applies only the selected overlay, resolves native sibling - inheritance, and strictly decodes the result. -3. The definition resolver applies recorded entry-level and inherited defaults, - runs deterministic domain normalization, records derived publish destinations, - and resolves ordinary paths relative to the root pipeline file. -4. The definition validator performs read-only checks that require only the - pipeline file. Checks requiring canonical-party family expansion remain at - the existing explicit pipeline/campaign resolution boundary. -5. Narratio retains public ConfigKit metadata for source ownership, attaches - runtime-only state, and computes the normalized effective digest. Artifact - and family validation applies to the composed effective entries. - -This ordering preserves monolithic configuration behavior. Moving a field to -an imported fragment changes its source ownership, not its path base, default, -or schema semantics. +Resource resolution consults native entry origins and source contexts before +normalization. An effective source supplies the base when usable; otherwise the +main source, then the captured invocation directory, is the fallback. Explicit +CLI file references use the invocation directory. Derived values record their +dependencies. See the [configuration reference](../config.md#pipeline-imports-and-overlays) +for the resource classifications and native authoring contract. ## Loaded Context Resolution -`LoadedPipelineCampaign` carries one already composed pipeline and its selected -campaign into session resolution. `LoadSessionWithPipelineCampaignOptions` -loads a local session against that context, while -`ResolveLoadedPipelineCampaign` also accepts an already loaded remote session -or no session while a caller retrieves one. Compatibility loaders route through -these functions after their initial pipeline and campaign reads. +`LoadedPipelineCampaign` carries the selected pipeline and campaign into local +or remote session resolution. Local campaign, session, and party loads enable +imports through their own ConfigKit definitions; only the pipeline selects an +overlay. Native optional search distinguishes an absent local session from an +invalid selected file. Remote session bytes have no incidental download-directory +context, and retain object provenance. Effective session strings are checked for +unresolved template tokens, and campaign/session identity and stable-input rules +remain Narratio-owned. -Application commands own pipeline and campaign discovery, campaign-file versus -registry selection, and the corresponding mutual-exclusion rules. Once they -have a `LoadedPipelineCampaign`, local session discovery and remote-session -download retain that exact pipeline object and its private provenance. Removing -a temporary downloaded session file therefore cannot invalidate the resolved -pipeline or campaign context. +Campaign and session loaders retain detached document values after their own +resolution. The session document is captured before campaign defaults are copied +into runtime session state. Configuration-owned renderers produce standalone +campaign and session YAML, preserving omitted session overrides. Prepare uses +those renderers and canonical party rendering for materialization and resume +comparison. -Campaign and session files use independent single-source ConfigKit definitions -with imports and selected overlays disabled; native sibling inheritance remains -available within each document. Canonical-party input follows the same -single-source rule. Campaign files normally load directly from their source; -session files are first read through the 8 MiB bounded-read boundary because the -application must reject unresolved template placeholders before passing an exact -byte snapshot to ConfigKit. Local and remotely downloaded session bytes then use -the same typed loader and retain the existing post-load identity expectations, -stable-input precedence, and source-relative path semantics. +## Inspection And Provenance -The application also has a separate read-only inspection resolver for `config -validate`, `config show`, `config sources`, and `config artifacts`. It uses the -same production root, overlay, and campaign selection functions, but never routes -through session discovery, -remote-session download, adapter composition, workspace -initialization, manifest access, or cleanup. A pipeline with retained artifact -family declarations must resolve its selected campaign before ordinary pipeline -validation, which expands its canonical-party members and generated publish -rules. A pipeline without those declarations may be validated by itself. -Secret-directory bootstrap remains invocation-scoped and happens before the -ordinary ConfigKit load because explicit typed references must resolve even for -inspection. +Read-only inspection shares production pipeline and campaign selection without +session discovery, workspace initialization, adapters, or manifest access. After +applicable campaign and family resolution, a safe typed final-domain DTO contains +concrete artifacts and generated publish rules. It stores credential reference +names rather than typed secrets. A defaults-backed native result freezes this +DTO without resolving credentials again; `WriteEffective` supplies show and +prepared pipeline YAML, and deterministic JSON of the same DTO supplies the +versioned semantic digest. Resolved resources participate in that digest, while +source layout and secret values do not. -`MarshalEffectivePipeline` is the configuration-owned projection for `config -show`. It serializes one explicit typed safe projection with the existing YAML -encoder. The projection has no slots for resolution-only artifact family -declarations or private provenance. The result has one trailing newline; -commands do not marshal runtime objects directly. - -`EffectivePipelineSources` and `EffectiveCampaignSources` provide the separate -safe provenance projection for `config sources`. Pipeline ownership begins with -the complete logical field paths projected from the schema and classifies -each contributor as root, import, overlay, or centralized default. It reads -ConfigKit's effective origins and inheritance locations when building each -projection, so inherited values retain their contributors and controlling -directives without a second cached leaf-ownership table. The -projection replaces generated concrete member paths with paired family and -canonical-party records, and does the same for generated publish rules. -Campaign records identify campaign-owned fields and party inputs; canonical -derived players point to the party source. The application command only joins these sorted -records with selection metadata and never reparses configuration files. - -`EffectiveArtifactCatalog` projects the same finalized concrete artifact map for -`config artifacts`. It derives source identity from each map key, carries sorted -effective dependencies, aggregates ConfigKit definition contributors, and adds -family/character/party provenance for generated members. Family source evidence -can name multiple contributing files. It does not inspect files named after -artifacts or infer another runtime graph. - -`config diff` uses a thin paired loader that performs two independent ConfigKit -loads. When campaign -resolution is needed, the command loads one selected campaign and party and -expands both effective pipelines from that same party value. The configuration -owner projects each normalized effective mapping directly from that same typed -projection into sorted logical paths using JSON number preservation; -mapping leaves are compared individually while sequence values remain atomic. -Values are compact deterministic JSON representations for command output, not -raw YAML fragments, ownership records, or secret material. A differing digest -with no projected difference is treated as an internal consistency error. - -Campaign context construction reads the campaign-owned party source through -`ParseParty`. The required canonical party retains its raw bytes and normalized -roster in runtime-only `ResolvedParty` provenance. Canonical resolution creates -a deterministic `derived_from_party` players projection. The party is the only -configured roster authority; player and session-party fields are unsupported. +Authoring results separately supply native origins, source histories, and empty +optional-import evidence to source and artifact inspection. Family and party +context identifies generated entries without reparsing sources. Diff fully +resolves each side against the same selected campaign and party, then uses +ConfigKit `Diff` on the safe snapshots. The command presents native JSON Pointer +paths and safe compact values. ## Canonical Party Domain -`ParseParty` is the package-owned boundary for parsing a party source. Its -single-source ConfigKit definition disables imports and overlays, requires -`schema_version` through `VersionRule`, and strictly decodes the -character-keyed typed schema before applying Narratio's party-domain checks. -Successful metadata coordinates recover authored character-map order without a -second YAML parse; class and alias slice order comes directly from typed decode. -The canonical value retains a separate exact byte copy of its source so -consumers can materialize the authored party document without reserializing it. -Its `PlayersYAML` method deterministically derives the ID-sorted, versioned -players-only projection. - -An unversioned source is rejected before resolution, family expansion, or -prepare-stage materialization. +File-backed party loading enables imports; `ParseParty` accepts contextless bytes. +The shared definition checks the main document's version and Narratio validates +the effective character roster. Native acquisition order and source coordinates +establish roster order, favoring each child's own declaration. A deterministic +renderer writes standalone canonical party YAML from that roster; the players +projection and family expansion use the same resolved roster. The external format +is defined in the [party contract](../integrations/party.md). ## Validation Ownership -Configuration validation remains within `internal/config`, with the public -entry points and pipeline-domain sequencing separated from the private owners -for core pipeline/session rules, external adapters, publish/storage, Notarius, -Scriptorium, and shared scalar/path checks. These are package-local ownership -boundaries rather than extension points or a general validation framework. - -ConfigKit supplies composition, inheritance, presence, typed-secret, and -provenance mechanics; Narratio supplies the domain rules applied to the -effective decoded entries. There is no separate authoring-history acceptance -gate: contributors and declaration history remain inspection evidence, not an -additional validity condition. -The [configuration reference](../config.md#pipeline-imports-and-overlays) -documents sibling syntax, reserved-key migration, and the operator-visible -inspection behavior. - -Validation is read-only, fails fast, and does not define which of several -independent invalid fields is reported first. Map-backed declarations are -nevertheless traversed deterministically. Reusable domain normalization is an -explicit preceding operation for both loaded and programmatically constructed -pipelines. Notarius normalization commits canonical scalar, reference, and output -values only after the complete normalization succeeds; its validator then checks -that candidate without changing it. Scriptorium validates artifact declarations -and outputs, dependencies and inputs, variables, referenced outputs, and the final -dependency graph in deterministic artifact order. - -Diagnostics identify the relevant semantic field. Merge conflicts also -retain every claiming source needed to correct an additive ownership conflict. -Complete prose, punctuation, formatting, and private validation choreography are -not compatibility contracts. - -## Diagnostics And Runtime Metadata - -Syntax, duplicate-key, merge, overlay, and schema failures include the -relevant source name and full field path. Additive conflicts report every -claiming source so operators can repair the split without repeatedly -rediscovering additional conflicts. - -The loaded pipeline retains private runtime metadata for the absolute root -path, ordered imports, selected overlay name, source, and path, original -ConfigKit metadata and source snapshots, and effective digest. Leaf source -records are projected from that retained native metadata when requested. This -metadata does not participate in YAML decoding or alter the public -configuration model. - -The effective digest is SHA-256 over a versioned deterministic JSON encoding of -the same typed safe projection used by show and diff. Runtime Notarius paths remain absolute for -execution, but the digest substitutes their normalized logical values captured -before root-relative resolution, so relocating an equivalent configuration -bundle does not change provenance. Because ConfigKit and resolution metadata -are private, the digest excludes source layout, overlay name, and ownership. -Configuration stores credentials as ConfigKit typed references. A successful -load resolves their values inside the opaque typed boundary, while this safe -projection serializes only `EnvName()` identities. Credential-value changes are -therefore never hashed; reference-name changes remain ordinary provenance. -The optional Notarius credential follows the same path even for disabled or -resumable extraction; normalization copies the opaque reference without reading -or resolving it again. Direct YAML serialization has a configuration-owned safe -projection so only the reference name can leave this boundary. -`recomputePipelineEffectiveDigest` is the single package-owned refresh point -for later runtime expansion. - -## Test Surfaces - -`pipeline_configkit_test.go` exercises real imports and fixed-path overlays, -base selection, unselected-overlay isolation, source attribution, cross-file -inheritance, and effective composition. Existing focused configuration -tests protect defaults, explicit-value semantics, strict decoding, validation, -metadata, and root-relative path behavior. Application configuration-loader tests protect the -single-read boundary by changing the pipeline file after its initial load and -confirming local session resolution retains the original pipeline. Other -configuration tests continue to protect defaults and validation after assembly. -`party_test.go` protects the versioned typed party schema, domain invariants, -authored character order, retained bytes, and deterministic players projection -without involving campaign or runtime wiring. -`party_resolution_test.go` protects campaign-owned party loading, canonical -input restrictions, source provenance, and virtual players input selection. +ConfigKit owns composition, inheritance, presence, typed-secret resolution, and +provenance. Narratio owns validation and normalization of effective domain +values. Source histories are inspection evidence, not an additional acceptance +gate. Focused configuration and application tests exercise imports, resource +contexts, snapshots, secret-safe inspection, party resolution, and materialized +inputs through the production boundaries. ## Artifact Family Resolution diff --git a/docs/internal/overview.md b/docs/internal/overview.md index 6fcf6d8..1e7c55b 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -107,7 +107,7 @@ so any earlier replacement can stale a prior cleanup result. ## Focused Documentation - [Configuration Internals](configuration.md): pipeline imports and overlays, - field ownership, decoding, and root-relative path semantics. + field ownership, decoding, and source-relative resource semantics. - [Adapter Internals](adapters.md): external adapter boundaries, composition, failure behavior, and test surfaces. - [Artifact Internals](artifacts.md): source identities, runtime catalog, diff --git a/docs/internal/stage-prepare.md b/docs/internal/stage-prepare.md index e642c00..a37652d 100644 --- a/docs/internal/stage-prepare.md +++ b/docs/internal/stage-prepare.md @@ -39,8 +39,8 @@ Materialize canonical current-session inputs before processing stages. - materializes a configured spell catalog with checksum and provenance, or safely removes an obsolete canonical spell catalog and its manifest record when the effective input is omitted. -- copies the validated canonical party bytes unchanged and deterministically - generates the prepared players projection. +- renders standalone effective campaign, session, and canonical party documents, + and deterministically generates the prepared players projection. - scans enabled configured artifact inputs for `narratio.previous_session.artifact.*` requirements. - clears managed `previous/` state on every invocation, then, when requirements exist: - resolves the pointer-selected previous source through the shared resolver; @@ -71,15 +71,15 @@ previous-artifact requirement set. A change reruns prepare and stales its fixed descendants. Existing successful records without this evidence rerun once when selected. -Workspace, spool, and cache placement and absolute source relocation are not -semantic when logical selection, canonical names, and bytes are equivalent. +Workspace, spool, and cache placement are not semantic when logical selection, +canonical names, and effective bytes are equivalent. The fingerprint deliberately does not read or rehash large audio. Prepared input checksums remain the content provenance. Before reusing success, prepare validates every durable prepared copy and compares current stable-input bytes, -canonical party and derived-player bytes, local audio membership/checksums, or -S3 key/size/entity-tag identity with that provenance. Source relocation with -equivalent names and bytes remains reusable; changed or unavailable evidence -causes a normal prepare rerun. +effective campaign/session, canonical party and derived-player bytes, local audio +membership/checksums, or S3 key/size/entity-tag identity with that provenance. +Source relocation with equivalent effective documents, names, and bytes remains +reusable; changed or unavailable evidence causes a normal prepare rerun. ## Related Contracts And Tests diff --git a/docs/internal/stage-publish.md b/docs/internal/stage-publish.md index 1d81b9d..6889820 100644 --- a/docs/internal/stage-publish.md +++ b/docs/internal/stage-publish.md @@ -15,8 +15,8 @@ Upload run/session outputs to object storage and atomically advance remote curre ## Outputs -- uploaded session manifest, declared run-local diagnostics, and selected - publish outputs; +- uploaded session manifest, prepared campaign/session/party inputs, declared + run-local diagnostics, and selected publish outputs; - uploaded durable previous-session cache files when present; - immutable run-scoped commit manifest; and - current commit pointer, written last. @@ -34,7 +34,9 @@ and optional rules, destination resolution, and collision inputs live in remote verification live in `publish_transfer.go`; committed-manifest, immutable-commit, and current-pointer encoding and writes live in `publish_commit.go`. `publish_output_plan.go` owns the deterministic, I/O-free -five-outcome model and its secret-free metadata projection. +five-outcome model and its secret-free metadata projection. Prepared input +objects are committed under the immutable run; remote session loading selects +the committed standalone session through the current pointer. - when publishing or run upload is disabled, completes successfully with no outputs and records explanatory metadata. This is not an explicit self-skip: diff --git a/docs/operations.md b/docs/operations.md index 6d54f53..aa20ef8 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -626,6 +626,8 @@ Publish commit model: - uploads eligible run files under `{session_prefix}/runs/{run_id}/`, excluding audio and the run-local Notarius staging bundle; +- uploads prepared campaign, session, and party documents under the run's + `prepare/inputs/` prefix, when recorded by the manifest; - uploads configured published outputs and `previous/**` cache files into the same immutable run scope, including only explicitly configured extraction lanes; @@ -714,8 +716,8 @@ plan, so it is not a network-free operation. Default restore scope: -- the committed session manifest and the committed transcript/artifact objects - declared by the selected remote commit +- the committed session manifest, prepared campaign/session/party documents, + and transcript/artifact objects declared by the selected remote commit - `previous/**` when needed by configured previous-session artifact inputs; each is selected from its exact committed `published_output` declaration @@ -821,7 +823,7 @@ ConfigKit. Every pipeline-loading command bootstraps credentials once, before the first configuration load and before any adapter is constructed. `--secrets-dir` selects an explicit directory (relative paths use caller CWD). Otherwise a -present `secrets/` beside the resolved root pipeline is used, while an absent +present `/usr/local/etc/narratio/secrets` directory is used, while an absent conventional directory is skipped. Explicit missing/empty selections and invalid existing directories fail the command. ConfigKit follows a symlink used as the selected directory, but immediate entries must be regular non-symlink diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index d62e77c..75f513c 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -183,7 +183,7 @@ fail-fast validation of the effective configuration. values remain present and are evaluated by Narratio's domain rules. - Validation rejects invalid configuration before stage execution where practical. -- Root-owned imports and one selected overlay resolve deterministically through +- Local document imports and one selected pipeline overlay resolve through the configuration owner; commands do not implement their own merge rules. - Native ConfigKit metadata and supported resolution callbacks provide presence, inheritance, and provenance information without a parallel loading or diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 220c5ec..ef086c9 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -365,7 +365,7 @@ Load examples through production definitions. Check dependent logical IDs, concrete family members, testing differences, and standalone prepared inputs. Keep examples synthetic and credential-free. -## Stage 11: Align canonical documentation and project-specific policy +## Stage 11: Align canonical documentation and project-specific policy [x] Read documentation/OSS/release policies, affected references, and the approved roadmap. Finish current-behavior documentation for the assembled implementation. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index b85c033..cd60011 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -90,12 +90,12 @@ Symptom: Likely cause: -- the campaign `party.yml` is an unversioned legacy document, or names a +- the campaign party main document is unversioned, or names a schema other than `narratio.party.v1`. Safe fix: -- replace the file with one exact, single-document +- replace the main file with a versioned [`narratio.party.v1` party](./integrations/party.md); do not add a version marker to an otherwise incompatible legacy structure. The versioned party is the only configured roster authority. @@ -118,7 +118,8 @@ narratio config sources --config /path/pipeline.yml --overlay testing Add `--campaign` or `--campaign-file` when the pipeline has party-driven artifact families. The output identifies each effective logical field's root, import, overlay, default, campaign, party, or family source without printing -the field value or credential contents. +the field value or credential contents. Empty optional import selections are +reported separately; selected invalid files still fail. Safe fix: @@ -958,7 +959,7 @@ Symptom: Likely causes: - missing, unreadable, or invalid explicit `--secrets-dir`; -- invalid entries in the conventional `secrets/` directory beside the root pipeline; +- invalid entries in `/usr/local/etc/narratio/secrets`; - missing credential environment variables; - invalid S3 endpoint/bucket settings. diff --git a/internal/app/config_commands.go b/internal/app/config_commands.go index 42035a1..35199fd 100644 --- a/internal/app/config_commands.go +++ b/internal/app/config_commands.go @@ -222,7 +222,7 @@ func parseInspectionFlags(command string, args []string, out io.Writer) (inspect flags.overlay.name = "overlay" fs.Var(&flags.overlay, "overlay", "named pipeline overlay") flags.secretsDir.name = "secrets-dir" - fs.Var(&flags.secretsDir, "secrets-dir", "secrets directory (optional; defaults beside pipeline.yml)") + fs.Var(&flags.secretsDir, "secrets-dir", "secrets directory (optional; defaults to /usr/local/etc/narratio/secrets)") fs.Usage = func() { fmt.Fprintf(out, "Usage: narratio %s [--config ] [--campaign | --campaign-file ] [--overlay ] [--secrets-dir ]\n\n", command) fmt.Fprintln(out, "Resolves configuration without a session, workspace, manifest, adapters, or external services.") diff --git a/internal/app/config_diff.go b/internal/app/config_diff.go index 0f6e58c..d55360c 100644 --- a/internal/app/config_diff.go +++ b/internal/app/config_diff.go @@ -105,7 +105,7 @@ func parseConfigDiffRequest(args []string, out io.Writer) (configDiffRequest, er fs.Var(&request.flags.pipelinePath, "config", "path to pipeline.yml (optional; defaults searched)") fs.Var(&request.flags.campaignPath, "campaign", "campaign ID") fs.Var(&request.flags.campaignFilePath, "campaign-file", "path to campaign.yml") - fs.Var(&request.flags.secretsDir, "secrets-dir", "secrets directory (optional; defaults beside pipeline.yml)") + fs.Var(&request.flags.secretsDir, "secrets-dir", "secrets directory (optional; defaults to /usr/local/etc/narratio/secrets)") fs.Usage = func() { fmt.Fprintln(out, "Usage: narratio config diff [--config ] [--campaign | --campaign-file ] [--secrets-dir ]") fmt.Fprintln(out) diff --git a/internal/app/operator_helpers.go b/internal/app/operator_helpers.go index 861d9f1..653e90c 100644 --- a/internal/app/operator_helpers.go +++ b/internal/app/operator_helpers.go @@ -37,7 +37,7 @@ func addCommonConfigFlags(fs *flag.FlagSet, flags *commonConfigFlags) { flags.overlay.name = "overlay" fs.Var(&flags.overlay, "overlay", "named pipeline overlay") flags.secretsDir.name = "secrets-dir" - fs.Var(&flags.secretsDir, "secrets-dir", "secrets directory (optional; defaults beside pipeline.yml)") + fs.Var(&flags.secretsDir, "secrets-dir", "secrets directory (optional; defaults to /usr/local/etc/narratio/secrets)") } func (f commonConfigFlags) sessionOptions() config.SessionLoadOptions { diff --git a/internal/app/operator_session_init.go b/internal/app/operator_session_init.go index 987e875..bdf4f2e 100644 --- a/internal/app/operator_session_init.go +++ b/internal/app/operator_session_init.go @@ -43,7 +43,7 @@ func sessionInitWithDependencies(ctx context.Context, args []string, out io.Writ overlay.name = "overlay" fs.Var(&overlay, "overlay", "named pipeline overlay") secretsDir.name = "secrets-dir" - fs.Var(&secretsDir, "secrets-dir", "secrets directory (optional; defaults beside pipeline.yml)") + fs.Var(&secretsDir, "secrets-dir", "secrets directory (optional; defaults to /usr/local/etc/narratio/secrets)") fs.BoolVar(&remote, "remote", false, "write session.yml to S3 session prefix") fs.BoolVar(&force, "force", false, "overwrite existing target") if err := parseSessionAwareFlags("session init", fs, args, &sessionID); err != nil {