Update native operations, consumer migration, and version checks
This commit is contained in:
@@ -10,6 +10,9 @@ Notarius uses native CommandKit commands with ckconfig and ckprompt. The
|
||||
and smoke semantics. The examples below show Notarius-specific selections.
|
||||
Use help for every installed command's current options.
|
||||
|
||||
For retired command/configuration mappings and receipt-consumer changes, see
|
||||
[native migration guidance](consumers/native-migration.md).
|
||||
|
||||
## Command Summary
|
||||
|
||||
~~~text
|
||||
|
||||
@@ -6,6 +6,9 @@ source selection, and selectable modules. The sole application version is
|
||||
[Operations](operations.md) owns physical state and recovery; integration
|
||||
contracts own artifact payloads.
|
||||
|
||||
Upgrading an older deployment or caller? Use the
|
||||
[native migration guide](consumers/native-migration.md).
|
||||
|
||||
## Configuration Discovery And Precedence
|
||||
|
||||
Configuration-consuming commands select one main file:
|
||||
|
||||
@@ -1,142 +1,83 @@
|
||||
# Consuming The Complete D&D Pipeline
|
||||
|
||||
Use this workflow when an orchestrator runs the maintained complete D&D
|
||||
pipeline and consumes its structured JSON artifacts. The generic
|
||||
[subprocess consumer guide](subprocess.md) owns process-level responsibilities;
|
||||
this guide connects that workflow to the complete D&D configuration, its
|
||||
Seriatim input, and its artifact inventory.
|
||||
This guide applies the [subprocess workflow](subprocess.md) to the maintained
|
||||
complete D&D configuration. [CLI Reference](../cli.md) owns invocation and the
|
||||
native report; [Configuration](../config.md) owns composition;
|
||||
[Published JSON Output](../integrations/json-output.md) owns logical paths.
|
||||
[Migration guidance](native-migration.md) covers older orchestrators.
|
||||
|
||||
The [CLI reference](../cli.md), [configuration reference](../config.md),
|
||||
[run-result receipt](../integrations/run-result.md), and
|
||||
[published JSON output contract](../integrations/json-output.md) remain the
|
||||
canonical definitions of those public interfaces.
|
||||
## Prepare The Deployment
|
||||
|
||||
## Prepare And Validate The Deployment
|
||||
Start from [examples/dnd-complete/config.yml](../../examples/dnd-complete/config.yml).
|
||||
Preserve its imported pipeline fragments, native companion, and reference files,
|
||||
or update the paths deliberately. It uses every implemented D&D family, ordered
|
||||
generated handoffs, and selected chunk-map/evidence output.
|
||||
|
||||
Start from the maintained
|
||||
[complete D&D configuration](../../examples/dnd-complete.config.yml). It uses
|
||||
the `dnd-session` pipeline and demonstrates every implemented D&D lane, ordered
|
||||
artifact handoffs, campaign references, chunk-map publication, and evidence
|
||||
context.
|
||||
Native embedded defaults and the maintained companion already provide profile/
|
||||
backend definitions. A deployment can supply variants rather than being required
|
||||
to recreate them. YAML paths retain the contributing declaration's origin;
|
||||
CLI reference paths use the process working directory. Absolute caller paths
|
||||
are useful for services. Keep credentials in the intended environment, not
|
||||
files, arguments, sessions, or logs.
|
||||
|
||||
A deployment must provide its own PromptKit profile and campaign reference
|
||||
files. Use absolute paths for service and subprocess deployments. In
|
||||
particular, observe these different resolution rules:
|
||||
Preflight without credentials or inference:
|
||||
|
||||
- reference and PromptKit profile paths in YAML are resolved relative to the
|
||||
file that supplies their surviving declaration;
|
||||
- reference paths passed with `--reference` are resolved relative to the
|
||||
Notarius process working directory.
|
||||
~~~sh
|
||||
notarius config validate --config /absolute/path/to/notarius.yml
|
||||
notarius pipeline validate dnd-session --config /absolute/path/to/notarius.yml
|
||||
~~~
|
||||
|
||||
Do not copy the repository example's relative profile path into a deployment
|
||||
without preserving its configuration-relative layout. The complete path and
|
||||
profile rules are defined in [Configuration](../config.md).
|
||||
## Supply The Transcript And References
|
||||
|
||||
Preflight the deployed configuration before processing sessions and whenever
|
||||
it changes:
|
||||
Use the [Seriatim JSON contract](../integrations/seriatim.md). Preserve segment
|
||||
IDs because artifact citations use those source-unit ranges. If the caller
|
||||
maintains a final trimmed transcript tier, select that same session content
|
||||
consistently for later consumers.
|
||||
|
||||
```sh
|
||||
notarius config validate \
|
||||
--config /absolute/path/to/notarius.yml \
|
||||
--pipeline dnd-session
|
||||
```
|
||||
Run with explicit input/output and pipeline-level references:
|
||||
|
||||
Provide credentials through the environment or the documented configuration
|
||||
mechanism. Do not put credentials in command arguments, generated
|
||||
configuration, or logs.
|
||||
|
||||
## Supply The Transcript
|
||||
|
||||
The complete pipeline consumes a Seriatim JSON document. The
|
||||
[Seriatim input contract](../integrations/seriatim.md) defines its required
|
||||
metadata, segments, and validation rules. Preserve segment IDs: D&D artifact
|
||||
citations use those segment IDs as source-unit ranges.
|
||||
|
||||
When the caller maintains several transcript tiers, use the final trimmed JSON
|
||||
transcript so extraction operates on the same session content presented to
|
||||
later consumers. For example, Narratio identifies this implemented artifact as
|
||||
`narratio.transcript.final_trimmed` and normally stores it at
|
||||
`transcripts/final.trimmed.json`.
|
||||
|
||||
Notarius generates a stable prompt session from the resolved input module and
|
||||
the exact input bytes. An ordinary orchestrator should not pass `--session-id`.
|
||||
Use that override only when intentionally changing the routing relationship
|
||||
between invocations; it is not a credential or output identity.
|
||||
|
||||
## Run Notarius
|
||||
|
||||
Invoke the pipeline with explicit absolute paths and request its
|
||||
machine-readable receipt:
|
||||
|
||||
```sh
|
||||
~~~sh
|
||||
notarius run dnd-session \
|
||||
--config /absolute/path/to/notarius.yml \
|
||||
--input /absolute/path/to/transcripts/final.trimmed.json \
|
||||
--input-file /absolute/path/to/transcripts/final.trimmed.json \
|
||||
--output-dir /absolute/path/to/notarius-output \
|
||||
--reference party=/absolute/path/to/references/party.txt \
|
||||
--reference players=/absolute/path/to/references/players.txt \
|
||||
--reference glossary=/absolute/path/to/references/glossary.txt \
|
||||
--reference spell_catalog=/absolute/path/to/references/spells.json \
|
||||
--json
|
||||
```
|
||||
--format json
|
||||
~~~
|
||||
|
||||
Each unqualified reference is pipeline-scoped: Notarius supplies it to every
|
||||
selected D&D target that declares the slot. A deployment may omit an optional
|
||||
reference it does not maintain, and may use the lane- or binding-qualified
|
||||
forms from the [CLI reference](../cli.md#reference-selectors) for an exceptional
|
||||
override. The registry, scene-description, combat-turn, and NPC-occurrence
|
||||
references declared between ordered steps in the complete configuration are
|
||||
generated artifacts. Do not pass those handoffs on the CLI.
|
||||
Only supply optional references the deployment maintains. Unqualified selectors
|
||||
reach every selected target declaring the slot; narrower overrides follow the
|
||||
[selector contract](../cli.md#reference-selectors). Generated registry, scene,
|
||||
combat, and NPC-occurrence handoffs stay in pipeline configuration, not CLI file
|
||||
bindings.
|
||||
|
||||
The caller should:
|
||||
The ordinary input-based session is suitable without an override. Use
|
||||
`--llm-session-id` only to intentionally change its native selection.
|
||||
Sessions correlate provider routing; they are not credential or output identities.
|
||||
|
||||
- capture stdout and stderr separately;
|
||||
- propagate cancellation and impose an operator-appropriate timeout;
|
||||
- wait for process completion before interpreting stdout; and
|
||||
- retain stderr for diagnosis without copying secrets or transcript content
|
||||
into other logs.
|
||||
Capture stdout/stderr separately, propagate cancellation, impose an appropriate
|
||||
caller deadline, and wait for completion. Parse a successful report only after
|
||||
status zero; failed writes can leave partial stdout. Nonzero status after
|
||||
publication can still leave valid artifacts for deliberate recovery.
|
||||
|
||||
Only exit status 0 permits decoding stdout as a receipt. Ignore stdout after a
|
||||
nonzero exit because a failed receipt write can leave partial bytes. The
|
||||
[CLI reference](../cli.md#output-streams-and-exit-statuses) defines the complete
|
||||
stream and exit-status contract.
|
||||
## Locate And Read Artifacts
|
||||
|
||||
## Discover The Published Bundle
|
||||
Use the report's absolute `output_directory`, not a newest-directory scan.
|
||||
The report's exact fields and fallback qualification belong in
|
||||
[CLI Reference](../cli.md#completion-report). There is no receipt schema,
|
||||
index/manifest traversal, or descriptor inventory to load.
|
||||
|
||||
Decode the successful stdout document as a supported run-result schema. For
|
||||
the current contract, `schema_version` is `notarius.run-result.v2`. Tolerate
|
||||
unknown fields allowed by that version, but reject an unsupported schema
|
||||
version.
|
||||
|
||||
Use the receipt's absolute `output_directory` as the exact run-specific bundle
|
||||
root. Do not scan the output root for its newest directory, guess a run ID, or
|
||||
construct a bundle path. Resolve `index_file` beneath `output_directory` and
|
||||
reject an absolute logical path or any result that escapes the bundle root.
|
||||
|
||||
The complete configuration uses the application validation defaults. A caller
|
||||
that requires fully validated D&D artifacts must also require receipt
|
||||
`validation_status: approved`; a successful `incomplete` result reflects the
|
||||
configured validator-failure continuation policy and carries its bounded
|
||||
validator provenance in `validation_summaries`.
|
||||
|
||||
Read `index.json` and locate each requested lane in `output_files` by its exact
|
||||
`lane_id`. Do not guess a lane filename. Before decoding a payload:
|
||||
|
||||
1. resolve its descriptor's relative `file` beneath the bundle root with the
|
||||
same confinement check;
|
||||
2. verify the descriptor's media type and schema identity against the linked
|
||||
artifact contract; and
|
||||
3. decode the payload according to that contract.
|
||||
|
||||
The [published JSON output contract](../integrations/json-output.md) defines
|
||||
the index and bundle layout. Treat all paths obtained from a decoded external
|
||||
document as untrusted until confined to their documented root.
|
||||
The complete configuration uses safe literal lane IDs: each table entry below
|
||||
is at `lanes/<lane-id>.json` beneath that run directory. Decode the configured
|
||||
domain contract directly. All ten lane artifacts plus the two enabled/available
|
||||
auxiliaries give twelve deliverables; a count is not a substitute for checking
|
||||
the files required by the caller.
|
||||
|
||||
## Complete Artifact Inventory
|
||||
|
||||
When every configured lane is accepted, the complete example publishes these
|
||||
lane artifacts:
|
||||
|
||||
| Lane ID | Purpose | Canonical contract |
|
||||
| --- | --- | --- |
|
||||
| `item-registry` | Canonical registry of encountered items and currency. | [Item registry](../integrations/dnd-item-registry-artifacts.md) |
|
||||
@@ -150,72 +91,52 @@ lane artifacts:
|
||||
| `location-occurrences` | Source-grounded location occurrences. | [Location occurrences](../integrations/dnd-location-occurrence-artifacts.md) |
|
||||
| `enemy-events` | Source-grounded enemy combat events. | [Enemy events](../integrations/dnd-enemy-event-artifacts.md) |
|
||||
|
||||
The JSON encoder always publishes these bundle-management files:
|
||||
The complete configuration also selects two pipeline-wide files:
|
||||
|
||||
| File | Purpose |
|
||||
| --- | --- |
|
||||
| `index.json` | Discovery document for lane and pipeline-wide artifacts. |
|
||||
| `manifest.json` | Run provenance and result summaries. |
|
||||
| `rejected.json` | Rejected pipeline outputs. |
|
||||
| `warnings.json` | Actionable process-degradation warnings. |
|
||||
| `diagnostics.json` | Advisory and observation findings for accepted artifacts. |
|
||||
- [`chunk-map.json`](../integrations/chunk-map.md), the materialized plan/chunks.
|
||||
- [`evidence-context.json`](../integrations/evidence-context.md), the union of
|
||||
selected cited units and their configured surrounding window.
|
||||
|
||||
The complete configuration also requests two pipeline-wide artifacts:
|
||||
They are not lanes and need no discovery descriptor. Their intrinsic payload
|
||||
schemas are unchanged. Evidence is reading context, not authoritative lane
|
||||
attribution; surrounding units do not widen citations in lane artifacts.
|
||||
No index, manifest, rejection/warning/diagnostic sidecars are required.
|
||||
|
||||
- [`chunk-map.json`](../integrations/chunk-map.md), the accepted chunk plan and
|
||||
chunk metadata; and
|
||||
- [`evidence-context.json`](../integrations/evidence-context.md), a reading
|
||||
excerpt containing the union of selected cited source units and the
|
||||
configured surrounding window.
|
||||
## Decide Consumer Acceptance
|
||||
|
||||
Discover both from their top-level `index.json` descriptors rather than
|
||||
treating them as lanes. Evidence context is convenient reading material, not
|
||||
authoritative provenance; citations in the normalized lane payloads remain the
|
||||
evidence contract.
|
||||
Process completion does not establish the caller's required artifact set,
|
||||
semantic quality, or fresh validation under changed policy. A complete-workflow
|
||||
caller should check all ten required lane files against their contracts. A
|
||||
subset consumer can choose a narrower set deliberately. Empty valid domain
|
||||
collections remain distinguishable from missing/incompatible files.
|
||||
|
||||
Every optional or lane file is published only when its corresponding artifact
|
||||
is available. A successful process does not guarantee that all configured
|
||||
lanes were accepted.
|
||||
Explicit permissive execution can complete with qualified fallback. Decide
|
||||
whether to accept that and inspect positive `fallback_count` when relevant.
|
||||
The count identifies emitting operations, not individual affected lanes.
|
||||
Fallbacks satisfy intrinsic/downstream invariants but not ordinary semantic
|
||||
approval; valid downstream results affected by them also avoid cache reuse.
|
||||
|
||||
## Decide What Counts As Consumer Success
|
||||
Default cache reuse does not repeat semantic assessment. For a deliberate model/
|
||||
policy experiment requiring fresh inference, select bypass/refresh under
|
||||
[configuration controls](../config.md#producer-controls). Do not look for retired
|
||||
validation ledgers or treat optional debug serialization as an API.
|
||||
|
||||
Exit status 0 means Notarius completed the pipeline and published its result
|
||||
bundle. The receipt or bundle may still report warnings, rejected outputs, or
|
||||
missing lane descriptors. A downstream consumer must define its own required
|
||||
artifact set explicitly.
|
||||
|
||||
A caller that claims to consume the complete D&D workflow should normally
|
||||
require all ten lane IDs in the table and verify each descriptor's expected
|
||||
contract. If any required lane is missing, rejected, or incompatible, fail the
|
||||
caller's extraction step while retaining the Notarius bundle for diagnosis. A
|
||||
consumer that needs only a subset may define and document a narrower policy.
|
||||
|
||||
Keep the successful receipt with the complete published bundle. Retain
|
||||
`manifest.json`, `rejected.json`, `warnings.json`, and captured process logs as
|
||||
required by the caller's provenance, diagnosis, and retention policies. Avoid
|
||||
selectively copying payload files without also preserving enough index and
|
||||
manifest information to identify their originating run and contracts.
|
||||
|
||||
The transcript, lane artifacts, evidence context, manifest, debug data, and
|
||||
logs can all contain private campaign information. Apply the same access,
|
||||
publication, and retention controls used for the source transcript.
|
||||
Retain useful domain files and caller-owned invocation/report evidence. Capture
|
||||
failure after publication does not invalidate completed artifact writes, but
|
||||
requires deliberate caller recovery rather than parsing partial success output.
|
||||
A failed publication can leave an incomplete directory. Transcript-derived
|
||||
artifacts, evidence, cache, debug, and streams can contain private campaign data;
|
||||
apply the caller's sharing and retention choices.
|
||||
|
||||
## Consumer Checklist
|
||||
|
||||
- Validate the deployed Notarius configuration and `dnd-session` pipeline.
|
||||
- Pass the final trimmed Seriatim JSON transcript with stable segment IDs.
|
||||
- Use absolute configuration, input, output-root, profile, and reference paths
|
||||
in service deployments.
|
||||
- Capture stdout and stderr separately and enforce cancellation and timeout.
|
||||
- Parse stdout only after exit status 0.
|
||||
- Accept only supported receipt, index, and artifact schema versions while
|
||||
tolerating permitted unknown fields.
|
||||
- Use the receipt's `output_directory`; never guess the run directory.
|
||||
- Confine `index_file` and every descriptor path to the published bundle root.
|
||||
- Discover lanes by `lane_id` and verify descriptor compatibility before
|
||||
decoding payloads.
|
||||
- Enforce an explicit required-lane policy and inspect rejections and warnings.
|
||||
- Preserve the receipt and sufficient bundle provenance for every retained
|
||||
artifact.
|
||||
- Protect all transcript-derived files and diagnostic streams as sensitive
|
||||
campaign data.
|
||||
- Validate application configuration and the selected pipeline offline.
|
||||
- Preserve Seriatim source-unit IDs.
|
||||
- Pass source, output root, and optional campaign references as first-class inputs.
|
||||
- Keep generated handoffs configured; avoid external/generated collisions.
|
||||
- Separate streams, propagate cancellation, and check status before parsing.
|
||||
- Use the native report directory and configured domain/auxiliary paths.
|
||||
- Decode supported payload contracts and check the caller's explicit required set.
|
||||
- Decide whether qualified fallback and cached acceptance meet caller needs.
|
||||
- Retain only the artifact/report/evidence records useful to the caller.
|
||||
- Treat source-derived data as sensitive; do not rely on a sanitized debug bundle.
|
||||
|
||||
143
docs/consumers/native-migration.md
Normal file
143
docs/consumers/native-migration.md
Normal file
@@ -0,0 +1,143 @@
|
||||
# Migrating To Native Configuration And Commands
|
||||
|
||||
This guide maps retired Notarius interfaces to the implemented native workflow.
|
||||
It is migration material: the old spellings below are not accepted aliases.
|
||||
[Configuration](../config.md), [CLI Reference](../cli.md), and
|
||||
[Operations](../operations.md) own the current contracts. Domain artifact schemas
|
||||
and lane filename mapping are unchanged.
|
||||
|
||||
## Update Caller Arguments
|
||||
|
||||
| Former interface | Current equivalent / intent |
|
||||
| --- | --- |
|
||||
| `notarius --version` | `notarius version`. |
|
||||
| `pipelines list` | `pipeline list`. |
|
||||
| `config validate --pipeline ID` | `pipeline validate ID`; native `config validate` checks the application document. |
|
||||
| `config effective` | `config show`; scopes are `app` and `promptkit`. |
|
||||
| Positional `config explain POINTER` | `config explain --path POINTER`; use native list/history options for discovery. |
|
||||
| `config profile ID` | `promptkit show profile QUALIFIED_ID`; this reports stored fields, not a copied completed target. |
|
||||
| `--config-overlay NAME` | `--overlay NAME`. |
|
||||
| Run `--input PATH` | `--input-file PATH`. |
|
||||
| `--only a,b` | Repeat literal `--lane a --lane b`; commas are no longer separators. |
|
||||
| `--json` | `--format json`; YAML/text are also native report formats. |
|
||||
| `--session-id ID` | `--llm-session-id ID`. |
|
||||
| `--reasoning-effort VALUE` | `--llm-reasoning-effort VALUE`. |
|
||||
| `--clear-reasoning-effort` | Explicit `--llm-reasoning-effort=`; omission retains native inheritance. |
|
||||
| `--resume`, `--recompute-step`, separate `--chunk_cache`/`--chunk-cache` | Ordinary accepted-result reuse, with invocation-wide `--cache auto\|bypass\|refresh`. No explicit resume or selective checkpoint barrier. |
|
||||
|
||||
`run` and `pipeline run` are two native routes to the same command. Put flags
|
||||
after the complete command path. Singleton occurrences are all parsed, with the
|
||||
last successfully parsed value retained. False/zero/empty supplied values are not
|
||||
omission. Canonical `--llm-*` execution overrides reach producers, correction,
|
||||
and validators; shorter ckprompt aliases apply only to their installed PromptKit
|
||||
commands. Help works offline on subcommands.
|
||||
|
||||
## Migrate Configuration And Resources
|
||||
|
||||
Version 6 is the sole application version; bumping the header alone does not
|
||||
translate old fields. Use the maintained
|
||||
[minimal](../../examples/dnd-minimal/config.yml) or
|
||||
[complete](../../examples/dnd-complete/config.yml) configuration as a starting point.
|
||||
|
||||
- Replace `promptkit.profile_file`, `profile_dir`, and `local_backend` wrappers
|
||||
with native settings beneath the main file's `promptkit`, or wrapperless
|
||||
`prompts.yml` / recursive `prompts/` companions beside that selected main file.
|
||||
Companion files form strict peers, not filename-priority layers.
|
||||
- Use identified `backends`, `profiles`, and `prompts` lists with literal `id`
|
||||
values and collections. The embedded profile is
|
||||
`notarius/dnd-extraction`; initial/correction prompt IDs are also qualified,
|
||||
for example `notarius/dnd.spells` and `notarius/dnd.spells.repair`.
|
||||
- The root bundled backend is `openrouter`. Retired external provider catalogs
|
||||
are not loaded as parallel sources; declare needed backends/profiles through
|
||||
native resources. Do not infer backend identity from an endpoint or model.
|
||||
- Use native root `config` imports and preserve declaration-relative paths.
|
||||
Remove old per-prompt resource-version wrappers such as `v1`; resource IDs no
|
||||
longer select those versions.
|
||||
- A later same-ID resource replaces the whole earlier record. Use a distinct
|
||||
sibling with `extends: {id: parent}` to retain fields, not a partial same-ID
|
||||
replacement or self-inheritance. Profile/backend references to collection
|
||||
resources use qualified IDs; sibling inheritance is local to its list.
|
||||
- Preserve deliberate null/empty/zero choices under native semantics. Null
|
||||
endpoint/key-environment values can allow backend fallback; empty strings can
|
||||
clear those sources. Inspect native effective values/history rather than
|
||||
assuming old wrapper null behavior.
|
||||
- Keep prompt-declared schemas. Inline schemas, source-relative `schema_file`,
|
||||
and explicit-root `schema_path` are native alternatives. Replacing a collection
|
||||
schema root does not retain missing-file fallback to embedded resources.
|
||||
|
||||
Review [native resource and schema rules](../config.md#promptkit-profiles) before
|
||||
reusing an old profile file. The complete example's
|
||||
[companion](../../examples/dnd-complete/prompts.yml) demonstrates current inheritance.
|
||||
|
||||
## Separate Retry Intent, Mode, And Capacity
|
||||
|
||||
Producer `retries` intended for semantic correction becomes
|
||||
`max_additional_turns`, excluding the initial response. Binding overrides pipeline;
|
||||
omission inherits and explicit zero disables semantic correction. Native growing
|
||||
continuation replaces latest-only manual replay, and each revised candidate gets
|
||||
the full applicable assessment chain, including its final permitted turn.
|
||||
|
||||
Remove validator `retries`, generic deterministic retries,
|
||||
`structured_output_repair_attempts`, and the application `promptkit.transient_retries`
|
||||
wrapper. PromptKit owns transient failures and structural repair. Run-wide
|
||||
transient customization uses `--llm-transient-retries`; structural customization
|
||||
belongs in native prompt output contracts. Bundled prompts leave repair counts
|
||||
unspecified. Former total provider-attempt parity is not promised.
|
||||
|
||||
Replace failure-category `validation_policy` and its `fail_run`, `reject_output`,
|
||||
and `warn_continue` choices with producer `mode: strict|permissive`. Strict is
|
||||
the default, including normalization. Permissive explicitly requests a supported
|
||||
module fallback, not incomplete ordinary acceptance. Cancellation does not fall
|
||||
back; unsupported fallback construction fails. Completion discloses actual
|
||||
fallback emitters independently of warning visibility.
|
||||
|
||||
Replace `concurrency.total_llm` and `concurrency.stage_workers.extract` with
|
||||
`concurrency.workers`, covering ready extraction and lane completion together.
|
||||
Choose native backend capacity independently. There is no cross-backend global
|
||||
LLM ceiling, inferred per-backend translation, or extra capacity queue/retry.
|
||||
The [configuration reference](../config.md#concurrency-output-cache-and-debug)
|
||||
owns current defaults and controls.
|
||||
|
||||
## Replace Both Legacy Caches
|
||||
|
||||
Replace nested `cache.chunk_plans` / `cache.checkpoints` directories and recording
|
||||
switches with root `cache_directory` and pipeline/binding `cache`. An explicit
|
||||
invocation `--cache` overrides every binding. Auto reads/stores acceptance;
|
||||
bypass does neither; refresh skips reads and replaces only after new acceptance.
|
||||
|
||||
The initial prepared native message-content hash selects final accepted artifacts
|
||||
and LLM plans. A hit is decoded/usability-checked, not semantically revalidated.
|
||||
Model/settings/policy changes can still hit; use refresh/bypass for experiments
|
||||
requiring fresh work. Source identity alone no longer fixes chunk boundaries.
|
||||
Fallback and actually affected consumers skip reuse; unaffected predecessors
|
||||
remain reusable after downstream failure. Failed refresh preserves old acceptance.
|
||||
|
||||
Do not copy, convert, or delete legacy records as an upgrade prerequisite. They
|
||||
remain ignored and untouched, and the new cache populates through ordinary work.
|
||||
[Operations](../operations.md#accepted-result-cache-and-recovery) owns recovery
|
||||
and deployment-permission guidance.
|
||||
|
||||
## Replace Receipt And Sidecar Traversal
|
||||
|
||||
Stop expecting `notarius.run-result.v2`, `index_file`, validation histories,
|
||||
warning counts, `index.json`, `manifest.json`, or reporting-only rejection/
|
||||
warning/diagnostic files. Check process status, optionally decode the
|
||||
[native completion report](../cli.md#completion-report), then read configured
|
||||
lane/auxiliary [domain paths](../integrations/json-output.md).
|
||||
|
||||
Use the report's absolute run directory, not a guessed run ID or newest-folder
|
||||
scan. Count actual deliverables rather than old normalized-output totals.
|
||||
The report does not establish fresh validation or model quality; decide whether
|
||||
positive fallback qualification is acceptable. Failed stdout can be partial,
|
||||
and post-publication failure can leave valid artifacts with a nonzero status.
|
||||
|
||||
Debug remains explicitly selected and independent of native visibility modes.
|
||||
Directory-only configuration does not enable it; explicit empty disables it.
|
||||
The relative `latest` link updates at start, not completion. Full selected
|
||||
evidence replaces the old summary/trace hierarchy; filenames/JSON shapes remain
|
||||
private and arbitrary application content is not heuristically sanitized.
|
||||
Do not replace receipt traversal with a debug-layout dependency or a new manifest.
|
||||
|
||||
Update callers using the [subprocess guide](subprocess.md) or
|
||||
[complete D&D workflow](dnd-pipeline.md). These changes do not publish a new release
|
||||
or rewrite historical release notes.
|
||||
@@ -1,115 +1,96 @@
|
||||
# Using Notarius As A Subprocess
|
||||
|
||||
Use this workflow when an orchestrator runs Notarius and consumes its published
|
||||
artifacts. The [CLI reference](../cli.md) owns invocation syntax and exit
|
||||
statuses, while the [run-result receipt](../integrations/run-result.md) and
|
||||
[Published JSON Output contract](../integrations/json-output.md) own the
|
||||
durable result formats.
|
||||
Use this workflow to run Notarius and consume domain files. [CLI Reference](../cli.md)
|
||||
owns commands, reports, and statuses; [Published JSON Output](../integrations/json-output.md)
|
||||
owns logical paths and payload links; [Operations](../operations.md) owns state
|
||||
and recovery. See the [complete D&D workflow](dnd-pipeline.md) for its inventory
|
||||
or [migration guidance](native-migration.md) for older callers.
|
||||
|
||||
For the maintained complete D&D workflow, including its transcript input,
|
||||
configured lane inventory, and downstream acceptance checklist, see
|
||||
[Consuming The Complete D&D Pipeline](dnd-pipeline.md).
|
||||
## Prepare And Run
|
||||
|
||||
## Run And Check The Process
|
||||
Use an argument vector with explicit paths appropriate to the deployment.
|
||||
Absolute paths avoid dependence on a service's working directory. Keep stdout
|
||||
and stderr separate, and choose caller cancellation/deadline behavior.
|
||||
|
||||
Optionally preflight a selected configuration and pipeline before work starts:
|
||||
Validate the application and pipeline offline:
|
||||
|
||||
```sh
|
||||
notarius config validate --config /path/to/notarius.yml --config-overlay production --pipeline pipeline-id
|
||||
```
|
||||
~~~sh
|
||||
notarius config validate --config /absolute/path/to/notarius.yml
|
||||
notarius pipeline validate pipeline-id \
|
||||
--config /absolute/path/to/notarius.yml --overlay production
|
||||
~~~
|
||||
|
||||
Omit `--config-overlay` when the composed base configuration is the intended
|
||||
selection; it is independent from any `--llm-profile` run override.
|
||||
Omit the overlay for the composed base. Pipeline validation needs no transcript,
|
||||
reference bytes, credentials, provider call, or state directory.
|
||||
|
||||
Invoke the run with explicit paths and machine-readable output. Capture
|
||||
standard output and standard error separately; do not combine them before
|
||||
processing the result.
|
||||
Then run with first-class source and output inputs:
|
||||
|
||||
```sh
|
||||
~~~sh
|
||||
notarius run pipeline-id \
|
||||
--config /path/to/notarius.yml \
|
||||
--config-overlay production \
|
||||
--input /path/to/source.json \
|
||||
--output-dir /path/to/output-root \
|
||||
--json
|
||||
```
|
||||
--config /absolute/path/to/notarius.yml --overlay production \
|
||||
--input-file /absolute/path/to/source.json \
|
||||
--output-dir /absolute/path/to/output-root --format json
|
||||
~~~
|
||||
|
||||
Use absolute paths for supplied input, configuration, output-root, and
|
||||
reference files. Pass each external reference as its own argument-vector pair;
|
||||
do not construct and invoke a shell command. An unqualified reference selector
|
||||
supplies that file to every compatible selected target. Lane and stage
|
||||
qualification are available for exceptional overrides, while generated
|
||||
same-run references remain part of configured pipeline composition. The
|
||||
[CLI reference](../cli.md#reference-selectors) owns the exact selector and
|
||||
precedence contract.
|
||||
Pass external references as separate arguments, for example
|
||||
`--reference party=/absolute/path/to/party.txt`. Unqualified selectors reach
|
||||
compatible selected targets; narrower selectors are exceptional overrides.
|
||||
Generated same-run handoffs stay configured, not CLI files. The
|
||||
[reference selector contract](../cli.md#reference-selectors) owns matching and
|
||||
precedence.
|
||||
|
||||
The maintained D&D subprocess workflow uses this facility for campaign context:
|
||||
Supply credentials through the intended process environment, not arguments,
|
||||
session IDs, generated files, or logs. Omitted session selection uses the
|
||||
input-based default; use the [native override](../cli.md#run-wide-llm-overrides)
|
||||
only when intentionally changing it.
|
||||
|
||||
```sh
|
||||
notarius run dnd-session \
|
||||
--config /absolute/path/to/notarius.yml \
|
||||
--input /absolute/path/to/transcripts/final.trimmed.json \
|
||||
--output-dir /absolute/path/to/notarius-output \
|
||||
--reference party=/absolute/path/to/references/party.txt \
|
||||
--reference players=/absolute/path/to/references/players.txt \
|
||||
--reference glossary=/absolute/path/to/references/glossary.txt \
|
||||
--reference spell_catalog=/absolute/path/to/references/spells.json \
|
||||
--json
|
||||
```
|
||||
## Check Completion Before Parsing
|
||||
|
||||
Only pass the external references available to and desired by the deployment.
|
||||
Notarius generates a stable prompt session for the resolved input module and
|
||||
exact input bytes; reference changes do not change it. Pass **--session-id**
|
||||
only when intentionally grouping different invocations under a different
|
||||
session. Supply credentials through Notarius's documented configuration and
|
||||
environment mechanisms, never as command-line arguments or generated
|
||||
secret-bearing configuration. In particular, a session identifier is
|
||||
provider-visible and is not a credential mechanism.
|
||||
Wait for completion. Only successful status permits interpreting stdout as a
|
||||
successful native report. Failed delivery can leave partial bytes; retain stderr
|
||||
and available locations rather than parsing a structured error envelope.
|
||||
|
||||
Wait for the process before interpreting standard output. Only an exit status
|
||||
of 0 permits decoding the receipt. On a nonzero exit, retain standard error for
|
||||
diagnosis and ignore all standard-output bytes: a failed receipt write may have
|
||||
left a partial document.
|
||||
Optionally decode the [small completion report](../cli.md#completion-report).
|
||||
Its `output_directory` is the exact run root; do not scan for the newest folder
|
||||
or guess a run ID. The report is not a persisted receipt, schema-version envelope,
|
||||
index, or validation/execution ledger.
|
||||
|
||||
## Discover Required Artifacts
|
||||
A completed run may contain supported fallback. The caller decides whether that
|
||||
is acceptable; inspect positive `fallback_count` when relevant. It counts
|
||||
emitting operations, not affected files. Reuse performs ordinary usability checks,
|
||||
not fresh semantic assessment. Callers needing new inference/assessment after
|
||||
policy changes deliberately select bypass/refresh under the
|
||||
[cache controls](../config.md#producer-controls).
|
||||
|
||||
Decode the successful receipt and accept the schema versions supported by the
|
||||
caller. Use its `output_directory` as the bundle root. For the production JSON
|
||||
output, resolve `index_file` under that root with a confinement check and reject
|
||||
an absolute path or a result that escapes the root.
|
||||
## Read Required Domain Files
|
||||
|
||||
Read the resulting `index.json` and locate each artifact by `lane_id`, not by a
|
||||
guessed filename. Before decoding a selected payload, verify its descriptor's
|
||||
media type and schema identity against the relevant published artifact
|
||||
contract. The JSON bundle contract links to the available lane contracts.
|
||||
Define required/optional lanes from the selected pipeline. Read their
|
||||
[documented paths](../integrations/json-output.md#bundle-layout) beneath the
|
||||
report directory and decode using the configured artifact contracts. For custom
|
||||
lane IDs, apply the unchanged filename mapping; collisions are command failures.
|
||||
No receipt/index traversal, manifest descriptors, or reporting sidecars are needed.
|
||||
|
||||
If `index.json` has an `evidence_context` descriptor, treat it as a
|
||||
pipeline-wide artifact rather than a lane entry. Verify its six descriptor
|
||||
fields before decoding the linked file according to the [Published Evidence
|
||||
Context contract](../integrations/evidence-context.md). Decode its top-level
|
||||
source-unit array as a reading excerpt. Obtain authoritative citations and lane
|
||||
provenance from the normalized lane artifacts; the excerpt has neither and its
|
||||
nearby units do not widen a lane artifact's cited source reference.
|
||||
Optional chunk-map/evidence files use their documented paths when enabled and
|
||||
available. Decode supported payload schemas. Evidence is a reading excerpt, not
|
||||
authoritative lane attribution or a widened citation; normalized lane payloads
|
||||
retain those contracts.
|
||||
|
||||
A zero exit status may still report rejected outputs, warnings, or absent
|
||||
lanes. The caller decides which lane IDs are required for its own work and
|
||||
which are optional; it should make that decision explicitly rather than infer
|
||||
failure from the receipt counts alone.
|
||||
Process completion, required artifact availability/compatibility, and caller
|
||||
acceptance requirements are separate checks. A count alone cannot identify
|
||||
which files were produced.
|
||||
|
||||
When complete validation is required, also require receipt
|
||||
`validation_status: approved` and inspect `validation_summaries`. A successful
|
||||
run with `validation_status: incomplete` contains a structurally valid result
|
||||
that advanced after validator execution could not complete under the configured
|
||||
`warn_continue` policy. It is not reusable checkpoint state and should not be
|
||||
silently treated as fully reviewed by the caller.
|
||||
## Preserve Useful Evidence
|
||||
|
||||
## Preserve Provenance And Handle Data Carefully
|
||||
Keep domain files and caller-owned invocation/report records useful for provenance.
|
||||
Reports can be explicitly redirected; Notarius does not automatically persist a
|
||||
second report or require a metadata bundle. Debug is optional private evidence,
|
||||
not a consumer protocol.
|
||||
|
||||
Keep the receipt with the published `manifest.json`, and retain
|
||||
`rejected.json`, `warnings.json`, and `diagnostics.json` when review or later provenance requires
|
||||
them. Treat the input, output bundle, cache, debug bundle, and captured process
|
||||
logs as potentially sensitive data. Apply the caller's access controls and
|
||||
retention policy, and avoid copying secrets into arguments, logs, or
|
||||
provenance records. An evidence-context artifact contains source-unit text and
|
||||
metadata and can cover most of an input; preserve and share it only when that
|
||||
source content is authorized for the recipient.
|
||||
Post-publication capture/report failure can return nonzero while leaving valid
|
||||
artifacts. Preserve available files for deliberate recovery, but do not promote
|
||||
partial stdout or incomplete publication to ordinary consumer success.
|
||||
[Operations](../operations.md#output-bundles) explains those boundaries.
|
||||
|
||||
Inputs, artifacts, evidence, cache, debug, and streams may contain sensitive data.
|
||||
Apply intended access, sharing, and retention controls. Evidence can cover the
|
||||
whole source and is not inherently less sensitive.
|
||||
|
||||
@@ -40,7 +40,7 @@ implementing changes. It owns implementation discipline and local commit rules.
|
||||
| --- | --- | --- |
|
||||
| Finding the package or component that owns current behavior | [Internal Overview](internal/overview.md) | It is the implemented component inventory and routes to focused internals. |
|
||||
| CLI composition or command behavior | [CLI Internals](internal/cli.md) and [CLI Reference](cli.md) | The internal guide owns composition and command flow; the reference owns public syntax. |
|
||||
| Building a subprocess caller or changing its result protocol | [Subprocess Consumer Guide](consumers/subprocess.md), [Complete D&D Consumer Guide](consumers/dnd-pipeline.md), [Run Result Receipt](integrations/run-result.md), and [CLI Internals](internal/cli.md) | These separate generic caller workflow, the complete D&D workflow, the durable receipt contract, and CLI implementation behavior. |
|
||||
| Building a subprocess caller or changing its result protocol | [Subprocess Consumer Guide](consumers/subprocess.md), [Complete D&D Consumer Guide](consumers/dnd-pipeline.md), [Native Completion Report](cli.md#completion-report), and [CLI Internals](internal/cli.md) | These separate generic caller workflow, the complete D&D workflow, the native completion report, and CLI implementation behavior. |
|
||||
| Configuration loading, resolution, or user-visible configuration behavior | [Configuration Internals](internal/configuration.md) and [Configuration](config.md) | The internal guide owns loading and resolution mechanics; the reference owns the configuration contract. |
|
||||
| Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. |
|
||||
| Production modules or validators | [Module Internals](internal/modules.md), [D&D Module Internals](internal/dnd.md), and [D&D integration contracts](integrations/) | The generic guide owns extension mechanics, the D&D guide owns shared family conventions, and the contracts own durable output shapes. |
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Accepted Chunk Map
|
||||
|
||||
This document defines the optional durable `chunk-map.json` artifact in a
|
||||
[published JSON bundle](json-output.md). It describes the accepted,
|
||||
materialized chunk plan used by one run. It is not a lane payload and is never
|
||||
an input to a later pipeline step.
|
||||
[published JSON bundle](json-output.md). It describes the materialized chunk
|
||||
plan used by one run, including a supported qualified fallback. It is not a lane
|
||||
payload and is never an input to a later pipeline step.
|
||||
|
||||
## Contract Identity
|
||||
|
||||
@@ -16,8 +16,8 @@ an input to a later pipeline step.
|
||||
| Schema name | `notarius_source_chunk_map_v1` |
|
||||
| Schema version | `v1` |
|
||||
|
||||
The optional `chunk_map` descriptor in `index.json` identifies this artifact.
|
||||
Export is controlled by the JSON output binding described in
|
||||
Read this logical path directly when enabled and available; no discovery
|
||||
descriptor is required. Export is controlled by the JSON output binding in
|
||||
[Configuration](../config.md#module-bindings-and-validators).
|
||||
|
||||
## Wire Shape
|
||||
@@ -30,7 +30,7 @@ Every payload has these required fields:
|
||||
| `source_digest` | Lower-case `sha256:` digest of that source document. |
|
||||
| `plan_digest` | Lower-case `sha256:` digest of the logical chunk plan. |
|
||||
| `requested_chunker` | Chunk module selected by the resolved pipeline. |
|
||||
| `producer` | Original accepted-plan producer. `input_module` and `chunk_module` are required; `llm_profile` is optional. |
|
||||
| `producer` | Current run's plan component selections. `input_module` and `chunk_module` are required; `llm_profile` is optional and applies only to an LLM-backed chunker. Not historical cache execution provenance. |
|
||||
| `plan_annotations` | Plan-level annotation namespace map; `{}` when none are present. |
|
||||
| `chunks` | Non-empty execution-order chunk collection. |
|
||||
|
||||
@@ -47,9 +47,11 @@ valid JSON and are retained without interpreting a module-specific namespace.
|
||||
|
||||
`chunks` are in execution order. Their indexes are contiguous, start at zero,
|
||||
and equal their array positions; chunk IDs are unique. The emitted map is built
|
||||
only after the selected plan has been accepted and materialized against the
|
||||
source document, so its ranges, unit counts, annotations, and digests describe
|
||||
that exact plan.
|
||||
after the selected plan has been materialized against the source document,
|
||||
so its ranges, unit counts, annotations, and digests describe that exact plan.
|
||||
Supported fallback still satisfies those invariants without asserting ordinary
|
||||
semantic approval; run-level qualification belongs to the
|
||||
[CLI report](../cli.md#completion-report).
|
||||
|
||||
The codec rejects malformed JSON, trailing content, unknown fixed-object
|
||||
fields, invalid identities or digests, invalid annotations, duplicate chunk
|
||||
@@ -67,12 +69,13 @@ annotations, producer identity, and ordered chunks.
|
||||
|
||||
## Publication And Compatibility
|
||||
|
||||
The map is present only when a chunk plan was accepted and its export is
|
||||
enabled. It remains publishable if a later lane is rejected, but is absent when
|
||||
chunk-plan validation rejects the plan. `requested_chunker` identifies the
|
||||
current pipeline selection, while `producer` identifies the component that
|
||||
originally produced the accepted plan; they may differ when an accepted plan is
|
||||
reused.
|
||||
The map is published when export is enabled, a materialized plan is available,
|
||||
and the run reaches publication. Payload fields and schema are unchanged.
|
||||
`requested_chunker` and `producer` describe the current composition; unified
|
||||
cache entries retain the logical plan, not an earlier run's producer/profile
|
||||
ledger. Reuse rematerializes against the current source without semantic
|
||||
reassessment or claims of historical model execution. A strict failure does not
|
||||
publish a successful bundle merely to expose this auxiliary.
|
||||
|
||||
The map contains structure rather than source content: it excludes transcript
|
||||
bytes, source-unit metadata, chunk text, private model output, reference
|
||||
|
||||
@@ -9,9 +9,8 @@ logical-file discovery is owned by [Published JSON Output](json-output.md).
|
||||
|
||||
## Identity And Discovery
|
||||
|
||||
When enabled, the JSON bundle contains `evidence-context.json` and an
|
||||
`index.json` `evidence_context` descriptor with the same six fields as other
|
||||
pipeline-wide artifact descriptors.
|
||||
When enabled and available, the JSON bundle contains `evidence-context.json`
|
||||
at that documented logical path; no index descriptor is required.
|
||||
|
||||
| Property | Value |
|
||||
| --- | --- |
|
||||
@@ -22,16 +21,16 @@ pipeline-wide artifact descriptors.
|
||||
| Schema version | `v1` |
|
||||
| Logical file | `evidence-context.json` |
|
||||
|
||||
Consumers must discover the file from the descriptor, verify all six descriptor
|
||||
fields, and decode only a supported schema version. The descriptor is optional:
|
||||
its absence means evidence publication was not enabled for that bundle.
|
||||
Consumers select this path from the configured output policy and decode the
|
||||
supported payload contract below. Disabled or unavailable evidence is not
|
||||
published. The identity and JSON shape above remain unchanged.
|
||||
|
||||
## Payload
|
||||
|
||||
The v1 payload is a top-level JSON array of generic source units. There is no
|
||||
wrapper, source-level metadata, context grouping, lane identifier, or evidence
|
||||
reference in the payload. An enabled configuration with no contributing
|
||||
accepted evidence publishes `[]`.
|
||||
evidence publishes `[]`.
|
||||
|
||||
```json
|
||||
[
|
||||
@@ -73,14 +72,16 @@ annotations or reshape source-owned metadata.
|
||||
## Selection And Citations
|
||||
|
||||
The framework obtains direct source references only through typed evidence
|
||||
projections of accepted normalized artifacts in the configured lane allowlist.
|
||||
projections of published normalized artifacts in the configured lane allowlist,
|
||||
including supported fallbacks and reused results.
|
||||
It validates each reference against the current source document, expands its
|
||||
range by `window_units` source-unit positions on each side, clamps at document
|
||||
boundaries, and takes the union of all expanded ranges. The output contains
|
||||
each selected source unit once in source-document position order, regardless
|
||||
of numeric unit IDs. Repeated references, overlapping windows, and citations
|
||||
from multiple lanes do not duplicate a unit. Rejected, failed, absent,
|
||||
inactive, and unselected lanes contribute nothing.
|
||||
from multiple lanes do not duplicate a unit. Failed, absent, inactive, and
|
||||
unselected lane results contribute nothing. Evidence from a supported fallback
|
||||
does not imply ordinary semantic approval; the completion report qualifies the run.
|
||||
|
||||
Normalized lane artifacts remain authoritative for citations and for which lane
|
||||
cited a range. The excerpt has no lane attribution and must not be used to
|
||||
@@ -94,10 +95,10 @@ the framework does not truncate the excerpt to meet an arbitrary size limit.
|
||||
|
||||
## Consumer Responsibilities And Data Handling
|
||||
|
||||
The artifact is additive to the JSON bundle and is not a lane payload,
|
||||
normalized-output count, checkpoint, or generated reference. Consumers that
|
||||
do not need it must tolerate an absent descriptor. Consumers that do use it
|
||||
should validate the descriptor and payload before use, retain the artifact with
|
||||
The artifact is additive to the JSON bundle and is not a lane payload or
|
||||
generated reference. It counts as a deliverable when published, not as a lane.
|
||||
Consumers that do not need it can ignore it. Consumers that do use it should
|
||||
validate the payload before use, retain the artifact with
|
||||
its schema identity when needed for a run record, and read citations from the
|
||||
corresponding normalized lane artifacts.
|
||||
|
||||
|
||||
@@ -1,196 +1,79 @@
|
||||
# Published JSON Output
|
||||
|
||||
This document defines the logical JSON bundle emitted by the production JSON
|
||||
output encoder. The bundle’s physical destination, atomic publication, and
|
||||
retention are operational concerns; see [Operations](../operations.md#output-bundles).
|
||||
Output configuration, including chunk-map and evidence-context publication, belongs in
|
||||
[Configuration](../config.md#module-bindings-and-validators).
|
||||
This contract defines logical files emitted by the production JSON output
|
||||
encoder. [Operations](../operations.md#output-bundles) owns physical placement,
|
||||
publication failures, permissions, and retention. [Configuration](../config.md#module-bindings-and-validators)
|
||||
owns output options; the [CLI completion report](../cli.md#completion-report)
|
||||
identifies a successful run's directory.
|
||||
|
||||
## Bundle Layout
|
||||
|
||||
All paths below are logical, relative, slash-separated bundle paths. The
|
||||
encoder always emits the first five JSON files below and adds lane or
|
||||
pipeline-wide artifact files when their corresponding artifacts are available:
|
||||
|
||||
A subprocess caller first obtains the physical bundle root from the
|
||||
[run-result receipt](run-result.md), then resolves `index.json` beneath that
|
||||
root for the logical discovery described here.
|
||||
Paths are relative, slash-separated paths beneath the run-specific directory:
|
||||
|
||||
| Path | Purpose |
|
||||
| --- | --- |
|
||||
| `index.json` | Entry point that names the other published files and lane payloads. |
|
||||
| `manifest.json` | Run provenance and result summaries. |
|
||||
| `rejected.json` | Rejected pipeline outputs. |
|
||||
| `warnings.json` | Actionable process-degradation warnings. |
|
||||
| `diagnostics.json` | Accepted-artifact quality advisories and normalization observations. |
|
||||
| `lanes/<safe-lane-id>.json` | One normalized artifact payload for each lane. |
|
||||
| `chunk-map.json` | Optional accepted chunk map, when its export is enabled and available. |
|
||||
| `evidence-context.json` | Optional selected source-unit excerpt, when evidence publication is enabled. |
|
||||
| `lanes/<safe-lane-id>.json` | One normalized domain artifact per available selected lane, including supported qualified fallback. |
|
||||
| `chunk-map.json` | Optional materialized chunk map, when enabled and available. |
|
||||
| `evidence-context.json` | Optional selected source-unit excerpt, when enabled and available. |
|
||||
|
||||
JSON files are pretty-printed with a trailing newline. Lane payloads are
|
||||
accepted only when their media type is `application/json`.
|
||||
There is no required index, manifest, persisted receipt, rejection file, warning
|
||||
file, or diagnostic sidecar. Expected lanes come from pipeline configuration;
|
||||
optional auxiliaries use the paths above. Consumers need no metadata bundle or
|
||||
debug data to decode artifacts.
|
||||
|
||||
## `index.json`
|
||||
The lane filename mapping is unchanged: trim the ID, replace characters outside
|
||||
ASCII letters, digits, periods, underscores, and hyphens with underscores,
|
||||
neutralize `..` sequences, and trim leading/trailing periods and underscores.
|
||||
An empty resulting name or colliding paths fails encoding. Maintained D&D lane
|
||||
IDs already map directly to their filenames.
|
||||
|
||||
`index.json` is the bundle’s discovery document. An approved run with no
|
||||
normalized lanes has this valid minimal index:
|
||||
|
||||
```json
|
||||
{
|
||||
"manifest_file": "manifest.json",
|
||||
"output_files": [],
|
||||
"rejected_file": "rejected.json",
|
||||
"warnings_file": "warnings.json",
|
||||
"diagnostics_file": "diagnostics.json"
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Required | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `manifest_file` | Yes | Always `manifest.json`. |
|
||||
| `output_files` | Yes | Lane descriptors sorted by `lane_id`. |
|
||||
| `rejected_file` | Yes | Always `rejected.json`. |
|
||||
| `warnings_file` | Yes | Always `warnings.json`. |
|
||||
| `diagnostics_file` | Yes | Always `diagnostics.json`. |
|
||||
| `chunk_map` | No | Descriptor for the pipeline-wide `chunk-map.json`; never a lane descriptor. |
|
||||
| `evidence_context` | No | Descriptor for the pipeline-wide `evidence-context.json`; never a lane descriptor. |
|
||||
|
||||
Each lane descriptor has required `lane_id` and `file`. It may also include
|
||||
`media_type`, `module_key`, `schema_id`, `schema_name`, and `schema_version`
|
||||
when supplied by the normalized artifact. Each pipeline-wide artifact
|
||||
descriptor (`chunk_map` or `evidence_context`) contains `artifact_kind`,
|
||||
`file`, `media_type`, `schema_id`, `schema_name`, and `schema_version`. Their
|
||||
payloads are defined by the [Accepted Chunk Map contract](chunk-map.md) and
|
||||
[Published Evidence Context](evidence-context.md), respectively.
|
||||
|
||||
The lane path is derived from its lane ID. Characters outside letters, digits,
|
||||
periods, underscores, and hyphens become underscores; `..` sequences are
|
||||
neutralized; leading and trailing periods and underscores are removed. A lane
|
||||
that produces an empty name, or two lanes that produce the same path, makes
|
||||
output encoding fail.
|
||||
Files are pretty-printed JSON with a trailing newline. Lane payloads require
|
||||
`application/json`; the encoder validates the artifact boundary without
|
||||
wrapping payloads in reporting metadata. Report presentation format does not
|
||||
change artifact encoding.
|
||||
|
||||
## Lane Payloads
|
||||
|
||||
Each `lanes/<safe-lane-id>.json` file is the codec-owned normalized JSON for
|
||||
that lane. Consumers should use the index descriptor’s schema identity rather
|
||||
than infer a lane schema from its name. The current D&D payload contracts are
|
||||
[spells](dnd-spell-artifacts.md), [NPC registry](dnd-npc-registry-artifacts.md),
|
||||
[NPC occurrences](dnd-npc-occurrence-artifacts.md),
|
||||
[combat turns](dnd-combat-turn-artifacts.md),
|
||||
[item registry](dnd-item-registry-artifacts.md),
|
||||
[item occurrences](dnd-item-occurrence-artifacts.md),
|
||||
[scene descriptions](dnd-scene-description-artifacts.md),
|
||||
[enemy events](dnd-enemy-event-artifacts.md),
|
||||
[location registry](dnd-location-registry-artifacts.md), and
|
||||
[location occurrences](dnd-location-occurrence-artifacts.md).
|
||||
Each lane file is its codec-owned normalized JSON value. Select the contract
|
||||
from the configured lane/module, not an absent discovery descriptor. Existing
|
||||
D&D payload contracts remain unchanged:
|
||||
|
||||
## `manifest.json`
|
||||
- [Spells](dnd-spell-artifacts.md).
|
||||
- [NPC registry](dnd-npc-registry-artifacts.md) and
|
||||
[NPC occurrences](dnd-npc-occurrence-artifacts.md).
|
||||
- [Combat turns](dnd-combat-turn-artifacts.md).
|
||||
- [Item registry](dnd-item-registry-artifacts.md) and
|
||||
[item occurrences](dnd-item-occurrence-artifacts.md).
|
||||
- [Scene descriptions](dnd-scene-description-artifacts.md).
|
||||
- [Enemy events](dnd-enemy-event-artifacts.md).
|
||||
- [Location registry](dnd-location-registry-artifacts.md) and
|
||||
[location occurrences](dnd-location-occurrence-artifacts.md).
|
||||
|
||||
`manifest.json` is published provenance, not a copy of lane payloads or a
|
||||
checkpoint store. Fields without a value may be omitted. Its top-level fields
|
||||
group into the following externally observable summaries:
|
||||
Intrinsic fields, identities, source references, and schemas are preserved.
|
||||
Supported fallback meets intrinsic/downstream invariants without asserting
|
||||
ordinary semantic approval. Run qualification belongs in the native completion
|
||||
report, not new payload fields or sidecars.
|
||||
|
||||
| Group | Fields |
|
||||
| --- | --- |
|
||||
| Run identity and result | `run_id`, `pipeline_id`, `pipeline_digest`, `schema_version`, `validation_status`, `started_at`, `completed_at` |
|
||||
| Resolved components | `input_module`, `chunker`, `extractors`, `merger`, `normalizer`, `output_encoder`, `artifact_lanes`, `validator_chains`, `module_metadata` |
|
||||
| Source and references | `source_digests`, `references` |
|
||||
| Published result summaries | `normalized_outputs`, `rejected_outputs`, `validation_summaries` |
|
||||
| Execution summaries | `chunk_plan`, `checkpoint_decisions`, `llm_profiles`, `metadata` |
|
||||
## Auxiliary Artifacts
|
||||
|
||||
`references` records provenance such as the target, slot, origin, digest,
|
||||
media type, size, and generated-artifact identity. It does not contain
|
||||
reference content. `normalized_outputs` and `rejected_outputs` likewise
|
||||
summarize results without embedding lane payload bytes. A chunk-plan summary is
|
||||
provenance for the plan used by this run; cache records, debug artifacts, and
|
||||
other operational state are not published as bundle files.
|
||||
[Chunk map](chunk-map.md) and [evidence context](evidence-context.md) are
|
||||
pipeline-wide artifacts, not lanes. Their existing payload schemas and logical
|
||||
paths remain; no index descriptor is required.
|
||||
|
||||
Each `validation_summaries` entry is a bounded outcome for one producer result.
|
||||
It has required `status`, `producer_attempt_count`, and `terminal_action`;
|
||||
the stage and affected step, lane, module, or chunk identity are present when
|
||||
applicable. `status` is `complete`, `rejected`, or `incomplete`.
|
||||
`rejecting_validators`, `reason_codes`, and `incomplete_validators` preserve
|
||||
configured validator order and omit later duplicates. Entries contain no raw
|
||||
candidate response, correction guidance, validator diagnostic message, or
|
||||
artifact payload. The same shape may appear as `validation` on an affected
|
||||
rejection entry.
|
||||
Enabled, available auxiliaries count as deliverables, including when built
|
||||
from reused results or supported fallback. Disabled/unavailable auxiliaries are
|
||||
not published. Evidence can cover the complete source; authoritative citations
|
||||
remain in normalized lane payloads.
|
||||
|
||||
When present, `metadata.session_id` is the effective non-secret routing
|
||||
correlation identifier used for the run. It can be visible to providers and is
|
||||
not a substitute for a cache or checkpoint identity. Its generation and
|
||||
override behavior are defined by the [CLI reference](../cli.md#run).
|
||||
## Consumer Use And Compatibility
|
||||
|
||||
Each `llm_profiles` entry identifies effective, non-secret LLM execution
|
||||
provenance:
|
||||
Check process status before decoding a successful native report. Use its absolute
|
||||
run directory, then the configured lane and selected auxiliary paths. Decode
|
||||
each file against its domain contract. The [subprocess guide](../consumers/subprocess.md)
|
||||
and [complete D&D workflow](../consumers/dnd-pipeline.md) explain caller tasks.
|
||||
|
||||
| Field | Required | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `id` | Yes | Selected PromptKit profile identifier. |
|
||||
| `provider` | No | Notarius adapter provider identifier. |
|
||||
| `model` | No | Effective provider model identifier. |
|
||||
| `backend_id` | No | Effective PromptKit backend registration identifier. Endpoint-only profiles omit it. |
|
||||
| `reasoning_effort` | No | Effective opaque provider reasoning setting. An empty or explicitly cleared setting is omitted. |
|
||||
|
||||
These values describe observed execution; they are not a backend-registration
|
||||
interface. Entries that differ by backend or effective reasoning remain
|
||||
distinct even when their profile, provider, and model are otherwise equal.
|
||||
|
||||
## Rejections, Warnings, And Diagnostics
|
||||
|
||||
`rejected.json` is always an object with a `rejected` array. Each entry has
|
||||
required `stage` and `message`; `step_id`, `lane_id`, `module_key`, `chunk_id`,
|
||||
`chunk_index`, `validator_name`, `reason_code`, `attempt_count`, and
|
||||
`diagnostic_artifact_path` are present only when applicable. An entry may also
|
||||
contain the bounded `validation` summary described above; the existing singular
|
||||
validator and reason fields remain the first configured rejection for
|
||||
compatibility.
|
||||
|
||||
`warnings.json` is always the `notarius.warnings.v2` envelope:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": "notarius.warnings.v2",
|
||||
"group_count": 0,
|
||||
"occurrence_count": 0,
|
||||
"groups": []
|
||||
}
|
||||
```
|
||||
|
||||
It contains only process warnings. `group_count` is exact, and
|
||||
`occurrence_count` is the exact sum of its group occurrence counts.
|
||||
|
||||
`diagnostics.json` is always the `notarius.diagnostics.v1` envelope:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": "notarius.diagnostics.v1",
|
||||
"group_count": 0,
|
||||
"occurrence_count": 0,
|
||||
"truncated": false,
|
||||
"unrepresented_occurrence_count": 0,
|
||||
"groups": []
|
||||
}
|
||||
```
|
||||
|
||||
It contains only advisory and observation groups. `group_count` counts groups
|
||||
represented in `groups`; `occurrence_count` includes both represented and
|
||||
unrepresented occurrences. When `truncated` is true,
|
||||
`unrepresented_occurrence_count` is the exact number omitted from group
|
||||
representation.
|
||||
|
||||
Each group has `disposition`, `category`, `reason_code`, framework-owned
|
||||
`origin`, exact `occurrence_count`, bounded `samples`, and
|
||||
`omitted_sample_count`. Samples carry safe `scope` and `message`, plus a chunk
|
||||
ID and zero-based chunk index when applicable. A group retains at most three
|
||||
distinct samples. The framework fails rather than truncating actionable
|
||||
warnings beyond 128 groups; it represents at most 256 advisory/observation
|
||||
groups and records further occurrences through the diagnostic truncation
|
||||
fields above.
|
||||
|
||||
## Compatibility
|
||||
|
||||
The index is the authoritative map from a logical lane to its published
|
||||
payload. Consumers must tolerate omitted optional manifest and descriptor
|
||||
fields, and should rely on the linked artifact contract for each lane’s JSON
|
||||
shape. This contract describes the published logical bundle only; it does not
|
||||
promise a filesystem layout or expose internal state formats.
|
||||
Removing reporting/discovery files is intentional, not a domain schema or
|
||||
filename-mapping change. [Migration guidance](../consumers/native-migration.md)
|
||||
replaces receipt/index traversal. Debug and cache are not public output contracts.
|
||||
Post-publication failure can leave valid files; failed publication can leave a
|
||||
partial directory. Neither supplies a successful completion report.
|
||||
|
||||
@@ -1,93 +0,0 @@
|
||||
# Run Result Receipt
|
||||
|
||||
`notarius run --json` writes this receipt to standard output when a run
|
||||
completes successfully. It lets a subprocess caller discover the physical root
|
||||
of the published output bundle without parsing interactive command output.
|
||||
Command syntax, streams, and exit statuses are defined in the
|
||||
[CLI reference](../cli.md); logical files within the bundle are defined in the
|
||||
[Published JSON Output contract](json-output.md).
|
||||
|
||||
## Schema
|
||||
|
||||
The current schema version is `notarius.run-result.v2`.
|
||||
|
||||
| Field | Required | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `schema_version` | Yes | Exactly `notarius.run-result.v2`. |
|
||||
| `run_id` | Yes | The finalized Notarius run identifier. |
|
||||
| `pipeline_id` | Yes | The effective pipeline identifier. |
|
||||
| `output_directory` | Yes | Absolute path to the published, run-specific output bundle. |
|
||||
| `index_file` | For the production JSON output | Logical path `index.json`; omitted for other output modules. |
|
||||
| `normalized_output_count` | Yes | Number of final normalized outputs. |
|
||||
| `rejected_output_count` | Yes | Number of recorded rejected outputs. |
|
||||
| `warning_group_count` | Yes | Exact number of actionable warning groups. |
|
||||
| `warning_occurrence_count` | Yes | Exact occurrences represented by actionable warning groups. |
|
||||
| `diagnostic_group_count` | Yes | Number of represented advisory and observation groups. |
|
||||
| `diagnostic_occurrence_count` | Yes | Advisory and observation occurrences, including unrepresented occurrences. |
|
||||
| `diagnostics_truncated` | Yes | Whether advisory/observation group representation was truncated. |
|
||||
| `validation_status` | Yes | The final run manifest validation status. |
|
||||
| `validation_summaries` | No | Bounded per-producer validation outcomes; present when producer work ran. |
|
||||
| `debug_directory` | No | Absolute path to the run-specific debug bundle when requested debug capture completed. |
|
||||
|
||||
For the production `json` output module, `index_file` is present only when the
|
||||
completed run returned exactly one logical output file named `index.json`.
|
||||
For another output module, its absence does not indicate a failed run.
|
||||
`validation_status` is `approved`, `rejected`, or `incomplete`; `incomplete`
|
||||
means one or more otherwise accepted results advanced under validator-failure
|
||||
`warn_continue`.
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": "notarius.run-result.v2",
|
||||
"run_id": "run-1770000000000000000-0123456789abcdef0123456789abcdef",
|
||||
"pipeline_id": "dnd-session",
|
||||
"output_directory": "/work/results/run-1770000000000000000-0123456789abcdef0123456789abcdef",
|
||||
"index_file": "index.json",
|
||||
"normalized_output_count": 6,
|
||||
"rejected_output_count": 2,
|
||||
"warning_group_count": 1,
|
||||
"warning_occurrence_count": 2,
|
||||
"diagnostic_group_count": 3,
|
||||
"diagnostic_occurrence_count": 5,
|
||||
"diagnostics_truncated": false,
|
||||
"validation_status": "incomplete",
|
||||
"validation_summaries": [
|
||||
{
|
||||
"stage": "extract",
|
||||
"lane_id": "spells",
|
||||
"status": "incomplete",
|
||||
"incomplete_validators": ["dnd/spells/source_refs"],
|
||||
"producer_attempt_count": 1,
|
||||
"terminal_action": "warn_continue"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Paths And Bundle Discovery
|
||||
|
||||
`output_directory` and `debug_directory`, when present, are lexical absolute
|
||||
paths. They identify the paths used by Notarius and do not resolve symlinks.
|
||||
`output_directory` is the run-specific bundle, not the configured output root.
|
||||
|
||||
The receipt is a summary and discovery document. Its optional validation
|
||||
summaries contain only stable status, identity, validator names, reason codes,
|
||||
attempt counts, and terminal actions. It does not contain lane descriptors,
|
||||
payloads, manifest payloads, rejection messages, warnings, raw model responses,
|
||||
correction guidance, or file contents.
|
||||
For the production JSON output, resolve `index_file` beneath
|
||||
`output_directory`, reject path escapes, and use the
|
||||
[Published JSON Output contract](json-output.md) to discover logical files and
|
||||
lane payloads.
|
||||
|
||||
## Delivery And Compatibility
|
||||
|
||||
Notarius writes the receipt only after the output bundle has been published and
|
||||
any requested debug terminal reporting has completed. Standard output is not
|
||||
transactional: a result-write failure returns a nonzero status and can leave
|
||||
partial bytes. Consumers must ignore standard output unless the process exits
|
||||
with status 0.
|
||||
|
||||
Future versions may add optional fields to this schema. Consumers must tolerate
|
||||
unknown fields. An incompatible field or semantic change requires a new
|
||||
`schema_version` value.
|
||||
@@ -152,7 +152,7 @@ reporting remains successfully terminalized. The CLI reports a bounded
|
||||
command-owned error and does not repeat terminal reporting. The receipt remains
|
||||
a CLI reporting concern rather than a framework or output-module responsibility;
|
||||
its public contract is the
|
||||
[run-result receipt](../integrations/run-result.md).
|
||||
[native completion report](../cli.md#completion-report).
|
||||
|
||||
## Failure Mapping And Terminal Reporting
|
||||
|
||||
|
||||
@@ -1,475 +1,292 @@
|
||||
# Operations
|
||||
|
||||
This is the canonical guide for operating Notarius runtime state. The
|
||||
[CLI reference](cli.md) owns command syntax and exit statuses, while
|
||||
[Configuration](config.md) owns fields, defaults, and precedence. Maintainers
|
||||
who need implementation mechanics should read [Run State Internals](internal/state.md).
|
||||
This guide owns runtime filesystem layout, publication, cache recovery, capture,
|
||||
provisioning, and retention. [CLI Reference](cli.md) owns invocation and reports;
|
||||
[Configuration](config.md) owns settings/defaults and source selection.
|
||||
The [native migration guide](consumers/native-migration.md) maps retired interfaces
|
||||
to the current workflow.
|
||||
|
||||
## Source Deployment
|
||||
|
||||
Linux is the supported deployment platform. Install a pinned source release
|
||||
with the Go version declared in `go.mod` (currently Go 1.27.0):
|
||||
Linux is supported; macOS is best-effort for development and Windows is
|
||||
unsupported. Install a pinned source tag with the Go version required by
|
||||
`go.mod`:
|
||||
|
||||
~~~sh
|
||||
GOWORK=off go install \
|
||||
gitea.maximumdirect.net/eric/notarius/cmd/notarius@vMAJOR.MINOR.PATCH
|
||||
notarius version
|
||||
~~~
|
||||
|
||||
Pin the exact tag in deployment automation rather than following a branch.
|
||||
Use [`notarius --version`](cli.md#command-summary) as a diagnostic after
|
||||
installation; its syntax and semantics are owned by the [CLI reference](cli.md).
|
||||
The maintainer publication process, including tag guards and verification,
|
||||
belongs to [Source Releases](release.md). macOS builds are best-effort for
|
||||
development, and Windows is unsupported.
|
||||
Dependency acquisition/private-module access is separate from offline validation.
|
||||
[Source Releases](release.md) owns maintainer publication commands. This workflow
|
||||
does not prepare or authorize a release.
|
||||
|
||||
## State Surfaces
|
||||
|
||||
Each run can use independent roots with different retention and access-control
|
||||
needs.
|
||||
Output, cache, and debug have independent roots and lifecycles.
|
||||
|
||||
| Surface | Purpose | Created when | Retention |
|
||||
| --- | --- | --- | --- |
|
||||
| Output | Durable user-facing result bundle | A pipeline completes and returns logical output files | Keep until consumers no longer need it. |
|
||||
| Chunk-plan cache | Reconstructible source-addressed plan | The configured cache mode permits cache I/O | Keep while reuse is useful. |
|
||||
| Checkpoint cache | Reconstructible execution and recovery state | Checkpoint recording is enabled | Keep only while recovery or reuse is useful. |
|
||||
| Debug bundle | Explicit diagnostic record | A run requests debug collection | Keep only under an intentional sensitive-data retention policy. |
|
||||
| Surface | Purpose | Allocation / reuse |
|
||||
| --- | --- | --- |
|
||||
| Output | Durable domain artifacts | A successful runner result publishes into a new run directory. |
|
||||
| Accepted-result cache | Reconstructible artifacts and LLM chunk plans | Eligible model-backed work reads/writes according to its resolved mode. |
|
||||
| Debug | Explicit run/operation evidence | Explicit capture selection with a nonempty effective destination. |
|
||||
|
||||
Output, cache, and debug roots are never merged or cleaned automatically. Use
|
||||
separate locations and permissions for operators or services that must not
|
||||
share application data.
|
||||
Directories do not imply one another's enablement or permissions. Configure
|
||||
separate roots when access or retention needs differ. Notarius performs no
|
||||
automatic output/debug deletion or cache collection.
|
||||
|
||||
## Roots And Permissions
|
||||
|
||||
The configured output and debug directories are exact roots. An empty cache
|
||||
directory selects a per-user root:
|
||||
Publication and capture use run-specific directories:
|
||||
|
||||
~~~
|
||||
<os.UserCacheDir>/notarius/chunk-plans
|
||||
<os.UserCacheDir>/notarius/checkpoints
|
||||
~~~text
|
||||
<output-root>/<run-id>/
|
||||
<debug-root>/<run-id>/
|
||||
<debug-root>/latest -> <run-id>
|
||||
~~~
|
||||
|
||||
The field definitions and configuration examples are in [Configuration](config.md).
|
||||
On supported Unix systems, output directories and files are created with
|
||||
requested modes **0755** and **0644**; debug directories and files use
|
||||
**0700** and **0600**. The operating system's umask may impose stricter output
|
||||
or debug modes. Chunk-plan and checkpoint caches instead request maximum modes
|
||||
of **0777** for new directories and **0666** for new files, including atomic
|
||||
temporary files. The kernel applies the service's umask and applicable default
|
||||
ACLs when those entries are created.
|
||||
The unified cache uses its selected exact root independently of run IDs:
|
||||
|
||||
## Cache Provisioning
|
||||
~~~text
|
||||
<cache-directory>/<opaque-message-content-hash>.json
|
||||
~~~
|
||||
|
||||
Cache locations select where Notarius writes, not who can access them. A
|
||||
private service can use a restrictive umask:
|
||||
Omitted/empty cache selection uses the per-user location defined in
|
||||
[Configuration](config.md#file-shape-and-defaults), normally under Linux's
|
||||
`XDG_CACHE_HOME` or `~/.cache`. `/var/cache/notarius` is the convention for
|
||||
an explicitly configured system-wide installation, not an automatically selected
|
||||
or provisioned root. Output/debug defaults, empty values, and path origins also
|
||||
belong in that reference.
|
||||
|
||||
Output creation requests directory/file modes `0755`/`0644`; debug requests
|
||||
`0700`/`0600`. The operating system may restrict effective access. Cache creation
|
||||
requests conventional maxima `0777`/`0666`, including atomic temporary files,
|
||||
so deployment umask, setgid inheritance, and default ACLs select effective access.
|
||||
Notarius does not converge existing ownership or modes.
|
||||
|
||||
### Cache Provisioning
|
||||
|
||||
A private deployment can use a restrictive process umask. A trusted-group
|
||||
deployment may provision a setgid root or default ACL. These are deployment
|
||||
choices, not requirements imposed on every operator:
|
||||
|
||||
~~~sh
|
||||
umask 0077
|
||||
notarius run example --config /etc/notarius/config.yml --input /srv/input.txt
|
||||
~~~
|
||||
|
||||
A trusted-group deployment can provision a setgid cache root and use a
|
||||
group-oriented umask (or an equivalent default ACL):
|
||||
|
||||
~~~sh
|
||||
install -d -m 2770 -g notarius-operators /srv/notarius/chunk-plans /srv/notarius/checkpoints
|
||||
# Example provisioning by an authorized administrator.
|
||||
install -d -m 2770 -g notarius-operators /var/cache/notarius
|
||||
umask 0007
|
||||
notarius run example --config /etc/notarius/config.yml --input /srv/input.txt
|
||||
~~~
|
||||
|
||||
Setgid and default ACLs control inherited group ownership and access policy;
|
||||
Notarius does not copy a parent directory's literal mode bits. Cache state is
|
||||
reconstructible but can contain source-derived data, so operators own its
|
||||
access and retention policy. Existing cache inodes retain their prior modes
|
||||
after an upgrade. Recreate a cache beneath correctly provisioned parents, or
|
||||
stop writers and perform a deliberate administrative migration when a
|
||||
whole-tree change is required.
|
||||
Configure the cache root to use that directory. Setgid/default ACLs govern
|
||||
inheritance; Notarius does not copy parent literal mode bits, manage accounts/
|
||||
groups, or chmod/chown old entries. If existing access is unsuitable, operators
|
||||
can recreate reconstructible state beneath provisioned parents or deliberately
|
||||
perform an administrative migration.
|
||||
|
||||
Cache publication remains atomic and stored records remain validated. Those
|
||||
guarantees do not add cross-process locking or transactional coordination for
|
||||
simultaneous writers to one checkpoint identity.
|
||||
|
||||
Output bundles can also contain source content when JSON output enables
|
||||
evidence publication. Apply an appropriate umask and output-root access policy
|
||||
before enabling that option; the requested output modes alone may not be
|
||||
suitable for transcript-bearing bundles.
|
||||
Cache, outputs, and evidence excerpts may contain source-derived data. Select
|
||||
access and retention accordingly; requested modes alone are not a sharing policy.
|
||||
Existing confined-path, symlink, atomic-write, and validation boundaries do not
|
||||
promise cross-process transactions or power-loss durability.
|
||||
|
||||
## PromptKit Profile Deployment
|
||||
|
||||
Profile deployment has four distinct layers:
|
||||
One selected application load includes bundled provider resources, embedded
|
||||
Notarius resources, main-relative operator companions, imports, and an optional
|
||||
overlay. Resource replacement/inheritance and source origins are defined in
|
||||
[Configuration](config.md#promptkit-profiles).
|
||||
|
||||
| Layer | Owner | Operational role |
|
||||
| --- | --- | --- |
|
||||
| Prompts and schemas | Notarius module families, with optional operator prompt overlays | Embedded request and structured-output definitions; operator configuration may recursively overlay prompt fields. Schemas remain application assets. |
|
||||
| Fallback profiles | Notarius module families | Embedded application defaults, including D&D's `dnd-extraction` profile. |
|
||||
| Built-in profiles | PromptKit | Upstream catalog entries available when no higher-precedence source defines an ID. |
|
||||
| Operator configuration | Deployment filesystem | Native profile/backend definitions and prompt overlays selected by `promptkit.profile_file` or `promptkit.profile_dir`. |
|
||||
Preserve the [complete example's](../examples/dnd-complete/config.yml)
|
||||
declaration-relative layout or deliberately update paths. Its
|
||||
[companion](../examples/dnd-complete/prompts.yml) adds native profile/backend
|
||||
variants; minimal needs no operator companion. Select qualified collection IDs.
|
||||
Supply credential values through the intended environment, not files, arguments,
|
||||
session IDs, or logs.
|
||||
|
||||
The maintained D&D pipeline uses the workload ID `dnd-extraction`. The
|
||||
embedded fallback makes that ID usable without an operator file. Production,
|
||||
development, and local deployments can each overlay settings for the same ID,
|
||||
retaining inherited values they do not change. Notarius's registered fallback
|
||||
configuration precedes the selected operator source, and groups compose by
|
||||
native recursive overlays; a configured Notarius local backend is applied
|
||||
after the operator source. The configuration fields and source precedence are
|
||||
owned by [Configuration](config.md#promptkit-profiles).
|
||||
|
||||
Use a profile source owned by the service account, keep it readable only by
|
||||
the intended operator, and supply provider credentials through the service
|
||||
environment—not in the Notarius configuration or profile YAML. The maintained
|
||||
[LLM profiles](../examples/llm-profiles/dnd-extraction.yml) are secret-free
|
||||
and can be copied as a format starting point. Validate a deployment without a
|
||||
provider call or credentials:
|
||||
Review the deployment before model work:
|
||||
|
||||
~~~sh
|
||||
notarius config validate --config /etc/notarius/config.yml --pipeline dnd-session
|
||||
notarius config validate --config /etc/notarius/config.yml
|
||||
notarius pipeline validate dnd-session --config /etc/notarius/config.yml
|
||||
notarius config show --scope promptkit --config /etc/notarius/config.yml
|
||||
notarius promptkit show profile notarius/dnd-extraction \
|
||||
--config /etc/notarius/config.yml --format json
|
||||
~~~
|
||||
|
||||
An unset optional `api_key_env` reaches the provider without authorization and
|
||||
may receive a 401 or 403 response.
|
||||
Offline preflight/inspection needs no credential values or provider calls.
|
||||
Use sources/explain reports for provenance and prepared preview for rendered
|
||||
request details. Profile inspection describes stored fields, not a completed
|
||||
execution target. New invocations observe source edits; captured engines do not
|
||||
reread configuration/message files. Schema/input reads occur during preparation.
|
||||
|
||||
Configured PromptKit profile paths resolve from the file that supplies their
|
||||
surviving declaration. The complete example's
|
||||
`./llm-profiles/dnd-extraction.yml` path is therefore relative to that example.
|
||||
Use absolute paths such as `/etc/notarius/profiles/dnd-extraction.yml` for
|
||||
services and containers when that is clearer for the deployment.
|
||||
|
||||
### Inspect Before Deployment
|
||||
|
||||
Use the three configuration reports together when reviewing an operator
|
||||
source:
|
||||
|
||||
~~~sh
|
||||
notarius config effective --scope promptkit --config /etc/notarius/config.yml
|
||||
notarius config explain --scope promptkit --config /etc/notarius/config.yml \
|
||||
/profiles/dnd-extraction/model
|
||||
notarius config profile --config /etc/notarius/config.yml dnd-extraction
|
||||
~~~
|
||||
|
||||
Each report command constructs an inspection engine from the selected
|
||||
configuration and registered application assets. Construction validates and
|
||||
captures the effective definitions; after editing a source, run the command
|
||||
again to inspect a newly constructed engine. These commands do not resolve
|
||||
credential values, call a provider, or prepare a request. `config profile`
|
||||
reports the named profile's resolved target and effective credential metadata,
|
||||
not whether a credential is available or which profile a particular prompt or
|
||||
run will select.
|
||||
|
||||
Effective and explanation reports can include ordinary inline prompt and
|
||||
provider-parameter data. Credential-shaped provider parameter keys are
|
||||
redacted, but the rest of the content is not treated as secret-free. Reports
|
||||
are written only when requested; Notarius does not automatically persist them
|
||||
in run output or debug bundles. Operators own their destination, access, and
|
||||
retention when redirecting or storing reports.
|
||||
|
||||
Configuration reports are distinct from a prepared execution or a requested
|
||||
debug trace. They describe captured definitions and source history, not
|
||||
rendered request messages, runtime credential values, or an individual
|
||||
provider interaction. The [CLI reference](cli.md#config-effective) documents
|
||||
the report syntax and output contracts.
|
||||
Reports can contain ordinary prompt/provider data. Secret-aware inspection and
|
||||
native credential omissions do not make arbitrary content safe to share.
|
||||
Operators select destinations, access, and retention when redirecting reports.
|
||||
|
||||
## Run Lifecycle
|
||||
|
||||
Use the [run command](cli.md#run) to start a pipeline. A valid invocation loads
|
||||
and resolves configuration before module preparation and source parsing. It
|
||||
then performs any permitted cache lookup, executes the pipeline, and publishes
|
||||
logical output files only after a successful runner result.
|
||||
A run loads configuration and obtains its identity. Requested capture allocates
|
||||
before engine construction/preflight. The run resolves/preflights, materializes
|
||||
references, prepares modules, parses source, performs chunk/lane work with
|
||||
eligible reuse, encodes domain files, and publishes into a new run directory.
|
||||
Requested capture finalizes before successful reporting.
|
||||
|
||||
On success, the command reports the output bundle path. A run with actionable
|
||||
process warnings still succeeds and reports warning-group and occurrence counts
|
||||
on standard error; advisory and observation findings do not produce a warning
|
||||
line. Errors and
|
||||
their exit classes are defined in the [CLI reference](cli.md#output-streams-and-exit-statuses).
|
||||
Argument/configuration failures may have no capture. Later failures retain
|
||||
available evidence. Publication, required capture, and presentation are separate
|
||||
boundaries: later failure does not undo completed publication.
|
||||
[CLI Reference](cli.md#completion-report) owns the completion value and outcomes.
|
||||
|
||||
## Validation Retries And Terminal Outcomes
|
||||
### Acceptance And Fallback
|
||||
|
||||
When debug recording is enabled, per-call response artifacts include the
|
||||
terminal execution summary for retry, repair, timing, and presence-aware usage
|
||||
diagnosis. Normal output, receipts, and warnings do not aggregate or publish
|
||||
this diagnostic metadata.
|
||||
PromptKit owns transient retries and structural repair. Application semantic
|
||||
continuation carries a growing conversation and assesses each usable candidate
|
||||
through the complete chain, including the final allowed response. No outer
|
||||
producer/validator retry loop extends native budgets.
|
||||
|
||||
Each producer binding has one outer **retries** budget. It covers complete
|
||||
producer attempts for operational failures, invalid structured output,
|
||||
normalizer fallback retry, and semantic correction. It is independent from
|
||||
PromptKit's structural-repair calls inside one completion, its configured
|
||||
`promptkit.transient_retries` built-in HTTP attempts, and an LLM-backed
|
||||
validator's own retry budget. The transient setting defaults to two additional
|
||||
attempts for eligible provider statuses; set it explicitly to zero when an
|
||||
operator needs one HTTP attempt per generation. A semantic correction rebuilds
|
||||
the ordinary producer request and supplies only the latest rejected model
|
||||
response plus aggregated validator guidance; it is not a conversation replay.
|
||||
Explicit permissive handling may publish a supported qualified fallback, not
|
||||
ordinary semantic acceptance. Construction still checks intrinsic artifact,
|
||||
source, and downstream invariants. Failure to construct fallback fails normally;
|
||||
cancellation never initiates it. D&D scene planning can fall back to one
|
||||
whole-source chunk, preserving coverage without promising bounded downstream
|
||||
request size.
|
||||
|
||||
These budgets multiply only by their declared scopes. With two outer producer
|
||||
retries, two transient retries, and one structural repair, a single producer
|
||||
can issue up to 18 HTTP attempts: three producer attempts, each with an initial
|
||||
generation and one repair generation, each allowing three HTTP attempts. This
|
||||
is an illustration, not a total-deadline or exactly-once guarantee. Every
|
||||
built-in HTTP attempt has its own generation timeout; caller cancellation
|
||||
remains the overall authority and interrupts Retry-After waiting. The shared
|
||||
Notarius scheduler keeps its permit through those waits, so retry budgets can
|
||||
increase latency and provider cost. Observed usage can remain incomplete when
|
||||
providers omit accounting fields.
|
||||
Fallback and actually affected consumers skip cache lookup/storage; unrelated
|
||||
accepted predecessors remain reusable. The report discloses emitting operations
|
||||
even when warnings are hidden; affected files and validator calls are not extra
|
||||
fallbacks. A hit does not establish fresh acceptance under changed policy.
|
||||
Consumers needing new inference/assessment deliberately choose bypass/refresh.
|
||||
|
||||
After the applicable budgets are exhausted, the resolved
|
||||
[`validation_policy`](config.md#pipelines) determines the result. Structural
|
||||
failure and semantic rejection normally fail the run; an explicit
|
||||
`reject_output` records a rejection and allows unrelated work to finish. A
|
||||
validator execution failure normally uses `warn_continue`, which keeps an
|
||||
otherwise accepted result in the current run with `incomplete` validation
|
||||
provenance. It emits one bounded warning for every validator whose execution
|
||||
budget was exhausted. A corrected result that later passes validation does not
|
||||
retain diagnostics from abandoned attempts.
|
||||
|
||||
Treat a successful process exit as a completed run, not as proof that every
|
||||
candidate was fully validated. Inspect the receipt's `validation_status`,
|
||||
`validation_summaries`, rejection count, and warning group and occurrence
|
||||
counts when an orchestrator requires complete validation. The durable fields
|
||||
and their meanings are owned by the
|
||||
[run-result receipt](integrations/run-result.md) and
|
||||
[published JSON output contract](integrations/json-output.md).
|
||||
Process warnings retain native visibility; quality advisories are not promoted
|
||||
to warnings. [Configuration](config.md#producer-controls) owns selected modes,
|
||||
turn allowances, defaults, and precedence.
|
||||
|
||||
## Output Bundles
|
||||
|
||||
Each successful run receives a generated safe run identifier and writes beneath:
|
||||
The encoder defines [logical artifact files](integrations/json-output.md);
|
||||
the CLI places them beneath the run directory. It checks paths before allocation
|
||||
and refuses an existing destination without changing it. Individual writes are
|
||||
atomic, not an all-files transaction. A later write failure can leave a partial
|
||||
new directory for inspection.
|
||||
|
||||
~~~
|
||||
<output-root>/<run-id>/
|
||||
~~~
|
||||
Artifacts are deliverables, not an index, manifest, persisted receipt, or reporting
|
||||
sidecar. Read configured lane and auxiliary paths beneath the successful report's
|
||||
directory; do not scan for the newest folder.
|
||||
|
||||
The [JSON output contract](integrations/json-output.md) owns the logical files
|
||||
and their schemas. Before creating the run directory, Notarius validates every
|
||||
logical output path. It refuses an existing run directory without changing it.
|
||||
Files are written atomically; if a later write fails, the newly created partial
|
||||
run directory remains for inspection and is never removed automatically.
|
||||
Capture/report failure can occur after publication. The command returns nonzero
|
||||
with available locations; valid artifacts are not removed or regenerated.
|
||||
Failed stdout can be partial. Do not treat it as a success report or assume a
|
||||
failed publication directory is complete.
|
||||
|
||||
Treat an output bundle as durable user data. Do not use cache-cleanup policy to
|
||||
remove it. An optional accepted chunk map is also durable output and can carry
|
||||
source- or model-derived annotations; its content and compatibility contract
|
||||
are defined in [Accepted Chunk Map](integrations/chunk-map.md). An optional
|
||||
[evidence context](integrations/evidence-context.md) contains source-unit text
|
||||
and metadata. It is not a cache or debug artifact: retain it with the output
|
||||
bundle only for as long as consumers need it, and apply source-content access
|
||||
controls to the entire bundle. Its selected source-unit excerpt may include
|
||||
every source unit once when coverage is broad or its configured window is
|
||||
large, so do not assume a byte or token reduction or reduced sensitivity.
|
||||
Output is durable user data. Optional [chunk maps](integrations/chunk-map.md)
|
||||
and [evidence context](integrations/evidence-context.md) are output, not cache/
|
||||
debug. Evidence can cover every source unit and is not promised smaller or less
|
||||
sensitive; retain/share it according to consumer needs.
|
||||
|
||||
## Chunk-Plan Cache
|
||||
## Accepted-Result Cache And Recovery
|
||||
|
||||
Chunk plans live beneath the selected chunk-plan root:
|
||||
The cache stores final accepted typed artifacts, including applied normalization,
|
||||
and accepted LLM chunk plans, not native responses, conversations, or execution
|
||||
ledgers. Cheap deterministic work runs normally.
|
||||
|
||||
~~~
|
||||
<chunk-plan-root>/<source-sha256-hex>/plan.json
|
||||
~~~
|
||||
The key is the initial native prepared `MessageContentHash`, unchanged through
|
||||
correction. Matching ordered roles/text/boundaries can reuse final acceptance.
|
||||
Model, session, schema metadata, policy, and implementation add no dimensions;
|
||||
text actually rendered into messages participates. Use refresh/bypass when a
|
||||
model or policy experiment needs new inference/assessment.
|
||||
|
||||
One validated canonical plan is active for each source digest. The plan stores
|
||||
boundaries and provenance, not a second copy of the entire source. This
|
||||
source-addressed policy is recorded in [ADR-0005](adr/0005-cache-canonical-chunk-plans-by-source.md).
|
||||
Hits perform ordinary codec/current-source usability checks without semantic
|
||||
reassessment, provider calls, or historical usage claims. Plans rematerialize
|
||||
against the current source. Source identity alone no longer selects a canonical
|
||||
partition; changed initial content can select different boundaries.
|
||||
|
||||
The configured cache mode controls one invocation:
|
||||
|
||||
- **auto** looks for a valid active plan. Missing or invalid state causes a new
|
||||
plan to be generated; an accepted new plan is atomically published.
|
||||
- **refresh** skips lookup, generates a plan with the configured chunker, and
|
||||
atomically replaces the active plan after it is accepted.
|
||||
- **bypass** performs no chunk-plan cache I/O. It does not resolve or create a
|
||||
chunk-plan root.
|
||||
|
||||
A reused plan is still materialized and validated against the current source.
|
||||
If a prior plan no longer gives acceptable results, use a refresh run rather
|
||||
than editing cache files. Deleting a plan is recoverable but can repeat costly
|
||||
chunking work. A plan accepted only under incomplete validation is not
|
||||
published, and a rejected cache hit falls through to ordinary generation rather
|
||||
than becoming a correction candidate.
|
||||
|
||||
## Checkpoint Recording, Resume, And Recompute
|
||||
|
||||
Checkpoint recording is an explicit configuration choice and is disabled by
|
||||
default. When enabled, each run records stage transitions and the state needed
|
||||
for compatible recovery. A run records checkpoints even when it does not ask
|
||||
to reuse them. Checkpoint payloads can contain source-derived and intermediate
|
||||
application data, so treat the entire root as sensitive.
|
||||
|
||||
Checkpoint loading is separate: [**--resume**](cli.md#run) asks a run to reuse
|
||||
compatible recorded work. A resume request fails when checkpoint recording is
|
||||
disabled. Without **--resume**, a recording-enabled run executes normally and
|
||||
does not load checkpoint state. Compatibility includes the resolved pipeline,
|
||||
input, selected lanes, runtime overrides, reference provenance, LLM-profile
|
||||
provenance, the effective PromptKit profile-source fingerprint, and
|
||||
prepared-component fingerprints. When a local PromptKit backend is configured,
|
||||
compatibility also includes a non-secret fingerprint of its endpoint. Changing
|
||||
profile content or the local endpoint causes a cold miss; changing only the
|
||||
local concurrency limit does not. A changed identity produces a cold miss;
|
||||
Notarius does not migrate, rewrite, or delete older checkpoint directories.
|
||||
Reasoning-effort inheritance, replacement, and explicit clearing are distinct
|
||||
runtime identities, so checkpoints created under one state are not reused by
|
||||
either of the others.
|
||||
|
||||
Only accepted, completely validated chunk, extract, merge, and normalize
|
||||
results are checkpointed for reuse. Rejected, structurally invalid, and
|
||||
validation-incomplete producer results remain non-reusable, even when a
|
||||
`warn_continue` result advanced during its original run. A resumed invocation
|
||||
therefore reruns that producer rather than treating degraded state as accepted.
|
||||
|
||||
Checkpoint state is confined below an identity-specific path:
|
||||
|
||||
~~~
|
||||
<checkpoint-root>/<pipeline-id>/<input-key>-<source-or-input-digest-prefix>/<pipeline-digest-prefix>/<identity-digest-prefix>/
|
||||
~~~
|
||||
|
||||
### Selective Recompute
|
||||
|
||||
[**--recompute-step**](cli.md#run) requires both **--resume** and enabled
|
||||
checkpoint recording. It forces the selected ordered step and every lane that
|
||||
depends on it through generated artifact references. Unrelated lanes remain
|
||||
eligible for reuse.
|
||||
|
||||
For an earlier producer required by a forced consumer, Notarius requires a
|
||||
compatible accepted normalized artifact. It validates that artifact before
|
||||
hydrating it and does not silently rerun the producer. If that state is
|
||||
missing, rejected, corrupt, non-canonical, or incompatible, the run stops
|
||||
before its dependent starts. Rerun the required producer deliberately instead
|
||||
of copying or editing checkpoint files.
|
||||
|
||||
## Checkpoint Decisions And Recovery
|
||||
|
||||
Checkpoint events classify work as **executed**, **reused**,
|
||||
**forced_recompute**, or **dependency_invalidated**. Their stable reason codes
|
||||
are written to run diagnostics and provenance. Use the code, not a copied
|
||||
error message, to decide what to repair.
|
||||
|
||||
| Reason code | Recovery meaning |
|
||||
| Mode | Recovery behavior |
|
||||
| --- | --- |
|
||||
| **loading_disabled** | This invocation did not permit checkpoint loading. |
|
||||
| **checkpoint_missing**, **checkpoint_path_invalid**, **checkpoint_read_failed**, **checkpoint_decode_failed** | The stored checkpoint could not be located or read safely; normal resume work can execute again. |
|
||||
| **workspace_schema_incompatible**, **identity_mismatch**, **stage_mismatch**, **step_mismatch**, **lane_mismatch**, **module_mismatch** | Stored state belongs to a different compatible scope or identity; allow a fresh run to create new state. |
|
||||
| **status_not_reusable** | The recorded operation did not end in reusable state. |
|
||||
| **dependency_mismatch** | A dependency changed; dependent work is invalidated rather than reused. |
|
||||
| **artifact_payload_invalid**, **artifact_digest_mismatch**, **artifact_codec_incompatible**, **artifact_not_canonical** | A stored artifact cannot safely be hydrated; rerun the producer instead of modifying the cache. |
|
||||
| **checkpoint_reused** | A normal checkpoint passed compatibility checks. |
|
||||
| **accepted_artifact_reused** | A required predecessor's accepted normalized artifact was safely hydrated. |
|
||||
| **recompute_step** | Selective recomputation deliberately forced this work. |
|
||||
| `auto` | Read accepted entries; execute misses and store final acceptance. |
|
||||
| `bypass` | Neither read nor write; no cache directory creation. |
|
||||
| `refresh` | Skip reads and replace only after new acceptance. |
|
||||
|
||||
Reason detail is bounded code-owned text. It is diagnostic information, not a
|
||||
path-discovery or data-recovery mechanism, and does not contain checkpoint,
|
||||
source, reference, credential, or environment content.
|
||||
[Configuration](config.md#producer-controls) owns precedence. No mode overrides
|
||||
eligibility: failure/rejection/incomplete/canceled/fallback results are not stored.
|
||||
Failed refresh preserves previous acceptance. Fallback incompleteness excludes
|
||||
actual dependent work, not unrelated siblings.
|
||||
|
||||
Missing entries are ordinary misses. Unreadable, malformed, or unusable entries
|
||||
miss with diagnostics. Cache I/O failures warn without failing otherwise successful
|
||||
work or repeating model calls. Artifact encoding, required capture, and invalid
|
||||
configuration remain their own failure boundaries.
|
||||
|
||||
After fixing failed work, rerun normally: accepted predecessors and the chunk
|
||||
plan can be reused while unsuccessful work executes. There is no resume request
|
||||
or checkpoint barrier. Refresh obtains new acceptance; bypass isolates an
|
||||
invocation without updating cache state.
|
||||
|
||||
### Legacy State
|
||||
|
||||
Legacy checkpoints and source-keyed chunk plans remain ignored and untouched,
|
||||
even below the new root. They do not supply content keys. No automatic import,
|
||||
conversion, rewrite, permission migration, or deletion occurs. New entries
|
||||
populate through eligible execution. Operators may retain/clean obsolete data
|
||||
deliberately; see [migration guidance](consumers/native-migration.md).
|
||||
|
||||
## Debug Bundles
|
||||
|
||||
Only a [debug-enabled run](cli.md#run) creates a bundle:
|
||||
Capture needs explicit selection and a nonempty effective destination.
|
||||
Directory configuration alone does not enable it. Explicit empty disables it;
|
||||
a nonempty invocation override can re-enable it. [Configuration](config.md#file-shape-and-defaults)
|
||||
owns defaults/origins; [CLI Reference](cli.md#run) owns mode ordering.
|
||||
|
||||
~~~
|
||||
<debug-root>/<run-id>/
|
||||
summary/
|
||||
trace/
|
||||
~~~
|
||||
Allocation creates the run directory and updates `latest` to a relative link
|
||||
at session start. It reflects the last successful start link update, including a
|
||||
run that later fails. Completion never retargets it. Concurrent starts retain
|
||||
separate directories; disabled capture leaves the link untouched. This link is
|
||||
not an output-discovery API.
|
||||
|
||||
The summary contains redacted invocation and resolution information, including
|
||||
`effective-config.yml` and the separately redacted resolved pipeline, plus run,
|
||||
final grouped diagnostic, checkpoint, chunk-plan, and terminal reporting artifacts. Attempt
|
||||
terminal records contain bounded attempt kinds, validator outcomes, policy,
|
||||
decision, PromptKit repair count, and usage; they do not contain assistant
|
||||
responses or complete correction messages. The trace contains allowlisted
|
||||
application diagnostic records and can include source, model, and correction
|
||||
content. Neither surface is a cache input. Do not treat a debug bundle as safe
|
||||
to share merely because its configuration summary is redacted. Invocation
|
||||
metadata omits reasoning effort when it is inherited, records the replacement
|
||||
value when one is supplied, and records an empty value when inherited reasoning
|
||||
was explicitly cleared. It records `config_overlay` only when a non-base
|
||||
configuration overlay was selected.
|
||||
One recorder writes available evidence incrementally: configuration/composition,
|
||||
parsed source/fixed inputs, prepared contracts, native events/results/full
|
||||
exchanges, typed candidates/findings, correction/normalization material, fallbacks,
|
||||
and cache decisions. Hits can retain prepared/current artifact evidence without
|
||||
invented executions. Unknown usage stays unknown. Result-less failures retain
|
||||
available facts; undelivered responses are unassessed.
|
||||
|
||||
Notarius never creates debug state without an explicit request and never
|
||||
automatically deletes a requested bundle. If allocation succeeds, the command
|
||||
reports its path on both success and later failure. A summary, trace, or
|
||||
terminal-report persistence failure fails the command while preserving any
|
||||
already-written diagnostic data for inspection.
|
||||
Exact files, nesting, and serialized shapes are private diagnostic choices,
|
||||
not public contracts or exhaustive provider transcripts. Debug never prepares
|
||||
or executes another request just to capture it.
|
||||
|
||||
Requested capture is required work. Allocation/link failure stops before model
|
||||
work, recording errors surface at normal boundaries, and finalization failure
|
||||
fails the invocation even after publication. Available evidence and primary/
|
||||
additional causes remain; capture failure never retries a model or deletes data.
|
||||
|
||||
Bundles can contain full source, reference, prompt, response, and intermediate
|
||||
data. Preserve marked-secret/native credential omissions, without assuming
|
||||
sanitized content: arbitrary application text is not heuristically scrubbed.
|
||||
Operators own access, sharing, and retention as for other application data.
|
||||
|
||||
## Cleanup
|
||||
|
||||
Cleanup is manual and destructive. First inspect the exact leaf directory,
|
||||
then remove only that leaf; do not use a glob or a parent root as the target.
|
||||
Inspect exact targets and coordinate with active writers before administrative
|
||||
cleanup. Prefer moving evidence aside or recoverable removal where practical.
|
||||
Target only intended run directories or entries, not globs or broad workspace,
|
||||
home, or root directories.
|
||||
|
||||
~~~
|
||||
rm -rf -- /srv/notarius/output/run-1721300000000000000-0123456789abcdef0123456789abcdef
|
||||
rm -rf -- /srv/notarius/chunk-plans/0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
|
||||
rm -rf -- /srv/notarius/checkpoints/example/seriatim-0123456789abcdef/0123456789abcdef/0123456789abcdef
|
||||
rm -rf -- /srv/notarius/debug/run-1721300000000000000-0123456789abcdef0123456789abcdef
|
||||
~~~
|
||||
|
||||
Deleting output permanently removes user data. Deleting chunk plans or
|
||||
checkpoints is recoverable but may repeat expensive provider or pipeline work.
|
||||
Deleting a debug bundle removes troubleshooting evidence and a retained copy of
|
||||
application data. Notarius has no cache garbage collector, rollback operation,
|
||||
or automatic cleanup command.
|
||||
Output deletion loses user data. Cache deletion is reconstructible but may repeat
|
||||
expensive work. Debug deletion loses evidence and retained application data;
|
||||
account for its `latest` link when managing old sessions. Notarius supplies
|
||||
no automatic retention, cache migration, or collection command.
|
||||
|
||||
## Operational Limits
|
||||
|
||||
Provider execution settings and the generation timeout come from the selected
|
||||
PromptKit profile. The invocation-only **--reasoning-effort** and
|
||||
**--clear-reasoning-effort** controls may replace or clear that profile setting
|
||||
for all LLM-backed calls in one run without changing the profile. PromptKit
|
||||
also retries eligible transient HTTP failures with its built-in default
|
||||
of two additional attempts per generation. This retry work is separate from
|
||||
structural output repair, which happens within one structured-completion call. Its
|
||||
effective `structured_output_repair_attempts` limit is resolved from the
|
||||
selected binding, then the pipeline, then the prompt declaration; see
|
||||
[module bindings](config.md#module-bindings-and-validators). This is distinct
|
||||
from Notarius binding **retries**, which rerun the complete module operation
|
||||
and validation chain and do not consume or replenish the structural-repair
|
||||
limit. The maintained production prompts declare one repair attempt, paid only
|
||||
after a structural failure. One structured completion with repair budget **R**
|
||||
contains at most **R + 1** serial generations. Each generation can now contain
|
||||
more than one provider call when PromptKit retries an eligible transient
|
||||
failure. LLM-backed validators have their own corresponding invocation and
|
||||
repair budgets. These are bounds on attempted work, not a promise that every
|
||||
call reaches a provider.
|
||||
Choose caller deadlines and native execution settings for the workload.
|
||||
Cancellation remains authoritative; native timeout/capacity/provider failures
|
||||
are not interchangeable with caller cancellation. Retry, repair, and continuation
|
||||
can increase latency/cost without promising old-wrapper attempt parity.
|
||||
|
||||
Timeouts are layered. Caller cancellation is the outer authority. A positive
|
||||
effective generation timeout adds an inner request deadline to each built-in
|
||||
generation attempt, while zero disables only that generation deadline. The HTTP
|
||||
client timeout remains a transport-wide cap. Notarius does not add another
|
||||
timeout around PromptKit. Retries and repairs are serial within the same caller
|
||||
context, so their worst-case latency and cost can exceed the structural-repair
|
||||
bound above; provision run deadlines and provider budgets accordingly.
|
||||
Credentials remain optional unless the selected
|
||||
PromptKit profile requires one, in which case preparation fails before a
|
||||
provider call when its configured credential is unavailable.
|
||||
The pinned upstream boundary and profile-format links are in
|
||||
[PromptKit Integration](integrations/pkg-promptkit.md).
|
||||
Workers bound ready jobs; native engine/backend admission governs configured
|
||||
backend capacity independently. There is no global cross-backend scheduler or
|
||||
second admission queue. Nested assessment does not hold a completed producer's
|
||||
native permit. Endpoint-only/unlimited backends gain no inferred application
|
||||
backend limit. Capacity failure follows producer mode without another backoff.
|
||||
|
||||
Concurrency has two independent layers. Notarius **total_llm** defaults to 16
|
||||
and is the application-wide provider-call limit shared by all backends,
|
||||
modules, retries, and validators. PromptKit may impose a narrower admission
|
||||
limit for the selected backend. The effective active-generation bound is the
|
||||
intersection of the Notarius limit, any PromptKit backend limit, and work made
|
||||
available by the pipeline. Built-in OpenRouter profiles use PromptKit's
|
||||
upstream backend limit; endpoint-only profiles have no PromptKit backend limit
|
||||
and remain bounded by Notarius. For the configured local backend, a zero
|
||||
**concurrency_limit** leaves only the Notarius scheduler as a call limit. A
|
||||
positive value makes the effective active local-generation bound the smaller
|
||||
of **total_llm** and that local limit, so a local limit of four permits no more
|
||||
than four active local generations.
|
||||
|
||||
The Notarius scheduler admits one logical structured completion and holds that
|
||||
permit while PromptKit performs its serial corrective calls. PromptKit applies
|
||||
its selected-backend admission to each provider call; Notarius does not
|
||||
reacquire a permit or add another scheduler for a repair.
|
||||
|
||||
For a positive local limit, PromptKit owns its default waiting capacity and
|
||||
admission behavior. When a PromptKit backend has admitted all active and queued
|
||||
work, a new call fails as capacity exhaustion before generation. The adapter
|
||||
maps that failure to Notarius's existing provider-neutral capacity error and
|
||||
does not retry it. The calling stage's configured retry policy applies
|
||||
normally, and the run fails if those attempts are exhausted. Caller
|
||||
cancellation remains authoritative. Configuration contracts are documented
|
||||
under [PromptKit profiles](config.md#promptkit-profiles) and
|
||||
[concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker
|
||||
limits and actual provider-call limits are independent. Notarius writes local
|
||||
filesystem state only; remote storage, archival, and retention automation are
|
||||
outside the implemented CLI.
|
||||
|
||||
Every run has an effective prompt session used for provider routing and run
|
||||
provenance. The generated default is stable for the same input module and raw
|
||||
input bytes; use [**--session-id**](cli.md#run) only when intentionally grouping
|
||||
different invocations. Both generated and explicit values can be visible to
|
||||
providers, manifests, checkpoints, and requested debug bundles. Do not put
|
||||
credentials or other secrets in an explicit session identifier; command-line
|
||||
values are not a credential mechanism.
|
||||
Sessions are provider-visible routing correlations, not secrets. Their default
|
||||
and override semantics belong in [CLI Reference](cli.md#run-wide-llm-overrides).
|
||||
State is local filesystem data; remote storage, archival, retention automation,
|
||||
live model-quality guarantees, and release publication are outside this workflow.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Base Policy Version:** 1.7
|
||||
|
||||
**Last Updated**: 2026-11-17
|
||||
**Last Updated**: 2026-11-22
|
||||
|
||||
**Upstream Source**: https://gitea.maximumdirect.net/eric/policy/src/commit/593c9b52273c5c879785ae9e995d21026df467b4/docs/policy/release.md
|
||||
|
||||
@@ -169,8 +169,8 @@ Do not add routine confirmations for checks the command can perform itself.
|
||||
### Notarius Gate Selection
|
||||
|
||||
Run the default offline Go suite once and build the native `cmd/notarius` with
|
||||
release version injection, checking its `--version` output. The command check
|
||||
protects the actual executable entry point and linker-supplied version, which
|
||||
release version injection, checking its native `version` command output. The
|
||||
command check protects the actual executable entry point and linker-supplied version, which
|
||||
package tests alone do not verify. Build products are temporary, not published.
|
||||
Use `GOWORK=off` so a sibling checkout cannot substitute for declared dependencies.
|
||||
|
||||
|
||||
@@ -25,7 +25,7 @@ without putting secrets in the repository. Prepare the selected toolchain and
|
||||
dependencies before offline validation where necessary:
|
||||
|
||||
```sh
|
||||
GOWORK=off GOPRIVATE=gitea.maximumdirect.net go mod download
|
||||
GOWORK=off GOPRIVATE=gitea.maximumdirect.net,go.maximumdirect.net go mod download
|
||||
```
|
||||
|
||||
Choose an unused stable version under the policy's compatibility rules. Add
|
||||
@@ -103,5 +103,8 @@ development guide and testing policy govern change-specific verification.
|
||||
|
||||
A fresh source install or exact-tag checkout/build may be used to investigate
|
||||
consumer availability after release. Such checks are optional and do not delay
|
||||
or reverse release completion. `notarius --version` is diagnostic; subprocess
|
||||
compatibility is defined by the receipt and artifact contracts.
|
||||
or reverse release completion. `notarius version` is diagnostic; subprocess
|
||||
compatibility is defined by the [CLI report](cli.md#completion-report) and
|
||||
[domain artifact contracts](integrations/json-output.md). The
|
||||
[native migration guide](consumers/native-migration.md) describes the command
|
||||
and configuration cutover; it does not prepare or authorize a release.
|
||||
|
||||
@@ -1943,6 +1943,35 @@ explicit values through declarations, not prose-based change detectors.
|
||||
|
||||
## Stage 40: Update Operations And Downstream Consumer Contracts
|
||||
|
||||
**Status:** Complete. Updated operations for independent output/cache/debug roots,
|
||||
per-user and explicitly provisioned system cache placement, best-effort reuse,
|
||||
failed-refresh preservation, untouched legacy records, fallback qualification/
|
||||
eligibility, whole-source chunk fallback, and required capture with start-time
|
||||
`latest`. Replaced receipt/index/manifest workflows with native completion reports
|
||||
and configured domain paths, including partial stdout and post-publication failure
|
||||
handling. Retired the superseded receipt contract, repaired current navigation,
|
||||
and linked new native migration guidance for command/configuration/resource IDs,
|
||||
retry intent, workers/admission, caches, reporting, and private debug evidence.
|
||||
JSON/chunk-map/evidence contracts retain payload schemas and logical paths without
|
||||
requiring descriptors or historical execution provenance on reuse.
|
||||
|
||||
The maintained complete native CLI contract now supplies explicit input/output and
|
||||
pipeline-level campaign/catalog references and verifies all ten domain files plus
|
||||
both auxiliaries against the twelve-deliverable report using the existing provider
|
||||
fake. Existing executable fake-HTTP checks retain native format/fallback coverage.
|
||||
Found and corrected the release validator's retired version flag; its existing
|
||||
offline test double now rejects the wrong command, demonstrably failing before the
|
||||
fix and passing afterward. Updated only affected release documentation/command
|
||||
references, preserving the selected validation gates and adopted policy baseline.
|
||||
A temporary linked executable reports its injected version via the native command.
|
||||
|
||||
Whole-tree tests and vet, executable build, shell syntax, 147 local links/anchors,
|
||||
and `git diff --check` pass. Removal searches leave receipt-contract references
|
||||
only in the active plan and preserved historical release/archive records. No
|
||||
runtime data, domain schema, tagged note, or archive was changed; no real provider,
|
||||
credential, release, push, or new provisioning/retention requirement was involved.
|
||||
Full internal/integration/orientation rewrites remain assigned to the next stages.
|
||||
|
||||
**Goal:** explain the intentional persistence and subprocess breaking changes.
|
||||
|
||||
**Inspect/change:** `docs/operations.md`, `docs/consumers/{subprocess,dnd-pipeline}.md`,
|
||||
|
||||
Reference in New Issue
Block a user