Files
narratio/docs/roadmap/workspace.md

281 lines
15 KiB
Markdown

# Session workspace organization
## Purpose
Give operators predictable locations for session inputs, completed outputs,
logs, and debug evidence. Apply the taxonomy in
[ADR 0001](../adr/0001-classify-session-content.md) and the ownership boundary in
[ADR 0002](../adr/0002-assign-subprocess-content-ownership.md).
This roadmap defines the target behavior and required changes. Implementation
sequencing belongs in a subsequent implementation plan. The ADRs remain
Proposed pending their acceptance.
## Confirmed Requirements
- Completed artifacts occupy a flat `artifacts/` directory. Intermediate
products such as `session_bounds.json` belong here.
- Combined transcripts occupy `transcripts/` directly. Initial per-speaker
transcripts occupy a subdirectory beneath it.
- Prepared external audio occupies `audio/`; other prepared external inputs
occupy `inputs/`. Outputs consumed by later stages retain their output paths.
- Logs and Debug use matching latest-attempt views: `logs/<stage>` and
`debug/<stage>` are symlinks to the producing run's directories. Both views
advance when a stage starts an actual attempt, including attempts that fail.
A skipped stage leaves its previous views intact.
- Execution reports and invocation records are Logs. Historical Logs and Debug
remain under their producing runs, subject to cleanup.
- Narratio can selectively request supported subprocess debug capture through
configuration. Omission preserves application configuration and defaults;
Narratio needs no explicit downstream debug-disable override.
- Prepared session audio is the single retained Narratio-managed audio copy.
Remove Narratio's separate persistent audio cache and spool duplicates.
- Configurations that produce colliding artifact names or destinations fail
validation before execution. Validation covers the effective pipeline,
including expanded character families and built-in artifact destinations.
- Narratio selects input, output, log, and debug paths. Subprocess applications
own their working State and Cache through their supported configuration.
- A dedicated cache root such as `/var/cache/notarius` is a deployment
recommendation. The application owns its cache defaults, internal layout,
identity, and reuse semantics. Narratio session cleanup respects that boundary.
- Narratio retains its own manifest, recovery records, and managed staging State.
- This is a clean cutover for newly prepared sessions. Existing-session migration
and backward compatibility shims are outside the feature.
- Treatment of older completed files after invalidation or failed replacement
remains unchanged, including their availability through manifest-backed
resolution. Directory reorganization does not authorize new stale-file
deletion, automatic fallback, or rollback behavior.
## Current Baseline
The [operator guide](../operations.md#local-state-layout) owns the current
physical layout. Relevant differences from the target are:
- Extraction moves the complete Notarius bundle to
`artifacts/notarius/<run_id>/`. Resume evidence and artifact resolution depend
on that bundle and its index.
- Merge and normalize promote Seriatim execution reports into `artifacts/`.
- Analysis output paths and character-family patterns can introduce nested
artifact directories. Output-path configuration also permits locations that
need reconciliation with the selected directory categories.
- Standard combined transcript paths already match the target. Layout creation
also creates an unused `transcripts/trimmed/` directory.
- Most subprocess diagnostics live under individual run directories. Root-level
log paths lack a consistent latest-attempt view. Audita reports sit beside
its staged transcript, and restore writes a separate top-level report.
- Prepared previous-session inputs occupy `previous/`. Narratio-owned reference
snapshots supplied to Notarius occupy session `cache/notarius/references/`.
These snapshots are distinct from Notarius's own execution caches.
- Audita debug placement follows downstream configuration. Narratio does not
expose Notarius debug capture, and Scriptorium render-debug placement varies.
- Narratio maintains separate S3 audio cache and spool paths in addition to the
prepared session audio copy.
## Target Layout And Ownership
Retain the configurable workspace root and its current system-wide default,
`/var/lib/narratio`. Sessions remain under
`/var/lib/narratio/work/<campaign>/<session>` by default. The following paths
express the selected categories:
| Path | Contents and role |
| --- | --- |
| `manifest.json` | Narratio's durable progress and output references. |
| `artifacts/<filename>` | Completed artifact payloads, including session bounds. |
| `transcripts/<filename>` | Completed combined transcripts and rendered transcripts. |
| `transcripts/raw/<speaker>.json` | Completed initial per-speaker transcripts. |
| `audio/` | Prepared external audio. |
| `inputs/` | Prepared configuration, references, and previous-session inputs. |
| `logs/<stage>` | Symlink to the latest attempt's run-local logs and execution reports. |
| `debug/<stage>` | Symlink to the latest attempt's run-local debug directory, empty when no capture is produced. |
| `runs/<run_id>/<stage>/logs/` | Actual logs and execution reports for that attempt. |
| `runs/<run_id>/<stage>/debug/` | Actual debug captures for that attempt, including retained state snapshots. |
| `runs/<run_id>/` | Narratio staging, original tool bundles, and retained execution evidence. |
For example, the latest views use relative links within the session:
```text
logs/polish -> ../runs/<run_id>/polish/logs/
debug/polish -> ../runs/<run_id>/polish/debug/
```
Each stage's latest attempt can belong to a different run. Producers write to
the actual run paths; the symlinks provide operator navigation without duplicate
diagnostic storage.
Narratio's lock and restore marker remain its own State. Their current placement
can remain; introducing another root-level State directory is unnecessary for
this feature. Generated records that describe an invocation belong with Logs;
configuration actually supplied as an input follows its input or staging role.
Subprocesses may create internal working directories wherever their configuration
directs. Narratio-selected staging destinations for result bundles remain valid
under ADR 0002. Cache locations are documented as operator recommendations using
each application's existing configuration surface.
## Required Changes
### Output paths and configuration
Centralize the revised paths in the existing path owner. Apply the flat artifact
rule to explicit artifacts, expanded families, extraction outputs, and session
bounds. Preserve logical source identities used by dependencies and publication.
Keep explicit analysis filenames and require flat destinations. Derive extraction
filenames from configured extraction keys with a `.json` extension, with an
explicit extraction filename override for naming preferences or collisions.
Retain `session_bounds.json` as the default bounds filename. Do not silently
rename conflicting files. Report conflicting producers and the shared destination
in configuration errors. Validate the effective configuration after composition
and family expansion.
Keep combined transcript destinations directly beneath `transcripts/`. Remove
obsolete directory scaffolding and reconcile configurable output paths with
their category. Local path changes do not require flattening explicitly
configured remote publish destinations.
### Notarius result handling and reuse
Receive and validate the complete Notarius bundle in Narratio staging. Promote
selected accepted payloads to their flat artifact destinations. Classify the
receipt, warnings, rejection summaries, and diagnostics as Logs. Keep bundle
indexes and other management metadata with the bundle or Narratio State according
to their use.
Revise extraction evidence, catalog hydration, and resume checks to resolve
promoted products independently of the old `artifacts/notarius/` layout. Retain
the identity, provenance, contract, and checksum evidence needed for current
reuse behavior. Historical bundles may remain available for inspection under
their existing retention semantics; relocating them beneath `runs/` must not
silently shorten their lifetime through run cleanup.
Preserve ordinary resume and forced-extraction behavior. Subprocess caches remain
under Notarius's control.
### Inputs
Place prepared previous-session material beneath `inputs/previous/`. Place the
stable reference snapshots Narratio supplies to Notarius beneath
`inputs/references/`, retaining their content identities where needed for reuse.
Update preparation, input resolution, manifest references, and restore mappings.
Keep source configuration ownership and prepared-file validation unchanged.
### Audio acquisition
Simplify acquisition to one prepared session copy under `audio/`. Remove the
separate Narratio S3 audio cache and routine retained spool copies, together with
their obsolete configuration, manifest bookkeeping, and cleanup controls.
Preparation may use temporary download files before installing validated audio.
Preserve reuse of valid prepared audio and update restore to use the same layout.
Recreating a session after its prepared audio is removed requires fetching the
source again. Original recordings remain outside this cleanup change. Removing
spool/cache handling does not authorize new deletion of prepared session audio.
Subprocess caches remain independently configured and managed.
### Logs and execution reports
Provide one consistent latest-attempt view per stage. Include failed attempts;
a skipped stage leaves its previous view intact. Distinguish multiple applications
or artifact invocations within a stage through their filenames. Route Seriatim
reports, Audita reports, Notarius diagnostic records, and Narratio invocation
records through this model. Treat restore execution as its own diagnostic group
under `logs/restore/`, using run-local evidence and a latest-attempt view.
Prevent a new attempt from presenting old diagnostic files as newly generated
ones. At the start of each actual stage attempt, create its run-local log and
debug directories and refresh both latest symlinks. An attempt without debug
output exposes an empty debug directory rather than an older capture. Preserve
subprocess errors and available evidence on failure. Retain existing
captured-stream redaction and limits.
Keep historical files at their run paths; rotating the links does not move files.
Run cleanup must preserve directories targeted by current latest links. Do not
introduce an automatic history age or count limit. Publication and manifest
diagnostic references use actual run paths. Adjust path handling for these
Narratio-managed navigation links without relying on arbitrary symlink traversal.
### Debug controls and placement
Add selective Narratio configuration for Audita and Notarius debug capture using
their supported enabling CLI flags. An enabled Narratio selector requests
capture; an omitted or false selector passes no debug override and preserves the
application's configured/default behavior. Do not add a downstream disable flag.
Supply the actual run-local debug directories, exposed through `debug/polish`
and `debug/extract`. Control the destination when capture can be enabled through
downstream configuration as well, including Audita overlays, using supported
destination controls without changing capture enablement. Audita's debug
work-directory option selects diagnostic storage and belongs to this placement
responsibility.
Route Scriptorium's existing analysis and bounds render-debug products to the
run-local directories exposed through `debug/analyze` and `debug/trim`. Preserve
its existing per-artifact selection capability. Seriatim requires no invented
debug mode; its ordinary reports are
Logs. Subprocess-defined bundle contents and nested run directories are allowed
beneath the selected run-local debug root. Logs and Debug share the same
latest-attempt semantics and cleanup protection.
Keep new full Audita and Notarius debug bundles local in this feature. Preserve
publication of currently declared diagnostics, including existing render-debug
products. Enabling capture does not implicitly add full debug bundles to remote
archives.
Debug controls govern capture when a stage executes. They do not independently
require regeneration of valid substantive outputs. Document the existing force
workflow for obtaining fresh diagnostic evidence when a stage would be reused.
Preserve downstream failure semantics for diagnostic writes and the existing
boundary of Narratio's captured-stream redaction.
### Audita destination support
Request upstream support for independently selecting the transcript output,
ordinary report/log directory, and debug/retained working directory, then adopt
a supporting release. Audita currently ignores `--report-dir` when transcript
output is a file; honoring that destination is the principal missing capability.
Narratio continues to capture subprocess stdout and stderr itself.
Direct ordinary reports, including failure reports, to the actual polish log
directory, the transcript to its staged product destination, and retained debug
material to the actual polish debug directory. This feature uses the supported
upstream controls in a clean cutover without version-based compatibility branches.
### Manifest, publication, restore, and cleanup
Update all consumers of changed paths and file categories together. Preserve
per-speaker and per-artifact completed-work reuse. Keep publication driven by
declared products and diagnostics, with new full debug bundles remaining local.
Restore new-format products and inputs to their new locations.
Define an explicit way to identify supported session state for the clean
cutover. Document regeneration with the new version and manual configuration
updates. Avoid accepting old state through path fallbacks or silently rewriting
old local sessions or remote snapshots.
Preserve existing invalidation and failed-replacement behavior. Review retention
by content role before changing cleanup targets, especially when durable products
or historical bundles move. Run cleanup preserves the current log and debug
link targets; explicit removal of an entire session removes its local views
with it. Subprocess working State and Cache remain outside Narratio's cleanup
responsibility.
### Documentation and verification
Update configuration examples, the operator guide, adapter contracts, workspace
internals, and the architecture policy when implementation lands. Replace the
current architectural assumption that extraction requires a complete immutable
bundle beneath `artifacts/` with the selected product-promotion model. Retire
policy and documentation for Narratio's removed audio cache/spool lifecycle and
document the managed latest-attempt symlinks. Keep per-application cache locations
as deployment recommendations owned by each application's configuration.
Use focused behavioral coverage for flat output placement, collision rejection,
latest-attempt diagnostics on success and failure, debug selection and routing,
unchanged completed-work reuse, failed replacement, and new-format publish/restore.
Cover empty debug views after attempts without capture, preserved views for
skipped stages, cleanup protection of latest targets, and audio acquisition,
reuse, and restore without persistent spool/cache duplicates. Exercise cleanup
against products and separately owned caches. Use temporary filesystems and
subprocess doubles; default tests remain offline and independent
of sibling checkouts or paid providers.