Plan the ConfigKit upgrade and workspace development
This commit is contained in:
280
docs/roadmap/workspace.md
Normal file
280
docs/roadmap/workspace.md
Normal file
@@ -0,0 +1,280 @@
|
||||
# Session workspace organization
|
||||
|
||||
## Purpose
|
||||
|
||||
Give operators predictable locations for session inputs, completed outputs,
|
||||
logs, and debug evidence. Apply the taxonomy in
|
||||
[ADR 0001](../adr/0001-classify-session-content.md) and the ownership boundary in
|
||||
[ADR 0002](../adr/0002-assign-subprocess-content-ownership.md).
|
||||
|
||||
This roadmap defines the target behavior and required changes. Implementation
|
||||
sequencing belongs in a subsequent implementation plan. The ADRs remain
|
||||
Proposed pending their acceptance.
|
||||
|
||||
## Confirmed Requirements
|
||||
|
||||
- Completed artifacts occupy a flat `artifacts/` directory. Intermediate
|
||||
products such as `session_bounds.json` belong here.
|
||||
- Combined transcripts occupy `transcripts/` directly. Initial per-speaker
|
||||
transcripts occupy a subdirectory beneath it.
|
||||
- Prepared external audio occupies `audio/`; other prepared external inputs
|
||||
occupy `inputs/`. Outputs consumed by later stages retain their output paths.
|
||||
- Logs and Debug use matching latest-attempt views: `logs/<stage>` and
|
||||
`debug/<stage>` are symlinks to the producing run's directories. Both views
|
||||
advance when a stage starts an actual attempt, including attempts that fail.
|
||||
A skipped stage leaves its previous views intact.
|
||||
- Execution reports and invocation records are Logs. Historical Logs and Debug
|
||||
remain under their producing runs, subject to cleanup.
|
||||
- Narratio can selectively request supported subprocess debug capture through
|
||||
configuration. Omission preserves application configuration and defaults;
|
||||
Narratio needs no explicit downstream debug-disable override.
|
||||
- Prepared session audio is the single retained Narratio-managed audio copy.
|
||||
Remove Narratio's separate persistent audio cache and spool duplicates.
|
||||
- Configurations that produce colliding artifact names or destinations fail
|
||||
validation before execution. Validation covers the effective pipeline,
|
||||
including expanded character families and built-in artifact destinations.
|
||||
- Narratio selects input, output, log, and debug paths. Subprocess applications
|
||||
own their working State and Cache through their supported configuration.
|
||||
- A dedicated cache root such as `/var/cache/notarius` is a deployment
|
||||
recommendation. The application owns its cache defaults, internal layout,
|
||||
identity, and reuse semantics. Narratio session cleanup respects that boundary.
|
||||
- Narratio retains its own manifest, recovery records, and managed staging State.
|
||||
- This is a clean cutover for newly prepared sessions. Existing-session migration
|
||||
and backward compatibility shims are outside the feature.
|
||||
- Treatment of older completed files after invalidation or failed replacement
|
||||
remains unchanged, including their availability through manifest-backed
|
||||
resolution. Directory reorganization does not authorize new stale-file
|
||||
deletion, automatic fallback, or rollback behavior.
|
||||
|
||||
## Current Baseline
|
||||
|
||||
The [operator guide](../operations.md#local-state-layout) owns the current
|
||||
physical layout. Relevant differences from the target are:
|
||||
|
||||
- Extraction moves the complete Notarius bundle to
|
||||
`artifacts/notarius/<run_id>/`. Resume evidence and artifact resolution depend
|
||||
on that bundle and its index.
|
||||
- Merge and normalize promote Seriatim execution reports into `artifacts/`.
|
||||
- Analysis output paths and character-family patterns can introduce nested
|
||||
artifact directories. Output-path configuration also permits locations that
|
||||
need reconciliation with the selected directory categories.
|
||||
- Standard combined transcript paths already match the target. Layout creation
|
||||
also creates an unused `transcripts/trimmed/` directory.
|
||||
- Most subprocess diagnostics live under individual run directories. Root-level
|
||||
log paths lack a consistent latest-attempt view. Audita reports sit beside
|
||||
its staged transcript, and restore writes a separate top-level report.
|
||||
- Prepared previous-session inputs occupy `previous/`. Narratio-owned reference
|
||||
snapshots supplied to Notarius occupy session `cache/notarius/references/`.
|
||||
These snapshots are distinct from Notarius's own execution caches.
|
||||
- Audita debug placement follows downstream configuration. Narratio does not
|
||||
expose Notarius debug capture, and Scriptorium render-debug placement varies.
|
||||
- Narratio maintains separate S3 audio cache and spool paths in addition to the
|
||||
prepared session audio copy.
|
||||
|
||||
## Target Layout And Ownership
|
||||
|
||||
Retain the configurable workspace root and its current system-wide default,
|
||||
`/var/lib/narratio`. Sessions remain under
|
||||
`/var/lib/narratio/work/<campaign>/<session>` by default. The following paths
|
||||
express the selected categories:
|
||||
|
||||
| Path | Contents and role |
|
||||
| --- | --- |
|
||||
| `manifest.json` | Narratio's durable progress and output references. |
|
||||
| `artifacts/<filename>` | Completed artifact payloads, including session bounds. |
|
||||
| `transcripts/<filename>` | Completed combined transcripts and rendered transcripts. |
|
||||
| `transcripts/raw/<speaker>.json` | Completed initial per-speaker transcripts. |
|
||||
| `audio/` | Prepared external audio. |
|
||||
| `inputs/` | Prepared configuration, references, and previous-session inputs. |
|
||||
| `logs/<stage>` | Symlink to the latest attempt's run-local logs and execution reports. |
|
||||
| `debug/<stage>` | Symlink to the latest attempt's run-local debug directory, empty when no capture is produced. |
|
||||
| `runs/<run_id>/<stage>/logs/` | Actual logs and execution reports for that attempt. |
|
||||
| `runs/<run_id>/<stage>/debug/` | Actual debug captures for that attempt, including retained state snapshots. |
|
||||
| `runs/<run_id>/` | Narratio staging, original tool bundles, and retained execution evidence. |
|
||||
|
||||
For example, the latest views use relative links within the session:
|
||||
|
||||
```text
|
||||
logs/polish -> ../runs/<run_id>/polish/logs/
|
||||
debug/polish -> ../runs/<run_id>/polish/debug/
|
||||
```
|
||||
|
||||
Each stage's latest attempt can belong to a different run. Producers write to
|
||||
the actual run paths; the symlinks provide operator navigation without duplicate
|
||||
diagnostic storage.
|
||||
|
||||
Narratio's lock and restore marker remain its own State. Their current placement
|
||||
can remain; introducing another root-level State directory is unnecessary for
|
||||
this feature. Generated records that describe an invocation belong with Logs;
|
||||
configuration actually supplied as an input follows its input or staging role.
|
||||
|
||||
Subprocesses may create internal working directories wherever their configuration
|
||||
directs. Narratio-selected staging destinations for result bundles remain valid
|
||||
under ADR 0002. Cache locations are documented as operator recommendations using
|
||||
each application's existing configuration surface.
|
||||
|
||||
## Required Changes
|
||||
|
||||
### Output paths and configuration
|
||||
|
||||
Centralize the revised paths in the existing path owner. Apply the flat artifact
|
||||
rule to explicit artifacts, expanded families, extraction outputs, and session
|
||||
bounds. Preserve logical source identities used by dependencies and publication.
|
||||
Keep explicit analysis filenames and require flat destinations. Derive extraction
|
||||
filenames from configured extraction keys with a `.json` extension, with an
|
||||
explicit extraction filename override for naming preferences or collisions.
|
||||
Retain `session_bounds.json` as the default bounds filename. Do not silently
|
||||
rename conflicting files. Report conflicting producers and the shared destination
|
||||
in configuration errors. Validate the effective configuration after composition
|
||||
and family expansion.
|
||||
|
||||
Keep combined transcript destinations directly beneath `transcripts/`. Remove
|
||||
obsolete directory scaffolding and reconcile configurable output paths with
|
||||
their category. Local path changes do not require flattening explicitly
|
||||
configured remote publish destinations.
|
||||
|
||||
### Notarius result handling and reuse
|
||||
|
||||
Receive and validate the complete Notarius bundle in Narratio staging. Promote
|
||||
selected accepted payloads to their flat artifact destinations. Classify the
|
||||
receipt, warnings, rejection summaries, and diagnostics as Logs. Keep bundle
|
||||
indexes and other management metadata with the bundle or Narratio State according
|
||||
to their use.
|
||||
|
||||
Revise extraction evidence, catalog hydration, and resume checks to resolve
|
||||
promoted products independently of the old `artifacts/notarius/` layout. Retain
|
||||
the identity, provenance, contract, and checksum evidence needed for current
|
||||
reuse behavior. Historical bundles may remain available for inspection under
|
||||
their existing retention semantics; relocating them beneath `runs/` must not
|
||||
silently shorten their lifetime through run cleanup.
|
||||
|
||||
Preserve ordinary resume and forced-extraction behavior. Subprocess caches remain
|
||||
under Notarius's control.
|
||||
|
||||
### Inputs
|
||||
|
||||
Place prepared previous-session material beneath `inputs/previous/`. Place the
|
||||
stable reference snapshots Narratio supplies to Notarius beneath
|
||||
`inputs/references/`, retaining their content identities where needed for reuse.
|
||||
Update preparation, input resolution, manifest references, and restore mappings.
|
||||
Keep source configuration ownership and prepared-file validation unchanged.
|
||||
|
||||
### Audio acquisition
|
||||
|
||||
Simplify acquisition to one prepared session copy under `audio/`. Remove the
|
||||
separate Narratio S3 audio cache and routine retained spool copies, together with
|
||||
their obsolete configuration, manifest bookkeeping, and cleanup controls.
|
||||
Preparation may use temporary download files before installing validated audio.
|
||||
Preserve reuse of valid prepared audio and update restore to use the same layout.
|
||||
|
||||
Recreating a session after its prepared audio is removed requires fetching the
|
||||
source again. Original recordings remain outside this cleanup change. Removing
|
||||
spool/cache handling does not authorize new deletion of prepared session audio.
|
||||
Subprocess caches remain independently configured and managed.
|
||||
|
||||
### Logs and execution reports
|
||||
|
||||
Provide one consistent latest-attempt view per stage. Include failed attempts;
|
||||
a skipped stage leaves its previous view intact. Distinguish multiple applications
|
||||
or artifact invocations within a stage through their filenames. Route Seriatim
|
||||
reports, Audita reports, Notarius diagnostic records, and Narratio invocation
|
||||
records through this model. Treat restore execution as its own diagnostic group
|
||||
under `logs/restore/`, using run-local evidence and a latest-attempt view.
|
||||
|
||||
Prevent a new attempt from presenting old diagnostic files as newly generated
|
||||
ones. At the start of each actual stage attempt, create its run-local log and
|
||||
debug directories and refresh both latest symlinks. An attempt without debug
|
||||
output exposes an empty debug directory rather than an older capture. Preserve
|
||||
subprocess errors and available evidence on failure. Retain existing
|
||||
captured-stream redaction and limits.
|
||||
|
||||
Keep historical files at their run paths; rotating the links does not move files.
|
||||
Run cleanup must preserve directories targeted by current latest links. Do not
|
||||
introduce an automatic history age or count limit. Publication and manifest
|
||||
diagnostic references use actual run paths. Adjust path handling for these
|
||||
Narratio-managed navigation links without relying on arbitrary symlink traversal.
|
||||
|
||||
### Debug controls and placement
|
||||
|
||||
Add selective Narratio configuration for Audita and Notarius debug capture using
|
||||
their supported enabling CLI flags. An enabled Narratio selector requests
|
||||
capture; an omitted or false selector passes no debug override and preserves the
|
||||
application's configured/default behavior. Do not add a downstream disable flag.
|
||||
|
||||
Supply the actual run-local debug directories, exposed through `debug/polish`
|
||||
and `debug/extract`. Control the destination when capture can be enabled through
|
||||
downstream configuration as well, including Audita overlays, using supported
|
||||
destination controls without changing capture enablement. Audita's debug
|
||||
work-directory option selects diagnostic storage and belongs to this placement
|
||||
responsibility.
|
||||
|
||||
Route Scriptorium's existing analysis and bounds render-debug products to the
|
||||
run-local directories exposed through `debug/analyze` and `debug/trim`. Preserve
|
||||
its existing per-artifact selection capability. Seriatim requires no invented
|
||||
debug mode; its ordinary reports are
|
||||
Logs. Subprocess-defined bundle contents and nested run directories are allowed
|
||||
beneath the selected run-local debug root. Logs and Debug share the same
|
||||
latest-attempt semantics and cleanup protection.
|
||||
|
||||
Keep new full Audita and Notarius debug bundles local in this feature. Preserve
|
||||
publication of currently declared diagnostics, including existing render-debug
|
||||
products. Enabling capture does not implicitly add full debug bundles to remote
|
||||
archives.
|
||||
|
||||
Debug controls govern capture when a stage executes. They do not independently
|
||||
require regeneration of valid substantive outputs. Document the existing force
|
||||
workflow for obtaining fresh diagnostic evidence when a stage would be reused.
|
||||
Preserve downstream failure semantics for diagnostic writes and the existing
|
||||
boundary of Narratio's captured-stream redaction.
|
||||
|
||||
### Audita destination support
|
||||
|
||||
Request upstream support for independently selecting the transcript output,
|
||||
ordinary report/log directory, and debug/retained working directory, then adopt
|
||||
a supporting release. Audita currently ignores `--report-dir` when transcript
|
||||
output is a file; honoring that destination is the principal missing capability.
|
||||
Narratio continues to capture subprocess stdout and stderr itself.
|
||||
|
||||
Direct ordinary reports, including failure reports, to the actual polish log
|
||||
directory, the transcript to its staged product destination, and retained debug
|
||||
material to the actual polish debug directory. This feature uses the supported
|
||||
upstream controls in a clean cutover without version-based compatibility branches.
|
||||
|
||||
### Manifest, publication, restore, and cleanup
|
||||
|
||||
Update all consumers of changed paths and file categories together. Preserve
|
||||
per-speaker and per-artifact completed-work reuse. Keep publication driven by
|
||||
declared products and diagnostics, with new full debug bundles remaining local.
|
||||
Restore new-format products and inputs to their new locations.
|
||||
|
||||
Define an explicit way to identify supported session state for the clean
|
||||
cutover. Document regeneration with the new version and manual configuration
|
||||
updates. Avoid accepting old state through path fallbacks or silently rewriting
|
||||
old local sessions or remote snapshots.
|
||||
|
||||
Preserve existing invalidation and failed-replacement behavior. Review retention
|
||||
by content role before changing cleanup targets, especially when durable products
|
||||
or historical bundles move. Run cleanup preserves the current log and debug
|
||||
link targets; explicit removal of an entire session removes its local views
|
||||
with it. Subprocess working State and Cache remain outside Narratio's cleanup
|
||||
responsibility.
|
||||
|
||||
### Documentation and verification
|
||||
|
||||
Update configuration examples, the operator guide, adapter contracts, workspace
|
||||
internals, and the architecture policy when implementation lands. Replace the
|
||||
current architectural assumption that extraction requires a complete immutable
|
||||
bundle beneath `artifacts/` with the selected product-promotion model. Retire
|
||||
policy and documentation for Narratio's removed audio cache/spool lifecycle and
|
||||
document the managed latest-attempt symlinks. Keep per-application cache locations
|
||||
as deployment recommendations owned by each application's configuration.
|
||||
|
||||
Use focused behavioral coverage for flat output placement, collision rejection,
|
||||
latest-attempt diagnostics on success and failure, debug selection and routing,
|
||||
unchanged completed-work reuse, failed replacement, and new-format publish/restore.
|
||||
Cover empty debug views after attempts without capture, preserved views for
|
||||
skipped stages, cleanup protection of latest targets, and audio acquisition,
|
||||
reuse, and restore without persistent spool/cache duplicates. Exercise cleanup
|
||||
against products and separately owned caches. Use temporary filesystems and
|
||||
subprocess doubles; default tests remain offline and independent
|
||||
of sibling checkouts or paid providers.
|
||||
Reference in New Issue
Block a user