Files
notarius/docs/internal/pipeline.md

12 KiB

Pipeline Internals

This guide explains resolution, typed execution, candidate assessment, and ordering in internal/framework/pipeline. Configuration owns bindings and producer controls; Operations owns state/recovery workflows; integration contracts own deliverable shapes. Module Internals owns extension recipes.

Boundary And Fixed Shape

The framework receives prepared composition, source bytes, a supplied session, native execution options, and explicit cache/capture collaborators. It returns logical deliverable files, process diagnostics, and fallback count. Normalized artifacts and reuse eligibility remain private run state for handoffs/evidence; there is no reporting ledger in the output boundary.

input -> chunk -> ordered steps [extract -> merge -> normalize] -> output

Input and chunking are pipeline-wide. Each ordered step contains typed artifact lanes; the next step waits for the current step and its generated handoffs. This fixed pipeline is not a general workflow graph. The CLI owns arguments, configuration loading, session derivation, physical roots, publication, and native reporting; see CLI Internals.

Resolve, Materialize, Prepare

ResolvePipeline selects modules, capabilities, exact artifact kinds, validators, references, options, and concrete producer controls. It copies static native prompt dependencies from selected specifications. Effective profiles apply only to model-backed roles; an empty profile selection leaves native prompt defaults applicable. Control resolution settles presence-aware controls before execution; modules do not repeat inheritance. The composition digest describes resolved composition, not cache identity.

Resolution receives exact reference bindings/unbinds, not CLI selector grammar. It preserves required slots, external/generated conflicts, prior-step dependencies, and artifact compatibility. Reference materialization reads selected external bytes with declaration/invocation origins, media/UTF-8 and declared size checks. Generated selections remain declarations until their producer step completes. Public selection semantics belong in References And Ordered Handoffs.

Prepare clones resolved data and constructs input, chunker, validators, typed lanes, and output through supplied registries and the shared native engine. Builders receive independently owned build requests. Exact Go type, codec, evidence-projector, registration, and constructor checks happen before source parsing. There is no correction transport, asset registry, or persistent execution identity to collect. Offline known-resource preflight is a separate composition check, not request execution or module construction.

Typed Operations And Assessment

Typed contracts carry a ProducerOperation[T] with resolved controls, one assessor, native execution options, incoming reuse eligibility, and an optional typed cache binding. Runner operation completion assesses deterministic production directly. Model modules use RunProducer; the runner consumes its acceptance/fallback outcome without assessing it again.

Candidate assessment invokes each applicable validator once in configured order. It continues after rejections and unavailable/failed assessments; caller cancellation and framework/ capture failures interrupt it. Typed candidates are encoded once for assessment, then decoded separately for each typed validator. Serialized validators receive owned bytes/schema metadata; chunk validators receive detached materialized chunks. Source, references, annotations, and metadata views are isolated too.

Assessment values distinguish approval, semantic rejection, failure, and unavailable judgment. An empty configured chain adds no ordinary acceptance criteria, but cannot remove intrinsic codec or source-addressing checks. Rejection guidance is module-authored, ordered, and deduplicated separately from reason codes/operator messages. Invalid guidance is a contract failure, not invented correction text. LLM validators execute directly on the shared engine and do not own another semantic loop.

Native Continuation And Terminal Handling

For executed model work, RunProducer checks native errors and structural validation status before interpretation, including a failed result with no Go error. Module interpretation produces a usable typed candidate and any actionable domain guidance. The assessor runs the complete chain on every delivered usable candidate, including unchanged revisions and the final allowed response.

When correction is possible, the helper starts native Coordinate from the captured exchange. Its first callback reuses already-collected initial findings. Native coordination owns turn iteration and growing history; the module builds only new correction material from fixed operation inputs. Successful native execution, stopped/exhausted coordination, and domain acceptance are distinct. Native repair/transient retries are not extended by outer application retries. LLM Integration owns the native request/capture integration.

Terminal disposition either fails or invokes one supported module fallback. Fallback construction preserves intrinsic/source/downstream invariants without ordinary reassessment or semantic-approval claims. It may fail normally. A failed continuation cannot recover an earlier candidate; caller cancellation and undelivered responses produce no acceptance/fallback. Native operational timeouts retain typed causes rather than becoming caller cancellation solely by error spelling.

Deterministic work produces once, then uses direct assessment and any supported module fallback. An operation/parse/materialization/encoding error remains an error, not a generic rerun or permissive recovery mechanism.

One Ready-Job Worker Pool

The lane coordinator dispatches extraction jobs and ready lane continuations to the same bounded worker pool. A continuation becomes ready after all extracts for its lane finish; one worker runs its merge then normalization. Ready continuations can interleave with unrelated extraction, and prerequisite waits occupy no worker. Input parsing, chunking, step barriers, and final output remain sequential.

Results are restored to chunk/lane order before merging. Stable operation/lane/ chunk positions select the scoped terminal error among reported failures rather than response timing. Parent cancellation takes precedence. On failure/cancellation the coordinator stops dispatch, drains started results, closes jobs, and joins workers before returning. Output encoding does not run after terminal failure.

Nested validation and correction use the operation's worker, not another worker permit. Native backend admission is independent and released before candidate assessment, allowing a nested validator on the same limited backend. Worker configuration does not create another model scheduler, queue, or retry policy.

Handoffs And Reuse Eligibility

Generated handoffs locate one earlier normalized producer, decode/re-encode through its registered codec, check full artifact/schema/media compatibility, and clone bytes into the consumer's declared reference set. Valid qualified fallback is usable current-run context; it does not become ordinary acceptance through a handoff.

Eligibility travels separately from artifact bytes:

  • A chunk fallback disables reuse for all derived lane work.
  • An extract fallback disables reuse for its dependent merge/normalization.
  • A fallback-derived generated reference disables reuse from the consuming operation onward; unrelated siblings and earlier accepted work remain eligible.

An emitting fallback increments the run count once. Merely consuming its data or inheriting ineligibility adds no fallback. These are transient dependency facts, not persisted fingerprints or a second recovery protocol.

Typed cache bindings use the registered durable codec and available exact-type evidence projection for current-source usability. Chunk planning uses the same producer cache with plan decoding/rematerialization. Hits do not run semantic assessment. Deterministic work has no synthetic request/cache key. Run State Internals owns prepared-handle/store mechanics and recovery.

Output And Capture Boundaries

Runner parses/validates source, materializes the final plan, runs ordered steps, aggregates diagnostics, and builds optional chunk-map/evidence values. Normalized values are codec-checked before handoff/output. Evidence reconstruction also works from reused artifacts; it projects selected citations in source order, not a second typed output channel. Evidence internals and the evidence contract define those owners.

The encoder receives owned normalized/auxiliary values and returns logical files. It does not choose a directory, publish a report, or serialize execution ledgers. JSON output owns logical paths and payloads; State Internals owns physical publication and failure retention.

Optional scoped evidence records inputs, candidates, findings, outcomes, and cache decisions. Native observers enroll actual executions once in one invocation group, preserving unknown usage and terminal coverage. Capture failures are ordinary boundary failures, never semantic feedback or reasons to rerun a model. Debug layout is private; it is not a recovery source or consumer protocol.

Focused Verification

Use focused pipeline and CLI suites for boundary changes. Development owns repository-wide checks.