85 lines
4.1 KiB
Markdown
85 lines
4.1 KiB
Markdown
# Accepted Chunk Map
|
|
|
|
This document defines the optional durable `chunk-map.json` artifact in a
|
|
[published JSON bundle](json-output.md). It describes the materialized chunk
|
|
plan used by one run, including a supported qualified fallback. It is not a lane
|
|
payload and is never an input to a later pipeline step.
|
|
|
|
## Contract Identity
|
|
|
|
| Property | Value |
|
|
| --- | --- |
|
|
| Artifact kind | `source/chunk-map` |
|
|
| Logical file | `chunk-map.json` |
|
|
| Media type | `application/json` |
|
|
| Schema ID | `notarius.source.chunk_map` |
|
|
| Schema name | `notarius_source_chunk_map_v1` |
|
|
| Schema version | `v1` |
|
|
|
|
Read this logical path directly when enabled and available; no discovery
|
|
descriptor is required. Export is controlled by the JSON output binding in
|
|
[Configuration](../config.md#module-bindings-and-validators).
|
|
|
|
## Wire Shape
|
|
|
|
Every payload has these required fields:
|
|
|
|
| Field | Meaning |
|
|
| --- | --- |
|
|
| `source_id` | Accepted source-document identity. |
|
|
| `source_digest` | Lower-case `sha256:` digest of that source document. |
|
|
| `plan_digest` | Lower-case `sha256:` digest of the logical chunk plan. |
|
|
| `requested_chunker` | Chunk module selected by the resolved pipeline. |
|
|
| `producer` | Current run's plan component selections. `input_module` and `chunk_module` are required; `llm_profile` is optional and applies only to an LLM-backed chunker. Not historical cache execution provenance. |
|
|
| `plan_annotations` | Plan-level annotation namespace map; `{}` when none are present. |
|
|
| `chunks` | Non-empty execution-order chunk collection. |
|
|
|
|
Each `chunks` entry contains non-empty `id`, zero-based `index`, `source_ref`,
|
|
positive `unit_count`, and an explicit `annotations` map. `source_ref` contains
|
|
the same `source_id` as the top-level value plus positive inclusive
|
|
`start_unit_id` and `end_unit_id` values. Endpoints identify source units; their
|
|
numeric values do not by themselves establish source-document order.
|
|
|
|
Annotation namespaces are non-empty trimmed strings. Their values are arbitrary
|
|
valid JSON and are retained without interpreting a module-specific namespace.
|
|
|
|
## Ordering And Validation
|
|
|
|
`chunks` are in execution order. Their indexes are contiguous, start at zero,
|
|
and equal their array positions; chunk IDs are unique. The emitted map is built
|
|
after the selected plan has been materialized against the source document,
|
|
so its ranges, unit counts, annotations, and digests describe that exact plan.
|
|
Supported fallback still satisfies those invariants without asserting ordinary
|
|
semantic approval; run-level qualification belongs to the
|
|
[CLI report](../cli.md#completion-report).
|
|
|
|
The codec rejects malformed JSON, trailing content, unknown fixed-object
|
|
fields, invalid identities or digests, invalid annotations, duplicate chunk
|
|
IDs, non-contiguous indexes, and a `plan_digest` that does not match the
|
|
reconstructed logical plan. The checked-in
|
|
[schema](../../internal/framework/chunkmap/assets/schemas/source_chunk_map.v1.json)
|
|
defines the strict JSON shape.
|
|
|
|
## Valid Example
|
|
|
|
The compact
|
|
[source chunk-map fixture](../../internal/framework/chunkmap/testdata/source_chunk_map.v1.json)
|
|
is decoded by the production codec and demonstrates an accepted map with
|
|
annotations, producer identity, and ordered chunks.
|
|
|
|
## Publication And Compatibility
|
|
|
|
The map is published when export is enabled, a materialized plan is available,
|
|
and the run reaches publication. Payload fields and schema are unchanged.
|
|
`requested_chunker` and `producer` describe the current composition; unified
|
|
cache entries retain the logical plan, not an earlier run's producer/profile
|
|
ledger. Reuse rematerializes against the current source without semantic
|
|
reassessment or claims of historical model execution. A strict failure does not
|
|
publish a successful bundle merely to expose this auxiliary.
|
|
|
|
The map contains structure rather than source content: it excludes transcript
|
|
bytes, source-unit metadata, chunk text, private model output, reference
|
|
content, debug data, and filesystem paths. Treat the exported map with the
|
|
same care as other published output. Publication location and retention are
|
|
defined in [Operations](../operations.md#output-bundles).
|