Record the documentation consistency audit
This commit is contained in:
258
docs/roadmap/audit/13-documentation.md
Normal file
258
docs/roadmap/audit/13-documentation.md
Normal file
@@ -0,0 +1,258 @@
|
|||||||
|
# Documentation consistency audit
|
||||||
|
|
||||||
|
Status: five open P3 documentation findings, plus cross-references to previously
|
||||||
|
recorded implementation discrepancies. Reviewed source:
|
||||||
|
`efb5f774176526bc6de74eca5f7472d078ea1704`.
|
||||||
|
|
||||||
|
## Architecture and scope
|
||||||
|
|
||||||
|
The [internal overview](../../internal/overview.md) describes the implemented
|
||||||
|
owners. CommandKit consists of three independently versioned Go modules:
|
||||||
|
|
||||||
|
- Core owns captured command declarations, typed bindings and presence,
|
||||||
|
installation, alias routing, dispatch, completion, help/version, shared
|
||||||
|
text/YAML/JSON rendering, and invocation reporting and outcomes. Private
|
||||||
|
command metadata, path and alias packages connect the facade to an internal
|
||||||
|
Cobra/pflag adapter.
|
||||||
|
- ckconfig depends on core and ConfigKit v1.1.0. It assembles six standard
|
||||||
|
configuration commands from declared scope capabilities: scopes, validate,
|
||||||
|
show, sources, explain, and diff. Loaders supply configuration and evidence;
|
||||||
|
the integration selects report content and core renders it.
|
||||||
|
- ckprompt depends on core, ckconfig, ConfigKit v1.1.0, and PromptKit v0.20.0.
|
||||||
|
It provides optional catalogs, resource inspection, requirements discovery
|
||||||
|
and validation, prepared-request preview, smoke testing, reusable request
|
||||||
|
flags, and a configuration scope. Applications explicitly install these
|
||||||
|
capabilities and retain engine/client construction and executable lifecycle.
|
||||||
|
|
||||||
|
Core has no ConfigKit or PromptKit dependency. Upstream libraries own domain
|
||||||
|
semantics; applications own contexts, streams, runtime effects, credentials,
|
||||||
|
and final exit mapping. The release launchers delegate source-only module
|
||||||
|
publication and optional hosted-page delivery to ReleaseKit.
|
||||||
|
|
||||||
|
The review covered the 47 tracked Markdown files, public package/API comments,
|
||||||
|
maintained examples and fixtures, module metadata, release profiles and
|
||||||
|
launchers, and relevant implementation and tests. The
|
||||||
|
[documentation policy](../../policy/documentation.md), architecture, testing,
|
||||||
|
release, and applicable attribution rules supplied the review boundaries.
|
||||||
|
Codebase-memory-mcp discovery and source inspection were supplemented with
|
||||||
|
direct reads of documentation, examples, and tooling.
|
||||||
|
|
||||||
|
The current shared reporting and PromptKit v0.20.0 features are implemented.
|
||||||
|
The ckprompt v0.3.0 candidate remains unpublished according to the current
|
||||||
|
release records; implementation and publication are different states.
|
||||||
|
Older release notes were evaluated as historical records against their local
|
||||||
|
tags, not as current CLI instructions. Retired roadmaps, completed remediation
|
||||||
|
descriptions, and explicitly deferred features were not treated as promises of
|
||||||
|
current functionality.
|
||||||
|
|
||||||
|
## Validation evidence
|
||||||
|
|
||||||
|
Validation used the cached Go 1.27.0 toolchain with `GOTOOLCHAIN=local`,
|
||||||
|
`GOPROXY=off`, `-mod=readonly`, and a temporary writable build cache. The initial
|
||||||
|
attempt to use the default cache encountered the sandbox's read-only filesystem;
|
||||||
|
the temporary cache resolved that environment limitation.
|
||||||
|
|
||||||
|
- All default suites passed in the aligned workspace:
|
||||||
|
`go test -mod=readonly ./... ./integrations/ckconfig/... ./integrations/ckprompt/...`.
|
||||||
|
Each module's independent `GOWORK=off go test -mod=readonly ./...` also passed.
|
||||||
|
- All 45 `go run` examples in the three example READMEs and all 20 developer-guide
|
||||||
|
example invocations were exercised using binaries built from the aligned
|
||||||
|
workspace. Their selected arguments and working directories were preserved.
|
||||||
|
The documented invalid-input examples failed as intended, and the differing
|
||||||
|
`config diff --check` example returned status 3. Representative reports and
|
||||||
|
help were compared with the CLI reference.
|
||||||
|
- Temporary external-consumer probes substantiated D1 and D2 below. No
|
||||||
|
production files, existing documentation, or repository tests were changed.
|
||||||
|
- The local-link pass checked 334 Markdown link occurrences. The only apparent
|
||||||
|
missing path was the intentional `COMMIT_URL` attribution-format placeholder
|
||||||
|
in the OSS policy. Repository-hosted links were also checked against locally
|
||||||
|
available Git objects: their file paths existed; unpublished ckprompt footer
|
||||||
|
targets and the release URL template were excluded. Checking 11 referenced
|
||||||
|
section fragments in those objects found D4.
|
||||||
|
- Independent module graphs agreed with the versions listed in their attribution
|
||||||
|
entries. The declared Go requirements, release profiles, launchers, and
|
||||||
|
release-note dependency descriptions were checked against local metadata and
|
||||||
|
tags. No publication, delivery, credentialed network operation, or live model
|
||||||
|
call was attempted. External upstream URL availability was not tested.
|
||||||
|
- Default tests skip interactive shell completion. The consumer fixture's
|
||||||
|
script-generation step was run separately and reproduced the already recorded
|
||||||
|
reserved-flag failure C6; the full interactive shell matrix was not run.
|
||||||
|
|
||||||
|
Passing default suites do not establish that the previously recorded exceptional
|
||||||
|
paths below are correct. They also do not validate every possible consumer
|
||||||
|
provider or upstream configuration schema.
|
||||||
|
|
||||||
|
## D1 — P3: Configuration provider GoDoc omits the effective-output contract
|
||||||
|
|
||||||
|
**Documentation:** [contracts.go](../../../integrations/ckconfig/contracts.go),
|
||||||
|
EffectiveWriter at lines 66–69 and View at 82–87. EffectiveWriter promises a
|
||||||
|
native effective representation without specifying its encoding. View still
|
||||||
|
describes exposing facets without reconstructing their reports.
|
||||||
|
|
||||||
|
**Implementation:** [show.go](../../../integrations/ckconfig/show.go),
|
||||||
|
runConfigShow at lines 68–101, buffers WriteEffective, passes its bytes to
|
||||||
|
[DecodeYAMLReport](../../../report_value.go), and submits the decoded value to
|
||||||
|
the shared reporter. The
|
||||||
|
[CLI reference](../../cli.md#show-effective-configuration) already describes
|
||||||
|
conversion to a logical report value.
|
||||||
|
|
||||||
|
**Discrepancy and consequence:** a custom scope implementer reading the owning
|
||||||
|
GoDoc is not told that WriteEffective must supply one supported YAML document,
|
||||||
|
or that its formatting is not passed through. Valid YAML is not sufficient in
|
||||||
|
every case: the converter rejects multiple documents, unsupported tags, and
|
||||||
|
non-string mapping keys. This is an undocumented provider requirement, rather
|
||||||
|
than evidence that all native YAML must be accepted.
|
||||||
|
|
||||||
|
**Observed behavior:** a temporary custom EffectiveWriter emitting
|
||||||
|
`answer: 42` produced a JSON mapping under `config show --format=json` with
|
||||||
|
status 0. A two-document YAML stream and a mapping containing an explicitly
|
||||||
|
tagged `!!timestamp` both returned status 1, empty result output, and a rendering
|
||||||
|
error identifying the unsupported input. No acquisition error was needed.
|
||||||
|
|
||||||
|
**Correct information and potential remediation:** describe WriteEffective as
|
||||||
|
supplying safe effective YAML for conversion through DecodeYAMLReport. Link that
|
||||||
|
helper as the common owner of the conversion contract, and make its supported
|
||||||
|
value boundary sufficiently explicit there. Update View's comment to describe
|
||||||
|
evidence used to build reports, rather than imply native report passthrough.
|
||||||
|
Do not promise preservation of comments, styles, or original framing, or broaden
|
||||||
|
YAML support merely to accommodate the old wording.
|
||||||
|
|
||||||
|
The related unused WriteExplanation requirement is already tracked as
|
||||||
|
[M6](03-complexity.md#m6--p3-explanation-providers-must-implement-an-unused-rendering-capability).
|
||||||
|
Clarifying the current read-versus-render boundary does not require resolving
|
||||||
|
that API simplification in this documentation change.
|
||||||
|
|
||||||
|
## D2 — P3: Public flag metadata omits enforced declaration constraints
|
||||||
|
|
||||||
|
**Documentation:** [command.go](../../../command.go), FlagMeta.Name and Short
|
||||||
|
at lines 91–103, describe a name without leading dashes and an optional one-byte
|
||||||
|
short spelling. The [registered CLI syntax](../../cli.md#registered-command-syntax)
|
||||||
|
describes invocation grammar, not the additional declaration grammar.
|
||||||
|
|
||||||
|
**Implementation:** validateBindings and validFlagName in the same file,
|
||||||
|
lines 477–545. Long names must be nonempty segments of Unicode lowercase
|
||||||
|
letters or digits separated by single hyphens. Leading, trailing, and repeated
|
||||||
|
hyphens are rejected. Core also rejects reserved help/reporting names and `h`
|
||||||
|
as a short spelling; `-` is not a valid short spelling. Declaration errors are
|
||||||
|
retained until installation.
|
||||||
|
|
||||||
|
**Discrepancy and consequence:** the exact exported metadata contract leaves
|
||||||
|
consumers to discover ordinary naming restrictions through installation errors
|
||||||
|
or private implementation. Names such as `foo_bar` and `fooBar` satisfy the
|
||||||
|
published field description but cannot be installed. RenameFlag and flag-alias
|
||||||
|
declarations also depend on the long-name grammar.
|
||||||
|
|
||||||
|
**Observed behavior:** temporary public-API constructions accepted `foo-bar`,
|
||||||
|
`123`, and `équipe`, but rejected `foo_bar`, `fooBar`, and `foo--bar` during New
|
||||||
|
with the lowercase/hyphen-separated-name error.
|
||||||
|
|
||||||
|
**Correct information and potential remediation:** document the enforced naming
|
||||||
|
and reserved-spelling rules once with FlagMeta, and refer related public helpers
|
||||||
|
to that contract. Keep the CLI reference focused on operator syntax. Do not
|
||||||
|
accidentally introduce an ASCII-only or leading-letter restriction: neither is
|
||||||
|
enforced by the implementation. If the intended naming policy differs, surface
|
||||||
|
that as a deliberate API decision rather than silently changing validation.
|
||||||
|
|
||||||
|
## D3 — P3: Reporting ADR still directs readers to a planned implementation
|
||||||
|
|
||||||
|
**Documentation:** [ADR 0006](../../adr/0006-invocation-reporting.md), lines 5–7
|
||||||
|
say shared formatting is implemented; line 54 leaves implementation status and
|
||||||
|
format choices in the now-retired roadmap; lines 76–77 still describe the
|
||||||
|
extension as planned until implementation and canonical contracts land.
|
||||||
|
|
||||||
|
**Implementation and current contracts:** [report.go](../../../report.go)
|
||||||
|
exports Report, Format, and RenderReport;
|
||||||
|
[reporter.go](../../../reporter.go) integrates Report with invocation delivery;
|
||||||
|
the [CLI reporting contract](../../cli.md#reporting) and
|
||||||
|
[consumer guidance](../../consumers/guide.md#invocation-reporting) describe the
|
||||||
|
implemented formats. The default suites exercise these APIs.
|
||||||
|
|
||||||
|
**Discrepancy:** the ADR mixes current implementation status with pre-implementation
|
||||||
|
wording and points to a historical roadmap for information now owned by current
|
||||||
|
API and CLI documentation. This can make a completed prerequisite look pending.
|
||||||
|
|
||||||
|
**Correct information and potential remediation:** keep the accepted ownership
|
||||||
|
decision intact. A status clarification can explicitly identify the remaining
|
||||||
|
future-tense passages as historical and direct current behavior to GoDoc and
|
||||||
|
the CLI reference. If editorial cleanup of those passages is approved, remove
|
||||||
|
only stale lifecycle wording; the documentation policy's accepted-decision
|
||||||
|
immutability rule does not justify rewriting the architectural decision or
|
||||||
|
creating a new ADR for an unchanged decision.
|
||||||
|
|
||||||
|
## D4 — P3: A historical release note links to a nonexistent section
|
||||||
|
|
||||||
|
**Documentation:** [ckprompt v0.2.0 release note](../../../integrations/ckprompt/docs/releases/v0.2.0.md),
|
||||||
|
line 32, links to the consumer guide at the immutable
|
||||||
|
`integrations/ckprompt/v0.2.0` tag with fragment
|
||||||
|
`#promptkit-configuration-and-definitions`.
|
||||||
|
|
||||||
|
**Evidence:** `git show integrations/ckprompt/v0.2.0:docs/consumers/guide.md`
|
||||||
|
contains the heading `PromptKit configuration, definitions, and requirements`
|
||||||
|
at line 244. The correct fragment is
|
||||||
|
`#promptkit-configuration-definitions-and-requirements`. The linked document
|
||||||
|
exists, but the requested section does not. The tagged modules' dependency
|
||||||
|
versions agree with the note; this is a navigation defect, not a claim that
|
||||||
|
historical dependencies should match current source.
|
||||||
|
|
||||||
|
**Correct information and potential remediation:** provide the
|
||||||
|
[correct pinned section link](https://forge.maximumdirect.net/base-layer/commandkit/src/tag/integrations/ckprompt/v0.2.0/docs/consumers/guide.md#promptkit-configuration-definitions-and-requirements)
|
||||||
|
in a forward-only documentation correction or erratum where useful. Preserve
|
||||||
|
the published tag, note, and hosted release contents. This defect does not
|
||||||
|
warrant republishing the old version or backfilling hosted pages as part of the
|
||||||
|
audit. Future note preparation should check fragments against the target
|
||||||
|
revision as well as checking that the linked file exists.
|
||||||
|
|
||||||
|
## D5 — P3: The root README duplicates the compatibility matrix
|
||||||
|
|
||||||
|
**Documentation:** [README.md](../../../README.md), lines 25–30, repeats the
|
||||||
|
published core/integration pairings, the incompatible ckconfig v0.3.0 and
|
||||||
|
ckprompt v0.2.0 combination, and the current candidate dependency selection.
|
||||||
|
The [consumer guide](../../consumers/guide.md#module-selection), lines 18–29,
|
||||||
|
maintains those same details alongside installation guidance.
|
||||||
|
|
||||||
|
**Implementation and policy:** the module requirements in
|
||||||
|
[core](../../../go.mod), [ckconfig](../../../integrations/ckconfig/go.mod), and
|
||||||
|
[ckprompt](../../../integrations/ckprompt/go.mod), plus locally available
|
||||||
|
release tags, corroborate the current descriptions. However, the
|
||||||
|
[one-owner rule](../../policy/documentation.md#one-canonical-owner) permits only
|
||||||
|
a short stable summary in a non-owning document and requires links for volatile
|
||||||
|
contract details. README owns orientation and current status, not a second
|
||||||
|
historical module compatibility reference.
|
||||||
|
|
||||||
|
**Discrepancy:** the facts currently agree; the finding is duplicated ownership
|
||||||
|
under the adopted policy. The README's existing link does not remove the need
|
||||||
|
to update its separate multi-version matrix whenever compatibility guidance
|
||||||
|
changes.
|
||||||
|
|
||||||
|
**Potential remediation:** retain a brief current-status statement and the module
|
||||||
|
overview in README, and make the consumer guide the single maintained source of
|
||||||
|
exact version combinations and installation choices. This requires no dependency
|
||||||
|
change and no invented compatibility guarantee. Historical release notes remain
|
||||||
|
historical records rather than duplicate current reference documents.
|
||||||
|
|
||||||
|
## Previously recorded discrepancies affecting documented behavior
|
||||||
|
|
||||||
|
These remain relevant to this audit but are not additional documentation
|
||||||
|
findings. Their detailed triggers and remediation belong in the earlier audits.
|
||||||
|
Current implementation paths were inspected; the completion-fixture failure was
|
||||||
|
also reproduced here. Do not weaken the selected documentation contracts merely
|
||||||
|
to make them describe these defects.
|
||||||
|
|
||||||
|
| Current documentation or promised behavior | Existing implementation finding |
|
||||||
|
| --- | --- |
|
||||||
|
| Reusable parsed inputs and per-kind smoke seeds preserve the operator's selected values. | [C1](01-correctness.md#c1--p2-cli-provider-parameter-objects-leak-mutations-between-requests): mutation of parsed provider-parameter maps can cross replay/aggregate boundaries. |
|
||||||
|
| Concurrent reporter submissions and deferred delivery follow the documented invocation lifecycle. | [C2](01-correctness.md#c2--p2-finalization-can-discard-a-successfully-queued-deferred-message) and [N2](04-concurrency.md#n2--p2-deferred-message-acceptance-remains-open-after-the-final-batch-detaches): a deferred submission can return success during finalization and be discarded. |
|
||||||
|
| Prepared-request preview returns the prepared request; aggregate smoke success covers all requested kinds. | [C3](01-correctness.md#c3--p2-prepared-request-inspection-reports-success-without-a-result) permits null successful preview output; [N1](04-concurrency.md#n1--p2-cancellation-can-stop-aggregate-smoke-testing-with-a-success-outcome) can return success after cancellation stops an aggregate. |
|
||||||
|
| Operation cancellation and presentation failures remain separately observable. | [C4](01-correctness.md#c4--p2-show-preparation-labels-ordinary-cancellation-as-a-rendering-error) and [N3](04-concurrency.md#n3--p2-buffered-configuration-show-cancellation-becomes-a-presentation-failure): a buffered-show cancellation is also recorded as a rendering failure. |
|
||||||
|
| The developer guide's optional completion consumer check can reach shell validation. | [C6](01-correctness.md#c6--p3-the-interactive-completion-consumer-cannot-construct-its-app): the fixture declares the now-reserved `format` flag. Direct script generation still fails during App construction, before any interactive shell check. |
|
||||||
|
|
||||||
|
## Assessment
|
||||||
|
|
||||||
|
The principal current CLI descriptions, maintained README commands, module
|
||||||
|
requirements, report defaults, and release-launcher instructions are consistent
|
||||||
|
with the reviewed implementation on the exercised paths. The new findings are
|
||||||
|
documentation clarifications and maintenance issues, not reasons to redesign
|
||||||
|
the command suite or extend upstream functionality. Prioritize the public
|
||||||
|
provider and naming contracts, then reconcile status/navigation and duplicated
|
||||||
|
compatibility guidance. The previously recorded runtime defects remain separate
|
||||||
|
implementation work.
|
||||||
Reference in New Issue
Block a user