From 4a91f29bcd6583fb77d7283f955a0220e7291f8c Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 7 Nov 2026 13:20:54 +0000 Subject: [PATCH] Record the documentation consistency audit --- docs/roadmap/audit/13-documentation.md | 258 +++++++++++++++++++++++++ 1 file changed, 258 insertions(+) create mode 100644 docs/roadmap/audit/13-documentation.md diff --git a/docs/roadmap/audit/13-documentation.md b/docs/roadmap/audit/13-documentation.md new file mode 100644 index 0000000..27e57db --- /dev/null +++ b/docs/roadmap/audit/13-documentation.md @@ -0,0 +1,258 @@ +# Documentation consistency audit + +Status: five open P3 documentation findings, plus cross-references to previously +recorded implementation discrepancies. Reviewed source: +`efb5f774176526bc6de74eca5f7472d078ea1704`. + +## Architecture and scope + +The [internal overview](../../internal/overview.md) describes the implemented +owners. CommandKit consists of three independently versioned Go modules: + +- Core owns captured command declarations, typed bindings and presence, + installation, alias routing, dispatch, completion, help/version, shared + text/YAML/JSON rendering, and invocation reporting and outcomes. Private + command metadata, path and alias packages connect the facade to an internal + Cobra/pflag adapter. +- ckconfig depends on core and ConfigKit v1.1.0. It assembles six standard + configuration commands from declared scope capabilities: scopes, validate, + show, sources, explain, and diff. Loaders supply configuration and evidence; + the integration selects report content and core renders it. +- ckprompt depends on core, ckconfig, ConfigKit v1.1.0, and PromptKit v0.20.0. + It provides optional catalogs, resource inspection, requirements discovery + and validation, prepared-request preview, smoke testing, reusable request + flags, and a configuration scope. Applications explicitly install these + capabilities and retain engine/client construction and executable lifecycle. + +Core has no ConfigKit or PromptKit dependency. Upstream libraries own domain +semantics; applications own contexts, streams, runtime effects, credentials, +and final exit mapping. The release launchers delegate source-only module +publication and optional hosted-page delivery to ReleaseKit. + +The review covered the 47 tracked Markdown files, public package/API comments, +maintained examples and fixtures, module metadata, release profiles and +launchers, and relevant implementation and tests. The +[documentation policy](../../policy/documentation.md), architecture, testing, +release, and applicable attribution rules supplied the review boundaries. +Codebase-memory-mcp discovery and source inspection were supplemented with +direct reads of documentation, examples, and tooling. + +The current shared reporting and PromptKit v0.20.0 features are implemented. +The ckprompt v0.3.0 candidate remains unpublished according to the current +release records; implementation and publication are different states. +Older release notes were evaluated as historical records against their local +tags, not as current CLI instructions. Retired roadmaps, completed remediation +descriptions, and explicitly deferred features were not treated as promises of +current functionality. + +## Validation evidence + +Validation used the cached Go 1.27.0 toolchain with `GOTOOLCHAIN=local`, +`GOPROXY=off`, `-mod=readonly`, and a temporary writable build cache. The initial +attempt to use the default cache encountered the sandbox's read-only filesystem; +the temporary cache resolved that environment limitation. + +- All default suites passed in the aligned workspace: + `go test -mod=readonly ./... ./integrations/ckconfig/... ./integrations/ckprompt/...`. + Each module's independent `GOWORK=off go test -mod=readonly ./...` also passed. +- All 45 `go run` examples in the three example READMEs and all 20 developer-guide + example invocations were exercised using binaries built from the aligned + workspace. Their selected arguments and working directories were preserved. + The documented invalid-input examples failed as intended, and the differing + `config diff --check` example returned status 3. Representative reports and + help were compared with the CLI reference. +- Temporary external-consumer probes substantiated D1 and D2 below. No + production files, existing documentation, or repository tests were changed. +- The local-link pass checked 334 Markdown link occurrences. The only apparent + missing path was the intentional `COMMIT_URL` attribution-format placeholder + in the OSS policy. Repository-hosted links were also checked against locally + available Git objects: their file paths existed; unpublished ckprompt footer + targets and the release URL template were excluded. Checking 11 referenced + section fragments in those objects found D4. +- Independent module graphs agreed with the versions listed in their attribution + entries. The declared Go requirements, release profiles, launchers, and + release-note dependency descriptions were checked against local metadata and + tags. No publication, delivery, credentialed network operation, or live model + call was attempted. External upstream URL availability was not tested. +- Default tests skip interactive shell completion. The consumer fixture's + script-generation step was run separately and reproduced the already recorded + reserved-flag failure C6; the full interactive shell matrix was not run. + +Passing default suites do not establish that the previously recorded exceptional +paths below are correct. They also do not validate every possible consumer +provider or upstream configuration schema. + +## D1 — P3: Configuration provider GoDoc omits the effective-output contract + +**Documentation:** [contracts.go](../../../integrations/ckconfig/contracts.go), +EffectiveWriter at lines 66–69 and View at 82–87. EffectiveWriter promises a +native effective representation without specifying its encoding. View still +describes exposing facets without reconstructing their reports. + +**Implementation:** [show.go](../../../integrations/ckconfig/show.go), +runConfigShow at lines 68–101, buffers WriteEffective, passes its bytes to +[DecodeYAMLReport](../../../report_value.go), and submits the decoded value to +the shared reporter. The +[CLI reference](../../cli.md#show-effective-configuration) already describes +conversion to a logical report value. + +**Discrepancy and consequence:** a custom scope implementer reading the owning +GoDoc is not told that WriteEffective must supply one supported YAML document, +or that its formatting is not passed through. Valid YAML is not sufficient in +every case: the converter rejects multiple documents, unsupported tags, and +non-string mapping keys. This is an undocumented provider requirement, rather +than evidence that all native YAML must be accepted. + +**Observed behavior:** a temporary custom EffectiveWriter emitting +`answer: 42` produced a JSON mapping under `config show --format=json` with +status 0. A two-document YAML stream and a mapping containing an explicitly +tagged `!!timestamp` both returned status 1, empty result output, and a rendering +error identifying the unsupported input. No acquisition error was needed. + +**Correct information and potential remediation:** describe WriteEffective as +supplying safe effective YAML for conversion through DecodeYAMLReport. Link that +helper as the common owner of the conversion contract, and make its supported +value boundary sufficiently explicit there. Update View's comment to describe +evidence used to build reports, rather than imply native report passthrough. +Do not promise preservation of comments, styles, or original framing, or broaden +YAML support merely to accommodate the old wording. + +The related unused WriteExplanation requirement is already tracked as +[M6](03-complexity.md#m6--p3-explanation-providers-must-implement-an-unused-rendering-capability). +Clarifying the current read-versus-render boundary does not require resolving +that API simplification in this documentation change. + +## D2 — P3: Public flag metadata omits enforced declaration constraints + +**Documentation:** [command.go](../../../command.go), FlagMeta.Name and Short +at lines 91–103, describe a name without leading dashes and an optional one-byte +short spelling. The [registered CLI syntax](../../cli.md#registered-command-syntax) +describes invocation grammar, not the additional declaration grammar. + +**Implementation:** validateBindings and validFlagName in the same file, +lines 477–545. Long names must be nonempty segments of Unicode lowercase +letters or digits separated by single hyphens. Leading, trailing, and repeated +hyphens are rejected. Core also rejects reserved help/reporting names and `h` +as a short spelling; `-` is not a valid short spelling. Declaration errors are +retained until installation. + +**Discrepancy and consequence:** the exact exported metadata contract leaves +consumers to discover ordinary naming restrictions through installation errors +or private implementation. Names such as `foo_bar` and `fooBar` satisfy the +published field description but cannot be installed. RenameFlag and flag-alias +declarations also depend on the long-name grammar. + +**Observed behavior:** temporary public-API constructions accepted `foo-bar`, +`123`, and `équipe`, but rejected `foo_bar`, `fooBar`, and `foo--bar` during New +with the lowercase/hyphen-separated-name error. + +**Correct information and potential remediation:** document the enforced naming +and reserved-spelling rules once with FlagMeta, and refer related public helpers +to that contract. Keep the CLI reference focused on operator syntax. Do not +accidentally introduce an ASCII-only or leading-letter restriction: neither is +enforced by the implementation. If the intended naming policy differs, surface +that as a deliberate API decision rather than silently changing validation. + +## D3 — P3: Reporting ADR still directs readers to a planned implementation + +**Documentation:** [ADR 0006](../../adr/0006-invocation-reporting.md), lines 5–7 +say shared formatting is implemented; line 54 leaves implementation status and +format choices in the now-retired roadmap; lines 76–77 still describe the +extension as planned until implementation and canonical contracts land. + +**Implementation and current contracts:** [report.go](../../../report.go) +exports Report, Format, and RenderReport; +[reporter.go](../../../reporter.go) integrates Report with invocation delivery; +the [CLI reporting contract](../../cli.md#reporting) and +[consumer guidance](../../consumers/guide.md#invocation-reporting) describe the +implemented formats. The default suites exercise these APIs. + +**Discrepancy:** the ADR mixes current implementation status with pre-implementation +wording and points to a historical roadmap for information now owned by current +API and CLI documentation. This can make a completed prerequisite look pending. + +**Correct information and potential remediation:** keep the accepted ownership +decision intact. A status clarification can explicitly identify the remaining +future-tense passages as historical and direct current behavior to GoDoc and +the CLI reference. If editorial cleanup of those passages is approved, remove +only stale lifecycle wording; the documentation policy's accepted-decision +immutability rule does not justify rewriting the architectural decision or +creating a new ADR for an unchanged decision. + +## D4 — P3: A historical release note links to a nonexistent section + +**Documentation:** [ckprompt v0.2.0 release note](../../../integrations/ckprompt/docs/releases/v0.2.0.md), +line 32, links to the consumer guide at the immutable +`integrations/ckprompt/v0.2.0` tag with fragment +`#promptkit-configuration-and-definitions`. + +**Evidence:** `git show integrations/ckprompt/v0.2.0:docs/consumers/guide.md` +contains the heading `PromptKit configuration, definitions, and requirements` +at line 244. The correct fragment is +`#promptkit-configuration-definitions-and-requirements`. The linked document +exists, but the requested section does not. The tagged modules' dependency +versions agree with the note; this is a navigation defect, not a claim that +historical dependencies should match current source. + +**Correct information and potential remediation:** provide the +[correct pinned section link](https://forge.maximumdirect.net/base-layer/commandkit/src/tag/integrations/ckprompt/v0.2.0/docs/consumers/guide.md#promptkit-configuration-definitions-and-requirements) +in a forward-only documentation correction or erratum where useful. Preserve +the published tag, note, and hosted release contents. This defect does not +warrant republishing the old version or backfilling hosted pages as part of the +audit. Future note preparation should check fragments against the target +revision as well as checking that the linked file exists. + +## D5 — P3: The root README duplicates the compatibility matrix + +**Documentation:** [README.md](../../../README.md), lines 25–30, repeats the +published core/integration pairings, the incompatible ckconfig v0.3.0 and +ckprompt v0.2.0 combination, and the current candidate dependency selection. +The [consumer guide](../../consumers/guide.md#module-selection), lines 18–29, +maintains those same details alongside installation guidance. + +**Implementation and policy:** the module requirements in +[core](../../../go.mod), [ckconfig](../../../integrations/ckconfig/go.mod), and +[ckprompt](../../../integrations/ckprompt/go.mod), plus locally available +release tags, corroborate the current descriptions. However, the +[one-owner rule](../../policy/documentation.md#one-canonical-owner) permits only +a short stable summary in a non-owning document and requires links for volatile +contract details. README owns orientation and current status, not a second +historical module compatibility reference. + +**Discrepancy:** the facts currently agree; the finding is duplicated ownership +under the adopted policy. The README's existing link does not remove the need +to update its separate multi-version matrix whenever compatibility guidance +changes. + +**Potential remediation:** retain a brief current-status statement and the module +overview in README, and make the consumer guide the single maintained source of +exact version combinations and installation choices. This requires no dependency +change and no invented compatibility guarantee. Historical release notes remain +historical records rather than duplicate current reference documents. + +## Previously recorded discrepancies affecting documented behavior + +These remain relevant to this audit but are not additional documentation +findings. Their detailed triggers and remediation belong in the earlier audits. +Current implementation paths were inspected; the completion-fixture failure was +also reproduced here. Do not weaken the selected documentation contracts merely +to make them describe these defects. + +| Current documentation or promised behavior | Existing implementation finding | +| --- | --- | +| Reusable parsed inputs and per-kind smoke seeds preserve the operator's selected values. | [C1](01-correctness.md#c1--p2-cli-provider-parameter-objects-leak-mutations-between-requests): mutation of parsed provider-parameter maps can cross replay/aggregate boundaries. | +| Concurrent reporter submissions and deferred delivery follow the documented invocation lifecycle. | [C2](01-correctness.md#c2--p2-finalization-can-discard-a-successfully-queued-deferred-message) and [N2](04-concurrency.md#n2--p2-deferred-message-acceptance-remains-open-after-the-final-batch-detaches): a deferred submission can return success during finalization and be discarded. | +| Prepared-request preview returns the prepared request; aggregate smoke success covers all requested kinds. | [C3](01-correctness.md#c3--p2-prepared-request-inspection-reports-success-without-a-result) permits null successful preview output; [N1](04-concurrency.md#n1--p2-cancellation-can-stop-aggregate-smoke-testing-with-a-success-outcome) can return success after cancellation stops an aggregate. | +| Operation cancellation and presentation failures remain separately observable. | [C4](01-correctness.md#c4--p2-show-preparation-labels-ordinary-cancellation-as-a-rendering-error) and [N3](04-concurrency.md#n3--p2-buffered-configuration-show-cancellation-becomes-a-presentation-failure): a buffered-show cancellation is also recorded as a rendering failure. | +| The developer guide's optional completion consumer check can reach shell validation. | [C6](01-correctness.md#c6--p3-the-interactive-completion-consumer-cannot-construct-its-app): the fixture declares the now-reserved `format` flag. Direct script generation still fails during App construction, before any interactive shell check. | + +## Assessment + +The principal current CLI descriptions, maintained README commands, module +requirements, report defaults, and release-launcher instructions are consistent +with the reviewed implementation on the exercised paths. The new findings are +documentation clarifications and maintenance issues, not reasons to redesign +the command suite or extend upstream functionality. Prioritize the public +provider and naming contracts, then reconcile status/navigation and duplicated +compatibility guidance. The previously recorded runtime defects remain separate +implementation work.