diff --git a/docs/roadmap/example-testing.md b/docs/roadmap/example-testing.md new file mode 100644 index 0000000..c87550f --- /dev/null +++ b/docs/roadmap/example-testing.md @@ -0,0 +1,119 @@ +# Go-Based Testing For Maintained Examples + +## Purpose + +Move the maintained consumer-example checks into Promptkit's ordinary Go test +suite. Developers and ReleaseKit should exercise these workflows through the +same testing entry point, without a separate Python example runner or custom +release-validation command. + +Keep the examples useful as readable, copyable consumer guidance. The objective +is simpler ownership and execution of tests, not a language-only translation or +a particular reduction in line count. This is a tooling change, not a new public +library API or a new release gate. + +## Current Baseline + +The [preparation](../../examples/go-library/prepare/main.go), +[execution](../../examples/go-library/run/main.go), and +[observation](../../examples/go-library/observe/main.go) examples are Go programs. +Their deterministic clients and transport keep them independent of live +providers. Preparation and execution use repository-relative configuration +paths; observation embeds its assets and already has a workflow helper returning +a summary and error. + +[scripts/check-examples.py](../../scripts/check-examples.py) launches all three +programs, decodes their JSON summaries, and checks meaningful results. Ordinary +`go test ./...` compiles the example packages but currently has no example tests +to execute. The [ReleaseKit profile](../../releasekit.json) runs the Python +script as its one custom check after shared validation. + +The separate [release-wiring test](../../scripts/test-release.py) is also Python, +but is a development-only diagnostic, not a release gate. Its disposable fixture +currently assumes the example script is a configured product check. + +## Target State + +Ordinary Go tests alongside the examples exercise the actual example workflows. +They are discovered by `go test ./...` and can be selected with +`go test ./examples/go-library/...`. ReleaseKit's existing standard suite runs +them with its selected race and vet settings; no second example-validation pass +is needed. + +Preserve the useful coverage currently supplied by the Python runner: + +- Preparation: collection composition and overrides, scoped selection, + inspection/preparation, and the resulting identities and inventories. +- Execution: injected-client execution, expected output and structural validity, + model selection, and usage reporting. +- Observation: bounded transient retries, attempt metadata, incomplete usage + reporting, and application-owned interpretation of proposal/assessment results. + +Assertions should remain tied to the synthetic examples and meaningful results. +Their output formatting and fixture values do not become new public API +guarantees. Preserve relevant output coverage without freezing complete JSON +documents or introducing broad golden snapshots. + +## Example And Test Structure + +Prefer small, unexported workflow helpers that return errors or results and, +where useful, accept an output writer. Keep `main` responsible for process-level +output and exit handling. Reuse the observation example's existing separation +where it already meets the need; uniform helper signatures are not an objective. + +Tests should normally call the workflow directly, using local output buffers +and the existing deterministic collaborators. Handle resource paths explicitly +so package-directory test execution and the documented repository-root +`go run` commands both work. A small resource-base or filesystem input is +preferable to changing process-wide working directories or redirecting global +standard output merely to reach the workflow. + +Use a Go subprocess test only where launching the executable protects a distinct +risk or avoids making an example materially harder to understand. Do not add +one for every example automatically. Subprocesses need their own consideration +of toolchain, offline execution, and instrumentation; parent race or coverage +settings do not automatically instrument a separately launched program. + +Keep tests deterministic, offline, and independent of real credentials under the +[testing policy](../policy/testing.md). Avoid a new public package, shared +example framework, or expanded failure-case inventory solely for this migration. +Examples remain consumer programs rather than release artifacts or library +implementation dependencies, consistent with the +[architecture policy](../policy/architecture.md). + +## Integration And Documentation Work + +Once the Go tests cover the retained behavior, remove `scripts/check-examples.py` +and its custom command and environment overlay from `releasekit.json`. Keep the +existing built-in validation selections and source/hosted publication workflow. +There is no need for new ReleaseKit functionality or a dependency upgrade. + +Adapt the development-only release-wiring fixture so it no longer expects the +removed script or product-check marker. Retain checks relevant to the actual +launcher and profile; do not manufacture an unused custom hook merely to retain +old fixture assertions. The Python release-wiring diagnostic can remain Python. +Removing Python from every repository tool is outside this scope. + +Update the [development guide](../development.md), +[testing policy](../policy/testing.md), [release policy](../policy/release.md), +and [release procedure](../release.md) when implementation lands. Replace the +standalone example command with Go test guidance, describe example coverage as +part of the standard suite, and remove Python as a normal release prerequisite. +Continue documenting Python where the optional release-wiring test needs it. +Check other incoming references to the removed runner and retain the existing +consumer quickstart commands. + +## Verification + +Verify the example packages through ordinary Go tests and the selected +race-enabled suite and vet. Confirm the documented `go run` entry points still +work from the repository root, and exercise the adapted release-wiring test in +disposable repositories. These are migration checks, not new recurring release +steps. Check that a meaningful example assertion failure fails the Go suite +rather than relying only on successful compilation. + +Completion means useful example behavior remains covered, normal test execution +includes that coverage, and the obsolete runner, profile hook, and documentation +references are removed. No performance thresholds, coverage percentage targets, +new platform commitments, hosted CI requirements, or changes to Promptkit's +consumer contracts are introduced by this work.