base-layer/commandkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Migrate PromptKit reports to shared rendering

Browse Source
This commit is contained in:
Eric Rakestraw
2026-11-07 04:10:47 +00:00
parent 5ff0dd4ea2
commit 6378ceb3c5
15 changed files with 249 additions and 265 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
docs/cli.md
View File

@@ -577,7 +577,7 @@ The command accepts:
| `--profile` | string | empty | Pass the profile ID to PromptKit; empty uses its native selection rule. |
| `--session-id` | string | empty | Pass session context to PromptKit. |
| `--var` | repeatable `KEY=VALUE` | none | Supply caller-variable context for native discovery. |
| `--format` | `yaml` | `yaml` | Select the requirements report format. Other formats are usage errors for this operation. |
| `--format` | `text`, `yaml`, `json` | automatic | Select the requirements report format; automatic and text use YAML for this structured report. |
The `promptkit` namespace identifies the operation, so there is no built-in
`--scope` flag. An explicitly empty profile or session value retains PromptKit's
@@ -594,7 +594,7 @@ values, whitespace, commas, and additional `=` characters are retained. All
occurrences are parsed; duplicate keys use the last value, while any malformed
occurrence is a usage error even if a later assignment is valid.
The YAML report contains selected identities, native completeness, and input,
The report contains selected identities, native completeness, and input,
variable, and issue inventories. False values and empty inventories are
retained. Input metadata preserves null versus an empty string. Variable entries
contain `name`, `kind`, `supplied`, and `explanation`. `supplied` reports native
@@ -659,7 +659,7 @@ The command accepts:
| `--session-id` | string | empty | Pass session context to PromptKit. |
| `--var` | repeatable `KEY=VALUE` | none | Supply caller variables for the concrete request. |
| `--input` | repeatable `NAME=PATH` | none | Bind a file-backed PromptKit artifact. |
| `--format` | `yaml` | `yaml` | Select the success confirmation format. Other formats are usage errors. |
| `--format` | `text`, `yaml`, `json` | automatic | Select the success confirmation format; automatic and text use YAML for this structured report. |
The built-in flags other than repeatable `--var` and `--input` are singleton
bindings; when repeated, the last value is used. Explicit empty `--config` or
@@ -674,7 +674,7 @@ native validation. File paths become `promptkit.File(PATH)` references without
being read, normalized, expanded, or rebased during argument binding. Duplicate
keys or names use the last value, but every occurrence is parsed.
On success, the YAML confirmation contains only the requested prompt ID and
On success, the confirmation contains only the requested prompt ID and
`valid: true`:
```yaml
@@ -712,8 +712,10 @@ example promptkit list collections
```
All catalogs accept `--config`, `--overlay`, applicable flags from
`Suite.Selectors`, and `--format yaml` (the default). Other formats are usage
errors. Prompt, profile, and backend listings accept namespace selection:
`Suite.Selectors`, and the shared `--format` flag. Automatic output uses text
for nonempty string lists and YAML for other inventories; explicit `text` uses
the same rendering for structured inventories. Prompt, profile, and backend
listings accept namespace selection:
| Binding | Effect |
| --- | --- |
@@ -729,8 +731,8 @@ before provider work. Namespace inventory accepts repeatable `--kind KIND`;
omitting `--kind` forwards PromptKit's native default kind selection. Repeated
values are forwarded in order. Collections have no namespace or kind filter.
Each operation writes one bare YAML sequence, retaining provider order and
native identities:
Each operation reports one bare sequence, retaining provider order and native
identities. YAML examples follow:
- Prompt entries are `{id}`.
- Profiles, backends, and namespaces are strings.
@@ -785,7 +787,7 @@ example promptkit test authentication --backend local
| `--reasoning-effort` | string | omitted | Override execution reasoning effort. |
| `--api-key-env` | string | omitted | Supply an API-key environment-variable name to native configuration. |
| `--transient-retries` | integer | omitted | Set the native inference transient retry count. |
| `--format` | `yaml` | `yaml` | Select the report format. Other formats are usage errors. |
| `--format` | `text`, `yaml`, `json` | automatic | Select the report format; automatic and text use YAML for these structured reports. |
The execution flags populate PromptKit's native execution override. The
`--api-key-env` value is a variable name, not a credential value. Application
@@ -795,7 +797,7 @@ builder forwards the seed; a supplied builder's complete native request is
authoritative. The helper retains a native report when PromptKit returns both a
report and an error.
A single report projects the native smoke report into YAML. It preserves native
A single report projects the native smoke report. It preserves native
false, unknown/null, and incomplete values and does not add a readiness score.
The report includes `kind`, `supported`, `applicable`, `completed`, `reason`,
`target`, `facts`, `timing`, `attempts`, `usage`, and `failure`. Its target
@@ -865,7 +867,7 @@ usage:
failure: ""
```
Aggregate output is one YAML mapping with ordered `tests` entries. Each entry
Aggregate output is one mapping with ordered `tests` entries. Each entry
has selected `kind`, nullable `report`, and optional `error` text. The inner
report keeps its native `kind`. This cancellation example shows two attempted
operations with no reports; later kinds are absent because cancellation stops

6
docs/roadmap/implementation.md
View File

@@ -1,6 +1,6 @@
# Shared reporting and PromptKit v0.20.0 implementation plan
Status: implementation in progress; Stages 1–8 complete.
Status: implementation in progress; Stages 1–9 complete.
## Scope, authority, and execution
@@ -323,6 +323,10 @@ native evidence rather than rechecking upstream inheritance or rendering.
## Stage 9: Consolidate remaining ckprompt reports
**Complete.** Requirements, catalogs, and smoke reports use shared rendering.
The root, ckconfig, and ckprompt workspace suites passed with Go 1.27.0,
including smoke evidence, cancellation, quiet, and delivery-failure tests.
Move requirements show/validate, all catalogs, and smoke reports onto core
rendering. Remove remaining local format bindings, allowlists, encoders, and
format-specific report paths. Preserve selected report content with JSON tags

146
integrations/ckprompt/command.go
View File

@@ -1,17 +1,14 @@
package ckprompt
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"strings"
"go.maximumdirect.net/ckconfig"
commandkit "go.maximumdirect.net/commandkit"
promptkit "go.maximumdirect.net/promptkit"
"go.yaml.in/yaml/v4"
)
type definitionInput[I any] struct {
@@ -27,7 +24,6 @@ type catalogInput[I any] struct {
allNamespaces commandkit.Supplied[bool]
kinds []promptkit.ResourceKind
filter promptkit.NamespaceFilter
format string
captured commandkit.FlagValues[I]
selected I
}
@@ -35,7 +31,6 @@ type catalogInput[I any] struct {
type requirementsInput[I any] struct {
selection ckconfig.Selection
inspection promptkit.RequirementInspectionRequest
format string
captured commandkit.FlagValues[I]
selected I
}
@@ -138,8 +133,6 @@ func newCatalogListCommand[I, S any](suite Suite[I, S], name string) commandkit.
func(input **catalogInput[I], value commandkit.Supplied[string]) { (*input).selection.Config = value }),
commandkit.ValueFlag(commandkit.FlagMeta{Name: "overlay", Usage: "Configuration overlay path.", Completion: commandkit.CompleteFiles()}, (*string)(nil), parseDefinitionString,
func(input **catalogInput[I], value commandkit.Supplied[string]) { (*input).selection.Overlay = value }),
commandkit.ValueFlag(commandkit.FlagMeta{Name: "format", Usage: "Catalog format: yaml.", Completion: completionChoices("yaml")}, stringDefault("yaml"), parseDefinitionString,
func(input **catalogInput[I], value commandkit.Supplied[string]) { (*input).format = value.Value }),
)
if name == "prompts" || name == "profiles" || name == "backends" {
flags = append(flags,
@@ -169,9 +162,6 @@ func newCatalogListCommand[I, S any](suite Suite[I, S], name string) commandkit.
if input == nil {
return errors.New("PromptKit catalog input must not be nil")
}
if err := checkCatalogFormat(input.format); err != nil {
return err
}
if err := prepareCatalogFilter(input, name); err != nil {
return err
}
@@ -185,21 +175,21 @@ func newCatalogListCommand[I, S any](suite Suite[I, S], name string) commandkit.
Run: func(invocation commandkit.Invocation, input *catalogInput[I]) commandkit.Outcome {
switch name {
case "prompts":
return runCatalogList(invocation, suite, input, name, suite.ListPrompts, encodePromptCatalog)
return runCatalogList(invocation, suite, input, name, suite.ListPrompts, promptCatalogReport)
case "profiles":
return runCatalogList(invocation, suite, input, name, suite.ListProfiles, encodeProfileCatalog)
return runCatalogList(invocation, suite, input, name, suite.ListProfiles, stringCatalogReport)
case "backends":
return runCatalogList(invocation, suite, input, name, suite.ListBackends, encodeProfileCatalog)
return runCatalogList(invocation, suite, input, name, suite.ListBackends, stringCatalogReport)
case "namespaces":
list := func(ctx context.Context, request CatalogRequest, value I, state S) (commandkit.Acquired[[]string], error) {
return suite.ListNamespaces(ctx, NamespaceRequest{Selection: request.Selection, Kinds: input.kinds}, value, state)
}
return runCatalogList(invocation, suite, input, name, list, encodeProfileCatalog)
return runCatalogList(invocation, suite, input, name, list, stringCatalogReport)
case "collections":
list := func(ctx context.Context, request CatalogRequest, value I, state S) (commandkit.Acquired[[]promptkit.CollectionInfo], error) {
return suite.ListCollections(ctx, request.Selection, value, state)
}
return runCatalogList(invocation, suite, input, name, list, encodeCollectionCatalog)
return runCatalogList(invocation, suite, input, name, list, collectionCatalogReport)
default:
return definitionFailure("Unable to list the selected PromptKit catalog.", commandkit.Explanation{}, fmt.Errorf("unknown PromptKit catalog %q", name))
}
@@ -230,20 +220,6 @@ func prepareCatalogFilter[I any](input *catalogInput[I], name string) error {
return nil
}
func checkCatalogFormat(format string) error {
if format != "yaml" {
return fmt.Errorf("unsupported catalog format %q", format)
}
return nil
}
func checkRequirementsFormat(format string) error {
if format != "yaml" {
return fmt.Errorf("unsupported requirements format %q; expected yaml", format)
}
return nil
}
func prepareCatalogInput[I, S any](suite Suite[I, S], input *catalogInput[I]) (I, error) {
return prepareSuiteInput(suite, input.selection, input.captured)
}
@@ -313,8 +289,6 @@ func newRequirementsCommand[I, S any](suite Suite[I, S]) commandkit.Command {
func(input **requirementsInput[I], value commandkit.Supplied[string]) {
(*input).inspection.SessionID = value.Value
}),
commandkit.ValueFlag(commandkit.FlagMeta{Name: "format", Usage: "Report format: yaml.", Completion: completionChoices("yaml")}, stringDefault("yaml"), parseDefinitionString,
func(input **requirementsInput[I], value commandkit.Supplied[string]) { (*input).format = value.Value }),
commandkit.RepeatFlag(commandkit.FlagMeta{Name: "var", Usage: "Caller variable assignment KEY=VALUE."}, []variableAssignment(nil), parseVariableAssignment,
func(input **requirementsInput[I], value commandkit.Supplied[[]variableAssignment]) {
variables := make(map[string]string, len(value.Value))
@@ -339,9 +313,6 @@ func newRequirementsCommand[I, S any](suite Suite[I, S]) commandkit.Command {
}),
},
CheckInput: func(input *requirementsInput[I]) error {
if err := checkRequirementsFormat(input.format); err != nil {
return err
}
selected, err := prepareSuiteInput(suite, input.selection, input.captured)
if err != nil {
return err
@@ -382,7 +353,7 @@ func runCatalogList[T any, I, S any](
input *catalogInput[I],
catalog string,
list func(context.Context, CatalogRequest, I, S) (commandkit.Acquired[T], error),
encode func(T) ([]byte, error),
project func(T) any,
) commandkit.Outcome {
if invocation.Context != nil {
if err := invocation.Context.Err(); err != nil {
@@ -409,9 +380,8 @@ func runCatalogList[T any, I, S any](
return definitionFailure("PromptKit catalog listing was canceled.", acquired.Explanation, err)
}
}
return renderPreparedPromptReport(invocation, acquired.Explanation, func() ([]byte, error) {
return encode(acquired.Value)
})
_ = invocation.Reporter().Report(commandkit.Report{Value: project(acquired.Value)})
return commandkit.Success()
}
func runDefinitionInspection[I, S any](invocation commandkit.Invocation, suite Suite[I, S], input *definitionInput[I], kind string) commandkit.Outcome {
@@ -467,35 +437,35 @@ func runDefinitionInspection[I, S any](invocation commandkit.Invocation, suite S
}
type requirementsInspectionReport struct {
PromptID string `yaml:"prompt_id"`
ProfileID string `yaml:"profile_id"`
Complete bool `yaml:"complete"`
Inputs []requirementInputReport `yaml:"inputs"`
Variables []requirementVariableReport `yaml:"variables"`
Issues []requirementIssueReport `yaml:"issues"`
PromptID string `json:"prompt_id"`
ProfileID string `json:"profile_id"`
Complete bool `json:"complete"`
Inputs []requirementInputReport `json:"inputs"`
Variables []requirementVariableReport `json:"variables"`
Issues []requirementIssueReport `json:"issues"`
}
type requirementInputReport struct {
Name string `yaml:"name"`
Kind string `yaml:"kind"`
Declared bool `yaml:"declared"`
ContentType *string `yaml:"content_type"`
Description *string `yaml:"description"`
Explanation string `yaml:"explanation"`
Name string `json:"name"`
Kind string `json:"kind"`
Declared bool `json:"declared"`
ContentType *string `json:"content_type"`
Description *string `json:"description"`
Explanation string `json:"explanation"`
}
type requirementVariableReport struct {
Name string `yaml:"name"`
Kind string `yaml:"kind"`
Supplied bool `yaml:"supplied"`
Explanation string `yaml:"explanation"`
Name string `json:"name"`
Kind string `json:"kind"`
Supplied bool `json:"supplied"`
Explanation string `json:"explanation"`
}
type requirementIssueReport struct {
Code string `yaml:"code"`
InputName string `yaml:"input_name"`
VariableName string `yaml:"variable_name"`
Description string `yaml:"description"`
Code string `json:"code"`
InputName string `json:"input_name"`
VariableName string `json:"variable_name"`
Description string `json:"description"`
}
func requirementsReportFromInspection(inspection *promptkit.RequirementInspection) requirementsInspectionReport {
@@ -566,76 +536,40 @@ func runRequirementsInspection[I, S any](invocation commandkit.Invocation, suite
return definitionFailure(fmt.Sprintf("Requirements inspection for PromptKit prompt %q was canceled.", input.inspection.PromptID), acquired.Explanation, err)
}
}
return renderPreparedPromptReport(invocation, acquired.Explanation, func() ([]byte, error) {
return encodePromptYAMLBytes(requirementsReportFromInspection(acquired.Value))
})
}
// These reports were already prepared before delivery. Keep that boundary
// inside managed rendering, with cancellation owned by the command outcome.
func renderPreparedPromptReport(invocation commandkit.Invocation, explanation commandkit.Explanation, prepare func() ([]byte, error)) commandkit.Outcome {
var canceled error
_ = invocation.Reporter().RenderOutput(func(writer io.Writer) error {
output, err := prepare()
if err != nil {
return err
}
if canceled = promptKitContextError(invocation.Context); canceled != nil {
return nil
}
_, err = writer.Write(output)
return err
})
if canceled != nil {
return definitionFailure("PromptKit report preparation was canceled.", explanation, canceled)
}
_ = invocation.Reporter().Report(commandkit.Report{Value: requirementsReportFromInspection(acquired.Value)})
return commandkit.Success()
}
func encodePromptYAML(writer io.Writer, value any) error {
encoder := yaml.NewEncoder(writer)
encoder.SetIndent(2)
return encoder.Encode(value)
}
type promptCatalogEntry struct {
ID string `yaml:"id"`
ID string `json:"id"`
}
func encodePromptCatalog(prompts []promptkit.PromptIdentity) ([]byte, error) {
func promptCatalogReport(prompts []promptkit.PromptIdentity) any {
entries := make([]promptCatalogEntry, len(prompts))
for index, prompt := range prompts {
entries[index] = promptCatalogEntry{ID: prompt.ID}
}
return encodePromptYAMLBytes(entries)
return entries
}
func encodeProfileCatalog(profiles []string) ([]byte, error) {
if profiles == nil {
profiles = []string{}
func stringCatalogReport(values []string) any {
if values == nil {
return []string{}
}
return encodePromptYAMLBytes(profiles)
return values
}
type collectionCatalogEntry struct {
Namespace string `yaml:"namespace"`
Version *string `yaml:"version,omitempty"`
Namespace string `json:"namespace"`
Version *string `json:"version,omitempty"`
}
func encodeCollectionCatalog(collections []promptkit.CollectionInfo) ([]byte, error) {
func collectionCatalogReport(collections []promptkit.CollectionInfo) any {
entries := make([]collectionCatalogEntry, len(collections))
for index, collection := range collections {
entries[index] = collectionCatalogEntry{Namespace: collection.Namespace, Version: collection.ReleaseVersion}
}
return encodePromptYAMLBytes(entries)
}
func encodePromptYAMLBytes(value any) ([]byte, error) {
var output bytes.Buffer
if err := encodePromptYAML(&output, value); err != nil {
return nil, err
}
return output.Bytes(), nil
return entries
}
func queuePromptDiagnostics(reporter *commandkit.Reporter, diagnostics []commandkit.Diagnostic) {

26
integrations/ckprompt/command_test.go
View File

@@ -202,7 +202,7 @@ func TestRequirementsCommandUsageFailuresPrecedeApplicationPreparation(t *testin
}{
{name: "missing operand", args: []string{"promptkit", "requirements", "show"}},
{name: "extra operand", args: []string{"promptkit", "requirements", "show", "briefing", "extra"}},
{name: "unsupported format", args: []string{"promptkit", "requirements", "show", "briefing", "--format", "text"}},
{name: "unsupported format", args: []string{"promptkit", "requirements", "show", "briefing", "--format", "invalid"}},
{name: "malformed variable followed by valid assignment", args: []string{"promptkit", "requirements", "show", "briefing", "--var", "broken", "--var", "mode=full"}},
{name: "no built-in scope", args: []string{"promptkit", "requirements", "show", "briefing", "--scope", "all"}},
{name: "malformed variable on help", args: []string{"help", "promptkit", "requirements", "show", "--var", "broken"}},
@@ -402,7 +402,7 @@ func TestDiscoveryCommandsForwardFiltersAndRenderCollections(t *testing.T) {
if outcome.Kind != commandkit.OutcomeSuccess || backendRequest.Filter.Mode != promptkit.NamespaceAll {
t.Fatalf("backend list outcome/filter = (%#v, %#v)", outcome, backendRequest.Filter)
}
outcome, output = run("promptkit", "list", "namespaces", "--kind", "prompt", "--kind", "backend")
outcome, output = run("promptkit", "list", "namespaces", "--kind", "prompt", "--kind", "backend", "--format=yaml")
if outcome.Kind != commandkit.OutcomeSuccess || !reflect.DeepEqual(namespaceRequest.Kinds, []promptkit.ResourceKind{promptkit.ResourcePrompt, promptkit.ResourceBackend}) {
t.Fatalf("namespace list outcome/kinds = (%#v, %#v)", outcome, namespaceRequest.Kinds)
}
@@ -460,7 +460,7 @@ func TestCatalogListUsesPreparedInputAndPreflightsFormat(t *testing.T) {
}
app := newPromptTestApp(t, Command(suite))
var output bytes.Buffer
invalid := app.Execute(context.Background(), []string{"promptkit", "list", "prompts", "--format", "text"}, commandkit.Streams{Out: &output, Err: io.Discard})
invalid := app.Execute(context.Background(), []string{"promptkit", "list", "prompts", "--format", "invalid"}, commandkit.Streams{Out: &output, Err: io.Discard})
if invalid.Kind != commandkit.OutcomeUsage || newInputCalls != 0 || newStateCalls != 0 || providerCalls != 0 || output.Len() != 0 {
t.Fatalf("invalid format preflight = %#v, factories/providers=%d/%d/%d output=%q", invalid, newInputCalls, newStateCalls, providerCalls, output.String())
}
@@ -800,7 +800,7 @@ func TestCatalogAndInspectionRespectSuccessfulReportingModes(t *testing.T) {
if output.Len() != 0 {
t.Fatalf("quiet inspection wrote result output %q", output.String())
}
} else if !strings.Contains(output.String(), "Prompt ID: \"briefing\"") {
} else if !strings.Contains(output.String(), "id: briefing") {
t.Fatalf("%s inspection omitted result: %q", mode, output.String())
}
})
@@ -1011,7 +1011,7 @@ func TestPromptCatalogYAMLPreservesSchemaStringsAndOrder(t *testing.T) {
{ID: "001"},
{ID: " briefing "},
}
output, err := encodePromptCatalog(want)
output, err := renderTestYAML(promptCatalogReport(want))
if err != nil {
t.Fatal(err)
}
@@ -1038,10 +1038,12 @@ func TestCatalogYAMLEncodersRenderNilAndEmptyInventoriesAsSequences(t *testing.T
name string
data func() ([]byte, error)
}{
{name: "nil prompts", data: func() ([]byte, error) { return encodePromptCatalog(nil) }},
{name: "empty prompts", data: func() ([]byte, error) { return encodePromptCatalog(make([]promptkit.PromptIdentity, 0)) }},
{name: "nil profiles", data: func() ([]byte, error) { return encodeProfileCatalog(nil) }},
{name: "empty profiles", data: func() ([]byte, error) { return encodeProfileCatalog(make([]string, 0)) }},
{name: "nil prompts", data: func() ([]byte, error) { return renderTestYAML(promptCatalogReport(nil)) }},
{name: "empty prompts", data: func() ([]byte, error) {
return renderTestYAML(promptCatalogReport(make([]promptkit.PromptIdentity, 0)))
}},
{name: "nil profiles", data: func() ([]byte, error) { return renderTestYAML(stringCatalogReport(nil)) }},
{name: "empty profiles", data: func() ([]byte, error) { return renderTestYAML(stringCatalogReport(make([]string, 0))) }},
} {
t.Run(test.name, func(t *testing.T) {
output, err := test.data()
@@ -1057,7 +1059,7 @@ func TestCatalogYAMLEncodersRenderNilAndEmptyInventoriesAsSequences(t *testing.T
func TestProfileCatalogYAMLPreservesStringValuesAndOrder(t *testing.T) {
want := []string{" default ", "001", "same", "same"}
output, err := encodeProfileCatalog(want)
output, err := renderTestYAML(stringCatalogReport(want))
if err != nil {
t.Fatal(err)
}
@@ -1077,7 +1079,7 @@ func TestProfileCatalogYAMLPreservesStringValuesAndOrder(t *testing.T) {
}
func TestRequirementsReportRendersEmptyInventoriesAndFalseValues(t *testing.T) {
output, err := encodePromptYAMLBytes(requirementsReportFromInspection(&promptkit.RequirementInspection{}))
output, err := renderTestYAML(requirementsReportFromInspection(&promptkit.RequirementInspection{}))
if err != nil {
t.Fatal(err)
}
@@ -1135,7 +1137,7 @@ func TestRequirementsReportPreservesNativeFieldsAndMetadata(t *testing.T) {
for _, complete := range []bool{true, false} {
t.Run(map[bool]string{true: "complete", false: "partial"}[complete], func(t *testing.T) {
inspection.Complete = complete
output, err := encodePromptYAMLBytes(requirementsReportFromInspection(inspection))
output, err := renderTestYAML(requirementsReportFromInspection(inspection))
if err != nil {
t.Fatal(err)
}

2
integrations/ckprompt/composition_test.go
View File

@@ -225,7 +225,7 @@ profiles:
output.Reset()
outcome = command.Execute(context.Background(), []string{"promptkit", "show", "profile", "default", "--campaign", "spring", "--format", "yaml"}, commandkit.Streams{Out: &output, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeSuccess || appAcquisitions != 1 || engineScopeAcquisitions != 1 || loaderCalls != 2 || !bytes.Contains(output.Bytes(), []byte("profile_id: default")) {
if outcome.Kind != commandkit.OutcomeSuccess || appAcquisitions != 1 || engineScopeAcquisitions != 1 || loaderCalls != 2 || !bytes.Contains(output.Bytes(), []byte("id: default")) {
t.Fatalf("independent profile run = (%#v, output=%q, counts=%d/%d/%d)", outcome, output.String(), appAcquisitions, engineScopeAcquisitions, loaderCalls)
}
if newInputCalls != 3 || newStateCalls != 2 {

2
integrations/ckprompt/examples/composed/main_test.go
View File

@@ -537,7 +537,7 @@ func TestComposedAliasesReachCanonicalCommands(t *testing.T) {
} {
output.Reset()
outcome = app.Execute(context.Background(), args, commandkit.Streams{Out: &output, Err: io.Discard})
if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(output.String(), "prompt_id: briefing") {
if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(output.String(), "id: briefing") {
t.Fatalf("prompt inspection route %q outcome/report = (%#v, %q)", args, outcome, output.String())
}
}

13
integrations/ckprompt/render_test.go Normal file
View File

@@ -0,0 +1,13 @@
package ckprompt
import (
"bytes"
commandkit "go.maximumdirect.net/commandkit"
)
func renderTestYAML(value any) ([]byte, error) {
var output bytes.Buffer
err := commandkit.RenderReport(&output, commandkit.Report{Value: value}, commandkit.FormatYAML)
return output.Bytes(), err
}

2
integrations/ckprompt/requirements_command_test.go
View File

@@ -71,7 +71,7 @@ func executeRequirementsAgainstNative(t *testing.T, engine *promptkit.Engine, re
if err != nil {
t.Fatalf("direct native inspection: %v", err)
}
expectedBytes, err := encodePromptYAMLBytes(requirementsReportFromInspection(native))
expectedBytes, err := renderTestYAML(requirementsReportFromInspection(native))
if err != nil {
t.Fatalf("encode direct native report: %v", err)
}

66
integrations/ckprompt/shared_smoke_test.go Normal file
View File

@@ -0,0 +1,66 @@
package ckprompt
import (
"bytes"
"context"
"encoding/json"
"errors"
"io"
"testing"
commandkit "go.maximumdirect.net/commandkit"
promptkit "go.maximumdirect.net/promptkit"
)
func TestSmokeAggregateJSONRetainsPartialEvidence(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
cause := errors.New("authentication interrupted")
calls := 0
app := newPromptTestApp(t, Command(Suite[struct{}, struct{}]{SmokeTest: func(_ context.Context, request SmokeRequest, _, _ struct{}) (commandkit.Acquired[*promptkit.SmokeTestReport], error) {
calls++
report := &promptkit.SmokeTestReport{Kind: request.Request.Kind, Supported: true, Reason: promptkit.SmokeTestReasonAuthenticationResponseLimitExceeded}
if calls == 2 {
cancel()
return commandkit.Acquired[*promptkit.SmokeTestReport]{Value: report}, cause
}
return commandkit.Acquired[*promptkit.SmokeTestReport]{Value: report}, nil
}}))
var output bytes.Buffer
result := app.Execute(ctx, []string{"promptkit", "test", "all", "--format=json"}, commandkit.Streams{Out: &output, Err: io.Discard})
dec := json.NewDecoder(&output)
var report struct {
Tests []struct {
Kind string `json:"kind"`
Error *string `json:"error"`
Report struct {
Reason string `json:"reason"`
Facts struct {
CredentialsAccepted *bool `json:"credentials_accepted"`
EndpointReachable *bool `json:"endpoint_reachable"`
} `json:"facts"`
} `json:"report"`
} `json:"tests"`
}
if err := dec.Decode(&report); err != nil {
t.Fatal(err)
}
if err := dec.Decode(new(any)); err != io.EOF {
t.Fatalf("extra JSON: %v", err)
}
if calls != 2 || !errors.Is(result.Cause, cause) || !errors.Is(result.Cause, context.Canceled) || len(report.Tests) != 2 || report.Tests[1].Error == nil || report.Tests[1].Report.Reason != string(promptkit.SmokeTestReasonAuthenticationResponseLimitExceeded) || report.Tests[1].Report.Facts.CredentialsAccepted != nil || report.Tests[1].Report.Facts.EndpointReachable != nil {
t.Fatalf("result=%#v calls=%d report=%#v", result, calls, report)
}
}
func TestSmokePreparationErrorDoesNotInventReport(t *testing.T) {
cause := errors.New("request preparation failed")
app := newPromptTestApp(t, Command(Suite[struct{}, struct{}]{SmokeTest: func(context.Context, SmokeRequest, struct{}, struct{}) (commandkit.Acquired[*promptkit.SmokeTestReport], error) {
return commandkit.Acquired[*promptkit.SmokeTestReport]{}, cause
}}))
var output bytes.Buffer
result := app.Execute(t.Context(), []string{"promptkit", "test", "authentication", "--format=json"}, commandkit.Streams{Out: &output, Err: io.Discard})
if !errors.Is(result.Cause, cause) || output.Len() != 0 || result.Presentation.Output.State != commandkit.DeliveryNone {
t.Fatalf("result=%#v output=%q", result, output.String())
}
}

33
integrations/ckprompt/smoke_command.go
View File

@@ -15,7 +15,6 @@ type smokeCommandInput[I any] struct {
selection ckconfig.Selection
kind string
request promptkit.SmokeTestRequest
format string
captured commandkit.FlagValues[I]
selected I
}
@@ -36,8 +35,6 @@ func newSmokeTestCommand[I, S any](suite Suite[I, S]) commandkit.Command {
func(input **smokeCommandInput[I], value commandkit.Supplied[string]) {
(*input).selection.Overlay = value
}),
commandkit.ValueFlag(commandkit.FlagMeta{Name: "format", Usage: "Smoke report format: yaml.", Completion: completionChoices("yaml")}, stringDefault("yaml"), parseDefinitionString,
func(input **smokeCommandInput[I], value commandkit.Supplied[string]) { (*input).format = value.Value }),
commandkit.ValueFlag(commandkit.FlagMeta{Name: "backend", Usage: "PromptKit backend ID."}, (*string)(nil), parseDefinitionString,
func(input **smokeCommandInput[I], value commandkit.Supplied[string]) {
(*input).request.BackendID = value.Value
@@ -134,7 +131,7 @@ func newSmokeTestCommand[I, S any](suite Suite[I, S]) commandkit.Command {
Name: "test",
Summary: "Run one PromptKit smoke test.",
NewInput: func() *smokeCommandInput[I] {
return &smokeCommandInput[I]{format: "yaml"}
return &smokeCommandInput[I]{}
},
Flags: flags,
Arguments: []commandkit.Argument[*smokeCommandInput[I]]{
@@ -145,9 +142,6 @@ func newSmokeTestCommand[I, S any](suite Suite[I, S]) commandkit.Command {
if input == nil {
return errors.New("PromptKit smoke-test input must not be nil")
}
if err := checkSmokeFormat(input.format); err != nil {
return err
}
if input.kind != "all" {
input.request.Kind = promptkit.SmokeTestKind(input.kind)
}
@@ -175,13 +169,6 @@ func parseSmokeFloat(value string) (float64, error) {
return strconv.ParseFloat(value, 64)
}
func checkSmokeFormat(format string) error {
if format != "yaml" {
return fmt.Errorf("unsupported smoke report format %q; expected yaml", format)
}
return nil
}
func runSmokeTest[I, S any](invocation commandkit.Invocation, suite Suite[I, S], input *smokeCommandInput[I]) commandkit.Outcome {
if err := promptKitContextError(invocation.Context); err != nil {
return definitionFailure("PromptKit smoke testing was canceled.", commandkit.Explanation{}, err)
@@ -214,14 +201,7 @@ func runSmokeTest[I, S any](invocation commandkit.Invocation, suite Suite[I, S],
return definitionFailure(smokeFailureMessage(input.kind), acquired.Explanation, operationErr)
}
_ = invocation.Reporter().RenderOutput(func(writer io.Writer) error {
output, err := encodeSmokeReport(acquired.Value)
if err != nil {
return err
}
_, err = writer.Write(output)
return err
})
_ = invocation.Reporter().Report(commandkit.Report{Value: smokeReportFromNative(acquired.Value)})
if operationErr != nil {
return definitionFailure(smokeFailureMessage(input.kind), acquired.Explanation, operationErr)
}
@@ -281,14 +261,7 @@ func runSmokeAggregate[I, S any](invocation commandkit.Invocation, suite Suite[I
if len(results) == 0 {
return definitionFailure("PromptKit aggregate smoke testing did not produce any results.", combineSmokeExplanations(explanations), errors.Join(operationErrors...))
}
_ = invocation.Reporter().RenderOutput(func(writer io.Writer) error {
output, err := encodeSmokeResults(results)
if err != nil {
return err
}
_, err = writer.Write(output)
return err
})
_ = invocation.Reporter().Report(commandkit.Report{Value: smokeResultsReport(results)})
if len(operationErrors) > 0 {
return definitionFailure("One or more PromptKit smoke tests failed.", combineSmokeExplanations(explanations), errors.Join(operationErrors...))
}

2
integrations/ckprompt/smoke_lifecycle_test.go
View File

@@ -78,7 +78,7 @@ func TestSmokeTestCommandUsageAndInputFailuresStopProviderWork(t *testing.T) {
{name: "malformed top-p", args: []string{"promptkit", "test", "authentication", "--top-p=high"}},
{name: "malformed timeout", args: []string{"promptkit", "test", "authentication", "--timeout-seconds=soon"}},
{name: "malformed retries", args: []string{"promptkit", "test", "authentication", "--transient-retries=several"}},
{name: "unsupported format", args: []string{"promptkit", "test", "authentication", "--format=json"}},
{name: "unsupported format", args: []string{"promptkit", "test", "authentication", "--format=invalid"}},
{name: "unknown flag", args: []string{"promptkit", "test", "authentication", "--unknown=value"}},
} {
t.Run(test.name, func(t *testing.T) {

163
integrations/ckprompt/smoke_report.go
View File

@@ -7,100 +7,106 @@ import (
)
type smokeReportDTO struct {
Kind promptkit.SmokeTestKind `yaml:"kind"`
Supported bool `yaml:"supported"`
Applicable *bool `yaml:"applicable"`
Completed bool `yaml:"completed"`
Reason promptkit.SmokeTestReason `yaml:"reason"`
Target smokeTargetDTO `yaml:"target"`
Facts smokeFactsDTO `yaml:"facts"`
Timing smokeTimingDTO `yaml:"timing"`
Attempts []smokeAttemptDTO `yaml:"attempts"`
Usage smokeUsageDTO `yaml:"usage"`
Failure promptkit.ExecutionFailureKind `yaml:"failure"`
Kind promptkit.SmokeTestKind `json:"kind"`
Supported bool `json:"supported"`
Applicable *bool `json:"applicable"`
Completed bool `json:"completed"`
Reason promptkit.SmokeTestReason `json:"reason"`
Target smokeTargetDTO `json:"target"`
Facts smokeFactsDTO `json:"facts"`
Timing smokeTimingDTO `json:"timing"`
Attempts []smokeAttemptDTO `json:"attempts"`
Usage smokeUsageDTO `json:"usage"`
Failure promptkit.ExecutionFailureKind `json:"failure"`
}
type smokeTargetDTO struct {
BackendID string `yaml:"backend_id"`
ProfileID string `yaml:"profile_id"`
PromptID string `yaml:"prompt_id"`
DefaultSourceBackendID string `yaml:"default_source_backend_id"`
SelectionSource string `yaml:"selection_source"`
Endpoint string `yaml:"endpoint"`
EffectiveSettings smokeExecutionTargetDTO `yaml:"effective_settings"`
SettingsPresence smokeSettingsPresenceDTO `yaml:"settings_presence"`
BackendID string `json:"backend_id"`
ProfileID string `json:"profile_id"`
PromptID string `json:"prompt_id"`
DefaultSourceBackendID string `json:"default_source_backend_id"`
SelectionSource string `json:"selection_source"`
Endpoint string `json:"endpoint"`
EffectiveSettings smokeExecutionTargetDTO `json:"effective_settings"`
SettingsPresence smokeSettingsPresenceDTO `json:"settings_presence"`
}
type smokeSettingsPresenceDTO struct {
Temperature bool `yaml:"temperature"`
MaxTokens bool `yaml:"max_tokens"`
TopP bool `yaml:"top_p"`
TimeoutSeconds bool `yaml:"timeout_seconds"`
Temperature bool `json:"temperature"`
MaxTokens bool `json:"max_tokens"`
TopP bool `json:"top_p"`
TimeoutSeconds bool `json:"timeout_seconds"`
}
type smokeExecutionTargetDTO struct {
BackendID string `yaml:"backend_id,omitempty"`
Endpoint string `yaml:"endpoint"`
Model string `yaml:"model"`
Temperature float64 `yaml:"temperature"`
MaxTokens int `yaml:"max_tokens"`
TopP float64 `yaml:"top_p"`
TimeoutSeconds int `yaml:"timeout_seconds"`
ServiceTier string `yaml:"service_tier"`
ReasoningEffort string `yaml:"reasoning_effort"`
APIKeyEnv string `yaml:"api_key_env"`
ExtraParams map[string]any `yaml:"extra_params"`
BackendID string `json:"backend_id,omitempty"`
Endpoint string `json:"endpoint"`
Model string `json:"model"`
Temperature float64 `json:"temperature"`
MaxTokens int `json:"max_tokens"`
TopP float64 `json:"top_p"`
TimeoutSeconds int `json:"timeout_seconds"`
ServiceTier string `json:"service_tier"`
ReasoningEffort string `json:"reasoning_effort"`
APIKeyEnv string `json:"api_key_env"`
ExtraParams map[string]any `json:"extra_params"`
}
type smokeFactsDTO struct {
CredentialsSupplied *bool `yaml:"credentials_supplied"`
EndpointReachable *bool `yaml:"endpoint_reachable"`
CredentialsAccepted *bool `yaml:"credentials_accepted"`
ResponseReceived *bool `yaml:"response_received"`
GenerationCompleted *bool `yaml:"generation_completed"`
TextReturned *bool `yaml:"text_returned"`
ExpectedTextMatched *bool `yaml:"expected_text_matched"`
StructuredOutputRequested *bool `yaml:"structured_output_requested"`
ValidJSON *bool `yaml:"valid_json"`
SchemaMatched *bool `yaml:"schema_matched"`
CredentialsSupplied *bool `json:"credentials_supplied"`
EndpointReachable *bool `json:"endpoint_reachable"`
CredentialsAccepted *bool `json:"credentials_accepted"`
ResponseReceived *bool `json:"response_received"`
GenerationCompleted *bool `json:"generation_completed"`
TextReturned *bool `json:"text_returned"`
ExpectedTextMatched *bool `json:"expected_text_matched"`
StructuredOutputRequested *bool `json:"structured_output_requested"`
ValidJSON *bool `json:"valid_json"`
SchemaMatched *bool `json:"schema_matched"`
}
type smokeTimingDTO struct {
StartedAt time.Time `yaml:"started_at"`
Duration string `yaml:"duration"`
CapacityWait *string `yaml:"capacity_wait"`
RequestDuration *string `yaml:"request_duration"`
RetryDelay *string `yaml:"retry_delay"`
EvaluationDuration *string `yaml:"evaluation_duration"`
StartedAt time.Time `json:"started_at"`
Duration string `json:"duration"`
CapacityWait *string `json:"capacity_wait"`
RequestDuration *string `json:"request_duration"`
RetryDelay *string `json:"retry_delay"`
EvaluationDuration *string `json:"evaluation_duration"`
}
type smokeAttemptDTO struct {
Dispatched bool `yaml:"dispatched"`
InitialEndpoint string `yaml:"initial_endpoint"`
Endpoint string `yaml:"endpoint"`
HTTPStatus *int `yaml:"http_status"`
Facts smokeFactsDTO `yaml:"facts"`
Timing smokeTimingDTO `yaml:"timing"`
Failure promptkit.ExecutionFailureKind `yaml:"failure"`
Usage smokeUsageDTO `yaml:"usage"`
Dispatched bool `json:"dispatched"`
InitialEndpoint string `json:"initial_endpoint"`
Endpoint string `json:"endpoint"`
HTTPStatus *int `json:"http_status"`
Facts smokeFactsDTO `json:"facts"`
Timing smokeTimingDTO `json:"timing"`
Failure promptkit.ExecutionFailureKind `json:"failure"`
Usage smokeUsageDTO `json:"usage"`
}
type smokeUsageDTO struct {
PromptTokens promptkit.ObservedTokenCount `yaml:"prompt_tokens"`
CompletionTokens promptkit.ObservedTokenCount `yaml:"completion_tokens"`
TotalTokens promptkit.ObservedTokenCount `yaml:"total_tokens"`
CachedTokens promptkit.ObservedTokenCount `yaml:"cached_tokens"`
CacheWriteTokens promptkit.ObservedTokenCount `yaml:"cache_write_tokens"`
PromptTokens smokeTokenCount `json:"prompt_tokens"`
CompletionTokens smokeTokenCount `json:"completion_tokens"`
TotalTokens smokeTokenCount `json:"total_tokens"`
CachedTokens smokeTokenCount `json:"cached_tokens"`
CacheWriteTokens smokeTokenCount `json:"cache_write_tokens"`
}
type smokeTokenCount struct {
Value int `json:"value"`
Known bool `json:"known"`
Complete bool `json:"complete"`
}
type smokeAggregateDTO struct {
Tests []smokeResultDTO `yaml:"tests"`
Tests []smokeResultDTO `json:"tests"`
}
type smokeResultDTO struct {
Kind promptkit.SmokeTestKind `yaml:"kind"`
Report *smokeReportDTO `yaml:"report"`
Error *string `yaml:"error,omitempty"`
Kind promptkit.SmokeTestKind `json:"kind"`
Report *smokeReportDTO `json:"report"`
Error *string `json:"error,omitempty"`
}
type smokeResult struct {
@@ -109,23 +115,16 @@ type smokeResult struct {
Err error
}
func encodeSmokeReport(report *promptkit.SmokeTestReport) ([]byte, error) {
return encodePromptYAMLBytes(smokeReportFromNative(report))
}
func encodeSmokeResults(results []smokeResult) ([]byte, error) {
func smokeResultsReport(results []smokeResult) smokeAggregateDTO {
entries := make([]smokeResultDTO, len(results))
for index, result := range results {
entries[index] = smokeResultDTO{
Kind: result.Kind,
Report: smokeReportFromNative(result.Report),
}
entries[index] = smokeResultDTO{Kind: result.Kind, Report: smokeReportFromNative(result.Report)}
if result.Err != nil {
message := result.Err.Error()
entries[index].Error = &message
}
}
return encodePromptYAMLBytes(smokeAggregateDTO{Tests: entries})
return smokeAggregateDTO{Tests: entries}
}
func smokeReportFromNative(report *promptkit.SmokeTestReport) *smokeReportDTO {
@@ -224,10 +223,10 @@ func durationString(duration *time.Duration) *string {
func smokeUsageFromNative(usage promptkit.ObservedTokenUsage) smokeUsageDTO {
return smokeUsageDTO{
PromptTokens: usage.PromptTokens,
CompletionTokens: usage.CompletionTokens,
TotalTokens: usage.TotalTokens,
CachedTokens: usage.CachedTokens,
CacheWriteTokens: usage.CacheWriteTokens,
PromptTokens: smokeTokenCount(usage.PromptTokens),
CompletionTokens: smokeTokenCount(usage.CompletionTokens),
TotalTokens: smokeTokenCount(usage.TotalTokens),
CachedTokens: smokeTokenCount(usage.CachedTokens),
CacheWriteTokens: smokeTokenCount(usage.CacheWriteTokens),
}
}

8
integrations/ckprompt/smoke_report_test.go
View File

@@ -84,7 +84,7 @@ func TestSmokeReportYAMLProjectsNativeFieldsAndPresence(t *testing.T) {
Failure: promptkit.ExecutionFailureTransport,
}
encoded, err := encodeSmokeReport(report)
encoded, err := renderTestYAML(smokeReportFromNative(report))
if err != nil {
t.Fatal(err)
}
@@ -145,7 +145,7 @@ func TestSmokeReportYAMLProjectsNativeFieldsAndPresence(t *testing.T) {
}
func TestSmokeReportYAMLEmitsNilAttemptsAsEmptySequence(t *testing.T) {
encoded, err := encodeSmokeReport(&promptkit.SmokeTestReport{Attempts: nil})
encoded, err := renderTestYAML(smokeReportFromNative(&promptkit.SmokeTestReport{Attempts: nil}))
if err != nil {
t.Fatal(err)
}
@@ -161,11 +161,11 @@ func TestSmokeReportYAMLEmitsNilAttemptsAsEmptySequence(t *testing.T) {
func TestSmokeAggregateYAMLRetainsOrderErrorAndNativeReportKind(t *testing.T) {
operationErr := errors.New("operation failed")
report := &promptkit.SmokeTestReport{Kind: promptkit.SmokeTestReachability, Attempts: []promptkit.SmokeTestAttempt{}}
encoded, err := encodeSmokeResults([]smokeResult{
encoded, err := renderTestYAML(smokeResultsReport([]smokeResult{
{Kind: promptkit.SmokeTestTextInference, Report: report, Err: operationErr},
{Kind: promptkit.SmokeTestAuthentication},
{Kind: promptkit.SmokeTestStructuredOutput, Report: report, Err: errors.New("")},
})
}))
if err != nil {
t.Fatal(err)
}

19
integrations/ckprompt/validation_command.go
View File

@@ -11,14 +11,13 @@ import (
type validationCommandInput[I any] struct {
promptID string
request ValidationRequest
format string
captured commandkit.FlagValues[I]
selected I
}
type requestValidationReport struct {
PromptID string `yaml:"prompt_id"`
Valid bool `yaml:"valid"`
PromptID string `json:"prompt_id"`
Valid bool `json:"valid"`
}
func newValidationCommand[I, S any](suite Suite[I, S]) commandkit.Command {
@@ -45,10 +44,6 @@ func newValidationCommand[I, S any](suite Suite[I, S]) commandkit.Command {
func(input **validationCommandInput[I], value commandkit.Supplied[string]) {
(*input).request.Request.SessionID = value.Value
}),
commandkit.ValueFlag(commandkit.FlagMeta{Name: "format", Usage: "Confirmation format: yaml.", Completion: completionChoices("yaml")}, stringDefault("yaml"), parseDefinitionString,
func(input **validationCommandInput[I], value commandkit.Supplied[string]) {
(*input).format = value.Value
}),
commandkit.RepeatFlag(commandkit.FlagMeta{Name: "var", Usage: "Caller variable assignment KEY=VALUE."}, []variableAssignment(nil), parseVariableAssignment,
func(input **validationCommandInput[I], value commandkit.Supplied[[]variableAssignment]) {
variables := make(map[string]string, len(value.Value))
@@ -71,7 +66,7 @@ func newValidationCommand[I, S any](suite Suite[I, S]) commandkit.Command {
Name: "validate",
Summary: "Validate a concrete PromptKit request without executing it.",
NewInput: func() *validationCommandInput[I] {
return &validationCommandInput[I]{format: "yaml"}
return &validationCommandInput[I]{}
},
Flags: flags,
Arguments: []commandkit.Argument[*validationCommandInput[I]]{
@@ -85,9 +80,6 @@ func newValidationCommand[I, S any](suite Suite[I, S]) commandkit.Command {
if input == nil {
return errors.New("PromptKit validation input must not be nil")
}
if err := checkRequirementsFormat(input.format); err != nil {
return err
}
selected, err := prepareSuiteInput(suite, input.request.Selection, input.captured)
if err != nil {
return err
@@ -121,7 +113,6 @@ func runRequestValidation[I, S any](invocation commandkit.Invocation, suite Suit
if err := promptKitContextError(invocation.Context); err != nil {
return definitionFailure(fmt.Sprintf("Validation of the PromptKit request for prompt %q was canceled.", input.promptID), acquired.Explanation, err)
}
return renderPreparedPromptReport(invocation, acquired.Explanation, func() ([]byte, error) {
return encodePromptYAMLBytes(requestValidationReport{PromptID: input.promptID, Valid: true})
})
_ = invocation.Reporter().Report(commandkit.Report{Value: requestValidationReport{PromptID: input.promptID, Valid: true}})
return commandkit.Success()
}

2
integrations/ckprompt/validation_command_test.go
View File

@@ -254,7 +254,7 @@ func TestValidationCommandUsageAndInputFailuresStopProviderWork(t *testing.T) {
}{
{name: "missing operand", args: []string{"promptkit", "requirements", "validate"}},
{name: "extra operand", args: []string{"promptkit", "requirements", "validate", "briefing", "extra", "--campaign", "spring"}},
{name: "unsupported format", args: []string{"promptkit", "requirements", "validate", "briefing", "--campaign", "spring", "--format", "text"}},
{name: "unsupported format", args: []string{"promptkit", "requirements", "validate", "briefing", "--campaign", "spring", "--format", "invalid"}},
{name: "unknown flag", args: []string{"promptkit", "requirements", "validate", "briefing", "--campaign", "spring", "--unknown", "value"}},
{name: "removed prompt version flag", args: []string{"promptkit", "requirements", "validate", "briefing", "--campaign", "spring", "--version", "2"}},
{name: "malformed variable before valid assignment", args: []string{"promptkit", "requirements", "validate", "briefing", "--campaign", "spring", "--var", "broken", "--var", "mode=full"}},