Adopt native PromptKit resource inspection
This commit is contained in:
80
docs/cli.md
80
docs/cli.md
@@ -528,72 +528,32 @@ output.
|
||||
|
||||
## PromptKit definition inspection
|
||||
|
||||
Applications can install prompt and profile inspection independently with
|
||||
`ckprompt.Command(suite)`. The `ckprompt.FromEngine(loadEngine)` adapter
|
||||
supplies these callbacks alongside native requirements discovery and the five
|
||||
catalogs. Its loader receives invocation context, configuration selection,
|
||||
typed input, and shared state. Direct suite callbacks retain their own
|
||||
implementation and lifecycle.
|
||||
|
||||
Prompt inspection uses an ID only. The prompt format and native selection
|
||||
behavior remain owned by
|
||||
[PromptKit v0.18.0](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/d9f6aad30509e538fc73ea16710df3a3ec084502/docs/consumers/pkg-promptkit.md).
|
||||
Applications can install prompt, profile, and backend inspection independently
|
||||
with `ckprompt.Command(suite)`. `ckprompt.FromEngine(loadEngine)` supplies these
|
||||
callbacks alongside requirements discovery and the five catalogs. Its loader
|
||||
receives invocation context, configuration selection, typed input, and shared
|
||||
state. Direct callbacks retain their own lifecycle.
|
||||
|
||||
```text
|
||||
example promptkit show prompt customer/summary --format yaml
|
||||
example promptkit show prompt customer/summary
|
||||
example promptkit show profile customer-support --format yaml
|
||||
example promptkit show profile customer-support --format json
|
||||
example promptkit show backend local --format yaml
|
||||
```
|
||||
|
||||
Both commands require one ID operand and accept `--config`, `--overlay`, and
|
||||
`--format text|yaml` (default `text`). These commands have no built-in
|
||||
`--scope` selection; their namespace is `promptkit`.
|
||||
Each command takes one resource ID and accepts `--config` and `--overlay`.
|
||||
Reports use the native PromptKit inspection value through the shared reporting
|
||||
formats; automatic and explicit text use YAML for these structured values. Native
|
||||
JSON field names and omission rules apply. Absent optional fields are omitted;
|
||||
explicit zero values and empty collections remain present. Prompt messages are
|
||||
captured content, not rendered requests, and the prompt hash is opaque.
|
||||
|
||||
Prompt YAML preserves native input order. The prompt hash is an opaque
|
||||
owning-library equality value.
|
||||
|
||||
```yaml
|
||||
prompt_id: customer/summary
|
||||
prompt_hash: opaque-value
|
||||
default_profile_id: customer-support
|
||||
inputs:
|
||||
- name: topic
|
||||
required: true
|
||||
content_type: text/plain
|
||||
description: Topic to summarize
|
||||
output_contract:
|
||||
format: json
|
||||
validation_mode: basic
|
||||
schema_path: ""
|
||||
repair_attempts: 2
|
||||
```
|
||||
|
||||
Profile YAML preserves the effective values reported by PromptKit, including
|
||||
zero values and nested provider parameters. `api_key_env` is only an
|
||||
environment-variable name. `api_key_required` reports a direct credential
|
||||
requirement; neither field reports credential availability. A nil provider
|
||||
parameter map is displayed as `{}`.
|
||||
|
||||
```yaml
|
||||
profile_id: customer-support
|
||||
effective_model_params:
|
||||
backend_id: local
|
||||
endpoint: https://example.invalid/v1
|
||||
model: support-model
|
||||
temperature: 0
|
||||
max_tokens: 0
|
||||
top_p: 0
|
||||
timeout_seconds: 600
|
||||
service_tier: ""
|
||||
reasoning_effort: ""
|
||||
api_key_env: ""
|
||||
extra_params: {}
|
||||
api_key_required: true
|
||||
```
|
||||
|
||||
Text uses labeled fields and includes nested `extra_params` values. A missing ID
|
||||
is a usage error. Definition selection reaches the owning inspection API and
|
||||
preserves supported error identity and cancellation.
|
||||
Profile reports describe the selected profile before backend completion or
|
||||
request overrides. Backend reports describe configured backend fields. An
|
||||
`api_key_env` value names an environment variable; it is not a credential
|
||||
value or a claim that one is available. Neither report resolves references or
|
||||
performs a network probe. See the [PromptKit resource inspection contracts](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/8ff9aa3efdc9f16b9d2729e0a29ea083eb3c3eaf/resource_inspection.go).
|
||||
A missing ID is a usage error; native lookup errors and cancellation retain
|
||||
supported cause identity.
|
||||
|
||||
## PromptKit requirements
|
||||
|
||||
|
||||
@@ -258,12 +258,12 @@ engine's captured configuration. Add that scope to a `ckconfig.Suite` to expose
|
||||
it under `config` with validate/show/sources/explain and native diff operations;
|
||||
it does not install PromptKit catalog, definition, or requirements commands.
|
||||
|
||||
`ckprompt.FromEngine(loadEngine)` supplies eight callbacks: prompt, profile,
|
||||
backend, namespace, and collection catalogs; named prompt and profile
|
||||
`ckprompt.FromEngine(loadEngine)` supplies nine callbacks: prompt, profile,
|
||||
backend, namespace, and collection catalogs; named prompt, profile, and backend
|
||||
inspection; and native requirements discovery. Concrete request validation and
|
||||
smoke testing are separate capabilities, added explicitly with
|
||||
`ckprompt.WithRequestValidation` and `ckprompt.WithSmokeTesting`. The suite has
|
||||
ten independently optional callbacks in total. `ckprompt.Command(suite)` creates
|
||||
eleven independently optional callbacks in total. `ckprompt.Command(suite)` creates
|
||||
only operations whose callbacks are present; discovery, validation, and smoke
|
||||
testing can be installed independently. A composed suite can enable both
|
||||
additional operations:
|
||||
@@ -300,7 +300,7 @@ absent, so explicitly supplied empty values stay meaningful. Applications that
|
||||
validate and then execute a request are encouraged to share their request
|
||||
construction logic between those paths.
|
||||
|
||||
The eight catalog and inspection callbacks and the two explicit additions are
|
||||
The nine catalog and inspection callbacks and the two explicit additions are
|
||||
independently optional. For catalogs without named or requirements inspection,
|
||||
narrow the suite before command capture:
|
||||
|
||||
@@ -308,6 +308,7 @@ narrow the suite before command capture:
|
||||
definitions := ckprompt.FromEngine(loadEngine)
|
||||
definitions.InspectPrompt = nil
|
||||
definitions.InspectProfile = nil
|
||||
definitions.InspectBackend = nil
|
||||
definitions.InspectRequirements = nil
|
||||
definitionCommand := ckprompt.Command(definitions)
|
||||
```
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Shared reporting and PromptKit v0.20.0 implementation plan
|
||||
|
||||
Status: implementation in progress; Stages 1–7 complete.
|
||||
Status: implementation in progress; Stages 1–8 complete.
|
||||
|
||||
## Scope, authority, and execution
|
||||
|
||||
@@ -287,6 +287,14 @@ policy wording as behavior changes. Do not add domain comparison tests.
|
||||
|
||||
## Stage 8: Adopt PromptKit v0.20.0 and native resource inspection
|
||||
|
||||
**Complete.** ckprompt selects released PromptKit v0.20.0 at commit
|
||||
`8ff9aa3efdc9f16b9d2729e0a29ea083eb3c3eaf` with verified module sums and
|
||||
updated attribution. Native prompt, profile, and backend inspection tests,
|
||||
including format, callback, error, and cancellation cases, passed with Go
|
||||
1.27.0. Root and ckconfig workspace suites also passed. The full ckprompt suite
|
||||
still encounters the remaining local format bindings scheduled for Stage 9;
|
||||
those bindings are not part of the completed inspection migration.
|
||||
|
||||
Update ckprompt's PromptKit requirement and authentic sums to v0.20.0; keep
|
||||
ConfigKit v1.1.0 and existing Go minima. Review the released notices and update
|
||||
[module attribution](../../integrations/ckprompt/docs/oss/attribution.md) and
|
||||
|
||||
@@ -19,8 +19,8 @@ Add `ckprompt.ConfigScope(loadEngine)` to the scopes passed to
|
||||
`ckconfig.Command` when the application wants PromptKit engine configuration
|
||||
and native comparison in the `config` family. Install
|
||||
`ckprompt.Command(ckprompt.FromEngine(loadEngine))` separately for catalogs,
|
||||
named inspections, and requirements discovery. `FromEngine` supplies eight
|
||||
PromptKit-backed callbacks: three inspections and five catalogs. Applications
|
||||
named inspections, and requirements discovery. `FromEngine` supplies nine
|
||||
PromptKit-backed callbacks: four inspections and five catalogs. Applications
|
||||
may narrow, wrap, or replace them before capture or provide custom callbacks
|
||||
with `ckprompt.Suite`. Add concrete request validation and smoke testing
|
||||
explicitly with `WithRequestValidation` and `WithSmokeTesting`. Inspection,
|
||||
|
||||
@@ -29,6 +29,14 @@ type ProfileRequest struct {
|
||||
ID string
|
||||
}
|
||||
|
||||
// BackendRequest selects one named backend for inspection.
|
||||
type BackendRequest struct {
|
||||
// Selection carries configuration and overlay values supplied to the command.
|
||||
Selection ckconfig.Selection
|
||||
// ID is the selected named backend.
|
||||
ID string
|
||||
}
|
||||
|
||||
// CatalogRequest selects the configuration and native namespace filter for a
|
||||
// prompt, profile, or backend catalog operation. A zero Filter preserves
|
||||
// PromptKit's native default selection.
|
||||
@@ -96,6 +104,8 @@ type Suite[I, S any] struct {
|
||||
// preparation. Its acquisition carries independent loader diagnostics and
|
||||
// terminal context alongside any native inspection result.
|
||||
InspectProfile func(context.Context, ProfileRequest, I, S) (commandkit.Acquired[*promptkit.ProfileInspection], error)
|
||||
// InspectBackend independently enables inspection of one named backend.
|
||||
InspectBackend func(context.Context, BackendRequest, I, S) (commandkit.Acquired[*promptkit.BackendInspection], error)
|
||||
// InspectRequirements inspects native PromptKit requirements after input and
|
||||
// state preparation. Its acquisition carries independent loader diagnostics
|
||||
// and terminal context alongside any native inspection result. It is
|
||||
@@ -300,9 +310,9 @@ func promptKitContextError(ctx context.Context) error {
|
||||
return ctx.Err()
|
||||
}
|
||||
|
||||
// FromEngine supplies eight PromptKit-backed callbacks: prompt, profile,
|
||||
// backend, namespace, and collection catalogs, plus named prompt and profile
|
||||
// inspection and requirements discovery.
|
||||
// FromEngine supplies nine PromptKit-backed callbacks: prompt, profile,
|
||||
// backend, namespace, and collection catalogs; named prompt, profile, and
|
||||
// backend inspection; and requirements discovery.
|
||||
// The requirements callback forwards RequirementsRequest.Inspection to
|
||||
// Engine.InspectRequirements. Concrete request validation is an independent
|
||||
// capability and can be composed explicitly with WithRequestValidation. Smoke
|
||||
@@ -384,6 +394,15 @@ func FromEngine[I, S any](load func(context.Context, ckconfig.Selection, I, S) (
|
||||
messages = messages.withOperationError(err)
|
||||
return commandkit.Acquired[*promptkit.ProfileInspection]{Value: inspection, Diagnostics: messages.diagnostics, Explanation: messages.explanation}, err
|
||||
},
|
||||
InspectBackend: func(ctx context.Context, request BackendRequest, input I, state S) (commandkit.Acquired[*promptkit.BackendInspection], error) {
|
||||
engine, _, messages, err := loadPromptKitEngine(ctx, request.Selection, input, state, "", load)
|
||||
if err != nil {
|
||||
return commandkit.Acquired[*promptkit.BackendInspection]{Diagnostics: messages.diagnostics, Explanation: messages.explanation}, err
|
||||
}
|
||||
inspection, err := engine.InspectBackend(ctx, request.ID)
|
||||
messages = messages.withOperationError(err)
|
||||
return commandkit.Acquired[*promptkit.BackendInspection]{Value: inspection, Diagnostics: messages.diagnostics, Explanation: messages.explanation}, err
|
||||
},
|
||||
InspectRequirements: func(ctx context.Context, request RequirementsRequest, input I, state S) (commandkit.Acquired[*promptkit.RequirementInspection], error) {
|
||||
engine, _, messages, err := loadPromptKitEngine(ctx, request.Selection, input, state, "", load)
|
||||
if err != nil {
|
||||
|
||||
@@ -6,7 +6,6 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"go.maximumdirect.net/ckconfig"
|
||||
@@ -18,7 +17,6 @@ import (
|
||||
type definitionInput[I any] struct {
|
||||
selection ckconfig.Selection
|
||||
id string
|
||||
format string
|
||||
captured commandkit.FlagValues[I]
|
||||
selected I
|
||||
}
|
||||
@@ -63,11 +61,10 @@ type inputAssignment struct {
|
||||
// callback-referenced objects and effects remain caller-owned. Suite documents
|
||||
// invocation input and state initialization, zero values, and caller ownership
|
||||
// of returned objects.
|
||||
// Catalog commands accept config/overlay selection and suite selectors, and
|
||||
// render YAML with `yaml` as the only format and default. Named-show formats are
|
||||
// unchanged.
|
||||
// Named resource inspections return native reports through shared formatting.
|
||||
// Catalog commands accept config/overlay selection and suite selectors.
|
||||
func Command[I, S any](suite Suite[I, S]) commandkit.Command {
|
||||
if suite.InspectPrompt == nil && suite.InspectProfile == nil && suite.InspectRequirements == nil && suite.ValidateRequest == nil && suite.SmokeTest == nil && suite.ListPrompts == nil && suite.ListProfiles == nil && suite.ListBackends == nil && suite.ListNamespaces == nil && suite.ListCollections == nil {
|
||||
if suite.InspectPrompt == nil && suite.InspectProfile == nil && suite.InspectBackend == nil && suite.InspectRequirements == nil && suite.ValidateRequest == nil && suite.SmokeTest == nil && suite.ListPrompts == nil && suite.ListProfiles == nil && suite.ListBackends == nil && suite.ListNamespaces == nil && suite.ListCollections == nil {
|
||||
command := commandkit.NoCommand()
|
||||
for _, issue := range suite.issues {
|
||||
command = commandkit.WithError(command, issue)
|
||||
@@ -79,10 +76,13 @@ func Command[I, S any](suite Suite[I, S]) commandkit.Command {
|
||||
rootCommands := make([]commandkit.Command, 0, 3)
|
||||
showCommands := make([]commandkit.Command, 0, 2)
|
||||
if suite.InspectPrompt != nil {
|
||||
showCommands = append(showCommands, newDefinitionCommand(suite, "prompt", true))
|
||||
showCommands = append(showCommands, newDefinitionCommand(suite, "prompt"))
|
||||
}
|
||||
if suite.InspectProfile != nil {
|
||||
showCommands = append(showCommands, newDefinitionCommand(suite, "profile", false))
|
||||
showCommands = append(showCommands, newDefinitionCommand(suite, "profile"))
|
||||
}
|
||||
if suite.InspectBackend != nil {
|
||||
showCommands = append(showCommands, newDefinitionCommand(suite, "backend"))
|
||||
}
|
||||
if len(showCommands) != 0 {
|
||||
rootCommands = append(rootCommands, commandkit.Group("show", "Show a named PromptKit definition.", showCommands...))
|
||||
@@ -248,7 +248,7 @@ func prepareCatalogInput[I, S any](suite Suite[I, S], input *catalogInput[I]) (I
|
||||
return prepareSuiteInput(suite, input.selection, input.captured)
|
||||
}
|
||||
|
||||
func newDefinitionCommand[I, S any](suite Suite[I, S], name string, prompt bool) commandkit.Command {
|
||||
func newDefinitionCommand[I, S any](suite Suite[I, S], name string) commandkit.Command {
|
||||
flags := commandkit.CaptureFlags(suite.Selectors, func(input **definitionInput[I]) *commandkit.FlagValues[I] {
|
||||
if input == nil || *input == nil {
|
||||
return nil
|
||||
@@ -262,8 +262,6 @@ func newDefinitionCommand[I, S any](suite Suite[I, S], name string, prompt bool)
|
||||
func(input **definitionInput[I], value commandkit.Supplied[string]) {
|
||||
(*input).selection.Overlay = value
|
||||
}),
|
||||
commandkit.ValueFlag(commandkit.FlagMeta{Name: "format", Usage: "Report format: text or yaml.", Completion: completionChoices("text", "yaml")}, stringDefault("text"), parseDefinitionString,
|
||||
func(input **definitionInput[I], value commandkit.Supplied[string]) { (*input).format = value.Value }),
|
||||
)
|
||||
spec := commandkit.CommandSpec[*definitionInput[I]]{
|
||||
Name: name,
|
||||
@@ -273,13 +271,10 @@ func newDefinitionCommand[I, S any](suite Suite[I, S], name string, prompt bool)
|
||||
},
|
||||
Flags: flags,
|
||||
Arguments: []commandkit.Argument[*definitionInput[I]]{
|
||||
commandkit.ValueArgument(commandkit.ArgumentMeta{Name: "id", Usage: "Prompt or profile ID."}, true, parseDefinitionString,
|
||||
commandkit.ValueArgument(commandkit.ArgumentMeta{Name: strings.ToUpper(name) + "_ID", Usage: "Resource ID."}, true, parseDefinitionString,
|
||||
func(input **definitionInput[I], value commandkit.Supplied[string]) { (*input).id = value.Value }),
|
||||
},
|
||||
CheckInput: func(input *definitionInput[I]) error {
|
||||
if err := checkDefinitionFormat(input.format); err != nil {
|
||||
return err
|
||||
}
|
||||
selected, err := prepareDefinitionInput(suite, input)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -288,7 +283,7 @@ func newDefinitionCommand[I, S any](suite Suite[I, S], name string, prompt bool)
|
||||
return nil
|
||||
},
|
||||
Run: func(invocation commandkit.Invocation, input *definitionInput[I]) commandkit.Outcome {
|
||||
return runDefinitionInspection(invocation, suite, input, prompt)
|
||||
return runDefinitionInspection(invocation, suite, input, name)
|
||||
},
|
||||
}
|
||||
return commandkit.NewCommand(spec)
|
||||
@@ -419,77 +414,56 @@ func runCatalogList[T any, I, S any](
|
||||
})
|
||||
}
|
||||
|
||||
func runDefinitionInspection[I, S any](invocation commandkit.Invocation, suite Suite[I, S], input *definitionInput[I], prompt bool) commandkit.Outcome {
|
||||
if invocation.Context != nil {
|
||||
if err := invocation.Context.Err(); err != nil {
|
||||
return definitionFailure("PromptKit inspection was canceled.", commandkit.Explanation{}, err)
|
||||
}
|
||||
func runDefinitionInspection[I, S any](invocation commandkit.Invocation, suite Suite[I, S], input *definitionInput[I], kind string) commandkit.Outcome {
|
||||
if err := promptKitContextError(invocation.Context); err != nil {
|
||||
return definitionFailure("PromptKit inspection was canceled.", commandkit.Explanation{}, err)
|
||||
}
|
||||
var state S
|
||||
if suite.NewState != nil {
|
||||
state = suite.NewState()
|
||||
}
|
||||
if invocation.Context != nil {
|
||||
if err := invocation.Context.Err(); err != nil {
|
||||
return definitionFailure("PromptKit inspection was canceled.", commandkit.Explanation{}, err)
|
||||
}
|
||||
if err := promptKitContextError(invocation.Context); err != nil {
|
||||
return definitionFailure("PromptKit inspection was canceled.", commandkit.Explanation{}, err)
|
||||
}
|
||||
|
||||
if prompt {
|
||||
request := PromptRequest{Selection: input.selection, ID: input.id}
|
||||
acquired, err := suite.InspectPrompt(invocation.Context, request, input.selected, state)
|
||||
queuePromptDiagnostics(invocation.Reporter(), acquired.Diagnostics)
|
||||
if err != nil {
|
||||
return definitionFailure(fmt.Sprintf("Unable to inspect PromptKit prompt %q.", input.id), acquired.Explanation, err)
|
||||
var value any
|
||||
var diagnostics []commandkit.Diagnostic
|
||||
var explanation commandkit.Explanation
|
||||
var err error
|
||||
switch kind {
|
||||
case "prompt":
|
||||
var result commandkit.Acquired[*promptkit.PromptInspection]
|
||||
result, err = suite.InspectPrompt(invocation.Context, PromptRequest{Selection: input.selection, ID: input.id}, input.selected, state)
|
||||
if result.Value != nil {
|
||||
value = result.Value
|
||||
}
|
||||
if acquired.Value == nil {
|
||||
return definitionFailure("PromptKit prompt inspection returned no result.", acquired.Explanation, nil)
|
||||
diagnostics, explanation = result.Diagnostics, result.Explanation
|
||||
case "profile":
|
||||
var result commandkit.Acquired[*promptkit.ProfileInspection]
|
||||
result, err = suite.InspectProfile(invocation.Context, ProfileRequest{Selection: input.selection, ID: input.id}, input.selected, state)
|
||||
if result.Value != nil {
|
||||
value = result.Value
|
||||
}
|
||||
if invocation.Context != nil {
|
||||
if err := invocation.Context.Err(); err != nil {
|
||||
return definitionFailure(fmt.Sprintf("Inspection of PromptKit prompt %q was canceled.", input.id), acquired.Explanation, err)
|
||||
}
|
||||
diagnostics, explanation = result.Diagnostics, result.Explanation
|
||||
case "backend":
|
||||
var result commandkit.Acquired[*promptkit.BackendInspection]
|
||||
result, err = suite.InspectBackend(invocation.Context, BackendRequest{Selection: input.selection, ID: input.id}, input.selected, state)
|
||||
if result.Value != nil {
|
||||
value = result.Value
|
||||
}
|
||||
return renderPromptReport(invocation, acquired.Value, input.format, acquired.Explanation)
|
||||
diagnostics, explanation = result.Diagnostics, result.Explanation
|
||||
}
|
||||
|
||||
request := ProfileRequest{Selection: input.selection, ID: input.id}
|
||||
acquired, err := suite.InspectProfile(invocation.Context, request, input.selected, state)
|
||||
queuePromptDiagnostics(invocation.Reporter(), acquired.Diagnostics)
|
||||
queuePromptDiagnostics(invocation.Reporter(), diagnostics)
|
||||
if err != nil {
|
||||
return definitionFailure(fmt.Sprintf("Unable to inspect PromptKit profile %q.", input.id), acquired.Explanation, err)
|
||||
return definitionFailure(fmt.Sprintf("Unable to inspect PromptKit %s %q.", kind, input.id), explanation, err)
|
||||
}
|
||||
if acquired.Value == nil {
|
||||
return definitionFailure("PromptKit profile inspection returned no result.", acquired.Explanation, nil)
|
||||
if value == nil {
|
||||
return definitionFailure(fmt.Sprintf("PromptKit %s inspection returned no result.", kind), explanation, nil)
|
||||
}
|
||||
if invocation.Context != nil {
|
||||
if err := invocation.Context.Err(); err != nil {
|
||||
return definitionFailure(fmt.Sprintf("Inspection of PromptKit profile %q was canceled.", input.id), acquired.Explanation, err)
|
||||
}
|
||||
if err := promptKitContextError(invocation.Context); err != nil {
|
||||
return definitionFailure(fmt.Sprintf("Inspection of PromptKit %s %q was canceled.", kind, input.id), explanation, err)
|
||||
}
|
||||
return renderProfileReport(invocation, acquired.Value, input.format, acquired.Explanation)
|
||||
}
|
||||
|
||||
type promptInputReport struct {
|
||||
Name string `yaml:"name"`
|
||||
Required bool `yaml:"required"`
|
||||
ContentType string `yaml:"content_type"`
|
||||
Description string `yaml:"description"`
|
||||
}
|
||||
|
||||
type promptOutputContractReport struct {
|
||||
Format string `yaml:"format"`
|
||||
ValidationMode string `yaml:"validation_mode"`
|
||||
SchemaPath string `yaml:"schema_path"`
|
||||
RepairAttempts int `yaml:"repair_attempts"`
|
||||
}
|
||||
|
||||
type promptInspectionReport struct {
|
||||
PromptID string `yaml:"prompt_id"`
|
||||
PromptHash string `yaml:"prompt_hash"`
|
||||
DefaultProfileID string `yaml:"default_profile_id"`
|
||||
Inputs []promptInputReport `yaml:"inputs"`
|
||||
OutputContract promptOutputContractReport `yaml:"output_contract"`
|
||||
_ = invocation.Reporter().Report(commandkit.Report{Value: value})
|
||||
return commandkit.Success()
|
||||
}
|
||||
|
||||
type requirementsInspectionReport struct {
|
||||
@@ -524,45 +498,6 @@ type requirementIssueReport struct {
|
||||
Description string `yaml:"description"`
|
||||
}
|
||||
|
||||
type modelParamsReport struct {
|
||||
BackendID string `yaml:"backend_id"`
|
||||
Endpoint string `yaml:"endpoint"`
|
||||
Model string `yaml:"model"`
|
||||
Temperature float64 `yaml:"temperature"`
|
||||
MaxTokens int `yaml:"max_tokens"`
|
||||
TopP float64 `yaml:"top_p"`
|
||||
TimeoutSeconds int `yaml:"timeout_seconds"`
|
||||
ServiceTier string `yaml:"service_tier"`
|
||||
ReasoningEffort string `yaml:"reasoning_effort"`
|
||||
APIKeyEnv string `yaml:"api_key_env"`
|
||||
ExtraParams map[string]any `yaml:"extra_params"`
|
||||
}
|
||||
|
||||
type profileInspectionReport struct {
|
||||
ProfileID string `yaml:"profile_id"`
|
||||
EffectiveModelParams modelParamsReport `yaml:"effective_model_params"`
|
||||
APIKeyRequired bool `yaml:"api_key_required"`
|
||||
}
|
||||
|
||||
func promptReportFromInspection(inspection *promptkit.PromptInspection) promptInspectionReport {
|
||||
report := promptInspectionReport{
|
||||
PromptID: inspection.PromptID,
|
||||
PromptHash: inspection.PromptHash,
|
||||
DefaultProfileID: inspection.DefaultProfileID,
|
||||
Inputs: make([]promptInputReport, 0, len(inspection.Inputs)),
|
||||
OutputContract: promptOutputContractReport{
|
||||
Format: string(inspection.OutputContract.Format),
|
||||
ValidationMode: string(inspection.OutputContract.ValidationMode),
|
||||
SchemaPath: inspection.OutputContract.SchemaPath,
|
||||
RepairAttempts: inspection.OutputContract.RepairAttempts,
|
||||
},
|
||||
}
|
||||
for _, input := range inspection.Inputs {
|
||||
report.Inputs = append(report.Inputs, promptInputReport{Name: input.Name, Required: input.Required, ContentType: input.ContentType, Description: input.Description})
|
||||
}
|
||||
return report
|
||||
}
|
||||
|
||||
func requirementsReportFromInspection(inspection *promptkit.RequirementInspection) requirementsInspectionReport {
|
||||
report := requirementsInspectionReport{
|
||||
PromptID: inspection.PromptID,
|
||||
@@ -601,24 +536,6 @@ func requirementsReportFromInspection(inspection *promptkit.RequirementInspectio
|
||||
return report
|
||||
}
|
||||
|
||||
func profileReportFromInspection(inspection *promptkit.ProfileInspection) profileInspectionReport {
|
||||
params := inspection.EffectiveModelParams
|
||||
extra := params.ExtraParams
|
||||
if extra == nil {
|
||||
extra = map[string]any{}
|
||||
}
|
||||
return profileInspectionReport{
|
||||
ProfileID: inspection.ProfileID,
|
||||
EffectiveModelParams: modelParamsReport{
|
||||
BackendID: params.BackendID, Endpoint: params.Endpoint, Model: params.Model,
|
||||
Temperature: params.Temperature, MaxTokens: params.MaxTokens, TopP: params.TopP,
|
||||
TimeoutSeconds: params.TimeoutSeconds, ServiceTier: params.ServiceTier,
|
||||
ReasoningEffort: params.ReasoningEffort, APIKeyEnv: params.APIKeyEnv, ExtraParams: extra,
|
||||
},
|
||||
APIKeyRequired: inspection.APIKeyRequired,
|
||||
}
|
||||
}
|
||||
|
||||
func runRequirementsInspection[I, S any](invocation commandkit.Invocation, suite Suite[I, S], input *requirementsInput[I]) commandkit.Outcome {
|
||||
if invocation.Context != nil {
|
||||
if err := invocation.Context.Err(); err != nil {
|
||||
@@ -654,27 +571,6 @@ func runRequirementsInspection[I, S any](invocation commandkit.Invocation, suite
|
||||
})
|
||||
}
|
||||
|
||||
func renderPromptReport(invocation commandkit.Invocation, inspection *promptkit.PromptInspection, format string, explanation commandkit.Explanation) commandkit.Outcome {
|
||||
return renderPreparedPromptReport(invocation, explanation, func() ([]byte, error) {
|
||||
report := promptReportFromInspection(inspection)
|
||||
if format == "yaml" {
|
||||
return encodePromptYAMLBytes(report)
|
||||
}
|
||||
return []byte(formatPromptText(report)), nil
|
||||
})
|
||||
}
|
||||
|
||||
func renderProfileReport(invocation commandkit.Invocation, inspection *promptkit.ProfileInspection, format string, explanation commandkit.Explanation) commandkit.Outcome {
|
||||
return renderPreparedPromptReport(invocation, explanation, func() ([]byte, error) {
|
||||
report := profileReportFromInspection(inspection)
|
||||
if format == "yaml" {
|
||||
return encodePromptYAMLBytes(report)
|
||||
}
|
||||
text, err := formatProfileText(report)
|
||||
return []byte(text), err
|
||||
})
|
||||
}
|
||||
|
||||
// These reports were already prepared before delivery. Keep that boundary
|
||||
// inside managed rendering, with cancellation owned by the command outcome.
|
||||
func renderPreparedPromptReport(invocation commandkit.Invocation, explanation commandkit.Explanation, prepare func() ([]byte, error)) commandkit.Outcome {
|
||||
@@ -696,42 +592,6 @@ func renderPreparedPromptReport(invocation commandkit.Invocation, explanation co
|
||||
return commandkit.Success()
|
||||
}
|
||||
|
||||
func formatPromptText(report promptInspectionReport) string {
|
||||
var output strings.Builder
|
||||
fmt.Fprintf(&output, "Prompt ID: %s\n", strconv.Quote(report.PromptID))
|
||||
fmt.Fprintf(&output, "Prompt hash: %s\n", strconv.Quote(report.PromptHash))
|
||||
fmt.Fprintf(&output, "Default profile ID: %s\n", strconv.Quote(report.DefaultProfileID))
|
||||
output.WriteString("Inputs:\n")
|
||||
for _, input := range report.Inputs {
|
||||
fmt.Fprintf(&output, " - name: %s\n required: %t\n content_type: %s\n description: %s\n", strconv.Quote(input.Name), input.Required, strconv.Quote(input.ContentType), strconv.Quote(input.Description))
|
||||
}
|
||||
if len(report.Inputs) == 0 {
|
||||
output.WriteString(" []\n")
|
||||
}
|
||||
fmt.Fprintf(&output, "Output contract:\n format: %s\n validation_mode: %s\n schema_path: %s\n repair_attempts: %d\n", strconv.Quote(report.OutputContract.Format), strconv.Quote(report.OutputContract.ValidationMode), strconv.Quote(report.OutputContract.SchemaPath), report.OutputContract.RepairAttempts)
|
||||
return output.String()
|
||||
}
|
||||
|
||||
func formatProfileText(report profileInspectionReport) (string, error) {
|
||||
params := report.EffectiveModelParams
|
||||
var output strings.Builder
|
||||
fmt.Fprintf(&output, "Profile ID: %s\n", strconv.Quote(report.ProfileID))
|
||||
output.WriteString("Effective model params:\n")
|
||||
fmt.Fprintf(&output, " backend_id: %s\n endpoint: %s\n model: %s\n temperature: %s\n max_tokens: %d\n top_p: %s\n timeout_seconds: %d\n service_tier: %s\n reasoning_effort: %s\n api_key_env: %s\n", strconv.Quote(params.BackendID), strconv.Quote(params.Endpoint), strconv.Quote(params.Model), strconv.FormatFloat(params.Temperature, 'g', -1, 64), params.MaxTokens, strconv.FormatFloat(params.TopP, 'g', -1, 64), params.TimeoutSeconds, strconv.Quote(params.ServiceTier), strconv.Quote(params.ReasoningEffort), strconv.Quote(params.APIKeyEnv))
|
||||
output.WriteString(" extra_params:\n")
|
||||
var nested strings.Builder
|
||||
if err := encodePromptYAML(&nested, params.ExtraParams); err != nil {
|
||||
return "", err
|
||||
}
|
||||
for _, line := range strings.Split(strings.TrimSuffix(nested.String(), "\n"), "\n") {
|
||||
output.WriteString(" ")
|
||||
output.WriteString(line)
|
||||
output.WriteByte('\n')
|
||||
}
|
||||
fmt.Fprintf(&output, "API key required: %t\n", report.APIKeyRequired)
|
||||
return output.String(), nil
|
||||
}
|
||||
|
||||
func encodePromptYAML(writer io.Writer, value any) error {
|
||||
encoder := yaml.NewEncoder(writer)
|
||||
encoder.SetIndent(2)
|
||||
@@ -808,13 +668,6 @@ func definitionFailure(message string, explanation commandkit.Explanation, cause
|
||||
return outcome
|
||||
}
|
||||
|
||||
func checkDefinitionFormat(format string) error {
|
||||
if format != "text" && format != "yaml" {
|
||||
return fmt.Errorf("unsupported format %q; expected text or yaml", format)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func parseDefinitionString(value string) (string, error) { return value, nil }
|
||||
|
||||
func parseVariableAssignment(raw string) (variableAssignment, error) {
|
||||
|
||||
@@ -809,11 +809,11 @@ func TestCatalogAndInspectionRespectSuccessfulReportingModes(t *testing.T) {
|
||||
|
||||
func TestQuietInspectionStillFindsReportEncodingFailure(t *testing.T) {
|
||||
called := false
|
||||
extra := map[string]any{"unsupported": func() {}}
|
||||
app := newPromptTestApp(t, Command(Suite[struct{}, struct{}]{InspectProfile: func(context.Context, ProfileRequest, struct{}, struct{}) (commandkit.Acquired[*promptkit.ProfileInspection], error) {
|
||||
called = true
|
||||
return commandkit.Acquired[*promptkit.ProfileInspection]{Value: &promptkit.ProfileInspection{
|
||||
ProfileID: "broken",
|
||||
EffectiveModelParams: promptkit.ExecutionTarget{ExtraParams: map[string]any{"unsupported": func() {}}},
|
||||
ProfileID: "broken", ExtraParams: &extra,
|
||||
}}, nil
|
||||
}}))
|
||||
result := &countingWriter{}
|
||||
@@ -924,99 +924,6 @@ func TestDefinitionInputAndStateLifetimesFollowChecks(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestDefinitionReportsRetainNativeFieldsAndSafeCredentialReferences(t *testing.T) {
|
||||
t.Setenv("PROMPTKIT_ADAPTER_TEST_KEY", "credential-value-must-not-appear")
|
||||
engine := adapterEngine(t)
|
||||
loadCalls := 0
|
||||
suite := FromEngine(func(context.Context, ckconfig.Selection, struct{}, struct{}) (*promptkit.Engine, error) {
|
||||
loadCalls++
|
||||
return engine, nil
|
||||
})
|
||||
app := newPromptTestApp(t, Command(suite))
|
||||
var output bytes.Buffer
|
||||
outcome := app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "briefing", "--format", "yaml"}, commandkit.Streams{Out: &output, Err: io.Discard})
|
||||
if outcome.Kind != commandkit.OutcomeSuccess || loadCalls != 1 {
|
||||
t.Fatalf("prompt report outcome=%#v load calls=%d report=%s", outcome, loadCalls, output.String())
|
||||
}
|
||||
var prompt map[string]any
|
||||
if err := yaml.Unmarshal(output.Bytes(), &prompt); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, exists := prompt["version"]; exists {
|
||||
t.Fatalf("prompt version emitted: %#v", prompt)
|
||||
}
|
||||
inputs := prompt["inputs"].([]any)
|
||||
if len(inputs) != 2 || inputs[0].(map[string]any)["name"] != "topic" || inputs[1].(map[string]any)["name"] != "audience" || prompt["prompt_hash"] == "" {
|
||||
t.Fatalf("prompt fields/order = %#v", prompt)
|
||||
}
|
||||
contract := prompt["output_contract"].(map[string]any)
|
||||
if contract["repair_attempts"] != 0 {
|
||||
t.Fatalf("zero-valued contract field was lost: %#v", contract)
|
||||
}
|
||||
|
||||
output.Reset()
|
||||
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "briefing"}, commandkit.Streams{Out: &output, Err: io.Discard})
|
||||
if outcome.Kind != commandkit.OutcomeSuccess || loadCalls != 2 || strings.Contains(output.String(), "Prompt version:") || !strings.Contains(output.String(), `- name: "topic"`) {
|
||||
t.Fatalf("default text prompt report = (%#v, calls=%d, %s)", outcome, loadCalls, output.String())
|
||||
}
|
||||
|
||||
output.Reset()
|
||||
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "profile", "default", "--format", "yaml"}, commandkit.Streams{Out: &output, Err: io.Discard})
|
||||
if outcome.Kind != commandkit.OutcomeSuccess || loadCalls != 3 {
|
||||
t.Fatalf("profile report outcome=%#v calls=%d report=%s", outcome, loadCalls, output.String())
|
||||
}
|
||||
var profile map[string]any
|
||||
if err := yaml.Unmarshal(output.Bytes(), &profile); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
params := profile["effective_model_params"].(map[string]any)
|
||||
if params["api_key_env"] != "PROMPTKIT_ADAPTER_TEST_KEY" || params["temperature"] != 0 || params["max_tokens"] != 0 || strings.Contains(output.String(), "credential-value-must-not-appear") {
|
||||
t.Fatalf("credential or zero model fields = %#v / %s", params, output.String())
|
||||
}
|
||||
extra := params["extra_params"].(map[string]any)
|
||||
nested := extra["nested"].(map[string]any)
|
||||
sequence := extra["sequence"].([]any)
|
||||
if extra["token"] != "caller-owned-value" || nested["null_value"] != nil || nested["visible"] != true || len(sequence) != 3 || sequence[1] != nil || sequence[2] != 3 {
|
||||
t.Fatalf("nested ExtraParams changed: %#v", extra)
|
||||
}
|
||||
|
||||
output.Reset()
|
||||
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "profile", "default"}, commandkit.Streams{Out: &output, Err: io.Discard})
|
||||
if outcome.Kind != commandkit.OutcomeSuccess || loadCalls != 4 || !strings.Contains(output.String(), "API key required: false") || !strings.Contains(output.String(), `api_key_env: "PROMPTKIT_ADAPTER_TEST_KEY"`) {
|
||||
t.Fatalf("default text profile report = (%#v, calls=%d, %s)", outcome, loadCalls, output.String())
|
||||
}
|
||||
|
||||
output.Reset()
|
||||
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "profile", "requires-key", "--format", "yaml"}, commandkit.Streams{Out: &output, Err: io.Discard})
|
||||
if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(output.String(), "api_key_required: true") {
|
||||
t.Fatalf("credential requirement report = (%#v, %s)", outcome, output.String())
|
||||
}
|
||||
|
||||
output.Reset()
|
||||
outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "empty", "--format", "yaml"}, commandkit.Streams{Out: &output, Err: io.Discard})
|
||||
if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(output.String(), "inputs: []") {
|
||||
t.Fatalf("empty prompt input list = (%#v, %s)", outcome, output.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptReportOmitsRemovedVersion(t *testing.T) {
|
||||
report := promptReportFromInspection(&promptkit.PromptInspection{PromptID: "briefing", PromptHash: "hash"})
|
||||
encoded, err := encodePromptYAMLBytes(report)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var document map[string]any
|
||||
if err := yaml.Unmarshal(encoded, &document); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, exists := document["version"]; exists || document["prompt_id"] != "briefing" || document["prompt_hash"] != "hash" {
|
||||
t.Fatalf("prompt report = %#v", document)
|
||||
}
|
||||
if text := formatPromptText(report); strings.Contains(text, "Prompt version:") || !strings.Contains(text, `Prompt hash: "hash"`) {
|
||||
t.Fatalf("prompt text = %q", text)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNativeLookupErrorsAndBlankIDsReachCallbacks(t *testing.T) {
|
||||
nativeErr := errors.New("native lookup failure")
|
||||
calls := 0
|
||||
@@ -1075,7 +982,7 @@ func TestDefinitionFormatPreflightAndReportDeliveryFailure(t *testing.T) {
|
||||
return commandkit.Acquired[*promptkit.ProfileInspection]{Value: &promptkit.ProfileInspection{ProfileID: "default"}}, nil
|
||||
}}))
|
||||
var output bytes.Buffer
|
||||
outcome := app.Execute(context.Background(), []string{"promptkit", "show", "profile", "default", "--format", "json"}, commandkit.Streams{Out: &output, Err: io.Discard})
|
||||
outcome := app.Execute(context.Background(), []string{"promptkit", "show", "profile", "default", "--format", "invalid"}, commandkit.Streams{Out: &output, Err: io.Discard})
|
||||
if outcome.Kind != commandkit.OutcomeUsage || calls != 0 || output.Len() != 0 {
|
||||
t.Fatalf("bad format was not rejected before callback: outcome=%#v calls=%d output=%q", outcome, calls, output.String())
|
||||
}
|
||||
@@ -1098,17 +1005,6 @@ func (writer partialReportWriter) Write(content []byte) (int, error) {
|
||||
|
||||
var _ io.Writer = partialReportWriter{}
|
||||
|
||||
func TestProfileTextUsesCredentialNamesAndExplicitEmptyExtraParams(t *testing.T) {
|
||||
report := profileReportFromInspection(&promptkit.ProfileInspection{ProfileID: "default", EffectiveModelParams: promptkit.ExecutionTarget{APIKeyEnv: "APP_KEY", ExtraParams: nil}})
|
||||
text, err := formatProfileText(report)
|
||||
if err != nil || !strings.Contains(text, "api_key_env: \"APP_KEY\"") || !strings.Contains(text, "extra_params:\n {}\n") || !strings.Contains(text, "API key required: false") {
|
||||
t.Fatalf("profile text = (%q, %v)", text, err)
|
||||
}
|
||||
if strings.Contains(text, "APP_KEY_VALUE") {
|
||||
t.Fatalf("report contains a credential value: %q", text)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptCatalogYAMLPreservesSchemaStringsAndOrder(t *testing.T) {
|
||||
want := []promptkit.PromptIdentity{
|
||||
{ID: " briefing "},
|
||||
|
||||
@@ -2,16 +2,16 @@
|
||||
|
||||
## PromptKit
|
||||
|
||||
Selected module: `go.maximumdirect.net/promptkit` at `v0.18.0`.
|
||||
Selected module: `go.maximumdirect.net/promptkit` at `v0.20.0`.
|
||||
|
||||
Copyright (C) 2026 Eric Rakestraw
|
||||
|
||||
SPDX-License-Identifier: LGPL-3.0-or-later
|
||||
|
||||
License and notices: [LGPL v3.0](../../LICENSE) |
|
||||
[LICENSE](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/d9f6aad30509e538fc73ea16710df3a3ec084502/LICENSE).
|
||||
[LICENSE](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/8ff9aa3efdc9f16b9d2729e0a29ea083eb3c3eaf/LICENSE).
|
||||
|
||||
[Source](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/d9f6aad30509e538fc73ea16710df3a3ec084502).
|
||||
[Source](https://forge.maximumdirect.net/base-layer/promptkit/src/commit/8ff9aa3efdc9f16b9d2729e0a29ea083eb3c3eaf).
|
||||
|
||||
## ConfigKit
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ require (
|
||||
go.maximumdirect.net/ckconfig v0.3.0
|
||||
go.maximumdirect.net/commandkit v0.3.0
|
||||
go.maximumdirect.net/configkit v1.1.0
|
||||
go.maximumdirect.net/promptkit v0.18.0
|
||||
go.maximumdirect.net/promptkit v0.20.0
|
||||
go.yaml.in/yaml/v4 v4.0.0-rc.6
|
||||
)
|
||||
|
||||
|
||||
@@ -18,6 +18,8 @@ go.maximumdirect.net/configkit v1.1.0 h1:UzXN2ZMnEkcL0O4YSLqD9qd9M1gb/NWCHQR/H8A
|
||||
go.maximumdirect.net/configkit v1.1.0/go.mod h1:s1zLJ9GGB/1Q4a+aOHMLISonJxWFgNWpA6SpPlOafeA=
|
||||
go.maximumdirect.net/promptkit v0.18.0 h1:mrO+mangAPiR1xdtYjTHemH5xA//waddekVnhI5i0yE=
|
||||
go.maximumdirect.net/promptkit v0.18.0/go.mod h1:NTdAFcqISmmgiAGch2+s/mHrZTrkUXJYGKc1UfUMF24=
|
||||
go.maximumdirect.net/promptkit v0.20.0 h1:ZpwbjiCZWXdnnZ26RuWKoOco0jFnaAX8m2tQAX7ZlnM=
|
||||
go.maximumdirect.net/promptkit v0.20.0/go.mod h1:kCoyvI0xSax9K9aRPCEkpkyAP5YLs157oBLFLdzgRbM=
|
||||
go.yaml.in/yaml/v4 v4.0.0-rc.6 h1:1h7H1ohdUh93/FyE4YaDa1Zh64K6VVbjF4K6WUxMtH4=
|
||||
go.yaml.in/yaml/v4 v4.0.0-rc.6/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
|
||||
golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
|
||||
|
||||
@@ -2,33 +2,30 @@ package ckprompt
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
commandkit "go.maximumdirect.net/commandkit"
|
||||
promptkit "go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
type cancelDuringYAML struct{ cancel context.CancelFunc }
|
||||
type cancelDuringJSON struct{ cancel context.CancelFunc }
|
||||
|
||||
func (value cancelDuringYAML) MarshalYAML() (any, error) {
|
||||
func (value cancelDuringJSON) MarshalJSON() ([]byte, error) {
|
||||
value.cancel()
|
||||
return "prepared", nil
|
||||
return []byte(`"prepared"`), nil
|
||||
}
|
||||
|
||||
func TestInspectionCancellationDuringPreparationPreventsDelivery(t *testing.T) {
|
||||
func TestInspectionRetainsAcquiredReportWhenRenderingCancels(t *testing.T) {
|
||||
for _, format := range []string{"text", "yaml"} {
|
||||
for _, mode := range []string{"ordinary", "quiet"} {
|
||||
t.Run(format+"/"+mode, func(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
extra := map[string]any{"controlled": cancelDuringJSON{cancel}}
|
||||
app := newPromptTestApp(t, Command(Suite[struct{}, struct{}]{
|
||||
InspectProfile: func(context.Context, ProfileRequest, struct{}, struct{}) (commandkit.Acquired[*promptkit.ProfileInspection], error) {
|
||||
return commandkit.Acquired[*promptkit.ProfileInspection]{Value: &promptkit.ProfileInspection{
|
||||
ProfileID: "demo",
|
||||
EffectiveModelParams: promptkit.ExecutionTarget{ExtraParams: map[string]any{
|
||||
"controlled": cancelDuringYAML{cancel},
|
||||
}},
|
||||
ProfileID: "demo", ExtraParams: &extra,
|
||||
}}, nil
|
||||
},
|
||||
}))
|
||||
@@ -38,7 +35,7 @@ func TestInspectionCancellationDuringPreparationPreventsDelivery(t *testing.T) {
|
||||
}
|
||||
output := &countingWriter{}
|
||||
got := app.Execute(ctx, args, commandkit.Streams{Out: output})
|
||||
if got.Kind != commandkit.OutcomeOperational || got.Status != 1 || !errors.Is(got.Cause, context.Canceled) || output.calls != 0 {
|
||||
if got.Kind != commandkit.OutcomeSuccess || got.Status != 0 || ctx.Err() != context.Canceled || (mode == "ordinary" && output.calls == 0) || (mode == "quiet" && output.calls != 0) {
|
||||
t.Fatalf("outcome=%#v writes=%d", got, output.calls)
|
||||
}
|
||||
if got.Presentation.Output.RenderError != nil || got.Presentation.Output.WriteError != nil {
|
||||
|
||||
192
integrations/ckprompt/resource_inspection_test.go
Normal file
192
integrations/ckprompt/resource_inspection_test.go
Normal file
@@ -0,0 +1,192 @@
|
||||
package ckprompt
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"go.maximumdirect.net/ckconfig"
|
||||
commandkit "go.maximumdirect.net/commandkit"
|
||||
promptkit "go.maximumdirect.net/promptkit"
|
||||
)
|
||||
|
||||
func jsonValue(t *testing.T, value any) any {
|
||||
t.Helper()
|
||||
data, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
decoder := json.NewDecoder(bytes.NewReader(data))
|
||||
decoder.UseNumber()
|
||||
var result any
|
||||
if err := decoder.Decode(&result); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func TestResourceInspectionsForwardNativeReports(t *testing.T) {
|
||||
zero := 0
|
||||
no := false
|
||||
empty := ""
|
||||
inputs := []promptkit.PromptInputInspection{{Name: "topic", Required: &no}}
|
||||
extra := map[string]any{}
|
||||
prompt := &promptkit.PromptInspection{PromptID: "p", PromptHash: "hash", Description: &empty, Inputs: &inputs, Messages: []promptkit.PromptMessageInspection{{Role: "user", Content: "captured {{.topic}}"}}}
|
||||
profile := &promptkit.ProfileInspection{ProfileID: "p", Temperature: nil, MaxTokens: &zero, APIKeyRequired: &no, ExtraParams: &extra}
|
||||
backend := &promptkit.BackendInspection{BackendID: "b", Endpoint: "https://example.invalid", QueueCapacity: &zero, ExtraParams: &extra}
|
||||
var selections []ckconfig.Selection
|
||||
calls := 0
|
||||
suite := Suite[struct{}, struct{}]{
|
||||
InspectPrompt: func(_ context.Context, r PromptRequest, _, _ struct{}) (commandkit.Acquired[*promptkit.PromptInspection], error) {
|
||||
calls++
|
||||
selections = append(selections, r.Selection)
|
||||
if r.ID != "p" {
|
||||
t.Errorf("prompt ID %q", r.ID)
|
||||
}
|
||||
return commandkit.Acquired[*promptkit.PromptInspection]{Value: prompt}, nil
|
||||
},
|
||||
InspectProfile: func(_ context.Context, r ProfileRequest, _, _ struct{}) (commandkit.Acquired[*promptkit.ProfileInspection], error) {
|
||||
calls++
|
||||
selections = append(selections, r.Selection)
|
||||
if r.ID != "p" {
|
||||
t.Errorf("profile ID %q", r.ID)
|
||||
}
|
||||
return commandkit.Acquired[*promptkit.ProfileInspection]{Value: profile}, nil
|
||||
},
|
||||
InspectBackend: func(_ context.Context, r BackendRequest, _, _ struct{}) (commandkit.Acquired[*promptkit.BackendInspection], error) {
|
||||
calls++
|
||||
selections = append(selections, r.Selection)
|
||||
if r.ID != "b" {
|
||||
t.Errorf("backend ID %q", r.ID)
|
||||
}
|
||||
return commandkit.Acquired[*promptkit.BackendInspection]{Value: backend}, nil
|
||||
},
|
||||
}
|
||||
app := newPromptTestApp(t, Command(suite), commandkit.Completion("example"))
|
||||
for _, kind := range []struct {
|
||||
name, id string
|
||||
value any
|
||||
}{{"prompt", "p", prompt}, {"profile", "p", profile}, {"backend", "b", backend}} {
|
||||
for _, format := range []string{"text", "yaml", "json"} {
|
||||
t.Run(kind.name+"/"+format, func(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
outcome := app.Execute(t.Context(), []string{"promptkit", "show", kind.name, kind.id, "--config=fixture", "--format=" + format}, commandkit.Streams{Out: &out, Err: io.Discard})
|
||||
if outcome.Status != 0 {
|
||||
t.Fatalf("%#v, %s", outcome, out.String())
|
||||
}
|
||||
var got any
|
||||
var err error
|
||||
if format == "json" {
|
||||
d := json.NewDecoder(&out)
|
||||
d.UseNumber()
|
||||
err = d.Decode(&got)
|
||||
} else {
|
||||
got, err = commandkit.DecodeYAMLReport(&out)
|
||||
}
|
||||
if err != nil || !reflect.DeepEqual(got, jsonValue(t, kind.value)) {
|
||||
t.Fatalf("got %#v, %v; want %#v", got, err, jsonValue(t, kind.value))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
for _, selection := range selections {
|
||||
if !selection.Config.Set || selection.Config.Value != "fixture" {
|
||||
t.Fatalf("selection %#v", selection)
|
||||
}
|
||||
}
|
||||
before := calls
|
||||
for _, argv := range [][]string{{"promptkit", "show", "backend", "--help"}, {"__complete", "promptkit", "show", "backend", ""}} {
|
||||
var help bytes.Buffer
|
||||
result := app.Execute(t.Context(), argv, commandkit.Streams{Out: &help, Err: io.Discard})
|
||||
if result.Status != 0 || calls != before {
|
||||
t.Fatalf("%v: %#v calls %d", argv, result, calls)
|
||||
}
|
||||
if argv[0] == "promptkit" && !strings.Contains(help.String(), "BACKEND_ID") {
|
||||
t.Fatalf("backend help = %q", help.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackendInspectionOptionalAndNativeErrorIdentity(t *testing.T) {
|
||||
calls := 0
|
||||
engine := adapterEngine(t)
|
||||
suite := FromEngine(func(context.Context, ckconfig.Selection, struct{}, struct{}) (*promptkit.Engine, error) {
|
||||
calls++
|
||||
return engine, nil
|
||||
})
|
||||
backendOnly := Suite[struct{}, struct{}]{InspectBackend: suite.InspectBackend}
|
||||
app := newPromptTestApp(t, Command(backendOnly))
|
||||
result := app.Execute(t.Context(), []string{"promptkit", "show", "backend", "missing"}, commandkit.Streams{Err: io.Discard})
|
||||
if !errors.Is(result.Cause, promptkit.ErrBackendNotFound) || result.Kind != commandkit.OutcomeOperational || calls != 1 {
|
||||
t.Fatalf("outcome %#v calls %d", result, calls)
|
||||
}
|
||||
without := newPromptTestApp(t, Command(Suite[struct{}, struct{}]{InspectPrompt: suite.InspectPrompt}))
|
||||
result = without.Execute(t.Context(), []string{"promptkit", "show", "backend", "missing"}, commandkit.Streams{Err: io.Discard})
|
||||
if result.Kind != commandkit.OutcomeUsage || calls != 1 {
|
||||
t.Fatalf("outcome %#v calls %d", result, calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEngineInspectionReportsMatchNativeResults(t *testing.T) {
|
||||
t.Setenv("PROMPTKIT_ADAPTER_TEST_KEY", "private-credential-value")
|
||||
engine := adapterEngine(t)
|
||||
calls := 0
|
||||
native := FromEngine(func(context.Context, ckconfig.Selection, struct{}, struct{}) (*promptkit.Engine, error) {
|
||||
calls++
|
||||
return engine, nil
|
||||
})
|
||||
suite := Suite[struct{}, struct{}]{InspectPrompt: native.InspectPrompt, InspectProfile: native.InspectProfile, InspectBackend: native.InspectBackend}
|
||||
app := newPromptTestApp(t, Command(suite))
|
||||
ctx := t.Context()
|
||||
prompt, err := engine.InspectPrompt(ctx, "briefing")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
profile, err := engine.InspectProfile(ctx, "default")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
backend, err := engine.InspectBackend(ctx, "local")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
name, id string
|
||||
want any
|
||||
}{{"prompt", "briefing", prompt}, {"profile", "default", profile}, {"backend", "local", backend}} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
var out bytes.Buffer
|
||||
result := app.Execute(ctx, []string{"promptkit", "show", tc.name, tc.id, "--format=json"}, commandkit.Streams{Out: &out, Err: io.Discard})
|
||||
if bytes.Contains(out.Bytes(), []byte("private-credential-value")) {
|
||||
t.Fatal("resource report exposed a credential value")
|
||||
}
|
||||
var got any
|
||||
decoder := json.NewDecoder(&out)
|
||||
decoder.UseNumber()
|
||||
err := decoder.Decode(&got)
|
||||
if result.Status != 0 || err != nil || !reflect.DeepEqual(got, jsonValue(t, tc.want)) {
|
||||
t.Fatalf("result=%#v got=%#v err=%v want=%#v", result, got, err, jsonValue(t, tc.want))
|
||||
}
|
||||
})
|
||||
}
|
||||
if calls != 3 {
|
||||
t.Fatalf("loader calls %d", calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackendInspectionPreservesDiagnosticsAndCause(t *testing.T) {
|
||||
cause := errors.New("backend lookup failed")
|
||||
app := newPromptTestApp(t, Command(Suite[struct{}, struct{}]{InspectBackend: func(context.Context, BackendRequest, struct{}, struct{}) (commandkit.Acquired[*promptkit.BackendInspection], error) {
|
||||
return commandkit.Acquired[*promptkit.BackendInspection]{Diagnostics: []commandkit.Diagnostic{commandkit.MessageDiagnostic(commandkit.SeverityWarning, "loader warning")}}, cause
|
||||
}}))
|
||||
var diagnostics bytes.Buffer
|
||||
result := app.Execute(t.Context(), []string{"promptkit", "show", "backend", "example", "--verbose"}, commandkit.Streams{Out: io.Discard, Err: &diagnostics})
|
||||
if !errors.Is(result.Cause, cause) || !strings.Contains(diagnostics.String(), "loader warning") {
|
||||
t.Fatalf("result=%#v diagnostics=%q", result, diagnostics.String())
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user