diff --git a/README.md b/README.md index 2c5d0b2..8797d1c 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,8 @@ and operand bindings, Cobra-backed routing, invocation outcomes, generated help, application or CommandKit version reporting, configuration scope discovery, ConfigKit-backed validation, native effective YAML display, source inventory, and path explanation reports. Applications can register a PromptKit engine -configuration scope independently of PromptKit definition commands. +configuration scope independently of PromptKit definition commands, and can +install optional prompt and profile inspection commands. ## Documentation diff --git a/docs/cli.md b/docs/cli.md index 3b98a6b..3e9abcc 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -225,6 +225,76 @@ environment-variable name; effective and historical secret fragments remain redacted. Empty location, inheritance, history, reason, and dependency lists are serialized as `[]`. +## PromptKit definition inspection + +Applications can install prompt and profile inspection independently with +`RegisterPromptKit`. Each installed handler constructs the selected engine once +and asks PromptKit to inspect the named definition. Inspection reads the +engine's captured configuration; it does not generate a prompt, contact a +provider, or look up credentials. +Prompt definitions and profile resolution remain owned by +[PromptKit](https://pkg.go.dev/gitea.maximumdirect.net/eric/promptkit). + +```text +example promptkit show prompt customer/summary --version 2 --format yaml +example promptkit show prompt customer/summary +example promptkit show profile customer-support --format yaml +``` + +Both commands require one ID operand and accept `--config`, `--overlay`, and +`--format text|yaml` (default `text`). Prompt inspection also accepts singleton +`--version`, defaulting to the empty string. PromptKit owns version selection: +an omitted version works for a single-version ID, while an ambiguous ID is an +operational error. These commands do not accept `--scope`; their namespace is +`promptkit`. + +Prompt YAML has these fields. Input order is preserved and an empty input list is +`[]`. The prompt hash is an opaque owning-library equality value. + +```yaml +prompt_id: customer/summary +prompt_version: "2" +prompt_hash: opaque-value +default_profile_id: customer-support +inputs: + - name: topic + required: true + content_type: text/plain + description: Topic to summarize +output_contract: + format: json + validation_mode: basic + schema_path: "" + repair_attempts: 2 +``` + +Profile YAML preserves the effective values reported by PromptKit, including +zero values and nested provider parameters. `api_key_env` is only an +environment-variable name. `api_key_required` reports a direct credential +requirement; neither field reports credential availability. A nil provider +parameter map is displayed as `{}`. + +```yaml +profile_id: customer-support +effective_model_params: + backend_id: local + endpoint: https://example.invalid/v1 + model: support-model + temperature: 0 + max_tokens: 0 + top_p: 0 + timeout_seconds: 600 + service_tier: "" + reasoning_effort: "" + api_key_env: "" + extra_params: {} +api_key_required: true +``` + +Text uses labeled fields and includes nested `extra_params` values. A missing ID +is a usage error. Blank, unknown, ambiguous, and canceled selections are +operational inspection results from PromptKit and retain its error identity. + ## Outcome status CommandKit usage failures use status `2`; operational failures use status `1`. diff --git a/docs/development.md b/docs/development.md index 8b9a349..695635d 100644 --- a/docs/development.md +++ b/docs/development.md @@ -6,8 +6,8 @@ declarations, an instance-owned registry, a Cobra-backed routing and parsing adapter, public `App.Execute` coordination, and generated help/version commands. Configuration suite registration, scope discovery, ConfigKit integration, validation, native effective YAML display, source inventory, and path -explanation reports, and PromptKit engine configuration inspection are -implemented; named prompt/profile inspection remains pending. The +explanation reports, PromptKit engine configuration inspection, and optional +PromptKit prompt/profile inspection commands are implemented. The [README](../README.md) provides project orientation; the [documentation policy](policy/documentation.md#canonical-ownership) assigns canonical documentation owners. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index c9e1e5d..5bd303e 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,6 +1,6 @@ # Initial CommandKit implementation plan -Status: implementation underway; Stages 1-12 complete and Stages 13-14 pending. +Status: implementation underway; Stages 1-13 complete and Stage 14 pending. Date: 2026-10-07. @@ -750,7 +750,7 @@ PromptKit/ConfigKit error identities. `go test ./...` passes. ## Stage 13: implement named prompt/profile inspection -**Status:** pending. **Prerequisite:** Stage 12. +**Status:** complete. **Prerequisite:** Stage 12. ```go type PromptKitSuite[C any] struct { @@ -811,6 +811,11 @@ formats, nil/empty parameters, independently installed prompt/profile capability definition-only registration, and help without engine construction. Assert no handler enters generation rather than exercising paid/network execution paths. +**Completion evidence:** `promptkit_test.go` exercises real captured prompt and +profile definitions, both report formats, inheritance and provider parameters, +selection errors, optional capabilities, and help without engine construction. +`go test ./...` passes. + ## Stage 14: substantiate consumer fit and complete documentation **Status:** pending. **Prerequisite:** Stage 13. diff --git a/promptkit.go b/promptkit.go new file mode 100644 index 0000000..ee7c051 --- /dev/null +++ b/promptkit.go @@ -0,0 +1,463 @@ +package commandkit + +import ( + "context" + "errors" + "fmt" + "io" + "strconv" + "strings" + + configkit "gitea.maximumdirect.net/eric/ConfigKit" + "gitea.maximumdirect.net/eric/promptkit" +) + +// PromptKitSuite declares optional PromptKit prompt and profile inspection +// commands for an App. +type PromptKitSuite[C any] struct { + // NewContext creates command-local application context when an inspection runs. + NewContext func() C + // Selectors declares context flags accepted by the installed commands. Each + // explicit scope list must include "promptkit"; an empty list applies here. + Selectors []Selector[C] + // CheckInput validates standard configuration selections and selector values. + CheckInput func(Selection, C) error + // InspectPrompt inspects an ID and exact or omitted version. + InspectPrompt func(context.Context, Selection, C, string, string) (Acquired[*promptkit.PromptInspection], error) + // InspectProfile inspects one profile ID. + InspectProfile func(context.Context, Selection, C, string) (Acquired[*promptkit.ProfileInspection], error) +} + +// NewPromptKitSuite creates prompt and profile inspection callbacks backed by +// a PromptKit Engine loader. The loader runs once per selected command. Each +// callback captures configuration diagnostics before calling the corresponding +// PromptKit inspection method, and returns those diagnostics with lookup errors. +func NewPromptKitSuite[C any]( + load func(context.Context, Selection, C) (*promptkit.Engine, error), +) (PromptKitSuite[C], error) { + if load == nil { + return PromptKitSuite[C]{}, fmt.Errorf("PromptKit engine loader is nil") + } + return PromptKitSuite[C]{ + InspectPrompt: func(ctx context.Context, selection Selection, appContext C, id, version string) (Acquired[*promptkit.PromptInspection], error) { + engine, _, diagnostics, err := loadPromptKitInspection(ctx, selection, appContext, load) + if err != nil { + return Acquired[*promptkit.PromptInspection]{Diagnostics: diagnostics}, err + } + inspection, err := engine.InspectPrompt(ctx, id, version) + return Acquired[*promptkit.PromptInspection]{Value: inspection, Diagnostics: diagnostics}, err + }, + InspectProfile: func(ctx context.Context, selection Selection, appContext C, id string) (Acquired[*promptkit.ProfileInspection], error) { + engine, _, diagnostics, err := loadPromptKitInspection(ctx, selection, appContext, load) + if err != nil { + return Acquired[*promptkit.ProfileInspection]{Diagnostics: diagnostics}, err + } + inspection, err := engine.InspectProfile(ctx, id) + return Acquired[*promptkit.ProfileInspection]{Value: inspection, Diagnostics: diagnostics}, err + }, + }, nil +} + +func loadPromptKitInspection[C any](ctx context.Context, selection Selection, appContext C, load func(context.Context, Selection, C) (*promptkit.Engine, error)) (*promptkit.Engine, promptkit.ConfigurationInspection, []Diagnostic, error) { + engine, err := load(ctx, selection, appContext) + if err != nil { + var loadErr *configkit.LoadError + if errors.As(err, &loadErr) { + return nil, promptkit.ConfigurationInspection{}, ConfigKitDiagnostics("promptkit", loadErr.Sources(), loadErr.Diagnostics()), err + } + return nil, promptkit.ConfigurationInspection{}, nil, err + } + if engine == nil { + return nil, promptkit.ConfigurationInspection{}, nil, fmt.Errorf("PromptKit configuration loader returned a nil engine") + } + inspection, err := engine.InspectConfiguration() + if err != nil { + return nil, promptkit.ConfigurationInspection{}, nil, err + } + sources := inspection.Sources() + diagnostics := ConfigKitDiagnostics("promptkit", sources, inspection.Diagnostics()) + return engine, inspection, diagnostics, nil +} + +// RegisterPromptKit installs the supplied PromptKit inspection commands under +// promptkit show. Each nonnil callback installs only its matching command. A +// suite with no inspection callbacks installs no PromptKit namespace. +func RegisterPromptKit[C any](app *App, suite PromptKitSuite[C]) error { + if suite.InspectPrompt == nil && suite.InspectProfile == nil { + return nil + } + if app == nil { + return fmt.Errorf("cannot register PromptKit commands on a nil app") + } + normalized, err := normalizePromptKitSuite(suite) + if err != nil { + return err + } + + showCommands := make([]Command, 0, 2) + if normalized.InspectPrompt != nil { + command, err := newPromptKitPromptCommand(normalized) + if err != nil { + return err + } + showCommands = append(showCommands, command) + } + if normalized.InspectProfile != nil { + command, err := newPromptKitProfileCommand(normalized) + if err != nil { + return err + } + showCommands = append(showCommands, command) + } + show, err := Group("show", "Inspect PromptKit definitions.", showCommands...) + if err != nil { + return err + } + root, err := Group("promptkit", "Inspect PromptKit definitions.", show) + if err != nil { + return err + } + return app.Register(root) +} + +func normalizePromptKitSuite[C any](suite PromptKitSuite[C]) (PromptKitSuite[C], error) { + normalized := suite + normalized.Selectors = append([]Selector[C](nil), suite.Selectors...) + seenNames := make(map[string]struct{}, len(normalized.Selectors)) + seenShorts := make(map[string]struct{}, len(normalized.Selectors)) + for i, selector := range normalized.Selectors { + binding := selector.Binding.binding + meta := binding.meta + if !binding.valid || binding.clear == nil || meta.Name == "" { + return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector %d has an invalid flag binding", i) + } + if !validFlagName(meta.Name) { + return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector flag name %q is invalid", meta.Name) + } + switch meta.Name { + case "config", "overlay", "scope", "format", "version", "help": + return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector flag --%s conflicts with a standard flag", meta.Name) + } + if meta.Short == "h" { + return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector short flag -%s conflicts with help", meta.Short) + } + if meta.Short != "" && (len(meta.Short) != 1 || meta.Short == "-") { + return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector short spelling %q must be one byte", meta.Short) + } + if _, exists := seenNames[meta.Name]; exists { + return PromptKitSuite[C]{}, fmt.Errorf("duplicate PromptKit selector flag --%s", meta.Name) + } + seenNames[meta.Name] = struct{}{} + if meta.Short != "" { + if _, exists := seenShorts[meta.Short]; exists { + return PromptKitSuite[C]{}, fmt.Errorf("duplicate PromptKit selector short flag -%s", meta.Short) + } + seenShorts[meta.Short] = struct{}{} + } + applies := len(selector.Scopes) == 0 + for _, scope := range selector.Scopes { + if scope == "promptkit" { + applies = true + break + } + } + if !applies { + return PromptKitSuite[C]{}, fmt.Errorf("PromptKit selector --%s must apply to scope %q", meta.Name, "promptkit") + } + normalized.Selectors[i].Scopes = append([]string(nil), selector.Scopes...) + } + return normalized, nil +} + +type promptKitSelectorState[C any] struct { + scopes []string + apply func(*C) + clear func(*C) +} + +type promptKitInput[C any] struct { + Selection Selection + Format string + ID string + Version string + SelectorContext C + selectors []promptKitSelectorState[C] +} + +func promptKitInputFlags[C any](suite PromptKitSuite[C], prompt bool) []Flag[promptKitInput[C]] { + formatDefault := "text" + flags := []Flag[promptKitInput[C]]{ + ValueFlag[promptKitInput[C]](FlagMeta{Name: "format", Usage: "Output format."}, &formatDefault, parseString, func(input *promptKitInput[C], value Supplied[string]) { + input.Format = value.Value + }), + ValueFlag[promptKitInput[C]](FlagMeta{Name: "config", Usage: "Configuration source."}, nil, parseString, func(input *promptKitInput[C], value Supplied[string]) { + input.Selection.Config = value + }), + ValueFlag[promptKitInput[C]](FlagMeta{Name: "overlay", Usage: "Overlay source."}, nil, parseString, func(input *promptKitInput[C], value Supplied[string]) { + input.Selection.Overlay = value + }), + } + if prompt { + versionDefault := "" + flags = append(flags, ValueFlag[promptKitInput[C]](FlagMeta{Name: "version", Usage: "Exact prompt version; empty selects the owning default."}, &versionDefault, parseString, func(input *promptKitInput[C], value Supplied[string]) { + input.Version = value.Value + })) + } + for _, selector := range suite.Selectors { + flags = append(flags, adaptPromptKitSelector(selector)) + } + return flags +} + +func adaptPromptKitSelector[C any](selector Selector[C]) Flag[promptKitInput[C]] { + binding := selector.Binding.binding + meta := binding.meta + return Flag[promptKitInput[C]]{binding: flagBinding[promptKitInput[C]]{ + meta: meta, + defaultText: binding.defaultText, + repeated: binding.repeated, + boolean: binding.boolean, + valid: binding.valid, + prepare: func(values []string) (func(*promptKitInput[C]), error) { + apply, err := binding.prepare(values) + if err != nil { + return nil, err + } + state := promptKitSelectorState[C]{scopes: append([]string(nil), selector.Scopes...), apply: apply, clear: binding.clear} + return func(input *promptKitInput[C]) { + apply(&input.SelectorContext) + input.selectors = append(input.selectors, state) + }, nil + }, + }} +} + +func checkPromptKitInput[C any](suite PromptKitSuite[C], input promptKitInput[C]) error { + if _, err := parseConfigFormat(input.Format); err != nil { + return err + } + if suite.CheckInput != nil { + return suite.CheckInput(input.Selection, input.SelectorContext) + } + return nil +} + +func promptKitContext[C any](suite PromptKitSuite[C], input promptKitInput[C]) C { + var result C + if suite.NewContext != nil { + result = suite.NewContext() + } + for _, selector := range input.selectors { + if selectorApplies(selector.scopes, "promptkit") { + selector.apply(&result) + } else { + selector.clear(&result) + } + } + return result +} + +func newPromptKitPromptCommand[C any](suite PromptKitSuite[C]) (Command, error) { + inputSpec := CommandSpec[promptKitInput[C]]{ + Name: "prompt", + Summary: "Inspect one PromptKit prompt definition.", + Flags: promptKitInputFlags(suite, true), + Arguments: []Argument[promptKitInput[C]]{ValueArgument[promptKitInput[C]](ArgumentMeta{Name: "ID"}, true, parseString, func(input *promptKitInput[C], value Supplied[string]) { + input.ID = value.Value + })}, + CheckInput: func(input promptKitInput[C]) error { return checkPromptKitInput(suite, input) }, + Run: func(invocation Invocation, input promptKitInput[C]) Outcome { + return runPromptKitPrompt(invocation, suite, input) + }, + } + return NewCommand(inputSpec) +} + +func newPromptKitProfileCommand[C any](suite PromptKitSuite[C]) (Command, error) { + inputSpec := CommandSpec[promptKitInput[C]]{ + Name: "profile", + Summary: "Inspect one PromptKit profile definition.", + Flags: promptKitInputFlags(suite, false), + Arguments: []Argument[promptKitInput[C]]{ValueArgument[promptKitInput[C]](ArgumentMeta{Name: "ID"}, true, parseString, func(input *promptKitInput[C], value Supplied[string]) { + input.ID = value.Value + })}, + CheckInput: func(input promptKitInput[C]) error { return checkPromptKitInput(suite, input) }, + Run: func(invocation Invocation, input promptKitInput[C]) Outcome { + return runPromptKitProfile(invocation, suite, input) + }, + } + return NewCommand(inputSpec) +} + +func runPromptKitPrompt[C any](invocation Invocation, suite PromptKitSuite[C], input promptKitInput[C]) Outcome { + contextValue := promptKitContext(suite, input) + acquired, err := suite.InspectPrompt(invocation.Context, input.Selection, contextValue, input.ID, input.Version) + if err != nil { + return promptKitFailure(err, acquired.Diagnostics) + } + if acquired.Value == nil { + return promptKitFailure(fmt.Errorf("PromptKit prompt inspection returned no result"), acquired.Diagnostics) + } + report := promptReportFromInspection(acquired.Value) + return writePromptKitReport(invocation, input.Format, report, formatPromptText(report), acquired.Diagnostics) +} + +func runPromptKitProfile[C any](invocation Invocation, suite PromptKitSuite[C], input promptKitInput[C]) Outcome { + contextValue := promptKitContext(suite, input) + acquired, err := suite.InspectProfile(invocation.Context, input.Selection, contextValue, input.ID) + if err != nil { + return promptKitFailure(err, acquired.Diagnostics) + } + if acquired.Value == nil { + return promptKitFailure(fmt.Errorf("PromptKit profile inspection returned no result"), acquired.Diagnostics) + } + report := profileReportFromInspection(acquired.Value) + textReport, err := formatProfileText(report) + if err != nil { + return promptKitFailure(err, acquired.Diagnostics) + } + return writePromptKitReport(invocation, input.Format, report, textReport, acquired.Diagnostics) +} + +func promptKitFailure(err error, diagnostics []Diagnostic) Outcome { + outcome := Failure(err) + outcome.Diagnostics = diagnostics + return outcome +} + +func writePromptKitReport(invocation Invocation, format string, report any, textReport string, diagnostics []Diagnostic) Outcome { + var output strings.Builder + if format == "yaml" { + if err := encodeYAML(&output, report); err != nil { + return promptKitFailure(err, diagnostics) + } + } else if _, err := io.WriteString(&output, textReport); err != nil { + return promptKitFailure(err, diagnostics) + } + if err := writeReport(invocation.Streams.Out, []byte(output.String())); err != nil { + outcome := incompleteReport(err) + outcome.Diagnostics = diagnostics + return outcome + } + outcome := Success() + outcome.Diagnostics = diagnostics + return outcome +} + +type promptInputReport struct { + Name string `yaml:"name"` + Required bool `yaml:"required"` + ContentType string `yaml:"content_type"` + Description string `yaml:"description"` +} + +type promptOutputContractReport struct { + Format string `yaml:"format"` + ValidationMode string `yaml:"validation_mode"` + SchemaPath string `yaml:"schema_path"` + RepairAttempts int `yaml:"repair_attempts"` +} + +type promptInspectionReport struct { + PromptID string `yaml:"prompt_id"` + PromptVersion string `yaml:"prompt_version"` + PromptHash string `yaml:"prompt_hash"` + DefaultProfileID string `yaml:"default_profile_id"` + Inputs []promptInputReport `yaml:"inputs"` + OutputContract promptOutputContractReport `yaml:"output_contract"` +} + +func promptReportFromInspection(inspection *promptkit.PromptInspection) promptInspectionReport { + report := promptInspectionReport{ + PromptID: inspection.PromptID, + PromptVersion: inspection.PromptVersion, + PromptHash: inspection.PromptHash, + DefaultProfileID: inspection.DefaultProfileID, + Inputs: make([]promptInputReport, 0, len(inspection.Inputs)), + OutputContract: promptOutputContractReport{ + Format: string(inspection.OutputContract.Format), + ValidationMode: string(inspection.OutputContract.ValidationMode), + SchemaPath: inspection.OutputContract.SchemaPath, + RepairAttempts: inspection.OutputContract.RepairAttempts, + }, + } + for _, input := range inspection.Inputs { + report.Inputs = append(report.Inputs, promptInputReport{Name: input.Name, Required: input.Required, ContentType: input.ContentType, Description: input.Description}) + } + return report +} + +type modelParamsReport struct { + BackendID string `yaml:"backend_id"` + Endpoint string `yaml:"endpoint"` + Model string `yaml:"model"` + Temperature float64 `yaml:"temperature"` + MaxTokens int `yaml:"max_tokens"` + TopP float64 `yaml:"top_p"` + TimeoutSeconds int `yaml:"timeout_seconds"` + ServiceTier string `yaml:"service_tier"` + ReasoningEffort string `yaml:"reasoning_effort"` + APIKeyEnv string `yaml:"api_key_env"` + ExtraParams map[string]any `yaml:"extra_params"` +} + +type profileInspectionReport struct { + ProfileID string `yaml:"profile_id"` + EffectiveModelParams modelParamsReport `yaml:"effective_model_params"` + APIKeyRequired bool `yaml:"api_key_required"` +} + +func profileReportFromInspection(inspection *promptkit.ProfileInspection) profileInspectionReport { + params := inspection.EffectiveModelParams + extra := params.ExtraParams + if extra == nil { + extra = map[string]any{} + } + return profileInspectionReport{ + ProfileID: inspection.ProfileID, + EffectiveModelParams: modelParamsReport{ + BackendID: params.BackendID, Endpoint: params.Endpoint, Model: params.Model, + Temperature: params.Temperature, MaxTokens: params.MaxTokens, TopP: params.TopP, + TimeoutSeconds: params.TimeoutSeconds, ServiceTier: params.ServiceTier, + ReasoningEffort: params.ReasoningEffort, APIKeyEnv: params.APIKeyEnv, ExtraParams: extra, + }, + APIKeyRequired: inspection.APIKeyRequired, + } +} + +func formatPromptText(report promptInspectionReport) string { + var output strings.Builder + fmt.Fprintf(&output, "Prompt ID: %s\n", strconv.Quote(report.PromptID)) + fmt.Fprintf(&output, "Prompt version: %s\n", strconv.Quote(report.PromptVersion)) + fmt.Fprintf(&output, "Prompt hash: %s\n", strconv.Quote(report.PromptHash)) + fmt.Fprintf(&output, "Default profile ID: %s\n", strconv.Quote(report.DefaultProfileID)) + output.WriteString("Inputs:\n") + for _, input := range report.Inputs { + fmt.Fprintf(&output, " - name: %s\n required: %t\n content_type: %s\n description: %s\n", strconv.Quote(input.Name), input.Required, strconv.Quote(input.ContentType), strconv.Quote(input.Description)) + } + if len(report.Inputs) == 0 { + output.WriteString(" []\n") + } + fmt.Fprintf(&output, "Output contract:\n format: %s\n validation_mode: %s\n schema_path: %s\n repair_attempts: %d\n", strconv.Quote(report.OutputContract.Format), strconv.Quote(report.OutputContract.ValidationMode), strconv.Quote(report.OutputContract.SchemaPath), report.OutputContract.RepairAttempts) + return output.String() +} + +func formatProfileText(report profileInspectionReport) (string, error) { + params := report.EffectiveModelParams + var output strings.Builder + fmt.Fprintf(&output, "Profile ID: %s\n", strconv.Quote(report.ProfileID)) + output.WriteString("Effective model params:\n") + fmt.Fprintf(&output, " backend_id: %s\n endpoint: %s\n model: %s\n temperature: %s\n max_tokens: %d\n top_p: %s\n timeout_seconds: %d\n service_tier: %s\n reasoning_effort: %s\n api_key_env: %s\n", strconv.Quote(params.BackendID), strconv.Quote(params.Endpoint), strconv.Quote(params.Model), strconv.FormatFloat(params.Temperature, 'g', -1, 64), params.MaxTokens, strconv.FormatFloat(params.TopP, 'g', -1, 64), params.TimeoutSeconds, strconv.Quote(params.ServiceTier), strconv.Quote(params.ReasoningEffort), strconv.Quote(params.APIKeyEnv)) + output.WriteString(" extra_params:\n") + var nested strings.Builder + if err := encodeYAML(&nested, params.ExtraParams); err != nil { + return "", err + } + for _, line := range strings.Split(strings.TrimSuffix(nested.String(), "\n"), "\n") { + output.WriteString(" ") + output.WriteString(line) + output.WriteByte('\n') + } + fmt.Fprintf(&output, "API key required: %t\n", report.APIKeyRequired) + return output.String(), nil +} diff --git a/promptkit_scope.go b/promptkit_scope.go index 9fceb3f..40cd4d4 100644 --- a/promptkit_scope.go +++ b/promptkit_scope.go @@ -2,10 +2,8 @@ package commandkit import ( "context" - "errors" "fmt" - configkit "gitea.maximumdirect.net/eric/ConfigKit" "gitea.maximumdirect.net/eric/promptkit" ) @@ -23,24 +21,10 @@ func NewPromptKitScope[C any]( Summary: "PromptKit engine configuration.", Operations: []ConfigOperation{ConfigValidate, ConfigShow, ConfigSources, ConfigExplain}, Acquire: func(ctx context.Context, request ConfigRequest, appContext C) (Acquired[ConfigView], error) { - engine, err := load(ctx, request.Selection, appContext) + _, inspection, diagnostics, err := loadPromptKitInspection(ctx, request.Selection, appContext, load) if err != nil { - var loadErr *configkit.LoadError - if errors.As(err, &loadErr) { - return Acquired[ConfigView]{Diagnostics: ConfigKitDiagnostics("promptkit", loadErr.Sources(), loadErr.Diagnostics())}, err - } - return Acquired[ConfigView]{}, err + return Acquired[ConfigView]{Diagnostics: diagnostics}, err } - if engine == nil { - return Acquired[ConfigView]{}, fmt.Errorf("PromptKit configuration loader returned a nil engine") - } - - inspection, err := engine.InspectConfiguration() - if err != nil { - return Acquired[ConfigView]{}, err - } - sources := inspection.Sources() - diagnostics := ConfigKitDiagnostics("promptkit", sources, inspection.Diagnostics()) return Acquired[ConfigView]{ Value: ConfigView{ Effective: inspection, diff --git a/promptkit_test.go b/promptkit_test.go new file mode 100644 index 0000000..70ab40c --- /dev/null +++ b/promptkit_test.go @@ -0,0 +1,378 @@ +package commandkit_test + +import ( + "bytes" + "context" + "errors" + "io" + "reflect" + "strings" + "testing" + "testing/fstest" + + commandkit "gitea.maximumdirect.net/eric/CommandKit" + configkit "gitea.maximumdirect.net/eric/ConfigKit" + "gitea.maximumdirect.net/eric/promptkit" + "go.yaml.in/yaml/v4" +) + +const promptKitInspectionFixture = `prompts: + "team/summary": + "2": + default_profile: child + inputs: + - name: topic + required: true + content_type: text/plain + description: "Topic to summarize" + - name: audience + required: false + content_type: text/plain + description: "" + messages: + - role: user + content: "Summarize {{.topic}}" + output: + format: json + validation_mode: basic + repair_attempts: 2 + "3": + messages: + - role: user + content: "Version three" + output: + format: text + validation_mode: none + empty: + "1": + messages: + - role: user + content: "No inputs" + output: + format: text + validation_mode: none +` + +func promptKitInspectionEngine(t *testing.T) *promptkit.Engine { + t.Helper() + inputs, err := promptkit.FilesystemInputs("prompt fixture", fstest.MapFS{ + "config/prompts.yml": &fstest.MapFile{Data: []byte(promptKitInspectionFixture)}, + }, "config") + if err != nil { + t.Fatal(err) + } + engine, err := promptkit.NewEngine(promptkit.Config{Sources: promptkit.ConfigurationSources{ + Underlays: []configkit.Underlay{ + {Inputs: inputs}, + {Inputs: []configkit.Input{{Source: configkit.Mapping("profile fixture", map[string]any{ + "backends": map[string]any{"local": map[string]any{"endpoint": "https://example.invalid/v1"}}, + "profiles": map[string]any{ + "parent": map[string]any{ + "backend": "local", "model": "base-model", "temperature": 0.7, + "max_tokens": 100, "top_p": 0.8, + "extra_params": map[string]any{ + "token": "provider-private-sentinel", + "nested": map[string]any{"null_value": nil, "visible": "kept"}, + "sequence": []any{"first", nil, 3}, + }, + }, + "child": map[string]any{"extends": "parent", "temperature": 0, "max_tokens": 0, "api_key_env": "PROMPTKIT_TEST_KEY"}, + "requires-key": map[string]any{"backend": "local", "model": "key-model", "api_key_required": true}, + "no-extra": map[string]any{"backend": "local", "model": "empty-params"}, + }, + })}}}, + }, + }}) + if err != nil { + t.Fatalf("construct PromptKit fixture engine: %v", err) + } + return engine +} + +func TestPromptKitInspectionCommandsReportRealCapturedDefinitions(t *testing.T) { + t.Setenv("PROMPTKIT_TEST_KEY", "credential-private-sentinel") + engine := promptKitInspectionEngine(t) + loadCalls := 0 + suite, err := commandkit.NewPromptKitSuite(func(context.Context, commandkit.Selection, struct{}) (*promptkit.Engine, error) { + loadCalls++ + return engine, nil + }) + if err != nil { + t.Fatal(err) + } + app, err := commandkit.New(commandkit.Identity{Name: "example"}) + if err != nil { + t.Fatal(err) + } + if err := commandkit.RegisterPromptKit(app, suite); err != nil { + t.Fatal(err) + } + + var promptYAML bytes.Buffer + outcome := app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "team/summary", "--version", "2", "--format", "yaml"}, commandkit.Streams{Out: &promptYAML, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeSuccess || loadCalls != 1 { + t.Fatalf("prompt inspection outcome=%#v calls=%d report=%q", outcome, loadCalls, promptYAML.String()) + } + var promptReport map[string]any + if err := yaml.Unmarshal(promptYAML.Bytes(), &promptReport); err != nil { + t.Fatalf("decode prompt YAML: %v", err) + } + if !reflect.DeepEqual(sortedKeys(promptReport), []string{"default_profile_id", "inputs", "output_contract", "prompt_hash", "prompt_id", "prompt_version"}) { + t.Fatalf("prompt report keys = %v", sortedKeys(promptReport)) + } + if promptReport["prompt_id"] != "team/summary" || promptReport["prompt_version"] != "2" || promptReport["default_profile_id"] != "child" || promptReport["prompt_hash"] == "" { + t.Fatalf("prompt report identity = %#v", promptReport) + } + inputs := promptReport["inputs"].([]any) + if len(inputs) != 2 || inputs[0].(map[string]any)["name"] != "topic" || inputs[1].(map[string]any)["name"] != "audience" { + t.Fatalf("prompt inputs/order = %#v", inputs) + } + outputContract := promptReport["output_contract"].(map[string]any) + if outputContract["format"] != "json" || outputContract["validation_mode"] != "basic" || outputContract["repair_attempts"] != 2 { + t.Fatalf("output contract = %#v", outputContract) + } + + var promptText bytes.Buffer + outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "team/summary", "--version", "2"}, commandkit.Streams{Out: &promptText, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(promptText.String(), `Prompt hash: "`) || !strings.Contains(promptText.String(), `- name: "topic"`) || !strings.Contains(promptText.String(), `validation_mode: "basic"`) { + t.Fatalf("prompt text = (%#v, %q)", outcome, promptText.String()) + } + var omittedVersion bytes.Buffer + outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "empty", "--format", "yaml"}, commandkit.Streams{Out: &omittedVersion, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(omittedVersion.String(), "prompt_version: \"1\"") { + t.Fatalf("omitted singleton prompt version = (%#v, %q)", outcome, omittedVersion.String()) + } + + var profileYAML bytes.Buffer + outcome = app.Execute(context.Background(), []string{"promptkit", "show", "profile", "child", "--format", "yaml"}, commandkit.Streams{Out: &profileYAML, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeSuccess { + t.Fatalf("profile inspection outcome=%#v report=%q", outcome, profileYAML.String()) + } + var profileReport map[string]any + if err := yaml.Unmarshal(profileYAML.Bytes(), &profileReport); err != nil { + t.Fatalf("decode profile YAML: %v", err) + } + if !reflect.DeepEqual(sortedKeys(profileReport), []string{"api_key_required", "effective_model_params", "profile_id"}) || profileReport["profile_id"] != "child" || profileReport["api_key_required"] != false { + t.Fatalf("profile report = %#v", profileReport) + } + params := profileReport["effective_model_params"].(map[string]any) + if !reflect.DeepEqual(sortedKeys(params), []string{"api_key_env", "backend_id", "endpoint", "extra_params", "max_tokens", "model", "reasoning_effort", "service_tier", "temperature", "timeout_seconds", "top_p"}) { + t.Fatalf("model parameter keys = %v", sortedKeys(params)) + } + if params["temperature"] != 0 || params["max_tokens"] != 0 || params["api_key_env"] != "PROMPTKIT_TEST_KEY" { + t.Fatalf("effective owning values = %#v", params) + } + extra := params["extra_params"].(map[string]any) + nested := extra["nested"].(map[string]any) + sequence := extra["sequence"].([]any) + if nested["null_value"] != nil || nested["visible"] != "kept" || len(sequence) != 3 || sequence[1] != nil || sequence[2] != 3 { + t.Fatalf("nested provider parameters changed: %#v", extra) + } + if extra["token"] != "provider-private-sentinel" || strings.Contains(profileYAML.String(), "credential-private-sentinel") { + t.Fatalf("provider parameters or credential reference changed: %q", profileYAML.String()) + } + + var profileText bytes.Buffer + outcome = app.Execute(context.Background(), []string{"promptkit", "show", "profile", "child"}, commandkit.Streams{Out: &profileText, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(profileText.String(), "temperature: 0\n") || !strings.Contains(profileText.String(), "api_key_env: \"PROMPTKIT_TEST_KEY\"") || !strings.Contains(profileText.String(), "null_value: null") { + t.Fatalf("profile text = (%#v, %q)", outcome, profileText.String()) + } + + var requiredKey bytes.Buffer + outcome = app.Execute(context.Background(), []string{"promptkit", "show", "profile", "requires-key", "--format", "yaml"}, commandkit.Streams{Out: &requiredKey, Err: io.Discard}) + var requiredReport map[string]any + if outcome.Kind != commandkit.OutcomeSuccess || yaml.Unmarshal(requiredKey.Bytes(), &requiredReport) != nil || requiredReport["api_key_required"] != true { + t.Fatalf("credential requirement report = (%#v, %q)", outcome, requiredKey.String()) + } + + var emptyPrompt bytes.Buffer + outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "empty", "--version", "1", "--format", "yaml"}, commandkit.Streams{Out: &emptyPrompt, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(emptyPrompt.String(), "inputs: []") { + t.Fatalf("empty inputs report = (%#v, %q)", outcome, emptyPrompt.String()) + } + var zeroExtra bytes.Buffer + outcome = app.Execute(context.Background(), []string{"promptkit", "show", "profile", "no-extra", "--format", "yaml"}, commandkit.Streams{Out: &zeroExtra, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeSuccess || !strings.Contains(zeroExtra.String(), "extra_params: {}") { + t.Fatalf("nil extra params report = (%#v, %q)", outcome, zeroExtra.String()) + } +} + +func TestPromptKitInspectionSelectionErrorsHelpAndOptionalCapabilities(t *testing.T) { + engine := promptKitInspectionEngine(t) + loaderCalls := 0 + suite, err := commandkit.NewPromptKitSuite(func(context.Context, commandkit.Selection, struct{}) (*promptkit.Engine, error) { + loaderCalls++ + return engine, nil + }) + if err != nil { + t.Fatal(err) + } + app, _ := commandkit.New(commandkit.Identity{Name: "example"}) + if err := commandkit.RegisterPromptKit(app, suite); err != nil { + t.Fatal(err) + } + badTarget := suite + badTarget.Selectors = []commandkit.Selector[struct{}]{{ + Binding: commandkit.ValueFlag[struct{}](commandkit.FlagMeta{Name: "campaign"}, nil, func(value string) (string, error) { return value, nil }, func(*struct{}, commandkit.Supplied[string]) {}), + Scopes: []string{"app"}, + }} + invalidApp, _ := commandkit.New(commandkit.Identity{Name: "invalid"}) + if err := commandkit.RegisterPromptKit(invalidApp, badTarget); err == nil { + t.Fatal("selector for another scope was accepted") + } + for _, tc := range []struct { + args []string + want error + }{ + {args: []string{"promptkit", "show", "prompt", "team/summary"}, want: promptkit.ErrPromptLoad}, + {args: []string{"promptkit", "show", "prompt", "missing", "--version", "1"}, want: promptkit.ErrPromptNotFound}, + {args: []string{"promptkit", "show", "profile", "missing"}, want: promptkit.ErrProfileNotFound}, + {args: []string{"promptkit", "show", "prompt", "", "--version", "2"}, want: promptkit.ErrInvalidRequest}, + } { + outcome := app.Execute(context.Background(), tc.args, commandkit.Streams{Out: io.Discard, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeOperational || !errors.Is(outcome.Cause, tc.want) { + t.Errorf("execute %q = (%#v), want operational error matching %v", tc.args, outcome, tc.want) + } + } + cancelled, cancel := context.WithCancel(context.Background()) + cancel() + outcome := app.Execute(cancelled, []string{"promptkit", "show", "prompt", "empty", "--version", "1"}, commandkit.Streams{Out: io.Discard, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeOperational || !errors.Is(outcome.Cause, context.Canceled) { + t.Fatalf("cancelled prompt inspection = %#v", outcome) + } + beforeHelp := loaderCalls + outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "--help"}, commandkit.Streams{Out: io.Discard, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeSuccess || loaderCalls != beforeHelp { + t.Fatalf("help acquired engine: outcome=%#v calls=%d", outcome, loaderCalls) + } + beforeMissing := loaderCalls + outcome = app.Execute(context.Background(), []string{"promptkit", "show", "prompt"}, commandkit.Streams{Out: io.Discard, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeUsage || loaderCalls != beforeMissing { + t.Fatalf("missing ID = (%#v, calls=%d), want usage before acquisition", outcome, loaderCalls) + } + + profileOnlyApp, _ := commandkit.New(commandkit.Identity{Name: "profile-only"}) + if err := commandkit.RegisterPromptKit(profileOnlyApp, commandkit.PromptKitSuite[struct{}]{InspectProfile: func(ctx context.Context, _ commandkit.Selection, _ struct{}, id string) (commandkit.Acquired[*promptkit.ProfileInspection], error) { + value, err := engine.InspectProfile(ctx, id) + return commandkit.Acquired[*promptkit.ProfileInspection]{Value: value}, err + }}); err != nil { + t.Fatal(err) + } + outcome = profileOnlyApp.Execute(context.Background(), []string{"promptkit", "show", "profile", "child"}, commandkit.Streams{Out: io.Discard, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeSuccess { + t.Fatalf("profile-only capability was not installed: %#v", outcome) + } + outcome = profileOnlyApp.Execute(context.Background(), []string{"promptkit", "show", "prompt", "empty"}, commandkit.Streams{Out: io.Discard, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeUsage { + t.Fatalf("profile-only suite installed prompt inspection: %#v", outcome) + } + promptOnlyApp, _ := commandkit.New(commandkit.Identity{Name: "prompt-only"}) + if err := commandkit.RegisterPromptKit(promptOnlyApp, commandkit.PromptKitSuite[struct{}]{InspectPrompt: func(ctx context.Context, _ commandkit.Selection, _ struct{}, id, version string) (commandkit.Acquired[*promptkit.PromptInspection], error) { + value, err := engine.InspectPrompt(ctx, id, version) + return commandkit.Acquired[*promptkit.PromptInspection]{Value: value}, err + }}); err != nil { + t.Fatal(err) + } + outcome = promptOnlyApp.Execute(context.Background(), []string{"promptkit", "show", "prompt", "empty"}, commandkit.Streams{Out: io.Discard, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeSuccess { + t.Fatalf("prompt-only capability was not installed: %#v", outcome) + } + outcome = promptOnlyApp.Execute(context.Background(), []string{"promptkit", "show", "profile", "child"}, commandkit.Streams{Out: io.Discard, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeUsage { + t.Fatalf("prompt-only suite installed profile inspection: %#v", outcome) + } + + emptyApp, _ := commandkit.New(commandkit.Identity{Name: "empty"}) + if err := commandkit.RegisterPromptKit(emptyApp, commandkit.PromptKitSuite[struct{}]{}); err != nil { + t.Fatalf("empty PromptKit suite = %v", err) + } + outcome = emptyApp.Execute(context.Background(), []string{"help"}, commandkit.Streams{Out: io.Discard, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeSuccess { + t.Fatalf("empty PromptKit suite changed command family: %#v", outcome) + } + outcome = emptyApp.Execute(context.Background(), []string{"help", "promptkit"}, commandkit.Streams{Out: io.Discard, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeUsage { + t.Fatalf("empty PromptKit suite installed its namespace: %#v", outcome) + } +} + +func TestPromptKitSuitePassesSelectionAndProjectedContext(t *testing.T) { + type appContext struct { + Campaign commandkit.Supplied[string] + Runtime string + } + type contextKey struct{} + engine := promptKitInspectionEngine(t) + ctx := context.WithValue(context.Background(), contextKey{}, "caller") + wantSelection := commandkit.Selection{ + Config: commandkit.Supplied[string]{Value: "main.yml", Set: true}, + Overlay: commandkit.Supplied[string]{Value: "dev", Set: true}, + } + loaderCalls, contexts, checks := 0, 0, 0 + var gotSelection commandkit.Selection + var gotContext appContext + suite, err := commandkit.NewPromptKitSuite(func(gotCtx context.Context, selection commandkit.Selection, input appContext) (*promptkit.Engine, error) { + loaderCalls++ + gotSelection, gotContext = selection, input + if gotCtx != ctx || gotCtx.Value(contextKey{}) != "caller" { + t.Fatal("engine loader did not receive invocation context") + } + return engine, nil + }) + if err != nil { + t.Fatal(err) + } + suite.NewContext = func() appContext { contexts++; return appContext{Runtime: "invocation cache"} } + suite.Selectors = []commandkit.Selector[appContext]{{ + Binding: commandkit.ValueFlag[appContext](commandkit.FlagMeta{Name: "campaign"}, nil, func(value string) (string, error) { return value, nil }, func(input *appContext, value commandkit.Supplied[string]) { input.Campaign = value }), + Scopes: []string{"promptkit"}, + }} + suite.CheckInput = func(selection commandkit.Selection, input appContext) error { + checks++ + if selection != wantSelection || input.Campaign != (commandkit.Supplied[string]{Value: "spring", Set: true}) { + t.Errorf("preflight inputs = (%+v, %+v)", selection, input) + } + return nil + } + app, _ := commandkit.New(commandkit.Identity{Name: "example"}) + if err := commandkit.RegisterPromptKit(app, suite); err != nil { + t.Fatal(err) + } + outcome := app.Execute(ctx, []string{"promptkit", "show", "profile", "child", "--config", "main.yml", "--overlay", "dev", "--campaign", "spring"}, commandkit.Streams{Out: io.Discard, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeSuccess || loaderCalls != 1 || contexts != 1 || checks != 1 { + t.Fatalf("inspection = (%#v), loader/context/check calls = %d/%d/%d", outcome, loaderCalls, contexts, checks) + } + if gotSelection != wantSelection || gotContext.Campaign != (commandkit.Supplied[string]{Value: "spring", Set: true}) || gotContext.Runtime != "invocation cache" { + t.Fatalf("loader selection/context = (%+v, %+v)", gotSelection, gotContext) + } +} + +func TestPromptKitDirectSuiteRetainsDiagnosticsOnNamedLookupFailure(t *testing.T) { + warning := commandkit.Diagnostic{Severity: configkit.SeverityWarning, Write: func(w io.Writer) error { _, err := io.WriteString(w, "configuration warning\n"); return err }} + app, _ := commandkit.New(commandkit.Identity{Name: "example"}) + err := commandkit.RegisterPromptKit(app, commandkit.PromptKitSuite[struct{}]{InspectPrompt: func(context.Context, commandkit.Selection, struct{}, string, string) (commandkit.Acquired[*promptkit.PromptInspection], error) { + return commandkit.Acquired[*promptkit.PromptInspection]{Diagnostics: []commandkit.Diagnostic{warning}}, promptkit.ErrPromptNotFound + }}) + if err != nil { + t.Fatal(err) + } + outcome := app.Execute(context.Background(), []string{"promptkit", "show", "prompt", "missing"}, commandkit.Streams{Out: io.Discard, Err: io.Discard}) + if outcome.Kind != commandkit.OutcomeOperational || !errors.Is(outcome.Cause, promptkit.ErrPromptNotFound) || len(outcome.Diagnostics) < 1 || outcome.Diagnostics[0].Severity != configkit.SeverityWarning { + t.Fatalf("named lookup outcome did not retain acquisition diagnostic: %#v", outcome) + } +} + +func sortedKeys(values map[string]any) []string { + keys := make([]string, 0, len(values)) + for key := range values { + keys = append(keys, key) + } + for i := 0; i < len(keys); i++ { + for j := i + 1; j < len(keys); j++ { + if keys[j] < keys[i] { + keys[i], keys[j] = keys[j], keys[i] + } + } + } + return keys +}