base-layer/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity

Complete resource inspection validation and coverage

Browse Source
This commit is contained in:
Eric Rakestraw
2026-11-06 15:34:21 +00:00
parent 19a0604874
commit f083ad5309
6 changed files with 61 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
docs/roadmap/implementation.md
View File

@@ -6,11 +6,11 @@ Implement the accepted [resource inspection roadmap](inspect-backend.md). That
document owns intent, scope, and product decisions; this plan owns the concrete
steps and representation choices needed to deliver them.
Baseline: `951739e`. ConfigKit v1.1.0 composes and resolves definitions once.
Promptkit captures executable prompts and backend-completed profile targets.
`InspectPrompt` currently exposes limited metadata; `InspectProfile` exposes an
execution target. There is no `InspectBackend` or typed `PromptSource`, and
`BackendConfig` lacks `Extends`. Existing inspection values do not have the
At baseline `951739e`, ConfigKit v1.1.0 composed and resolved definitions once.
Promptkit captured executable prompts and backend-completed profile targets.
`InspectPrompt` exposed limited metadata; `InspectProfile` exposed an
execution target. There was no `InspectBackend` or typed `PromptSource`, and
`BackendConfig` lacked `Extends`. Existing inspection values did not have the
selected generic JSON/YAML contract.
Each numbered stage is intended for one Luna implementation prompt, in order.
@@ -571,6 +571,13 @@ ConfigKit composition contracts intact.
## Stage 12: Validate And Close The Sprint
**Completed:** Audited the selected resource, encoding, source, ownership, and
null-migration boundaries; added coverage for inherited diagnostics and explicit
serialization values. With Go 1.27.3, `GOTOOLCHAIN=local`, and `GOWORK=off`,
`go test -race ./...`, `go vet ./...`, and module tidy verification pass. Changed
Go files are formatted, local documentation targets and whitespace are checked,
and the feature roadmap and plan are retained for review.
**Scope:** Combined verification and final cleanup of this feature only.
1. Confirm that all three inspection operations, every selected result field,

18
docs/roadmap/inspect-backend.md
View File

@@ -6,8 +6,8 @@ Help operators using CommandKit-based applications inspect the configured
prompt, profile, or backend selected by ID. CommandKit owns commands and report
presentation; Promptkit exposes the selected resource's resolved configuration.
The direction is accepted for planning; implementation has not begun. Selected
work includes fuller prompt inspection, profile inspection of the resolved
The selected work has been implemented; this roadmap is retained for review.
It includes fuller prompt inspection, profile inspection of the resolved
profile itself, a new `InspectBackend`, `BackendConfig.Extends`, typed prompt
construction through `PromptConfig`/`PromptSource`, generically renderable
inspection results, and simplification of Promptkit's post-handoff null handling.
@@ -26,7 +26,7 @@ inspection assembles everything affecting execution.
| --- | --- |
| `InspectPrompt` | Resolved prompt definition, including captured imported message content. |
| `InspectProfile` | Resolved profile definition, before backend-to-profile execution completion. |
| Proposed `InspectBackend` | Resolved backend definition, including configured diagnostic metadata. |
| `InspectBackend` | Resolved backend definition, including configured diagnostic metadata. |
For example, given:
@@ -153,8 +153,8 @@ lookup. A captured prompt remains inspectable without a runnable default profile
## Resolved Profile Inspection
This is a deliberate change to existing `InspectProfile` behavior. It currently
returns backend-completed `EffectiveModelParams`; the new resource boundary
This is a deliberate change to the former `InspectProfile` behavior. It previously
returned backend-completed `EffectiveModelParams`; the resource boundary
requires the resolved profile's own fields instead.
Report fields supplied by the profile or inherited through ConfigKit, including
@@ -346,11 +346,11 @@ source round-tripping are not selected deliverables.
All five decisions below are resolved, including generic rendering and the
post-handoff null simplification in decision 4. Numbering is retained for
continuity with the discussion; implementation is pending.
continuity with the discussion; the selected implementation is complete.
### 1. How Should An Unknown Backend Be Classified?
**Status:** Accepted; implementation pending.
**Status:** Accepted and implemented.
**Decision:** Add public `ErrBackendNotFound` for an absent backend selected by
`InspectBackend`, following the existing prompt/profile sentinel pattern.
@@ -391,7 +391,7 @@ where to place those additions remains necessary for resource inspection.
### 3. Should Typed Prompt Construction Be Added In This Round?
**Status:** Accepted; implementation pending.
**Status:** Accepted and implemented.
**Decision:** Add `PromptConfig` and `PromptSource`, including inheritance,
following the thin mapping-constructor pattern described above. Retain the
@@ -411,7 +411,7 @@ is unnecessary; raw mappings continue to serve those cases.
### 4. How Should Resolved Resource Results And Presence Be Represented?
**Status:** Accepted; implementation pending.
**Status:** Accepted and implemented.
**Decision (Option A):** Use resource-oriented typed inspection results, reusing
suitable public nested values. Change `ProfileInspection` deliberately to report

14
internal/configuration/resources_test.go
View File

@@ -51,3 +51,17 @@ func TestResolvedBackendAndProfileResources(t *testing.T) {
t.Fatalf("cancellation = %v", err)
}
}
func TestBackendResourcePreservesInheritedDiagnostics(t *testing.T) {
state, err := load(mapping("backend diagnostics", map[string]any{"backends": []any{
map[string]any{"id": "parent", "endpoint": "https://provider.test/v1", "smoke_tests": map[string]any{"reachability": map[string]any{"endpoint": "https://probe.test/ready"}, "authentication": map[string]any{"kind": "none"}}},
map[string]any{"id": "child", "extends": map[string]any{"id": "parent"}},
}}), nil)
if err != nil {
t.Fatal(err)
}
got, err := state.InspectBackend(context.Background(), "child")
if err != nil || got.ID != "child" || got.ConcurrencyLimit != nil || got.QueueCapacity != nil || got.SmokeTests == nil || got.SmokeTests.Reachability == nil || got.SmokeTests.Reachability.Endpoint == nil || *got.SmokeTests.Reachability.Endpoint != "https://probe.test/ready" || got.SmokeTests.Authentication == nil || got.SmokeTests.Authentication.Kind == nil || *got.SmokeTests.Authentication.Kind != "none" || got.SmokeTests.Authentication.Endpoint != nil {
t.Fatalf("inherited diagnostics = %+v, %v", got, err)
}
}

12
prompt_sources_contract_test.go
View File

@@ -92,3 +92,15 @@ func TestTypedPromptSourceDefersValidation(t *testing.T) {
t.Fatalf("invalid final source: %v", err)
}
}
func TestNilTypedPromptSourceSuppliesEmptySequence(t *testing.T) {
source := promptkit.PromptSource("empty", nil)
engine, err := promptkit.NewEngine(promptkit.Config{Sources: promptkit.ConfigurationSources{Main: &configkit.Input{Source: source}}}, promptkit.WithBundledOpenRouter(false))
if err != nil {
t.Fatal(err)
}
prompts, err := engine.ListPrompts(context.Background())
if err != nil || len(prompts) != 0 {
t.Fatalf("empty source = %v, %v", prompts, err)
}
}

6
resource_inspection_api_test.go
View File

@@ -42,4 +42,10 @@ func TestResourceInspectionSelections(t *testing.T) {
if _, err := engine.InspectBackend(canceled, "local"); !errors.Is(err, context.Canceled) {
t.Fatalf("canceled = %v", err)
}
if _, err := engine.InspectPrompt(canceled, "prompt"); !errors.Is(err, context.Canceled) {
t.Fatalf("canceled prompt = %v", err)
}
if _, err := engine.InspectProfile(canceled, "partial"); !errors.Is(err, context.Canceled) {
t.Fatalf("canceled profile = %v", err)
}
}

13
resource_serialization_test.go
View File

@@ -13,8 +13,8 @@ import (
func TestResourceInspectionGenericEncoding(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{Sources: promptkit.ConfigurationSources{Main: &configkit.Input{Source: configkit.Mapping("resources", map[string]any{
"backends": []any{map[string]any{"id": "local", "endpoint": "https://example.test/v1", "concurrency_limit": 2, "extra_params": map[string]any{"null": nil, "array": []any{nil}}, "smoke_tests": map[string]any{"reachability": map[string]any{}}}, map[string]any{"id": "plain", "endpoint": "https://plain.test/v1"}},
"profiles": []any{map[string]any{"id": "parent", "backend": "local", "model": "model"}, map[string]any{"id": "profile", "extends": map[string]any{"id": "parent"}, "temperature": 0, "api_key_required": false, "endpoint": "", "extra_params": map[string]any{}}, map[string]any{"id": "unset", "extra_params": nil}},
"backends": []any{map[string]any{"id": "local", "endpoint": "https://example.test/v1", "api_key_env": "BACKEND_KEY", "concurrency_limit": 2, "extra_params": map[string]any{"null": nil, "array": []any{nil}}, "smoke_tests": map[string]any{"reachability": map[string]any{}, "authentication": map[string]any{"kind": "none"}}}, map[string]any{"id": "plain", "endpoint": "https://plain.test/v1"}},
"profiles": []any{map[string]any{"id": "parent", "backend": "local", "model": "model"}, map[string]any{"id": "profile", "extends": map[string]any{"id": "parent"}, "temperature": 0, "top_p": 0, "max_tokens": 0, "timeout_seconds": 0, "service_tier": "", "reasoning_effort": "", "api_key_env": "", "api_key_required": false, "endpoint": "", "extra_params": map[string]any{}}, map[string]any{"id": "unset", "extra_params": nil}},
"prompts": []any{map[string]any{"id": "prompt", "inputs": []any{map[string]any{"name": "a", "required": false, "description": ""}, map[string]any{"name": "b"}}, "messages": []any{map[string]any{"role": "user", "content": "", "cache_control": map[string]any{"type": "ephemeral", "ttl": ""}}}, "output": map[string]any{"format": "text", "validation_mode": "basic", "repair_attempts": 0}}, map[string]any{"id": "empty-inputs", "inputs": []any{}, "messages": []any{map[string]any{"role": "user", "content": "x"}}, "output": map[string]any{"format": "text", "validation_mode": "none"}}, map[string]any{"id": "omitted-inputs", "messages": []any{map[string]any{"role": "user", "content": "x"}}, "output": map[string]any{"format": "text", "validation_mode": "none"}}},
})}}})
if err != nil {
@@ -77,7 +77,7 @@ func TestResourceInspectionGenericEncoding(t *testing.T) {
}
}
b := decode(backend)
if b["id"] != "local" || fmt.Sprint(b["concurrency_limit"]) != "2" || fmt.Sprint(b["queue_capacity"]) != "1024" {
if b["id"] != "local" || b["api_key_env"] != "BACKEND_KEY" || fmt.Sprint(b["concurrency_limit"]) != "2" || fmt.Sprint(b["queue_capacity"]) != "1024" {
t.Errorf("backend = %#v", b)
}
params := b["extra_params"].(map[string]any)
@@ -85,16 +85,19 @@ func TestResourceInspectionGenericEncoding(t *testing.T) {
t.Errorf("provider nulls = %#v", params)
}
smoke := b["smoke_tests"].(map[string]any)
if smoke["authentication"].(map[string]any)["kind"] != "none" {
t.Errorf("authentication = %#v", smoke)
}
if reach := smoke["reachability"].(map[string]any); len(reach) != 0 {
t.Errorf("reachability = %#v", reach)
}
absent(b, "extends")
absent(decode(plain), "smoke_tests", "queue_capacity", "concurrency_limit")
p := decode(profile)
if p["id"] != "profile" || p["backend"] != "local" || p["endpoint"] != "" || p["api_key_required"] != false || fmt.Sprint(p["temperature"]) != "0" || len(p["extra_params"].(map[string]any)) != 0 {
if p["id"] != "profile" || p["backend"] != "local" || p["model"] != "model" || p["endpoint"] != "" || p["api_key_env"] != "" || p["service_tier"] != "" || p["reasoning_effort"] != "" || p["api_key_required"] != false || fmt.Sprint(p["temperature"]) != "0" || fmt.Sprint(p["top_p"]) != "0" || fmt.Sprint(p["max_tokens"]) != "0" || fmt.Sprint(p["timeout_seconds"]) != "0" || len(p["extra_params"].(map[string]any)) != 0 {
t.Errorf("profile = %#v", p)
}
absent(p, "extends", "effective_model_params", "timeout_seconds", "api_key_env")
absent(p, "extends", "effective_model_params")
absent(decode(unset), "extra_params", "endpoint")
q := decode(prompt)
inputs := q["inputs"].([]any)