Files
notarius/docs/roadmap/promptkit-architecture-discussion.md

12 KiB
Raw Blame History

PromptKit Architecture Discussion

Purpose And Status

Working notepad for reviewing the ownership boundaries in the PromptKit v0.23.2 adoption roadmap. Work through the topics below in order, recording questions, evidence, decisions, and follow-up work. Items 1 and 2 record selected architectural directions. Detailed design, read-only filesystem and schema decisions, and the remaining topics are still under discussion. The Audita comparison is a seventh topic and may inform earlier decisions.

The goal is to make Notarius fully PromptKit-native and adopt modern PromptKit capabilities as fully as practical. Application ownership of a policy does not necessarily require application ownership of its implementation. Evaluate remaining machinery against this goal, including opportunities for upstream features where PromptKit does not yet meet an application need.

No application changes have been implemented through this notepad. The adoption roadmap still needs reconciliation with the selected direction; its adapter preservation language is subject to this review. Significant changes to accepted architecture need successor ADRs under the documentation policy. Current boundaries are described in Architecture and LLM Runtime Internals.

1. LLM API And Diagnostic Boundary

Discussion / decision: Direction selected: remove the Notarius PromptKit adapter and make the application fully PromptKit-native. Detailed design and migration consequences remain to be worked through.

Background And Intent

Notarius and related applications, including Audita, originally developed their own LLM interfaces or used the OpenAI SDK directly. PromptKit emerged by extracting their duplicated mechanics into a shared library. Notarius adopted an early version and predates many of the library's modern capabilities.

The maintainer clarified that duplication of PromptKit capabilities, including slight behavioral differences, does not reflect an intentional requirement for an independent Notarius implementation. Existing wrappers, tests, and historical decisions are migration evidence, not sufficient reasons to preserve that duplication. Identify actual application needs before choosing what to retain.

Missing reusable capabilities may justify upstream feature requests. The discussion is not constrained to what v0.23.2 can replace immediately; any resulting change to the adoption target remains to be selected.

Design Consequences To Work Through

  • Use native PromptKit APIs and contracts for LLM execution instead of preserving a parallel provider-neutral completion API or rebuilding the adapter under another name.
  • Review copied request, result, error, inspection, observation, and debug types for removal. Keep application-owned representations only where they serve a concrete application need, with compatibility decisions made explicitly.
  • Let PromptKit provide reusable execution mechanics. Determine how Notarius supplies domain inputs, interprets typed outputs, and expresses its acceptance decisions through the resulting design.
  • Review application composition, test seams, diagnostic storage, and disclosure handling without assuming they require an LLM abstraction layer.
  • Evaluate each upstream gap as a requirement to understand and potentially address in PromptKit, rather than automatically preserving local machinery.

Open questions:

  • What application collaborators and helpers remain after the adapter is removed?
  • Which durable output or diagnostic contracts need deliberate migration?
  • Which existing behavioral differences should disappear through native adoption, and which reflect an application need that should be supplied to PromptKit?

Follow-up: Apply this direction to items 2–6, compare Audita under item 7, and reconcile the adoption roadmap. Record the significant boundary change in a successor ADR and update architecture and implementation documentation when the change lands. Do not treat the remaining topics as decided by this direction.

2. Prompt Assets And Schema Assembly

Discussion / decision: Direction agreed: use native PromptKit declarations as the sole prompt assembly mechanism and remove the duplicate application assembly plumbing. Resolution of the custom read-only filesystem and schemas is explicitly deferred; the next topic may clarify those choices.

Selected direction: Keep the embedded content tree and load its prompt and profile definitions as native PromptKit configuration in the application collection. Use native filesystem contexts and imports to reference shared content directly. Application composition assembles the embedded and operator sources once; modules select prompt IDs, supply domain inputs, and interpret typed results.

The spell extractor illustrates the current duplication: its prompt YAML declares ordered messages and a schema, while a separate Go asset manifest lists files, mounts shared fragments, registers filesystems, and hashes them. The selected direction makes native declarations the sole prompt assembly source.

Candidates for removal:

  • Prompt and fallback-profile registration plumbing in AssetRegistry and individual modules; schema responsibilities remain under review.
  • ModulePromptFS, PromptAssetManifest, shared-name lookup tables, synthetic sharedassets mounts.
  • Manual prompt-file enumeration for fingerprints, replaced by captured native prompt identities subject to the broader identity design in item 5.

Keep prompt prose, ordered messages, cache controls, workload profiles, private response schema documents, and domain interpretation. Durable artifact schemas remain separate. Content ownership does not require runtime asset registration.

Native schema paths resolve against the configured schema root, separately from message-file source contexts. PromptKit v0.23.2 loads schemas during preparation and requires self-contained schema documents. A captured prompt hash alone does not establish schema equality. Existing schema metadata in manifests and checkpoint inputs must be deliberately replaced or retired; its presence does not require preserving the complete wrapper system.

Deferred decisions:

  • Whether the custom read-only filesystem can be removed after reviewing its remaining responsibilities.
  • Schema assembly, root/path selection, wrapper removal, and schema metadata. Supplying the embedded filesystem as a native schema root remains a candidate, not a selected design.
  • How schema identity contributes to manifests and checkpoint reuse.

Implementation questions:

  • Can native sources represent the existing shared fragments and module assets?
  • Which registry, remounting, and hashing helpers could disappear?
  • What private schema identity is still needed for checkpoint reuse?
  • How do we preserve message ordering and cache controls while simplifying assembly?

Follow-up: Continue with item 3, then revisit the deferred filesystem and schema decisions with any new clarity. Confirm native source contexts for shared fragments and operator replacements. Resolve metadata and reuse identity with items 5–6. Verify the resulting definitions through native preparation and fake-provider execution; filesystem remounting tests can retire with the removed implementation. The agreed direction has not yet been implemented.

3. Retry And Semantic Correction Ownership

Consider narrowing outer stage retries once PromptKit has exhausted its own transport and structural-repair budgets. Notarius would retain application assessment, semantic correction, fallback, and acceptance decisions.

Questions:

  • Which operational failures justify repeating an entire stage?
  • Should semantic correction preserve only the latest response or accumulate conversation history?
  • How should retries for non-LLM operations differ from LLM execution retries?
  • Would a native orchestration API reduce machinery without obscuring typed validation and terminal disposition?

Discussion / decision: Pending.

Follow-up: Evaluate behavior changes against ADR-0014 and ADR-0016. PromptKit v0.23.2's Coordinate continues an exchange; it is not a direct replacement for the existing latest-response-only policy or full stage loop.

4. Global Scheduling And Admission

Consider whether backend capacity limits and pipeline worker bounds meet the operational need, or whether a separate cross-backend ceiling remains useful. If it remains useful, consider implementing that reusable mechanism upstream.

Questions:

  • What resource or operational budget does each limit protect?
  • Should a global permit remain occupied during backend queueing and repair?
  • Can the Notarius scheduler be removed, or should PromptKit gain engine-wide admission support?

Discussion / decision: Pending.

Follow-up: Account for the existing global-limit contract. PromptKit v0.23.2's per-backend limits do not directly replace it.

5. Execution Identity And Checkpoint Reuse

Consider moving identity for PromptKit-owned execution semantics upstream. Notarius would combine opaque upstream identities with its source, module, reference, schema, and policy identities and retain checkpoint eligibility and storage decisions.

Questions:

  • Which identity can be computed before a stage runs, and which needs a concrete prepared request?
  • Which changes should invalidate reuse, and at what granularity?
  • Should a conservative application fingerprint be an interim implementation rather than the architectural destination?
  • Does this justify upstream work and a later adoption target?

Discussion / decision: Pending.

Follow-up: Review PromptKit's deferred semantic execution-target fingerprint proposal. This capability is not supplied by the v0.23.2 adoption target.

6. Inspection And Execution Bookkeeping

Extend the roadmap's use of native inspection, Run, OnPrepared, observations, and ExecutionGroup to reconsider surrounding wrappers and copied report types. Notarius retains stage correlation, artifact provenance, storage, and disclosure policy; CommandKit is the planned owner of standardized operator presentation.

Questions:

  • Which reports and accounting mechanisms can use upstream results directly?
  • Which application projections remain necessary for durable contracts?
  • Can the PromptKit migration avoid building temporary machinery that the CommandKit adoption would immediately replace?

Discussion / decision: Pending.

Follow-up: Coordinate this topic with item 1 and the roadmap's CommandKit follow-up so report and adapter decisions are made once.

7. Compare Audita's PromptKit Workflow

Review the sibling Audita checkout at ../audita relative to the Notarius repository root. Audita uses PromptKit for a workflow the user identifies as very similar, potentially identical. Verify the similarities and differences before drawing architectural conclusions.

Questions:

  • What are Audita's PromptKit version, execution flow, and ownership boundaries?
  • How does it handle candidate assessment, correction, retries, scheduling, configuration, inspection, and diagnostics?
  • Which simpler patterns could Notarius adopt directly?
  • Which differences arise from actual product requirements?
  • Does duplication across the two applications reveal work that belongs in PromptKit, ConfigKit, or CommandKit?

Discussion / decision: Pending; Audita has not yet been reviewed for this comparison.

Follow-up: Record the reviewed Audita revision and evidence, then feed the findings back into items 1–6 and the adoption roadmap.