Outline prompt quality discussion roadmap

docs/roadmap/prompt-quality.md

@@ -0,0 +1,107 @@
+# Prompt Quality Roadmap
+
+**Updated:** 2026-11-11
+
+**Status:** Proposed discussion roadmap; decisions and implementation pending.
+Work through the topics below in order. Recommendations are starting points for
+discussion, not newly adopted restrictions or guarantees.
+
+## Baseline and Intended Direction
+
+The review of revision `6d34602` covered all eleven
+[prompt definitions](../internal/prompts.md), their shared instructions, and
+the context and response handling for proposals, validation, and repair.
+It was a design review, not a measurement of live-model quality.
+
+Retain the strengths: conservative transcript editing, explicit literal-span and
+deletion semantics, distinct glossary context and protection, full before/after
+validation, independent repair conversations with withdrawal, and native
+PromptKit/ConfigKit prompt replacement. Improve consistency across the three
+roles, useful rejection feedback, and maintainability.
+
+## Discussion and Work Order
+
+### 1. Align Validation with Module Scope
+
+The grammar proposal permits formatting and the narrow `a`/`an` exception,
+while its editorial validator also permits homophone corrections, dysfluency
+removal, and combinations of those changes. An edit can satisfy that validator
+while violating the proposing module's instructions.
+
+**Recommendation:** supply the originating module's allowed edits to editorial
+validation and assess them alongside meaning preservation. Prefer reusing the
+shared validator where practical. This needs request-context wiring as well as
+prompt changes; a prompt rewrite alone cannot supply missing scope information.
+Discuss the scope contract and shared versus specialized definitions first.
+See [modules](../internal/modules.md) and [validators](../internal/validators.md).
+
+### 2. Make Rejection Reasons Useful for Repair
+
+Validator explanations now become repair feedback, but current instructions give
+little guidance about producing actionable reasons.
+
+**Recommendation:** ask for a concise description of the specific change, the
+applicable criterion, and relevant evidence, within the validator's remit.
+For example: removing "might" turns a tentative statement into a definite one;
+preserve the speaker's uncertainty. Start with the existing `reason` field.
+Discuss examples and sufficient detail before considering schema expansion.
+
+### 3. Make Repair a Reconsideration of the Edit's Merits
+
+Withdrawal is already explicit. Clarify when the model should use it and how it
+should respond when the original evidence remains insufficient.
+
+**Recommendation:** ask the model to reassess the correction from the supplied
+evidence, preserve module scope, and withdraw unsupported proposals. Address the
+risk that numerical confidence feedback merely encourages raising the score.
+Do not automatically prohibit confidence-only revisions: legitimate reassessment
+is possible. Discuss and evaluate that behavior before adding a business rule.
+Retain the agreed full-history conversation model and existing repair lifecycle.
+
+### 4. Simplify Instructions and Improve Examples
+
+Restrictions and examples recur across proposal, validation, and repair assets;
+the grammar prompt repeats its word-substitution boundary several ways.
+
+**Recommendation:** distinguish shared editing principles, module scope, and
+task-specific response contracts. Use small shared fragments where they reduce
+drift. Replace some repetitive example lists with contextual contrasts, such as
+accidental repetition versus deliberate emphasis. Preserve useful distinctions;
+shorter prompts alone are not the objective. Discuss what belongs in shared
+fragments and what remains local to each definition.
+
+### 5. Evaluate Editorial Quality
+
+Rendering, schema, and controlled-response tests establish integration behavior,
+but do not establish the quality of model editorial judgments.
+
+**Recommendation:** assemble a modest evaluation set containing both needed edits
+and text that should remain unchanged. Include ambiguous glossary matches,
+intentional repetition, meaningful uses of apparent filler, punctuation that
+changes meaning, and repairs that should be withdrawn. Compare unnecessary edits,
+missed corrections, validator disagreement, and repair outcomes using independent
+editorial assessment; validator approval alone is insufficient evidence.
+
+Capture candidate examples during topics 1–4 and establish a baseline before
+changing prompts. Discuss representative models, expected outcomes, and how to
+run evaluations. Keep model-dependent evaluation separate from the default
+offline test suite; no numerical quality gate is selected here.
+
+### 6. Revisit Broader Product Choices
+
+Use the preceding decisions and evaluation evidence to discuss these in order:
+
+1. **Grammar naming and scope:** the current module is close to copy editing,
+   with an isolated article exception. Decide whether its name and scope convey
+   the intended product behavior.
+2. **Glossary/homophone overlap:** assess the benefit of their overlapping remit
+   and repeated glossary passes before changing the default sequence.
+3. **Language and examples:** decide whether spoken English and campaign-specific
+   examples are the right bundled defaults, and how broadly those defaults
+   should apply.
+4. **Meaning-reversal policy:** the guard can also exclude a genuine correction
+   of a mistranscription. Initially retain the conservative default while
+   explicitly evaluating that tradeoff.
+
+These are product discussions, not selected renames, broader editing permissions,
+language-support promises, or changes to validation policy.