AI Eval Plan Generator
Turns a feature description into a concrete evaluation plan: dimensions to score, a rubric, and a minimum viable test set.
When to use this
Use before shipping an LLM-powered feature, to decide what "good" means and how you'll know if a change made it better or worse.
Prompt template
You are an AI evaluation designer. Produce an evaluation plan for the following feature.
Feature description:
{{feature_description}}
Primary risk to guard against: {{primary_risk}}
Produce:
1. 3-6 evaluation dimensions relevant to this feature (e.g. correctness, safety, tone, latency-sensitivity).
2. A 1-5 scoring rubric for each dimension, with a one-sentence description of each score level.
3. A minimum viable test set: {{test_case_count}} concrete test cases (inputs + what a passing output looks like), covering typical use, edge cases, and the primary risk above.
4. A pass/fail threshold recommendation for a release gate.
Output format: {{output_format}}.Customize
Feature description is required
Primary risk to guard against is required
Preview
You are an AI evaluation designer. Produce an evaluation plan for the following feature.
Feature description:
{{feature_description}} missing
Primary risk to guard against: {{primary_risk}} missing
Produce:
1. 3-6 evaluation dimensions relevant to this feature (e.g. correctness, safety, tone, latency-sensitivity).
2. A 1-5 scoring rubric for each dimension, with a one-sentence description of each score level.
3. A minimum viable test set: 10 concrete test cases (inputs + what a passing output looks like), covering typical use, edge cases, and the primary risk above.
4. A pass/fail threshold recommendation for a release gate.
Output format: Markdown.Usage instructions
Describe the feature in plain language, name the one failure mode you're most worried about, and run this before your first release and again after any prompt or model change.
Example output
A Markdown document with dimensions (factual grounding, tone, completeness...), a 1-5 rubric per dimension, 12 test cases, and a suggested release gate (e.g. no case scores below 3 on factual grounding).
Limitations
The generated test cases are a starting set, not a complete suite; domain experts should review them before relying on the plan for a real release decision.
Rate this prompt
Not yet rated