RAG Evaluation Plan

When to use this

Use when you need to measure whether a RAG system's poor answers come from bad retrieval, bad generation, or both.

Prompt template

You are a RAG evaluation specialist. The system below retrieves from: {{corpus_description}}
and answers questions about: {{use_case}}

Design an evaluation plan that separately measures:
1. Retrieval quality (e.g. recall@k, precision@k, whether the gold passage is retrieved at all).
2. Answer faithfulness to retrieved context (does the answer only use what was retrieved?).
3. Answer relevance to the user's question.

For each metric, state how to compute it and what data it needs (labeled gold passages, reference answers, etc.).
Then produce {{question_count}} example evaluation questions spanning easy lookups, multi-passage synthesis, and questions with no good answer in the corpus (should trigger an "I don't know").

Customize

Preview

You are a RAG evaluation specialist. The system below retrieves from: {{corpus_description}} missing
and answers questions about: {{use_case}} missing

Design an evaluation plan that separately measures:
1. Retrieval quality (e.g. recall@k, precision@k, whether the gold passage is retrieved at all).
2. Answer faithfulness to retrieved context (does the answer only use what was retrieved?).
3. Answer relevance to the user's question.

For each metric, state how to compute it and what data it needs (labeled gold passages, reference answers, etc.).
Then produce 9 example evaluation questions spanning easy lookups, multi-passage synthesis, and questions with no good answer in the corpus (should trigger an "I don't know").

Usage instructions

Describe your corpus and use case, then use the plan to build a small labeled gold set before trusting any retrieval metric.

Example output

A metrics section (retrieval recall@5, faithfulness rubric, relevance rubric) plus 9 example questions labeled easy/multi-passage/unanswerable.

Limitations

Faithfulness and relevance scoring here assume a human or LLM-judge grader; the plan does not itself implement automated scoring code.

Rate this prompt