Practice
How to Write Evidence-Based AI Evaluation Rationales
A repeatable method for turning a judgment about an AI response into a clear, specific, and reviewable rationale.
A rationale is the bridge between a judgment and a reviewable decision. It should not merely announce a preference. It should show why the criterion applies and where the relevant evidence appears.
Use a four-part structure
For most comparison tasks, a strong first draft can follow this pattern:
- Judgment: identify the stronger response or the key issue.
- Criterion: name the relevant requirement, such as accuracy or instruction following.
- Evidence: cite the specific statement, omission, or structural choice.
- Impact: explain why it matters to the user or task outcome.
Here is the structure in one paragraph:
Response A is preferable because it follows the requested two-step format and distinguishes confirmed information from an assumption. Response B gives a plausible conclusion, but presents its estimate as established fact. That makes B less reliable for a user who needs to understand the uncertainty.
Avoid vague praise and vague criticism
Words such as better, clearer, weak, and confusing can be useful summaries, but they are not evidence. Replace them with observable details.
| Vague statement | Evidence-based revision |
|---|---|
| “It is more accurate.” | “It avoids the unsupported claim that the policy applies in every country.” |
| “It follows the prompt.” | “It provides the requested recommendation and two supporting reasons.” |
| “The tone is bad.” | “It dismisses the user’s concern instead of explaining the limitation.” |
Handle uncertainty explicitly
Not every flaw is equally decisive. If two answers are both acceptable, say what creates the close call. If a factual statement needs external verification, identify it as unverified rather than calling it false without evidence.
This is especially important in AI work because confident wording can make uncertainty easy to miss. A careful evaluator distinguishes among a statement contradicted by the reference, a statement not supported by the reference, and a statement that needs further checking.
Keep the rationale proportional
A rationale should match the decision. A small formatting issue does not need a long essay. A high-impact factual contradiction may need a clearer explanation and reference to the task materials. The goal is not to sound severe; it is to make the judgment reproducible.
That discipline is valuable far beyond evaluation tasks. It improves reviews, research notes, quality assurance, and everyday professional decisions.