How to evaluate AI outputs: test sets, rubrics and regression checks
A practical approach to measuring AI quality — building a test set, writing a scoring rubric, using human and automated review, and catching regressions.
By the AiTrain editorial team · Reviewed under our editorial standards
Whether you are rolling out an internal assistant or shipping an AI feature, "it seemed fine when I tried it" is not a quality process. Evaluation turns impressions into evidence.
Key takeaways
- Build a test set from real inputs, including the awkward ones.
- Score against a written rubric, not gut feel.
- Re-run the same tests after every change to prompts, models or data.
- Combine automated checks with periodic human review.
Step 1: build a test set
Collect 30–100 real examples of what users will ask or submit. Include:
- Typical cases (the bulk of real usage).
- Edge cases: very short, very long, ambiguous, multilingual.
- Cases where the correct answer is "I don't know" or "out of scope".
- Known past failures.
For each, write the expected result or the key facts a good answer must contain.
Step 2: write a rubric
| Criterion | 0 | 1 | 2 |
|---|---|---|---|
| Correctness | Wrong or invented | Partly correct | Fully correct |
| Completeness | Misses key points | Some missing | All key points |
| Grounding | No sources / wrong sources | Some cited | Correctly cited |
| Format | Unusable | Needs edits | Ready to use |
| Safety/tone | Problematic | Minor issues | Appropriate |
Keep criteria few and clear so two reviewers would score the same answer similarly.
Step 3: score
- Human review for nuance. Have two people score a sample and compare to keep the rubric honest.
- Automated checks for objective things: valid format, required fields present, banned phrases absent, length limits.
- Model-assisted grading can speed up review, but calibrate it against human scores first and spot-check it regularly.
Step 4: watch for regressions
Save scores with the date and what changed (prompt version, model, data). When you change anything, re-run the full set. A prompt tweak that fixes one problem can quietly break three others.
Step 5: monitor in production
- Let users flag bad answers easily.
- Sample real conversations (with appropriate privacy controls) for review.
- Add new failure cases to the test set.
Minimum viable evaluation
If you only have an afternoon: 30 real examples, a three-criterion rubric (correct, grounded, usable), and a spreadsheet. That alone puts you ahead of most AI rollouts.