How to Evaluate LLM Prompts Without Fooling Yourself

Prompt work has a measurement problem. Code has tests, design has review, and prompts have vibes. You change three words, the demo output looks better, and it ships. Three weeks later a user reports the thing you broke, and nobody can tell which of the last nine changes did it, because there is no record of what the prompt used to do. A prompt evaluation harness is a small investment that turns opinions into a number you can defend.

Start small and freeze it

Twenty to fifty cases is enough to start, and small is an advantage rather than a compromise. A set you can read in full gets used; a set of a thousand does not. Build it from real inputs, not imagined ones: production logs, support tickets, the queries your team actually typed. Include the awkward cases, the empty inputs, the ones in another language, and the ones where two people on your team disagreed about the right answer.

Two kinds of score, never blended

Most teams collapse every judgment into one number and then cannot tell what moved. Split them. Mechanical checks, like schema validity, required fields, or refusal correctness, are pass or fail and can be automated completely. Quality judgments need a rubric, and a rubric has to be specific enough that two reviewers would land in the same place.

Keep them separate in reporting. A change that lifts style scores and drops factuality is a regression wearing a disguise, and it will hide inside a blended average every single time. Report mechanical and judged scores on different lines and treat a drop in either as a failure.

Rubrics that survive disagreement

• Describe the failure, not the ideal. "Mentions the refund policy" beats "high quality".
• Make each level observable. If you cannot point to the sentence that proves the score, it is a vibe.
• Anchor with real examples from your own set, including one at each level.
• Allow "cannot tell" when it is true. Forcing a score on an ambiguous case adds noise.
• Have two people score ten cases independently before you trust the rubric. Their agreement rate is your rubric quality.

RUBRIC v4 — support ticket classification

For each ticket, output a JSON object:
  {"label": "pass|fail|unclear", "reason": "<one sentence>"}
  "reason" must quote the ticket text that decided the score.

PASS requires all three:
  1. The label matches the annotated label exactly.
  2. The reason cites words that appear in the ticket.
  3. The confidence field is not "low" when the label is not "other".

FAIL when the label is wrong OR the reason invents text not in the ticket.

UNCLEAR only when the ticket is genuinely ambiguous, meaning two
annotators disagreed. Do not use UNCLEAR for a hard case you got wrong.

Worked examples at each level are in eval/rubric-examples-v4.md.

Writing the rubric in the same declarative style you would use for the model is not a coincidence. The rubric is itself a prompt, so it inherits the same failure modes, which is another reason to keep it versioned and reviewed like one.

When two runs disagree

A non-deterministic model is not a flaky one, and reading disagreement is a core evaluation skill. If a single case flips between runs, the model is sitting on an unresolved decision boundary and the score is telling you the prompt has not settled it. Sampling several completions per case and reporting the mean alongside the spread separates "this prompt is reliable" from "this prompt is right on average", which are very different claims to make about a system.

Judge-based grading adds a second layer of variance and deserves to be treated as a measured instrument rather than an oracle. Report agreement between judge and human reviewers, hand-check a sample of disagreements, and keep the rubric short enough that a person can apply it consistently. A judge with a vague rubric is slower and less honest than a human reading the same output.

What to do with the score

A score without a decision rule becomes decoration. Write down the number that must hold before you ship, the number that triggers a rewrite, and who owns the decision. Most teams find that a 3 percent regression on a high-traffic case is worth more attention than a 15 percent regression on a rare one, and weighting the cases by real volume is the only way to know which is which.

• Set a ship threshold and a rewrite threshold, and put both in the pull request template.
• Weight cases by real traffic rather than treating every row as equal.
• Track the eval score in the same place as error rates, so a regression has somewhere to surface.
• Re-baseline deliberately when the model changes, and record why in the eval file.

Put it where changes happen

An evaluation that only runs when someone remembers is not a harness, it is a script. Run it in CI on the pull request that changes a prompt, and keep the prompt in version control as code rather than in a dashboard field, so the diff is reviewable and the history is real. Score against the frozen set on every run and store the numbers, because the trend is worth more than any single reading.

If you cannot state the before-and-after number, you made a change, not an improvement.

Pair it with production monitoring, because a frozen set tells you about the cases you thought of. Track disagreement rates, escalation rates, and user corrections on live traffic, and when one spikes, add that case to the frozen set with an annotation. The set then grows from evidence instead of from imagination.

What evaluation is not for

• It is not a way to pick a model by vibes. Compare models on the same set, in the same run, or the comparison means nothing.
• It is not a substitute for reading outputs. Skim the failures every time; the pattern in the failures usually names the next fix.
• It is not a license to optimize the metric. Once a number is a target, it stops being a measurement. Rotate in fresh cases.
• It is not exhaustive. Fifty cases cannot prove correctness. They can prove you did not break what already worked.