All learning paths Learn See Try Make Write the answer, then mark three tries What this lesson is about Write the reference answer your scoring guide is applied against, then mark three attempts with it.
Your project sends you forty replies to score by Friday, with nothing to hold them against. What do you do first?
A Write the answer first; a guide with no answer beside it drifts by the third case. B Score the forty in one sitting, so the same mood carries across all of them. C Ask the project for the marks it gave last week and copy the pattern.
Check my recall Why this matters Marking without a written answer beside you drifts: the third reply is judged against the second, not the task. Published work on model graders hands the grader a reference answer for exactly this reason. Your answer is also the gold standard the listing asks you to contribute to.
Write the reference answer first, then mark three attempts against it.
Marking without a written answer beside you drifts: the third reply is judged against the second, not the task. Published work on model graders hands the grader a reference answer for exactly this reason. Your answer is also the gold standard the listing asks you to contribute to.
Write the answer first; a guide with no answer beside it drifts by the third case. Mark against the answer, not against how the reply sounds. Write the one line that decided each mark, so a second marker can disagree. Task Three replies come back to the returned-cartons task, and all three refuse the credit.
Weak approach Reads all three and puts the longest one first, because it explains the policy at length. It sounds like a manager. Marks the four-line reply lowest for being blunt.
Stronger approach Holds each reply against the reference answer. The long one refuses on the wrong ground, so it loses the first line. The four-line reply names the sixty-day window and the restocking charge, and takes the top mark.
Why the stronger approach works The weak marker ranked the replies against each other. The strong one marked each against the answer, which is why a second marker could check the work.
Try a changed situation A media desk asks you to score two news summaries of the same press note. One is polished, the other plain.
Make something yourself Take your own task from the first lesson. Write its reference answer, then mark three replies against it. For each mark, write the one line that decided it.
Lesson 3 of 6 on Your own trade, as AI work next Flag what a model cannot know Part 1 ends in a work sample you can send The rules this work sample is read against, with a worked example
Read the primary or official source: Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv)