All learning paths Learn See Try Make Run a check round and reset the scale What this lesson is about Give a team the same ten items and measure how far apart their marks land. Then rewrite the one anchor that split them.
Four people in your team hand in batches that do not read alike. The client has said so. How do you open the round?
A Sit the four down together and go through ten replies out loud. B Take ten replies from each person's own last batch and compare those. C Send everyone the same ten items, marked alone, before anybody compares notes.
Check my recall Why this matters Two careful people rating the same thing differ more than anybody expects. One published safety project measured four raters on the same cases. Their agreement scored 0.32 once chance was taken out. It rose to 0.49 when the rating was cut to two levels. That number is about the written scale, and it is the only thing a second round can be compared with.
Run a round on the same items with a team. Measure how far apart the marks land, and rewrite the anchor that split them.
Two careful people rating the same thing differ more than anybody expects. One published safety project measured four raters on the same cases. Their agreement scored 0.32 once chance was taken out. It rose to 0.49 when the rating was cut to two levels. That number is about the written scale, and it is the only thing a second round can be compared with.
Send everyone the same ten items, marked alone, before anybody compares notes. Take out the agreement that chance alone would have produced, then report what is left. Rewrite the anchor the team split on, then send the same ten again. Task Four reviewers in a small team rate replies on the five-level scale. Their client says the four batches do not read like one batch.
Weak approach Calls the four together, reads out ten replies, and takes the level the room settles on. Everybody leaves agreeing, and next week's batches are as far apart as before.
Stronger approach Sends the same ten to each of the four, to be marked alone. Two items come back carrying every level, lowest to highest. Both sit on the line between Moderately Meets and Highly Meets. Rewrites that one anchor and sends the ten again. The spread on those two items closes to one level.
Why the stronger approach works A room agrees because somebody spoke first. Ten items marked alone say which anchor was unreadable, and the second round says whether rewriting it worked.
Try a changed situation Two health workers weigh the same twenty children on one morning, and their reports differ. What do you report upward?
Make something yourself Run one round. Send the same ten items to two or more people, and collect the marks made alone. Write the note: the spread, the anchor you rewrote, and the date.
Lesson 9 of 9 on AI trainer essentials next Judge meaning and naturalness Part 2 ends in a work sample you can send The rules this work sample is read against, with a worked example
Read the primary or official source: Ganguli et al., Red Teaming Language Models (arXiv)