All learning paths Learn See Try Make Build a small set of settled cases What this lesson is about Agree a handful of cases with a second reviewer, so the rest of the batch has something to sit against.
You and one other person have an hour before two thousand short Hindi reviews start. How do you spend it?
A Start the batch, and settle each hard case with him as it turns up. B Talk ten hard cases through out loud, and carry on once you both say yes. C A settled set is a few cases two reviewers agreed on, with the reason written down.
Check my recall Why this matters Two careful people agree less often than you expect. One published study measured agreement between its own researchers and its paid reviewers at about two answers in three. A small set of agreed cases is what the rest of a batch is measured against.
Agree a few hard cases with a second reviewer, with reasons, before the rest of a batch starts.
Two careful people agree less often than you expect. One published study measured agreement between its own researchers and its paid reviewers at about two answers in three. A small set of agreed cases is what the rest of a batch is measured against.
A settled set is a few cases two reviewers agreed on, with the reason written down. Pick the cases you argued about, because an easy case settles nothing. Re-open a settled case only in writing, and record who changed it and why. Task You and a second reviewer have one hour together before a batch of two thousand short Hindi reviews begins.
Weak approach Agrees ten cases you both found obvious, so the set looks clean and takes ten minutes. The cases you actually disagreed on are left for the batch itself.
Stronger approach Picks the cases the two of you actually fought over. Records what was settled and why. Puts a mark on the two hardest.
Why the stronger approach works The arguments are where the batch will drift later. Ten easy cases settle nothing, because nobody was going to differ on them.
Try a changed situation Two brothers are agreeing rules for a shared shop counter before the festival season. Which rule still applies?
Make something yourself Agree ten cases with a second reviewer on work you both do. Write the decision and the reason for each of them, and name the two that were hardest to settle.
Lesson 5 of 7 on Working well on data projects next Audit your own finished batch Part 2 ends in a work sample you can send
Read the primary or official source: Bai et al., Training a Helpful and Harmless Assistant (arXiv)