All learning paths Learn See Try Make Measure the people collecting it What this lesson is about Give two collectors the same ten items, count where they differ, and repair the guideline before the person.
Two people you trained yesterday hand in their first ten clips each, from the same ten questions. What now?
A Give two collectors the same ten items and count where their answers differ. B Listen to both sets yourself and keep the one whose work sounds surer. C Wait for a full week of clips, so there is enough of it to judge fairly.
Check my recall Why this matters Trained people still split. One published study put a lab's own researchers and its paid reviewers on the same comparisons. They agreed about 63% of the time. That is the ordinary state of two people reading one instruction. You cannot hear it in a batch. You see it only by giving two people the same items and counting.
Measure two collectors against each other and repair what the difference exposes.
Trained people still split. One published study put a lab's own researchers and its paid reviewers on the same comparisons. They agreed about 63% of the time. That is the ordinary state of two people reading one instruction. You cannot hear it in a batch. You see it only by giving two people the same items and counting.
Give two collectors the same ten items and count where their answers differ. Read a disagreement as a fault in the guideline before reading it as a fault in a person. Fix the re-check share and the stop line before the batch, and write both down. Task Two new collectors in Gorakhpur finish a first day on the same ten Bhojpuri questions.
Weak approach Listening to both sets yourself, and deciding the second collector is weaker than the first. Telling her on the phone to be more careful with short answers tomorrow.
Stronger approach Laying the ten side by side and counting: they differ on three. Two of the three are the same fault: a one-word answer both should have prompted again. The guideline never said so. That line goes into the guideline with an example. The third is a real miss and goes back to her with the clip attached.
Why the stronger approach works The weak lead may even be right about who is weaker. She still cannot name the three items, and the rule that caused two of them stays exactly as it was.
Try a changed situation The same two people now type out what they hear. Their two files of one clip do not match. What do you look at?
Make something yourself Give two people the same ten items from your own collection. Report where they differed, and which difference was the guideline's fault. Write the line you added to the guideline because of it.
Lesson 11 of 12 on Bharat-language data craft next Release it with a datasheet Part 4 ends in a work sample you can send
Read the primary or official source: Bai et al., Training a Helpful and Harmless Assistant (arXiv)