All learning paths Learn See Try Make Gold items, and what pauses your queue What this lesson is about Some items in your batch already carry a decided answer. How you agree with those decides whether the work keeps coming.
Six of the forty items in today's work are ones the team already decided. Nobody tells you which six. What changes?
A Guess which six they are and give those your best care. B A gold item carries a decision the project fixed before you ever saw it. C Work exactly as you would have, then go back over all forty at the end.
Check my recall Why this matters Projects hide items whose answer they already fixed, and score you on those. They do not say which ones they were. One published project measured its own reviewers like this. Reviewers kept outside the training agreed with each other about 77 times in 100. Nobody agrees 100 times in 100, and no threshold asks for it.
Know what a platform measures while you work, and what to do when it pauses your queue.
Projects hide items whose answer they already fixed, and score you on those. They do not say which ones they were. One published project measured its own reviewers like this. Reviewers kept outside the training agreed with each other about 77 times in 100. Nobody agrees 100 times in 100, and no threshold asks for it.
A gold item carries a decision the project fixed before you ever saw it. A paused queue is a message about the guideline, not a verdict on you. Ask which gold items you missed before you touch the next batch. Task Your work stops on a Tuesday. The message says your score on the hidden checked items fell to 62 in 100. This project's line is 80.
Weak approach Opens a fresh batch the same evening and works twice as fast. Takes the easy-looking items first and leaves the doubtful ones for later.
Stronger approach Stops working. Asks for the eight checked items that went wrong. Six of the eight have one shape. The user asked for Hindi in turn one, and the reply came in English. Re-reads that page of the guideline, then opens the next batch.
Why the stronger approach works Working faster after a pause moves the number the wrong way. Six failures of eight had one cause, and one page of the guideline covered it.
Try a changed situation A delivery app puts your account under review after two orders were marked wrong this week. What is your first move?
Make something yourself Write one page about a check you failed, at work or in a test. Name what was being measured. Say what you re-read, and the one question you asked afterwards.
Lesson 7 of 9 on AI trainer essentials next Write the answer others are rated against Part 2 ends in a work sample you can send
Read the primary or official source: Ouyang et al., Training language models to follow instructions (arXiv)