All learning paths Learn See Try Make Prepare for a sample test What this lesson is about Most sample tests measure whether you followed the written guideline, not whether you are clever.
An entry test asks you to rank ten answers, with one line of reason under each. How do you fill it in?
A A sample test measures whether you followed the written guideline. B Rank them as well as you can, and put a full paragraph under each. C Follow every instruction except the tie rule, which you can see is wrong.
Check my recall Why this matters The InstructGPT team picked reviewers with a screening test and then measured agreement. Reviewers kept outside the training agreed with each other 77% of the time. The measuring does not stop on the day you pass. Items whose answer is already settled sit unmarked inside a paid batch, beside plain attention checks. Your score on them is what keeps the queue open.
Prepare for a sample test by reading the guideline and following it exactly.
The InstructGPT team picked reviewers with a screening test and then measured agreement. Reviewers kept outside the training agreed with each other 77% of the time. The measuring does not stop on the day you pass. Items whose answer is already settled sit unmarked inside a paid batch, beside plain attention checks. Your score on them is what keeps the queue open.
A sample test measures whether you followed the written guideline. People lose by skipping an instruction they disagreed with. Settled items sit unmarked in the batch, and a low score on them pauses the queue. Task A sample test of ten rankings with a one-line reason under each, and the paid batch it opens.
Weak approach Writing a paragraph under each, and skipping the rule about ties because it seemed wrong. Then working twice as fast the week a message says agreement has dropped.
Stronger approach One line under each, exactly as asked. The tie rule is used even where it feels wrong, and the disagreement goes in the notes box. When a message says the queue is paused, nothing else is touched. The guideline and the settled cases are read again, and the failed items are asked for by number.
Why the stronger approach works The test is not looking for the best judgment. It is looking for a person who follows the written guideline on the item nobody marked as watched.
Try a changed situation You passed the test and the paid batch has started. Halfway through, one instruction still looks wrong to you. What do you do with the remaining items?
Make something yourself Do one practice test under its own time limit. Write a note beside every point where you were tempted to depart from the written guideline, and what you did. Name the items you would ask to see again if your score were paused.
Lesson 2 of 8 on Getting and keeping remote AI work next Read the terms before you sign Part 1 ends in a work sample you can send
Read the primary or official source: Ouyang et al., InstructGPT (arXiv)