All learning paths Learn See Try Make Write the item sheet What this lesson is about Write one question of a check set the way it is handed over. The input, the expected answer, the marking rule, the version.
Your team keeps its check set in a chat thread. A new reviewer joins on Monday and cannot run it.
A Move the questions into a sheet, since the team already knows what a good reply looks like. B Every item carries four things: the input, the answer you expect, the marking rule, and a date. C Ask the new reviewer to sit beside someone for a week, so the set is learnt the same way.
Check my recall Why this matters A check set is a document before it is a score. The published evaluation guide builds an item the same way. The input, the answer it expects, the rule that decides a pass. A run nobody else can repeat is an opinion with a number stuck on it.
Write a check-set item a stranger can run: the input, the answer you expect, the marking rule, the version.
A check set is a document before it is a score. The published evaluation guide builds an item the same way. The input, the answer it expects, the rule that decides a pass. A run nobody else can repeat is an opinion with a number stuck on it.
Every item carries four things: the input, the answer you expect, the marking rule, and a date. The marking rule says what counts as a pass, so two people reach the same verdict. Number the version, because a changed item makes last month's figure mean something else. Task Write one item for a check set on bus-timetable answers in Hindi.
Weak approach Writes: ask when the last bus to Lucknow leaves, and check that the reply looks right. Leaves out the expected time and the version.
Stronger approach Writes the input, the time printed on the timetable, and the rule that the reply must match to the minute. Adds the date and version one.
Why the stronger approach works The weak item leaves looks right to whoever is marking. The strong one gives the same verdict twice.
Try a changed situation A partner sends fifty items and last month's figure. Three of the items were reworded since then.
Make something yourself Write three items for a check set in a subject you know. Give the input, the answer you expect, the marking rule and the version.
Lesson 6 of 8 on Coding & reasoning evaluation next Check the model that marks Part 2 ends in a work sample you can send
Read the primary or official source: OpenAI evaluation guidance