All learning paths Learn See Try Make Turn a wrong answer into a test What this lesson is about Take one wrong answer and write the item that fails today and will pass after the repair.
You find the helpline giving a wrong fare for Gorakhpur. Your circle asks you to file it properly.
A Write down what went wrong in your own words, and attach a picture of the screen. B A test item needs the exact input, the output you expect, and the output you actually saw. C Send the whole conversation to the buyer and ask them to look at the fare question inside it.
Check my recall Why this matters A published code check set is built this way round. Each item carries a test that fails before the repair and passes after it. So the repair is proved, not claimed. A complaint cannot be proved and cannot be fixed. An item can be handed to anybody.
Turn one wrong answer into a test item somebody else can run and get the same failure.
A published code check set is built this way round. Each item carries a test that fails before the repair and passes after it. So the repair is proved, not claimed. A complaint cannot be proved and cannot be fixed. An item can be handed to anybody.
A test item needs the exact input, the output you expect, and the output you actually saw. Cut the input down until the smallest one that still fails is left. Say why your expected output is right, or the next person argues with the item instead of the model. Task A shop assistant returned 12.3499 where a bill must carry paise. Make it a test item.
Weak approach Files a note saying the assistant is poor with money and gets the paise wrong. Attaches a picture of the chat and asks for a repair.
Stronger approach Gives the exact request, the expected 12.35, the seen 12.3499, and the rule that a bill carries two places. Cuts the request to one line that still fails.
Why the stronger approach works Both report the same fault. Only the strong one can be run by somebody who never saw that chat.
Try a changed situation Your item fails on a four-page input. A colleague says they cannot tell which part breaks it.
Make something yourself Take one wrong answer you met this week. Write it as a test item: the input, the output you expect, the output you saw, and the reason.
Lesson 8 of 8 on Coding & reasoning evaluation next Worked examples and SFT Part 2 ends in a work sample you can send The rules this work sample is read against
Read the primary or official source: Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv)