Path: Coding & reasoning evaluation
Keep the held-out set held out
Spot when a test result has already been used to repair the system, and rebuild the held-out set honestly.
- Starting level
- Comfortable with the basics
- Time
- 20 minutes
- Public lesson guide
- EN + हिंदी
You will learn to
Tell which evaluation items still carry evidence, and rebuild a set once it has been used for fixing.
Why this matters
A held-out set is spent the first time someone reads its failures and edits the system.
Three things to remember
- An item used to fix the system stops being evidence about it.
- Record what each item has been used for, and when.
- Refill the held-out set before the next round; do not reuse it.
See one example
Fifty held-out questions; the twelve failures were read and used to change the instructions.
Try the judgement yourself
Both answers respond to the task above. One of them does the job better. Which, and why?
Make something yourself
Take any check you repeat — a mock test, a monthly report review. Write which items you have already used to fix things, and how you would collect a clean set.
Read further
The source is optional. The complete teaching and practice are available inside AI2Bharat.
Read the primary or official source: OpenAI GPT-4 technical report