Path: Coding & reasoning evaluation
Design an evaluation that can fail
Build a small check set with easy, hard and genuinely unclear items, so that a passing score means something.
- Starting level
- Comfortable with the basics
- Time
- 24 minutes
- Public lesson guide
- EN + हिंदी
You will learn to
Build a small check set whose score can drop, and say in advance what each item tests.
Why this matters
A check set chosen after the system was built usually measures the choices that were already made.
Three things to remember
- Include items the system should fail today.
- Write the expected answer before you run the check.
- Label each item easy, hard, or genuinely unclear.
See one example
Task
A ten-item set for a Hindi bus-timetable helpline assistant.
Try the judgement yourself
Both answers respond to the task above. One of them does the job better. Which, and why?
Make something yourself
Design six check items for a task you know well. Mark two easy, two hard and two genuinely unclear, and write the expected answer for each before any testing.
A useful starting pointFour parts per item: the input, the expected answer, the difficulty label, and what it tests.
Read further
The source is optional. The complete teaching and practice are available inside AI2Bharat.
Read the primary or official source: Holistic Evaluation of Language Models