Path
Coding & reasoning evaluation
Judge one code, maths or research answer. Then build the check set that judges the next thousand.
- Prepares you for
- It prepares you for the part of evaluation work that is written down. Check-set questions, marking rules, and an honest figure with its count beside it.
- What you learn
- You run generated code against what was really asked for. You mark the first wrong step in a piece of reasoning. You split an answer into claims and check each one. Then you go one rung up. You write a check set that can really fail. You keep a held-out set — questions the system never saw — held out. And you turn one wrong answer into an item somebody else can run.
- Why this order
- Judging one answer comes first, because a check set is only as good as the judgement inside its questions.
- Test and review generated code26 minutes
- Find the first invalid reasoning step25 minutes
- Verify claims and citations24 minutes
- Design an evaluation that can fail24 minutes
- Keep the held-out set held out20 minutes
- Write the item sheet22 minutes
- Check the model that marks24 minutes
- Turn a wrong answer into a test26 minutes
Curriculum reviewed: