Path: How post-training works
Spot reward hacking
Find the shortcut that raises a score without doing the real task, and the measure change that closes it.
- Starting level
- Comfortable with the basics
- Time
- 20 minutes
- Public lesson guide
- EN + हिंदी
You will learn to
Read a measure for the cheapest way to raise it, and name a second check that closes that gap.
Why this matters
Every measure is eventually optimised, including the part of it nobody meant.
Three things to remember
- Ask what raises the score without doing the task.
- Pair a measure with a check it cannot satisfy by itself.
- Watch the measure that stops moving with the outcome it stands for.
See one example
Task
A helpline assistant scored on conversations where the caller does not ask again.
Try the judgement yourself
Both answers respond to the task above. One of them does the job better. Which, and why?
Make something yourself
Take one number your workplace, college or team is judged on. Write the cheapest way to raise it without doing the real work, and the second check that would catch that.
A useful starting pointFour parts: the measure, the real goal, the shortcut, and the second check.
Read further
The source is optional. The complete teaching and practice are available inside AI2Bharat.
Read the primary or official source: Concrete Problems in AI Safety