Path: Teaching and reviewing
Handle honest disagreement
Tell a task-design problem apart from a reviewer mistake when two careful people reach different scores.
- Starting level
- Comfortable with the basics
- Time
- 20 minutes
- Public lesson guide
- EN + हिंदी
You will learn to
Tell a brief that does not decide the case apart from a reviewer who read it wrongly.
Why this matters
Two careful people reaching different scores is information about the instructions. Averaging the scores throws that information away.
Three things to remember
- A defensible disagreement is evidence about the brief, not about the reviewers.
- Write both readings down before deciding which one the brief supports.
- Never average two scores to make a disagreement disappear.
See one example
Two reviewers mark the same voice clip differently: one rejects it for background noise, the other accepts it.
Try the judgement yourself
Both answers respond to the task above. One of them does the job better. Which, and why?
Make something yourself
Describe a disagreement you have seen over the same piece of work. Write both readings, then the sentence in the instructions that allows both, and how you would rewrite it.
Read further
The source is optional. The complete teaching and practice are available inside AI2Bharat.
Read the primary or official source: Davani et al., Dealing with Disagreements (arXiv)