When two annotators label the same data point differently, it's tempting to count it as a mistake. It's the quickest and tidiest signal. But disagreement has at least four causes, and each one needs a different fix.
The four causes
The rule doesn't cover the case, or covers it in a way that can be read two ways. Experienced annotators disagree as often as new ones, and they split fairly evenly between the answers. For example, a guideline says to label an item only if it is “clearly visible”, and annotators can't agree on whether a half-hidden object qualifies.
The fixClarify the rule, add a visual example, then test the same cases again to check the clarification works.
The rule is clear, but people read it differently.
The fixUpdate training, then retest against the gold set to confirm everyone has reached the same understanding.
When misses cluster around one person, that's usually an execution issue. Retraining an entire team because of one person's mistakes costs time and can send the wrong message.
The fixAddress it with that annotator, not the whole team.
Reasonable people disagree, and forcing one answer hides information about your data. The disagreement isn't necessarily a problem to fix. It is information. Research on annotation agreement makes the same point: disagreement often reveals ambiguity in the task or data itself, not poor annotation (James, 2026).
How to tell the causes apart
A recent example
On a recent project, quality stalled despite several rounds of re-coaching and guideline updates. When we looked at where the errors fell, two things stood out. They clustered around two attributes and split almost evenly between the possible answers, with the most experienced annotators failing as often as the new ones. And the same item often appeared more than once, at a different angle or shape, and wasn't always labelled the same way. A change in viewpoint could change how an attribute looked, and the guideline didn't cover that grey area. The gap was in the rule, not the people.
Why this matters at volume
On a small project, you can get away with treating every mismatch as a mistake, because someone can look at each one. At volume, that's much harder. A QC process that only tracks agreement will mistake ambiguous data for poor performance and miss the rules that keep causing confusion. If that isn't caught, the numbers stay stubbornly flat.
Agreement rates need care too. Raw percentage agreement ignores agreement that happens by chance, so a high figure can look reassuring when one answer dominates the data (James, 2026). Low agreement, on the other hand, can point to unclear instructions, too little training or a subjective task, and a single score is not a verdict on annotation quality (James, 2026).
What good QC does
Good QC looks at where disagreement clusters, finds out why, and adjusts the guidelines and training. In practice:
- Track disagreement by rule, task type and annotator group, not just the overall rate.
- Each cycle, review a sample of disagreements and sort them into the four causes.
- Turn each finding into a change: a guideline edit, a training update, a conversation with an individual or group, or a data flag.
- Retest after each change, so you know the fix worked.
Check how the same item is labelled across views
When the same item shows up more than once, at a different angle, shape, colour or size, you have a free consistency test. Compare the labels each time. If they differ, check what changed: the angle, the colour, or how the guideline applies in that context. Differences point to edge cases the guideline needs to cover, especially in dynamic or ambiguous data.
For some tasks, the right outcome is still one accepted label, because the model needs one clear answer. For others, it's an escalation, an ambiguity flag or a record of plausible answers. The right answer depends on what each model needs.
Quality is not just agreement. It's knowing what disagreement actually means and responding to the cause.
Want us to look at where your task breaks?
Talk to Our TeamJames, J. (2026). Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation. arXiv:2603.06865. arxiv.org/abs/2603.06865