REVIEW PRACTICE · SEPTEMBER 2026
Find the point of difference
Two reviewers can give different ratings for reasons that need very different responses. One may have missed a fact. The instructions may leave a boundary unclear. Or the task may allow more than one reasonable interpretation.
Before deciding which rating should stand, compare the explanations. The explanation often shows whether the reviewers were evaluating the same thing.
Separate evidence from interpretation
Start with what is directly observable: the prompt requested two items; the response included one; the source stated a particular date. Then identify the judgement applied to that evidence.
If reviewers disagree about the observable detail, revisit the material. If they agree on the detail but rate it differently, examine the rubric.
Use the disagreement to improve the guide
A useful resolution records the original issue, the agreed interpretation and an example. It should help a third reviewer handle the next similar item without asking the same question again.
A rule such as “omit no requested item” is easier to apply than “prefer complete answers”, provided the project defines what counts as a requested item.
Do not revise silently
If a decision changes, retain the reason. A revised rating without a revision note makes the record look more consistent while removing the information needed to understand the improvement.
A simple disagreement record
- Item or example reference
- Original ratings and explanations
- Rubric criterion in question
- Resolved interpretation or remaining uncertainty
- Guide change and date, if applicable
Some disagreements remain valid. In those cases, the project lead should decide how to represent uncertainty rather than forcing a false consensus.