One brief, a shared interpretation
Review work becomes inconsistent when the instructions leave important decisions unstated. Our method begins with a small calibration set, so questions appear before a full batch is under way.
- Define the taskAgree what reviewers should assess, what they should ignore and the output format. Identify any specialist judgement the project requires.
- Calibrate with examplesReview a small representative set against the proposed rubric. Compare explanations and resolve unclear boundaries.
- Review the batchWork from the agreed guide. Record the decision, supporting reason and any case requiring clarification.
- Check a sampleA quality lead reviews selected work and flagged cases. Feedback is tied to the rubric, not a preference for different wording.
- Revise and hand overReturn agreed outputs with unresolved questions and documented changes. Confirm which decisions should update the project guide.
What a useful rationale contains
A short explanation should point to the part of the response or source that supports the rating. “Better quality” is too broad. “The first response answers both requested questions; the second omits the delivery-time question” can be checked.
Handling disagreement
Differences are separated into a missed detail, an ambiguous instruction or a judgement the rubric genuinely permits. The aim is to improve the shared guide, not to hide every disagreement behind a single final number.
Project access and materials
Reviewers use the tools and sources permitted for the assignment. Work instructions should say whether external source checking is allowed, what material is confidential and where notes are to be stored.
Changes during a project
A revised rubric needs a date and a decision about earlier work. The project lead confirms whether affected items should be revisited; reviewers do not silently apply new criteria to only the remainder of the batch.