Why do two assessors rate the same performance differently?
Two assessors rate the same performance differently because they are not measuring the same thing, even when they think they are. One anchors to "compared with last year," another to "compared with the best I have seen." One catches the strong opening, another the shaky middle. Add ordinary measurement error, and a gap of one or two points on a five-point scale is not a scandal. It is what disagreement looks like when humans judge complex work.
This matters for anyone trying to measure growth honestly. If you treat a single rating as fact, you will chase noise. If you understand why ratings diverge, you can design assessment that still tells you something true.
What actually causes the gap
Research on measurement error shows that any single assessment carries a margin of uncertainty, and a change smaller than that margin cannot be called real improvement (see the discussion of minimal detectable change in the PMC article on measurement error). On top of that baseline noise, several specific things push two raters apart.
| Source of disagreement | What it looks like |
|---|---|
| Different anchors | One rater compares to peers, another to the person's own past performance |
| Different standards | A strict rater and a generous rater score the same work differently, consistently |
| Partial observation | Each rater saw a different slice of the work, not the whole of it |
| Vague rubric | Criteria like "shows leadership" are read differently by different people |
| Recency and halo effects | A strong recent moment, or a strong reputation, colours the whole rating |
A gymnastics panel is a clean example. Two judges watching the same routine can differ by a few tenths of a point even with a written code of points, because one weighs a wobble on landing more heavily than the other. Federations handle this by averaging several judges and sometimes dropping the highest and lowest score, not by trusting any one judge's number. A founder pitching a board sees the same pattern: one director focuses on the model's assumptions, another on the team's delivery record, and their verdicts differ even though they read the same deck.
How to measure growth when assessors disagree
Locke and Latham's review of goal-setting research found that specific, well-defined standards produce more consistent performance and judgement than vague ones (Locke & Latham, 2002). The practical version of that finding is to fix what "good" means before anyone rates anything, not after.
A few steps help:
- Write the rubric in observable terms. Replace "communicates well" with something a stranger could check, such as "sent a written update within 24 hours of each milestone."
- Use more than one rater where the decision matters. A single manager's view of a first-time manager's progress, discussed in our piece on writing a plan for a first-time manager, is a starting point, not a verdict.
- Look at the spread, not just the average. If two raters are two points apart on a five-point scale, that gap is information. It usually means the rubric needs sharpening, not that someone is lying.
- Track the trend across cycles, not one sitting. Our piece on how many check-ins you need before trusting a trend sets out why three or more data points beat one.
- Separate self-rating from measured evidence. People often rate their own progress differently from what the record shows, for reasons explored in why self-ratings diverge from the evidence.
A performing artist rehearsing for a role faces this directly. A director's note after one run-through and a coach's note after another can read as contradictory, when both are accurate descriptions of different nights. The honest response is to ask what changed between the two performances, not to decide which assessor was right.
What this means for a development plan
A development goal that depends on a single assessor's opinion is fragile. It will rise and fall with that person's mood, standards, and vantage point, and the plan will feel unreliable even when the underlying work is steady. A sturdier goal defines the measure in advance, states who assesses it and how often, and treats any single reading as provisional until it is confirmed by a second observation or a second cycle.
This is also why "held steady" is a legitimate result rather than a failure. If two assessments a quarter apart differ by less than the normal margin of error for that measure, the honest report is that nothing moved, not that something went wrong.
Jobulary's check-ins are built around this idea. Each one takes about three minutes, and growth is reported only when the change is larger than the measurement error for that goal; otherwise the plan says the person held steady, which is itself useful information. Because the plan sits with the individual rather than the employer, a sponsoring organisation can see that a goal is live and its measure, without seeing the reflections or notes behind any single assessment, as described in how organisations protect privacy in sponsored IDPs.
None of this removes the need for judgement. Assessors will keep disagreeing, because they are watching different parts of the same complicated thing. The task is to design measurement that expects that disagreement and still produces a trend worth trusting.
Frequently asked questions
Is rater disagreement a sign that one assessor is wrong?
Not necessarily. Disagreement often reflects different vantage points or standards rather than error. The fix is a shared rubric and multiple observations, not picking the rater you prefer.
How many raters do you need before a score is trustworthy?
There is no fixed number that works everywhere, but a single rater's view should be treated as one data point. Where the stakes are high, use at least two independent raters and look at the spread between them, not just the average.
Should self-ratings count at all if they rarely match external evidence?
Yes, as context, not as the measure. A mismatch between self-rating and evidence is itself useful information about blind spots or unseen effort, covered in more detail in our piece on why self-ratings diverge from evidence.
Sources
Topics: Measurement · Individual Development Plan · Goal setting