The Curse of Expertise: The Effects of Expertise and Debiasing Methods on Predictions of Novice Performance

The question

Are experts good at predicting how a beginner will perform, and if not, can you fix it by warning them?

The method

Two studies on two hands-on tasks. In Study 1 — learning an advanced cellular handset — “salespeople (experts), customers (intermediate users), and novices estimated the novices’ performance” [2], and the estimates were scored against how long novices actually took. Study 2 moved to a different domain on purpose, building a 39-piece LEGO model, with expertise manipulated by practice rather than sampled. Both tested debiasing prompts: recall your own first attempt, or read a list of the problems novices hit.

The findings

The result is blunt: “in both studies, those with more expertise were worse predictors of novice performance times and were resistant to debiasing techniques intended to reduce underestimation” [1]. The size of it is what makes it useful. Novices actually took 31.5 minutes [3]. Unaided, “experts, on average, predicted that it would take 12.86 min” [3] — intermediate users said 20.09 and the novices themselves said 15.71. The people who knew the task best were furthest from the truth, and the people in the middle were closest.

Warning them did nothing. Across both debiasing conditions, “intermediate users improved their predictions by 20%, whereas experts gave predictions that were 2% worse in the debiased than in the unaided trials” [4]. Hinds tested why. Experts did not see the task as having fewer steps — “evidence from Study 2 suggests that experts did not perceive fewer steps in the task than did those with less expertise” [5] — so oversimplification is not it. What survives is memory: “experts underestimated their own novice performance times by over 3 min” [5], and since everyone estimates by anchoring on their own first attempt, an expert anchored to a beginner who never existed. “Evidence from these data suggest that the availability bias is the key contributor to experts’ relative inaccuracy in estimating novice performance times” [5]. She reads all of it as of a piece with Camerer, Loewenstein and Weber’s “curse of knowledge, a bias in which knowledgeable people are unable to ignore their own superior knowledge in trying to make naive predictions” [5].

Her own conclusion states the practical implication: “experts may have a cognitive handicap that leads to underestimating the difficulty novices face and that those with an intermediate level of expertise may be more accurate in predicting novices’ performance” [1].

The limits

Two procedural tasks and one kind of prediction — elapsed time to learn. Whether the same gap appears when an expert predicts the quality of novice work, or judges a candidate rather than a learner, is not tested here. The samples are modest and several of the debiasing contrasts are not statistically significant.

• You are asking an expert on a Calibration Call what a new hire should be able to do in ninety days. Without this paper you take the answer as a calibrated estimate. Hinds says expect it to be far too optimistic, and expect the expert to hold it after you point that out.
• You are choosing who to call. The most interesting number in the paper is the middle one. The person two years into doing the job well may describe the ramp more accurately than the person who has done it for fifteen years — not because they know more, but because they still remember not knowing.
• You are setting a bar for Knowing what great looks like from one expert’s account of what is basic. What an expert calls table stakes has been through the same compression. Ask for the last time they saw someone struggle with it.