Eitan Zemel passes on this wonderful story about psychologists Kahneman and Tversky. Early in their careers, back in the 1960s, they reviewed a performance reward system used by the Israeli air force. The air force adopted a consistent policy of praising trainees who performed well on a series of maneuvers. They found, to their surprise, that performance deteriorated, on average, in this group. The other group was criticized for poor performance and typically improved.
What was going on? Did we need to rethink the laws of human behavior? Kahneman and Tversky report that graduate students suggested explanations based on overconfidence of high-performing pilots and perceptual biases of instructors. What would you say? The answer involves more statistics than psychology and goes back to the 1880s. The idea is that any measurement system captures both performance (effort, ability) and luck. If good performance was mostly luck, then you’d expect exactly this result, as the lucky pilots today were (by pure chance) less lucky later on. There’s a long history of similar examples in statistics, generally referred to as regression to the mean. Friend and mentor Gary Smith adds: “There are few statistical facts more interesting than regression to the mean for two reasons. First, people encounter it almost every day of their lives. Second, almost nobody understands it.”