Abusing Statistics I

In general, the public is not very science-literate. The media often makes things worse by giving a voice to crackpots and ideologues who unwittingly or knowingly misrepresent the facts. Fortunately in this hyper-connected era, for every crank that generates rubbish commentary, there are always well-informed, scientific-minded people ready to shoot it down.

Here is a wonderful example: the Guardian’s climate reporter ripping into a shocker from the Daily Mail that proclaims the world is cooling because the arctic ice has grown by 60% over the past year.

As well as thoroughly debunking David Rose and providing a nice summary of the scientific consensus of what is going on with climate change, the Guardian article features two neat gifs that capture in diagrammatic form how one should and should not interpret the data.

There are many lessons to be learnt from this, here are the two main ones:

1. Evaluate the overall trend, instead of cherry-picking bits to suit your biases

Real life is messy and complicated. Science is messy and complicated. When researchers go out into the world to collect data, there’s bound to be a lot of variance (or noise, or fluctuation). Over relatively short periods of time, and in relatively small sample sizes, the fluctuations can seem compelling. Don’t be fooled. What does the overall picture say? Are global temperatures getting warmer or cooler? Is support for this candidate growing or shrinking?

As a corollary, when it comes to scientific findings — and this is especially true in the area of medicine and health care — evaluate the whole of the literature rather than accept the conclusion of any single study, or small group of studies. Place your trust only in things that are tried-and-true, as demonstrated in well-designed, large-scale and repeated trials. Examine everything else with a critical eye. Is this tonic the secret to anti-aging? Will this miracle fruit cure all that ails you? Is this drink the secret weapon in weight loss? Probably not — unless they can meet the justifiably high burden of proof.

2. Understand regression to the mean

This is such a simple yet critical concept. It is related to the above idea that real life is messy and complicated, and that anything we observe is inherently going to have random variations. Often, what we seek to explain by some plausible theory or story, or what we seek to attribute significance to, is simply a result of these natural variations.

A 60% increase in arctic ice coverage in one year does seem remarkable — until you realise that in the previous year it had shrunk to the smallest size in recorded history. After such a large decrease, an increase was inevitable, one that would seem relatively big in percentage terms. As an aside, you should also be aware of percentages. They don’t mean anything without context, especially the absolute values involved.

There are many great examples of poor reasoning stemming from ignorance of regression to the mean. You may have heard of the Sports Illustrated cover jinx, in which athletes who get featured on the front cover of SI are jinxed because they tend to suffer a major decline in performance afterwards. This seems spooky at first, until you realise that these athletes are only featured in the first place because they have performed at an incredible (and unsustainable) level. When they once again perform at their long-term average, it is a relative decline when viewed against the giddy heights of their SI-cover-worthy exploits.

For me, the best example comes from Daniel Kahneman (whose book Thinking Fast and Slow I cannot recommend highly enough). Here he recounts an aha moment:

I had the most satisfying Eureka experience of my career while attempting to teach flight instructors that praise is more effective than punishment for promoting skill-learning. When I had finished my enthusiastic speech, one of the most seasoned instructors in the audience raised his hand and made his own short speech, which began by conceding that positive reinforcement might be good for the birds, but went on to deny that it was optimal for flight cadets. He said, “On many occasions I have praised flight cadets for clean execution of some aerobatic maneuver, and in general when they try it again, they do worse. On the other hand, I have often screamed at cadets for bad execution, and in general they do better the next time. So please don’t tell us that reinforcement works and punishment does not, because the opposite is the case.” This was a joyous moment, in which I understood an important truth about the world: because we tend to reward others when they do well and punish them when they do badly, and because there is regression to the mean, it is part of the human condition that we are statistically punished for rewarding others and rewarded for punishing them. I immediately arranged a demonstration in which each participant tossed two coins at a target behind his back, without any feedback. We measured the distances from the target and could see that those who had done best the first time had mostly deteriorated on their second try, and vice versa. But I knew that this demonstration would not undo the effects of lifelong exposure to a perverse contingency.

Admit it, statistics is pretty cool.