Are exit polls an exercise in skill or luck?

The most eagerly awaited exit poll before Bihar results was Today’s Chanakya and it went on to become the most ridiculed. Axis, a relatively less know agency, became the toast of town. Naturally, we will now eagerly await predictions by Axis in the next state election. A common theme across many exit polls was close fight. JDU and NDA were neck and neck. As we all know, when the results came out, the margin was mile and a yard. We were all left wondering how pollsters didn’t pick up on such a one sided election.

Understandably there are many explanations for these differences between various exit polls. Sampling errors, selection bias as articulated by Yogendra Yadav, caste weights, inadequacies of the prediction models among others. I am no psephologist and am not going to analyse why the inconsistency. I am also not going to analyse what drove voter preferences.

Let’s just look at one aspect— What is role of skill in getting predictions right?

What strikes me is the fascination for a particular pollster, someone who we think is invincible. If you go back to our Twitter timelines, you will see many people saying things like “He predicted everything right since 2013”, "He always got the winner right", "They were spot on in 2014". Today’s Chanakya had the hot hand— they had success in the last event, so they were expected to in the future too. The mood in BJP changed after Chanakya’s prediction came in. A leading English news channel even decided not to reveal their exit polls which were very different from Chanakya's. One prominent Hindi channel changed their prediction the night before election results and called the election for BJP.

So is it really skill or is it just luck and chance, as Nobel Laurette Daniel Kahneman calls it. In his bestselling book Thinking Fast and Slow, Kahneman talked of a really interesting story. He was invited to talk to a investment advisory firm that provides financial advise to the super rich. The firm had 25 advisors and on Kahneman’s request, the firm provided eight year of data for investment results of these advisers. What their recommendations were and how it played out in the market. Needless to say, the results determine growth and bonus of these advisors. What he was looking for in this data is consistency and skill. Did the same set of advisers consistently achieve better returns every year? And did some advisers demonstrate that they had better skill than their colleagues? Kahneman calculated the correlations between the rankings of advisers in different years, comparing year one with year two, year one with year three.

The results surprised Kahneman. “While I was prepared to find little year-to-year consistency,” he writes, “I was still surprised to find that the average of the 28 correlations was .01. In other words, zero. The stability that would demonstrate differences in skill levels among different advisers was absent.” He further writes “The results resembled what you would expect from a dice-rolling contest, not a game of skill and the firm was rewarding luck as if it were skill”.

A useful exercise would be to check the consistency of pollsters over a 10-15 year period or across 10 such polls to know whether it is skill or just luck and chance. In a complex country like ours where there are multiple parties and all kinds of realignments are happening, it is hard to predict. We certainly cannot accuse pollsters of dishonesty or lack of effort. Maybe the crazy number of these factors make elections a very random event affair than one with clear, identifiable patterns.

The advisory firm didn’t take note of Kahneman’s advise. They said the bonuses were hard earned and he was making light of their achievements. Kahneman concludes “the illusion of skill is not only an individual aberration; it is deeply ingrained in the culture of the industry.”

There is a same danger here, where we are looking for the hot hand. The "hot-hand fallacy" is the belief that a person who has experienced success with a random event has a greater chance of further success in additional attempts. This is commonly used in gambling and sports like basketball. The law of averages caught up with Today’s Chanakya and Axis is the current hot hand. They may well get the next poll right, but when will their luck run out?

PS: The purpose of this piece is not to undermine the honesty and sincerity of pollsters. There is a great amount of skill involved in doing exit polls. This is merely a hypotheses that explains inconsistencies in predictions.