Data without discretion can sink your ship

One of the classic mistakes that people make when dealing with data is not asking the right questions. It doesn't matter how much data one has if the right questions aren't asked. Incorrect inferences drawn from data have sunk more companies and ships than perhaps any other cause. Among the many ways people say (post-facto of course, hindsight being 20-20 and all), the Titanic could have averted disaster given multiple messages from other ships in the vicinity of floating icebergs, if the captain had asked whether their risks would go down considerably were they to simply stop in the water and proceed come daylight.

Why am I bringing this up now? During one of the conferences that I attended, I met a senior HR leader of a large firm triumphantly announce that they were taking data-based decisions in his company. Intrigued, I asked him to explain how they were doing that. He mentioned that a recent analysis of the data showed that over 60% of the poor performers in his organisation came from college X, and hence, they had decided that they were not going to hire from that particular college. It seems fairly logical at first glance, doesn't it? An organisation using data to make decisions – the kind of decisions that make people feel and look good. However, there was one issue with this particular decision. The fact that even with the best of intentions, the organisation had not asked the right question given the data that they did have.

The question that the organisation had asked was: What is the probability that the employees were from college X if their performances were poor? However, the actual question they should have asked in order to make the decision that they did in refusing to recruit from the given college was: What is the probability that employees would perform poorly if they were from college X? While it could sound slightly confusing to start with, a little bit of thought shows that there these two are linked but different questions. These probabilities are known as conditional probabilities because, as the name suggests, they are dependent on another event happening. However, the probability of event A occurring given that another event B has occurred is not necessarily the same as the probability of event B occurring given that event A has occurred. There as a whole branch of statistics known as Bayesian statistics, named after the man who founded it – Reverend Bayes. The beauty of Bayesian statistics is that in the absence of information, we can start off with a prior estimate of the probabilities and then based on information that becomes available, adjust our initial probabilities to reflect the true probability.

Assuming that 5% of an organisation are rated as poor performers at any given point in time, the answer to the question originally posed by the manager on the probability that a person was from a given college provided they were poor performers works out to around 60%. That seems incredibly high and hence would necessitate dropping that college from any future recruitment plans. However, without going into the gory statistical details, the probability that a person from that same college becoming a poor performer is roughly 7.3%. While this is still more than double the percentage of people from other colleges becoming poor performers (at 3.4%), it is nowhere near as alarming as the original scenario made it out to be.

Hence, it is imperative that organisations take cognizance of the fact that while data-driven decision making is good, correct analysis of the given data ought to be a precursor to that. And correct data analysis can only be done if we ask the right questions. To quote Ronald Coase, "If you torture that data long enough, it will confess".

The writer is founder & CEO of HR analytics start-up nFactorial Analytical Sciences