Open data nowhere near its true potential as key players missing

Open data is going to be the next big thing. How much I wish I could say that. But alas, I have to instead spend the next few paragraphs explaining why it is nowhere near that. 

For the uninitiated, open data refers to data being released by various agencies with a licence that permits free use, and in re-usable, machine readable format. Though open data can come from any source, one of the biggest source are governments and multilateral global agencies such as World Bank, IMF, UNESCO, and the like who collect huge amounts of information for their various studies. 

Many governments have voluntarily decided to release the data to the public because they believe wide and open access to such data will result in innovative applications being created. To some extent, that has happened in countries like United States, which was one of the first countries to opt for this, and has since championed the concept globally. In many countries, open data is part of open governance initiatives.

But despite its huge potential, and limited pockets of success, open data movement is nowhere near realising its true potential when it comes to creating social or economic impact. 

Why? 
Many will invariably point fingers at the slow speed at which the government departments and agencies are releasing data. In India, for exampleafter more than five months of its launch, the open data site, data.gov.in, has just about 115 datasets; what’s worse, all those datasets are from only 11 government departments/agencies, who have participated so far.

While slow speed of releasing data may be a reason, it is surely not the only reason. Also, it is unfair to put the blame entirely on the government agencies. With little useful applications of the released datasets, they do not feel motivated enough to make that extra effort to release more and more, not because they have any objection to their being released—as many would like to believe—but because most of the data are not stored in machine readable format and has to be converted and cleaned before uploading them on the open data portal. 

Open data in India
India launched the its open data site, data.gov.in in September last year. The site is part of the country’s commitment to provide open and transparent access to data collected by various government departments and agencies, as outlined in the National Data Sharing and Accessibility Policy 2012 (NDSAP-12).

The stated advantages, as envisioned by the policy, include maximization of use of data, avoidance/minimization of duplication of efforts on collection, facilitating integration by leading to common standards, providing ownership information, faster and better decision making, and of course, equitable access to information by all citizens.

The data.gov.in site has debuted with 13 raw datasets provided by seven departments and today has 115 datasets, provided by 11 departments. As part of the plan, data management offices have been created in each of the departments, which are headed by senior officials called data controllers. Most  large  ministries/departments have already identified their data controllers, whose names are available in the site. The departments are responsible for directly uploading the datasets, in machine readable, re-usable format. National Informatics Centre (NIC) manages the site. 

The gaps
The open data site in India is managed by an enthusiastic set of people at NIC who have been constantly engaging with both the government departments by promoting, training and helping them as well as the outside stakeholders such as software companies, individual developers, advocacy groups and NGOs who want to use this data. That is the right approach as without real collaboration happening among all stakeholders, it is difficult to progress.  

Globally (as well as in India), these set of people—a few committed government personnel, techies, developers, small start-ups, and advocacy groups/activists/NGOs—are the ones who are active as far as open data movement is concerned. In some cases, media too gets involved, thanks to the new interest in what is called data journalism—getting stories from analysing available data.   

In essence, the gaps are two. 
1. These people have little understanding of the end user needs and they also are not driven by that. So, there are often major gaps between what they develop and what the “market” needs. 

2. They often struggle after a point because they fail to generate money for taking it forward because of the above reason. 

And those people who could actually help them in both—the marketers—are completely missing from the picture. Few marketing people that I follow in various media globally ever talk of how open data could change marketing, even though they keep talking about social media, mobile, and many other new technologies. Even big data for them mostly means either their internal data or social media data. While this often takes centre stage in all marketing discussions, many are blissfully unaware of the parallel phenomenon of open data. 

I do not blame them. Rather, I think that the blame should go to our community—the market researchers. Many marketers often depend on market researchers to provide them with what they call consumer insights. 

I would argue that the market researchers today either do not understand the value/potential of the whole phenomenon or they ignore it. It could be because of a variety of reasons. Some of these are as follows. 

l Lack of awareness: Many simply do not know about the phenomenon. They are too engrossed in their research work.

l Intimidation by the tech-lingo that dominates the space. Thanks to the dominance of techies and developers, the open data space today is full with tech jargon. That is often intimidating for a community, which has not been very tech savvy.  

l The mindset. Many have pre-conceived notions that anything secondary is inferior to primary research. 

l Fear of cannibalisation. Many fear that too much of secondary data analysis may actually make some marketer skip primary consumer research in some cases, because a primary research takes both more time and more money to do. 

While the open data evangelists and those government people and agencies who want open data to succeed to work towards ensuring that the first two issues are addressed, the last two are internal to the market research community. 

They have to believe that the force is disruptive and if they do not seize the opportunities, newer players such as analytics firms and pure play companies will take the game away from them. 

I believe that for the open data evangelists within government and multil-lateral agancies, it would be a better idea to reach out directly to marketers. Once they are convinced, market researchers can no longer ignore this. 

As far as open data community is concerned, once the marketing and market research professionals join it, the possibility of building something actually useful for the end users will grow manifold, as these people will not just bring in valuable consumer insights that will help build better applications but they can also take that to the actual consumers. 

Shyamanuja Das is director and head, business research, JuxtConsult