Big Data Not The End

3rd June, 2014

We have been somewhat unabashed in positive analysis of big data on the PFA-Research blog. Understanding the scope of big data analytics and the effect of big data application in real world settings are another aspect of evaluating the evolving role of big data in our digital lifestyles, but we must concede that big data is a catchall term that leaves nothing behind.

You will have noted the term big data being thrown around throughout multiple industry sectors; indeed, the data we generate and store is a vast money-spinner and has emerged as a potentially outstanding analytic tool for researchers to pore over. The rise of the term big data has the classic hallmarks as the biggest buzzword of the year, or perhaps even half decade – but is there substance to the claims of the many that it will entirely change our lives?

There is clear evidence that some forms of big data are becoming increasingly relevant in our day to day: consider the digital footprint most of us are continually updating. This forms much of the data stream that companies seek for analysis with the hope of improving services. This type of data can be considered ‘found,’ and our interactions with digital organisations find us leaving our data emissions for their usage. Companies that thrive on big data such as Facebook and Google have undoubtedly excelled at utilising found data to their advantage to expand and enhance their operations – good news for their billions of users and provision of ample evidence that these symbiotic data relationships will form the basis of the next evolution of our digital society.

Similarly, the onus has been placed on big data to locate the root of common problems through widespread statistical analysis. Many big data advocates believe that once there is enough data gathered for analysis the suspect correlations will become clear, anomalies will become easier to locate and most of all, it will all be easily fixable. In some circles it is estimated that the US healthcare system could save up to $300bn per year, or around $1,000 per citizen, through careful data analysis and widespread implementation and integration of information gathered. On UK shores, the NHS is facing a £20bn budget squeeze, set to affect productivity across all departments. Again, the answer is seen in understanding where inefficiencies lie and eradicating them.

But it is exactly this reliance on statistical analysis and mass dataset correlation that draws criticism. For example, take the Google Flu Trends data experiment. Google utilised search data for flu, its symptoms, preventatives and cures, and correlated the data by location enabling location specific flu maps to be updated in real-time. Results were positive and the service was widely used, up to 2012 when the big data service began predicting a major spread of the virus when in fact, the virus was moving almost twice as slow. Google attributed the statistical error to the number of media outlets reporting on flu outbreaks across the USA encouraging healthy individuals to search for flu related terminology, considerably boosting the collective search rate. But this is exactly where those who remain sceptical of big data exercise critical analysis: big data is an outstanding tool for establishing correlation between concurrent events, but has no direct method of establishing causation and in many cases it is the latter that provides detailed insight into a statistical phenomenon. As we have seen with Google Flu Trends, if you are unsure of why the data is correlating to begin with, how will you explain when the data pattern changes?

We also see that despite the advances in data pattern recognition between large datasets there is still an inherent need for direct marketing – that is the data giants such as Google and Facebook cannot always unpick the numbers to discern consumer insight and as such rely on the tried and tested methods of consumer surveying and online panels. Understanding the why of the consumer is evidently still encompassed in big data business plans, which is good news for those market researchers in full support of not only traditional methodologies, but ensuring that the human nature of market research remains, through even the allure of big data.

As we have said on this blog before, big data is here to stay, that is certain. It now seems a growing number of businesses are realising that it is not the final evolution of research – just another facet of the research armoury at our disposal.