Big data – quantifying management

Businesses are collecting more data and doing more with this data than ever before. Big data is about the commercial exploitation of the links between different data. To harness this new gold mine, businesses need new skills and a new look at data driven decision-making and management.


Reporting has always been an essential component of management but the notion of data-driven management is given a new lease on life through the tools and methodologies made available through Big Data. There was a rise of analytics in the early 2000’s but the amount of data being analysed and the types of insights that is produced by this data has radically changed the game. Nowhere is this more visible that with Facebook, Google, Twitter and others competing, through algorithms, to bring you more relevant news from people that you are interested in so that they can advertise to you.


The question is if you should care about the big data revolution in your business. A research study published in the article, Big Data – The Management Revolution (McAfee and Brynjolfsson, 2012:64) showed that companies that characterised themselves as data driven on average were 5% more productive and 6% more profitable than their competitors even after accounting for contributions of labour, capital, purchased services and traditional investment. This is a lot of money once you get to scale and a data oriented approach from the beginning of a new business can change the game substantively when you later start pushing out new initiatives. The idea of “growth hacking” – which is where a business grows very fast – is highly dependent on data driven reasoning and Peters (2014) makes the point that everything, including the business model is changeable when you want growth such as experienced by the large big-data companies.


So why do we get so stuck as businesses? One of the main theories is that decision making starts revolving around certain key people in organisations. It has been dubbed the Hippo phenomena in that most organisations gravitate towards respecting the HIghest Paid Person’s Opinion. Research on executive decision-making theory invariably shows that there is usually only negative correlation between remuneration and executive performance due to moral hazard factors. This can be mitigated – but it does not account for bad hunches by executives that do not pay out.


While it is important to realise that intuitive reasoning of a very experienced executive is often valid – there is a increasingly complex world that has to be analysed when making decisions and the most relevant question an executive can is “What does the data say?”.


This is then further qualified with:

  • What insights can be derived from this thinking?
  • Where did this data come from?
  • What analysis was conducted?
  • How confident are we with the data?


The power of this approach is that executives should allow themselves to innovate with and within the power of data and not get stuck in an old playbook that does not take the real world into consideration. That is easier said that done and one of the keys to big data is great change management and a flexibility mindset.


Some of the key management challenges facing anyone that adopts big data as an organisation strategy includes:

  • Leadership
  • Talent Management
  • Technology
  • Decision making
  • Company culture


McAfee and Brynjolfsson (2012:64) state the following to highlight the enormity of this business challenge: “Big data’s power does not erase the need for vision and human insight. On the contrary, we still must have business leaders who can spot a great opportunity, understand how a market is developing, think creatively and propose truly novel offerings, articulate a compelling vision, persuade people to embrace it (the idea) and work hard to realise it (the business), and deal with customers, employees, stockholders, and other stakeholders. The successful companies of the next decade will be the ones whose leaders can do all that while changing the way their organizations make many decisions.”


Gartner’s Svetlana Sicular has catalogued eight common causes of big data project failures, including:


  • Management resistance.
  • Asking the wrong questions.
  • Lacking the right skills.
  • Unanticipated problems beyond big data technology.
  • Disagreement on enterprise strategy.
  • Big data silos.
  • Problem avoidance.


One of the key issues with big data is an idea called “planned myopia”. We sometimes choose not to look at the interaction factors between variables simply because it would not look good. All data can be used to tell a story and planned lies and manipulation of results hurt the validity of what is found. We also choose not to ask the hard questions and to learn from what we suspect.


Another other issue is often that statistical correlations are sometimes not understood well, leading to poor decision-making (Leek, 2014). The science of analysing underlying contributing factors to statistical phenomena is difficult, even if you are trained to understand it. There is a growing host of “big data failures” where predictions are either over or under sensitive or make the wrong conclusions. This is fine with small-scale studies but when you are trying to use big-data to regulate resource allocation you can get into real trouble.


On the other side – you can get it spectacularly right. If you have not heard of Uber – a service where you can order taxi’s anywhere in the world, then you will soon. It is a classic case of technology meeting an old industry and creating a new player. Uber offers an application that uses GPS in your smartphone to hook you up with a taxi. You do not have to pay the driver physical cash as you link your credit card to the service. Started in 2009, this business is now valued at $ 4 billion USD and operates in most major cities in the world. How could this happen? First and foremost, Uber provides a solution to a real problem that impacts millions of people. In all sense of the word they have disrupted the monopoly of taxi-cab transportation that exists in many cities and reinvented the experience from top to bottom. Among the many problems Uber is tackling are: poor cab infrastructure in some cities, poor service and fulfilment–including dirty cabs, poor customer experience, late cars, drivers unwilling to accept credit cards, and more.


Uber set out to reimagine the entire experience to make it seamless and enjoyable across the board. They didn’t fix one aspect of the system (e.g. mobile payments for the existing taxi infrastructure), they tackled the whole experience from mobile hailing, seamless payments, better cars, to no tips and driver ratings. This has now made such a huge impact in key cities that it is forcing the historical cab industries to rethink their delivery and even “signing-up” to become Uber suppliers, while failing into new delivery standards.


By avoiding the trap of smaller thinking, and iterating on one element of the taxi experience (say, by making credit card payments more accessible in the car) they were able to create a wow experience that has totally redefined what it means to use a car service, sparking an avalanche of word of mouth and press.




Big data is the future. With increasing exposure it is critical that you look at data in your organisation and ask how this can be best used into the future. As managers we need to understand data driven reasoning and be prepared to have ourselves challenged when the numbers simply say that we live in a different world than before.




Peters, Robert (2014) Growth Hacking Techniques, Disruptive Technology – how 40 companies made it big, BLEP Publishing.


Mcafee, Andrew; Brynjolfsson, Erik (2012), Big Data: The Management Revolution, Harvard Business Review.