navigation-background arrow-down-circle Reply Icon Show More Heart Delete Icon wiki-circle wiki-square wiki arrow-up-circle add-circle add-square add arrow-down arrow-left arrow-right arrow-up calendar-circle chat-bubble-2 chat-bubble check-circle check close contact-us credit-card drag menu email embed facebook-circle snapchat-circle facebook-square facebook faq-circle faq film gear google-circle google-square googleplus history home instagram-circle instagram-square instagram linkedin-circle linkedin-square linkedin load monitor Video Player Play Icon person pinterest-circle pinterest-square pinterest play readlist remove-circle remove-square remove search share share2 sign-out star trailer trash twitter-circle twitter-square twitter youtube-circle youtube-square youtube

When is big data too big?

March 25, 2016 (Photo Credit: Courtesy Photo)

As our computers become more powerful and our reach becomes more ambitious, when do the benefits of big data begin to decline?  Surely there is a limit to the effectiveness of the ever-increasing amount of data stored on the web, in the cloud, and on our mobile devices. But where is that limit – the inflection point, at which identifying, collecting, and storing more data becomes counter-productive?

Since the early days of mainframes behind the doors, humans have energetically built processers with more power, designed larger and larger databases, and captured more and more data for those databases. At the same time, however, we have recognized that the abundance of stored data and information can result in information overload, “analysis paralysis”, and poor decisions. What we need is a metaphor or template for knowing when more data is too much data – for knowing when big data has become too big.

Malcolm Gladwell has discussed the differences in decisions that can be framed as puzzles and those framed as mysteries, and the challenges of each for decision makers. Puzzles are decisions frustrated by a lack of information. As with a jigsaw puzzle, once the final piece of information falls into its correct place, the picture is complete. The decision maker has solved the puzzle and found the answer. Puzzles resolve to nice, satisfying conclusions, and are typically lacking in ambiguity and uncertainty.

Mysteries, however, are frustrated by too much information. The decision maker is inundated with information, and more information merely leads to more ambiguity and more uncertainty. Somehow the decision maker must navigate through the fog of information, using his experience and insight to make a best guess at the answer, i.e. to connect the dots in a sea of data. Mysteries do not resolve to a satisfying conclusion, and sometimes the decision maker even senses that there may be other, and possibly better solutions.

The federal government’s drive toward big data, larger and larger databases, and more complex systems to manage the data and its databases continues.  Recent years have also brought increased emphasis on transparency, accountability, and access to the taxpayer – all good, sound policies and intentions. But even good intentions and policies can lead to declining returns.  The combined effect of big data and transparency has increased the amount of data and information that decision makers must contend with, moving decision making away from a puzzle-solving approach to a mystery-solving approach.  So, how does the decision maker – federal employee or taxpayer – navigate through the fog of information?

Well, there is no clear-cut, definitive answer, and there is no mathematical formula that can tell us when more information is too much information.  If there were, we would still be living in a puzzle-solving world. This, however, is the information age.

In the information age, the best way to navigate the fog of information overload, become a better mystery solver, and make better strategic decisions is to change the way one thinks.  A recognition of one’s cognitive biases is key. Many current authors have written about recent findings regarding cognitive biases such as overconfidence, anchoring, and rigid framing. With conscious, focused effort, each of these can be mediated by the decision maker.

Decision makers also need to leave their comfort zones more often – much more often -- and strive to think more creatively and imaginatively. Like a real-world Sherlock Holmes, they must come to recognize and acknowledge the dog that didn’t bark.  They must be able to connect the dots in a world of fog, ambiguity, and uncertainty.  And finally, maintaining one’s curiosity is critical; without an interest in what drives the world, and what wags its tail, wanna-be Sherlocks will be stuck at the puzzle stage, always looking for the missing puzzle piece when the answer is right in front of them amid the buzzing, overloaded information picture.

Gerry Gingrich is a professor at the iCollege at the National Defense University, where she specializes in leadership and executive decision making, cyber strategy, and innovation. With 25 years in the federal sector, she is a frequent speaker and writer on cyber leadership and behavioral and IT-driven change.

Next Article