Big Data, gold nuggets and the email abyss

Gold on Wikipedia

Big Data is a big buzz in the software world. It is an attempt to create small nuggets of gold from a steaming mass of information. It is not a product or a theory, more a collection of tools and platforms for organizing, analyzing and visualizing masses of data in new ways. This isn't a post on which vendor has the best visualization, the best data management, or whatever. It just provides a quick glimpse into what Big Data is, and how one of its biggest failings is common to small businesses as much as the huge research establishments that coined the term.

Big Data has sprung out of the desire for corporations to gain more meaning from all the data they collect every minute of every day. The information they are collecting about customers, about activities people perform, what they buy and the decisions they make. It is based on techniques grown in scientific research such as the Large Hadron Collider (that enormous “atom smasher”), that attempts to make the results of its 150 million sensors producing millions of sets of data every second into something that mere humans geniuses can understand. It provides medical research with a ways to make the human genome project into something more than a big experiment, developing drugs to address real diseases. And of course, government, with ways to meaningfully understand the requirements, trends (and tax evasion) of tens of millions of citizens.

Big Data is one big funnel, with megatons of data flowing in the top, and ounces of precious observation dripping out the bottom. And just like any organization, dealing with any insight, issue or lead it is at this point the Big Data analysis organization falls over and resorts to... email. All that effort in understanding an aspect of client behavior, drug interactions, or financial transactions takes real human effort. The care taken with a valuable result it is to dump it into a large abyss of junk mail and Facebook notifications.

Large corporations, governments and small businesses are all alike; everybody suffers from the same issue. They spend a lot of time working on problems, finding leads, understanding clients, but have no way of really organizing the useful information into something meaningful, to ensure that the value in the data doesn't get lost. That the potential new customer doesn't just forget she asked for information on your website. That your biggest client doesn't get upset at poor customer service and Tweet #fail about it to the world. That the analysis of your customer’s spending patterns doesn’t just leak out the bottom of a busy executive’s iPhone messages.

Sometimes email is good enough, but often we all need just a little more organization of information, a defined business process to follow and some simple management of who gets to see what, when. This combination of workflow and simple tools is all that is needed to prevent your own Big Data gold nuggets disappearing into the email abyss.

Follow more of my information management, Big Data and process rants: @consected on Twitter. Or ask me about how to prevent the precious information in your business leaking away.

A post from the Improving It blog

Let us help you improve your business today. Visit www.consected.com