To understand the enormity of the potential of data, let’s go back to the Industrial Revolution.  That’s when there was an exodus of people from agriculture into manufacturing and the country urbanized en masse.  We are now in the early throes of the Information Revolution and jobs are being reshuffled with higher value placed on those who can incorporate different kinds of information into every job. Who can use information to do their jobs?

There will be mistakes on that journey.

Our personal proclivities and psychographics are now in private hands.  Sort of.  Those private hands have a jaded view of what that is.  Historically, corporations used information to make judgments about you - and this was mostly done on paper and barely usable - now they reach into the (landmine) data caverns of third-party curators of our digital footprints.  If it can be monetized, it will be.  But the curation today is in its infancy.

This curation requires the cooperation of the owners of the data – quite often application companies and also the people implicated in the use - to agree to share.    

Are you a cigar smoker?  Well, you did subscribe to Cigar Afficiando.  Do you own a horse?  Well, you do live on a lot zoned for horses.  You get the point.  Scores of people across thousands of dimensions are being calculated and often with imprecise data.

Companies know that if they can accurately anticipate your next move, they have a tremendous advantage in the market. But stores greeting you by name and recalling your last purchase, like in the movie Minority Report, are the tip of the iceberg of possibilities for how people will be treated in the Information Revolution.

Companies also know this interest in data extends to other companies and increasingly create lines of business for their data.  It’s a ‘Wild West’ of data.

Our loose digital cues become sacrosanct in the mad rush to label us so companies can take informed actions.  At the same time, companies are building their data science to handle more nuances in our data so they can treat us, more or less, as individuals.  Until the data is accurate and the science is vastly improved, there will be errors.  And companies have repeatedly shown their willingness to accept this and take chances.

It makes you wonder what other data we might be wiling - or coerced - to give up to give business an edge.  We’ve only seen the first pitch of this game.  What else do we have to give?  We give our clicks, our corporate interactions go into that Wild West I spoke of and there’s a maelstrom of analysis over it. 

How about our DNA?  We could get very personal there.  At some level, we can skip all other data because the DNA is definitive about, well, just about everything.  Although I’m not sure that ship has to sail for there to be a vastly different human experience from what we have today.

We are a ways off from DNA harvesting and understanding.  It’s being worked on, but know that the commensurate technology is there to do anything a company wants to do with today’s data.  Companies are actually able to afford and store to process much more data than they are storing and processing today.  Business needs to be planning for that. What data could it use?

Business has clear upward trends of spending on big data.  It’s projected to be the top item of spend in many industries.  Companies are adopting Hadoop and NoSQL, although larger companies struggle to get them into production.  That will get fixed with the advent of more robust systems management tools and the increased pressure to save all data at lower costs.  Graph database is the fastest growing database category.  Streaming data is becoming more common for real-time data analysis.  All of these involve big data and all were barely spoke of 10 years ago.

Most businesses have to admit that no matter what business they are in, they are in the business of information and everything else simply allows them to pursue business-as-usual.

So data is valuable.  It gives business the view it needs to understand and improve itself.  This adoption has driven the improvements in hardware.  And while data is proving itself to be the next natural resource, there is a dark side.  Data misinterpretation.  Data misrepresentation.  Hacking is at an all-time high.  All the IoT devices are susceptible. 

In the last installment, I will discuss how as we push hard on new and advanced uses of data, that without a governed framework, it will be easy for data to overwhelm us.