I’ve spent my career looking at how large quantities of complex information affects every part of our lives and this is the most exciting time to be doing that.  Information affects finances.  Information affects your health.  It affects the life choices presented to you.  It cannot be overstated how important the accumulation of enormous sums of detailed data about all of us and every aspect of business is.

Ten years ago who imagined that so much of the planet would be photographed and those photographs made widely available, as in Google Earth? Or who would have imagined that we would be so willing to share large amounts of personal information through public and quasi-public outlets like Twitter, Facebook and Foursquare?

How much data are we talking about?  The earth generated about 4 zettabytes of digital data in 2013.  IDC forecasts that we will generate 40 zettabytes (ZB) by 2020.  Now, all that data isn’t used, but increasingly, more of it is. And it’s not just stored once.  Data with value is branched off into numerous databases across multiple companies.  In only the last few years, as much data has been generated as had previously ever existed. 

The Rise of Machine Data

It is machines that are primarily responsible.  Machine data contains critical insights.  It allows us to do unprecedented triangulation of physical objects,including all of us.

Unlike traditional structured data– for example data stored in a traditional relational database for batch reporting – machine data is non-standard, highly diverse, dynamic and high volume.  

For example, each of the underlying customer touch systems in a product purchase can generate millions of machine data events daily. When we look more closely at the data we see that it contains valuable information – customer, order, time waiting on hold, twitter id … what was tweeted.

If you can correlate and visualize related events across these disparate sources, you can build a picture of activity, behavior and experience. And if you can do all of this in real-time, you can respond more quickly to events that matter.

You can extrapolate this example to a wide range of use cases – security and fraud, transaction monitoring and analysis, web analytics, IT operations and so on.

The Future Sources of Digital Data

We’re on the way to 50 billion connected devices.  Processors are embedded in things everywhere.  Most physical objects are on the way to being online.  Internet Protocol 6 will allow for 78 octillion (billion billon billion) simultaneous connections.  The internet in a few years will swell from the size of a relative baseball to the sun.   Here are just a few of the advances to come, all of which will create digital data:

• The Pill Cam, which you swallow and it takes photos of your internals is one of many under skin devices. 
• The Bodytel, the Glucotel and the IHealth Oximeter for no-sting blood measurements. 
• Trash cans that sense their fullness so they can do compaction and call for emptying over wifi.
• Vending that takes pictures of you and shows images of what people like you buy – and what the vending will prefer you buy now.
• The Moticam, which captures everything a microscope looks at.
• The Bodyguardian, which allows physicians to monitor biometric data around the clock.
• Transportation companies prevent derailments caused by failing wheels and bearings. There are 20 million sensor readings that spot 1500 issues per day with the wheels and bearings at Union Pacific.

Companies that can harness this data can benefit accordingly.  Industries that are growing fastest are those that are adopting technology and using data to understand themselves and within an industry, companies that are best harnessing their data are growing fastest.

Companies are beginning to value information more highly than any individual transaction or supporting process.  What could be more important to a retailer?  Making a sale or understanding that customer x would make that sale at that time in that store?  It is the latter because that affects a life-long relationship – if you know what to do with the data.

“Software Eats the World”

All of this relates to Marc Andresson’s claim that “software eats the world.”  What it means is every industry is becoming a software industry and a data-intensive industry.  We really can’t afford not to store and process more data if we want to be successful and keep improving relative to peers.

For example, the data generated from sensor networks can be terabytes per second.  A trans-Atlantic flight generates 650 terabytes of data from the airplane engine sensors. There is a lot of value in this information informing the repair and optimal use of the aircraft.

Or consider a truck-based asset delivery customer with average journey times of 19.6 hours for delivery corridor. A combination of tracking and insight allowed the discovery of floating optimal departure time across each day of week. Scheduling departures during these times reduces the journey duration up to 48%. Missing them can add over 100% to journey times which doubles the cost.

However, when you have an abundance of something, your relationship with that something changes. In Parts 2 & 3, I will examine an emerging new model of relating to our next natural resource, data.