Technologies to deploy now:

In this third and final installment of my series (check out Part 1 and Part 2), I would like to discuss how the value of data demands that we capture its value in environments we control. “Let no data escape” must be the mantra of our systems development. However, we don’t need to store all data forever.  We don’t even need to store every piece of data, but we do need to glean every possible value out of every data element possible.

For example, high-volume data can be used in a streaming sense of determining if it is useful to real-time applications of next-best-offer, fraud detection, account verification, etc.  It doesn’t have to be stored anywhere. However, the analytic value of the transaction could be pulled into the profiles in master data management and the data could move on to the data warehouse for long-term storage for reporting.

Data can also be interesting from the third-party data marketplace.  It is well past time to think of data as an asset and think about what data you could use to advantage. Chances are that data is available. Is your team?

This mantra implies that we must grow the data science of our organization to deal with the many and varied forms of data. While everyone will not be a degreed data scientist, the individuals in the organization that can deal with the greatest amount of information will be most successful. Gone are the days when a valued job gets the “data drop” monthly and proceeds without new information for a month or longer. Gone are the days when a valued job deals with only one type of information.

Those jobs exist, but they are being devalued. The degree that one can capitalize on the next natural resource of information is the degree to which one will be valued in the information revolution that is upon us.

All of this cannot be accomplished without an intense focus on the many and growing technical bases that can be used to store, view and manage data.  There are many now, more than ever, that have merit in organizations today, which is why I advocate companies have a Chief Data Architect, or similar, position to govern the introduction of new data technologies.

The vendor market has kept up.  As these systems continue to double their price-performance, bandwidth and storage capabilities annually, all things become possible.

Technologies to deploy now:

• Hadoop/Spark Ecosystem – This ecosystem will evolve, but the foundation of scale-out file systems without overhead and moving towards stronger non-functionals, will not change.
• Master Data Management – Despite the intense resistance to sharing that these projects create, efficiently collecting or generating data to share in small and large ways is essential to the bottom line; the generation capabilities of MDM are increasingly being required.
• Internet of Things – Though not a technology, a consideration of using the internet as the processing backbone of new applications is increasingly compelling.
• Cloud – It’s hard to imagine just “cloud” as being a category, but at least starting there, it is a major disruptive force to IT-as-we-know-it.
• NoSQL – Perhaps the moniker will morph again, this time away from “not only SQL” to something that doesn’t imply its origination as the antithesis of a programming language; anyway, online digital strategies simply need to process too much information for any other operational approach.

We are in the data economy and it is the next natural resource.  Unlike other natural resources, every business must have a relationship with this one.  Also unlike other natural resources, it is not entirely evident what that relationship needs to be.  We need to figure it out in earnest.  And I hope we keep the balance in favor of the human experience. 

We will not run out of data, but we may be overwhelmed by it.