Big Data Fuels a ‘Data First’ Movement

My esteemed friend and colleague Tom Davenport recently wrote in these pages about The Shift to aNew Data Architecture. With his usual instinct for wry observation Tom noted, “so this architectura revolution won’t be televised, but it will be revolutionary.” And, there will be bodies. To quote the French revolutionary Robespierre, “The king must die so that the country can live!”

Having spent many years as a first hand practitioner on both the technology and the business side of the data fence, I can attest to how difficult it is to manage data well, and how frustrating it sometimes can be to be dependent upon the IT organization and the data warehouse for access. Can’t we just do it ourselves? Give us the data!

Well, things are changing, and changing fast. As Mr. Davenport notes, new data architectures like Hadoop represent “a classic disruptive innovation.” The question facing many organizations today is how to integrate newly developed Big Data architectural approaches into traditional legacy data environments. Business analysts have long clamored for first hand access and control over the data they need to do their jobs. The data warehouse has represented an institutional compromise, providing a centralized repository of standardized data.

While proponents of the data warehouse approach have pointed to the benefits of standardization, consistency, and control, critics lament that data warehouses can be slow, costly, and lacking in agility inresponse to change. This becomes more acute in rapidly changing environments or dynamic markets. With the growing and accelerating proliferation of data, some business users have begun to sound a drumbeat as they demand a more flexible and dynamic data environment that is responsive to their needs – and by extension, the needs of their end customers.

Many of my colleagues in industry are enthusiastically embracing emerging Big Data architectures as the first data architecture approaches that deliver the speed and agility that businesses need to compete in a digital era. Within the data management community the debate is on, with defenders of the old and proponents of the new engaged in a dialogue aimed at winning the hearts and minds of the mainstream.  Out of this debate has emerged the “Data First” movement, which has fashioned its own manifesto and call-to-action based on a set of core principles:

Businesses must have greater control over their data assets. The argument is that in the same way that the Internet has driven end-customer self-service, Big Data can drive business analyst self service. The times demand it. Victory goes to the fast and nimble.

Data discovery must be encouraged, and not penalized. Many of the most innovative firms have prospered by their ability to develop new products and services quickly, and validate them in the market. They have developed test-and-learn models which enable rapid analysis. In the past I have discussed the ability of leading edge firms to develop learning practices that enables them to “fail fast” and adapt quickly.

Data efforts must move toward decentralization of control. The pendulum swings between the benefits of centralization and organizational control, and decentralization and unit autonomy. New approaches aim to enable greater responsiveness to data discovery while ensuring lightweight data governance standards to maintain data integrity at a corporate level.

Inexpensive data storage and processing power have liberated data. Data can now be produced liberally and cost-effectively. Each data user is able to house and manage their own data environment. Data needs can be driven on-demand in the context of what information is required in the moment. Moving control of data to the business user means moving decision making closer to the customer.

As customers demand faster responsiveness to their needs and desires, pressure will mount on companies to find new ways to enable data discovery so that businesses can create the analytics they need at the pace they require. Brute-force processing power and the demand for agility will drive change. The data itself will dictate what information and patterns are relevant. Big Data gives us the power to learn from the data, and hypothesize based on actual results. Some will argue that we had forgotten about the data – and now, the data comes first, as it should be.

There will be honest debate and there will be resistance. So, in times like these, it’s worth recalling the legendary Apple marketing campaign, “Think Different” — “Here’s to the crazy ones. The misfits. The rebels. The troublemakers. The ones who see things differently. They’re not fond of rules. They change things.”

This article first appeared in The Wall Street Journal on June 15, 2015.

Barry Devlin

Dr. Barry Devlin is among the foremost authorities on business insight and one of the founders of data warehousing, having published the first architectural paper on the topic in 1988....

More About Barry Devlin