MDM has been broken for a long time.

Do you have an MDM project at your company? Is it the first time you’ve tried to create a golden record? Or have you tried and failed several times already? Most of you probably have tried and failed or have an MDM solution that you are dissatisfied with and are having trouble getting the folks at your company to comply with.

When I work with clients I often hear the following: “I don’t know that my customer in my Salesforce database is the same as my customer in my Client Services database.” This is the same problem that I heard in 1995, in 2005, in 2015 and now.

This problem was supposed to have been solved decades ago with a variety of solutions with MDM being one of the latest. Why hasn’t MDM solved it yet?  

Maybe classical MDM doesn’t work as well as we thought?  Maybe it suffers from many of the same problems as does the classical data warehouse?

Last week I had the pleasure of speaking with two big thinkers within the MDM space: Salah Kamel and Michael Hiskey of Semarchy. They had reached out to me because of my article:  “The Demise of the Data Warehouse”. They agreed that the requirements of flexibility and time to value outweighed the importance of having an all-knowing monolithic data warehouse. Salah and Michael’s vision has been to develop MDM tools that allow for rapid construction of the essential promises of the data warehouse by using MDM and data virtualization technologies.

Got a technology problem? Get a piece of paper or a whiteboard.

To look at the roots of the real problems plaguing MDM let me use a story as an analogy.

When I was an undergraduate at MIT and I took my first programming course (in LISP), our professor, Hal Abelson, warned us:

“Go home and write your program with paper and pencil first. Don’t sit down at the computer until you’ve designed your program on paper.”

This sounded crazy and old fashioned and most of us ignored his advice. Instead of logically thinking through a strategy for what we wanted to build, we sat down directly at the computer and began writing code and then tinkering endlessly to try to get it to work.

But in the long run, our ‘code first’ mentality made it take much longer to complete the program and it certainly wasn’t as elegantly designed. Professor Abelson was right.

Now this was in the 1980s before the dawn of time and computers only lived in the computer lab not in your backpack. But, the point was the same as it is today: to be successful you need to figure out what you want to do strategically (big picture) before you sit down to tactically implement.

I had been trying to solve a complicated problem that was somewhat ill defined by directly using technology. I didn’t stop to think and use old technology like paper and pencil to think things through before I started building. This “technology will solve all of our problems” mentality is one of the major issues with classical MDM methodologies.

Some requisite blasphemy about data

In master data management, fundamentally, your data problems are not technology problems. They are not even MDM problems. Your data problems aren’t even really well … data problems. They are business problems. They are the problem of getting four business people, three data stewards and several application managers into a room to formally agree on what revenue is for a customer record stored in the sales, marketing, ERP, and finance systems.

MDM problems are about getting the right people educated, motivated and in agreement. And this can be messy and difficult. (Side note: if you think this sounds like Data Governance you’re right – more on that in another post, soon.)

When you succeed with MDM you succeed by working from the business down. When you fail you fail because you design and implement something around a technology first and then you ‘release’ your master data solution to various practitioners around your company and expect them to comply.

Like my peers in my freshman programming course we race to implement without spending enough time planning, negotiating and understanding.

Four Ways to Fail at MDM

1. It’s the business, stupid. Today with our amazingly fast software development and prototyping tools I nonetheless still see too many data architects, stewards, scientist, and analysts rushing to code something before they’ve thought it out. They need to be in meetings first – productive meetings. In the immortal words of James Carville: “it’s the business, stupid…” (ok maybe Carville mentioned something about the economy not MDM but the point is the same).

The point of Carville’s phrase was that you need to focus on the most important part of the problem and don’t worry about all the other stuff. So here are some directional but completely made up numbers that show the difference between where you should spend your time on MDM and where you think you should:

• Where you think you need to spend time on MDM:
  • 10% finding, evaluating, installing awesome new MDM software
  • 10% recruiting, hiring and training a specialized team on the new software to build the MDM / golden record
  • 80% hunkering down and implementing it
• Where you should be spending 99% of your effort:
  • Every morning at 9am in a 15 minute standup with folks from around your company
  • Presenting to owners of different data sources the value of MDM, listening to their issues
  • On the phone and email tracking down the real meaning of that field that you are working on this week.

2. You try to solve all the problems at once. Trying to do too much all at once is the bane of the classical data warehouse. It is a similar problem for MDM. It results in projects that never get finished and project leads looking for their next project as they put together PowerPoints claiming victory.

3. You sacrifice the good for the perfect.  You may find that as you work together with your MDM committee that the complexity and special cases of your solution start to rapidly increase. Before this happens you need to check in your solution and deploy it so you have something that works pretty well. If you then want to improve it you can but if the improvement never happens then you still have something functional. Perhaps we can call this “pretty good MDM” that results in a silver not golden record.

4. Your data governance is dictatorial not collaborative. Before founding Semarchy Salah Kamel created and sold his previous data integration company to Oracle. He has seen many of the problems that big companies face in deploying database solutions and in particular MDM. One of his biggest insights is that too often data governance is dictatorial and top down rather than collaborative. This causes two problems:

• The people who know the most about the data and how it is used are not part of the team. They are not able to contribute key ideas that would have improved the data governance.
• Since these same people were not part of the process that defined the rules of MDM they do not fully understand them. They do not understand why they need to be implemented. This results in unnecessary friction and incorrect implementations as you start to deploy your MDM system.

Four Ways to Win at MDM

1. Focus on building understanding not on “MDM”. Focus on building understanding not implementing MDM. Look for tools that give you insights into your own data and help you to more quickly define master data that everyone will agree on and support.

2. Start small and grow. Hunt down a KPI or key metric every week or so. You will find that most companies are driven by a very limited set of key metrics and reports. The reason for this is that even though IT and reporting software can deliver near infinite numbers of numbers, the human brain can only process and remember a limited amount. This is especially true if senior management is only looking at those numbers monthly or quarterly. You’d be surprised how many companies flounder amongst a sea of metrics but lose track of two important ones: revenue and profit.

3. Look for tools that reduce complexity. Look for tools that reduce complexity not increase it. A surprisingly small set of rules and easy to use tools can provide an enormous amount of value. There are existing tools that are ready for massive adoption by the business with simple user interfaces and built-in guided collaboration workflows, Spend some effort researching the right tool that supports collaboration and reduces complexity.

4. Look for tools that can track MDM over time.  One of the interesting features I heard about from Semarchy is the ability to roll back MDM and see how decisions were made over time. The more I thought about it the more it seemed like this would be an important feature (similar to Domino Data Lab’s reproducibility engine for machine learning).

Four Years from Now

Thirteen years ago (nearly to the day) Bill Inmon, our data warehousing godfather, understood the problems of implementing a single version of the truth. He was talking about the classical data warehouse but his point is valid for MDM as well (see Inmon’s full article here):

There are a surprisingly small set of rules for creating an environment of data integrity across a large and complex environment. Implementation of these rules, however, requires organizational discipline and the attitude of all organizational units working in harmony with other organizational units, which is easier said than done. – Bill Inmon

Hmm… “easier said than done” … this might be the data warehousing understatement of the last decade and a half. The good news is that recognizing that you have a problem is the first step towards fixing that problem. This problem is in the business process not the technology.

Here are some vague but perhaps intriguing predictions. In four years, we might see:

1. Agile MDM. A well-defined agile-like or Six Sigma-like process for managing the business processes required to create a strong MDM. The breakthroughs in MDM will come from the organizational management strategies that surround the technology and tools that support them.

2. Dictatorial MDM. We will probably still see centralized, strong-armed, authoritarian attempts at data governance and MDM implementation. This is life, get over it. As the famous physicist Max Planck once stated: “A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather its opponents eventually die, and a new generation grows up that is familiar with it.”  Collaboratively designed MDM will succeed in the end but it may take more than four years.

3. Collaborative MDM. Some companies will have solved the golden record problem with nimble, lightweight solutions that are designed from the ground up to be collaborative. If the success of open source software has proved nothing else, it has shown us that collaboration results in rapid innovation for very complex problems. Those companies that have utilized this approach of collaborative MDM will have distinct market advantage over those that don’t.

4. DL+MDM. Though there are flaws in both MDM and the data lake, I still stand by my forecast of the demise of the classical data warehouse. But of course, what is really wrong with the classical data warehouse (and classical MDM) is not the technology but the philosophy and the process. Data warehouses and data marts that are successfully built will be important input sources that just happen to have tremendously high speed for certain types of queries. In four years data lakes, data warehouses and data marts will just be one part of a federated or virtual database landscape that is held together by the fabric of powerful MDM.

Further Reading:

• The Demise of the Data Warehouse  
• IT Activation Energy: Cloud Considerations in Retail and Insurance
• Cloud Data Warehousing: Producing the Infrastructureless Culture
• 17 Things a CDO Should Know: A Report from MIT’s CDO-IQ Symposium  
• Deep Learning, AI and Privacy
• Enterprise-Grade Data Science
• For Best Predictive Analytics Results: Don’t Move the Data