ABSTRACT: The Data Fabric and Data Mesh are complementary frameworks that can combine to create a unified data architecture. While the Data Mesh focuses on organizational structure and data ownership, the Data Fabric provides the technological foundation for integration and management. Together, they can offer a robust solution for seamless data sharing, fostering collaboration, and enhancing data-driven insights across domains.

Read time: 18 mins.

Originally published on the Promethium website in 2023

Part One: The Relationship Between the Data Fabric and Data Mesh

Two of the hottest topics in the last two years in the data management/analytics space are, without a doubt, the Data Fabric and the Data Mesh. There’s been a lot of confusion about what they are. Frameworks or products? Why do we need them? Which one will survive the infamous Gartner Hype Cycle? Well, let me try to make things a bit simpler for everyone in this series of blog posts. I realize the minute I said “series,” it probably didn’t sound simple. But bear with me as I believe this can be summarized in a non-self-promoting and direct manner so we can understand and figure out what we all need or want to do.

Definition: what is a Data Mesh?

The concept is based on four themes:

1. Decentralization
2. Domain-Oriented
3. Self-Service
4. Data Platform Scalability

Where do you usually apply a Data Mesh?

A Data Mesh is applied in large organizations with multiple departments using many large databases, data lakes, and data warehouses. Think complex ETL/ELT, with hundreds or thousands of data engineers/data scientists/and data analysts.

What’s the big concept the Data Mesh prompts?

Generally speaking, the Data Mesh promotes two major concepts--data products and domain-based ownership--which I’ll briefly discuss at a high level.

Data Products or Data as a Product. Consider this as data outputs such as datasets, queries, and models. They’re not too different from, say, a data mart or cube, but more for the modern world where these “products” are easier and faster to generate and/or refresh. Consequently, data products don’t require armies of people to refresh and are thus much more easily consumable.

Domain, Domain, and Domain. The Data Mesh emphasizes the importance of the notion of where/to whom the data belongs. Being isolated by domains, there is more agility in finding and accessing the data for self-service.

Now, what is a Data Fabric?

In short, the Data Fabric is a single product or framework previously known as the Modern Data Stack. There, I said it. The Modern Data Stack was a great idea in that it ushered in a new way to think about a decentralized or “best of breed” approach to architecting one’s data management environment.

However, after 2+ years, we’ve learned a painful and expensive lesson. In reality, only the biggest and richest companies can afford to buy and implement so many different products. Only these companies have hundreds or thousands of staff to handle the complexity of piecing together so many different products.

But, what if you could combine those modern approaches in…ready for it…one PRODUCT? Yes, one product.

That is the whole idea behind what the Data Fabric is. More specifically, the Data Fabric is an integrated data infrastructure that enables a seamless combination of:

• Data discovery
• Data access
• Data integration
• …and some AI or automation to help put it all together

The goal of the Data Fabric is to provide a unified and consistent view of data, simplifying data exploration, analysis, and collaboration for users.

That’s the product definition of a Data Fabric. In the framework definition, it’s a design that leverages various technologies and tools--such as data virtualization, data integration platforms, and data catalogs--to create a flexible, scalable, and adaptable data environment. In short, the Data Fabric, when defined as a framework, is that of a more tightly integrated and simpler Modern Data Stack.

Part Two: What Makes Up a Data Fabric vs. a Data Mesh, and Why It Matters

We’ve established what a Data Fabric and Data Mesh are at a high level. Now, let’s take a deeper look at how a Data Fabric and a Data Mesh are assembled. To do that, it’s important to first understand the components of both products.

The Data Mesh shifts the focus from centralized data lakes and data warehouses to a distributed architecture, where data is treated as a product and managed by cross-functional, domain-centric teams.

Components of a Data Mesh

The key components of a Data Mesh include:

• Data domains: Data domains are organized around specific business areas, functions, or product lines within the organization. Each domain has a dedicated team responsible for managing the associated data products.
• Data products: In a Data Mesh, data is treated as a product with a clear purpose, value, and usability for internal and external consumers. Each data domain is responsible for creating, maintaining, and evolving its data products, ensuring they meet quality, availability, and performance standards.
• Data product owners: Each data domain has a data product owner or data product manager responsible for the overall success of the data products within their domain. They ensure the data products are discoverable, accessible, and meet the needs of consumers.
• Self-serve data platform: A self-serve data platform provides tools, services, and infrastructure that enable domain teams to independently manage their data products. This includes capabilities for data ingestion, storage, processing, analytics, and data access.
• Data governance and compliance: Data Mesh enforces data governance, quality, and compliance standards across all domains. This includes setting up policies and processes for data cataloging, data lineage, data access control, and data security.
• Federated data catalog: A federated data catalog allows users to discover, understand, and access data products from various domains across the organization. The catalog should include metadata, data lineage information, and data quality metrics to help users make informed decisions about using the data.
• Cross-domain collaboration: Data Mesh encourages collaboration and knowledge-sharing between domain teams through regular meetings, workshops, and communication channels, enabling teams to learn from each other's experiences and identify opportunities for improvement.
• Data observability and monitoring: Data Mesh supports data observability and monitoring practices to ensure the health and performance of data products across all domains. This includes tracking data quality, freshness, and availability, as well as setting up alerts and notifications for any issues or anomalies.

Implementing these components and embracing the core principles of Data Mesh enables organizations to build a decentralized, domain-oriented, and self-serve data infrastructure that scales effectively and promotes collaboration and data sharing across the organization.

Components of a Data Fabric

Data Fabric is a framework, and sometimes, the productization of the Modern Data Stack consolidated into one product or integrated framework.

The core components and capabilities of a Data Fabric include:

• Data access and virtualization: Data Fabric allows users to access data from multiple sources, such as databases, data warehouses, data lakes, and APIs, as if they were a single, consolidated data source. Data virtualization techniques are often employed to create this unified view without the need for physical data movement or replication.
• Data integration: Data Fabric simplifies the process of integrating data from different sources by providing tools and services for data ingestion, transformation, and harmonization. This enables users to combine and analyze data from various sources more efficiently.
• Data governance and security: Data Fabric enforces data governance, security, and compliance policies across the organization. It provides mechanisms for data access control, data quality management, data lineage tracking, and data privacy enforcement.
• Data cataloging and discovery: Data Fabric includes a data catalog that captures metadata, data lineage, and data quality information for all data assets. This helps users to discover, understand, and access the right data for their needs.
• Scalability and adaptability: Data Fabric is designed to be scalable and adaptable to accommodate evolving data needs, technologies, and use cases. It supports the growing volume, variety, and velocity of data while being flexible enough to adapt to new data sources and integration requirements.
• Analytics and insights: Data Fabric provides a foundation for advanced analytics and insights by making it easier for users to access, integrate, and analyze data from various sources. This supports data-driven decision-making and innovation across the organization.

A Data Fabric is relevant to building a Data Mesh because it provides the underlying data management and integration framework that enables the Data Mesh's core principles to function effectively.

Here's how a Data Fabric contributes to building a Data Mesh:

• Decentralized data management: A Data Fabric enables seamless access, integration, and management of data across various sources and domains. This supports the Data Mesh's decentralized approach by allowing domain teams to work independently while still having access to data from other domains as needed.
• Domain-oriented architecture: Data Fabric's ability to provide a unified and consistent view of data across disparate sources supports the Data Mesh's domain-oriented architecture, allowing each domain team to focus on their specific data products while still having the ability to access and share data with other teams.
• Self-serve data platform: Data Fabric simplifies data access, integration, and management by abstracting the underlying complexities. This makes it easier for domain teams to access and work with data, promoting the self-serve data platform capabilities emphasized in the Data Mesh paradigm.
• Data discoverability: Data Fabric can help improve data discoverability through metadata management, data cataloging, and data lineage capabilities. These features are essential for a Data Mesh, as they enable domain teams to find, understand, and use data from other domains effectively.
• Data governance and compliance: Data Fabric provides a foundation for maintaining data governance and compliance within a Data Mesh. It supports data quality, security, and privacy, ensuring that data usage adheres to organizational policies and regulatory requirements.
• Scalability and flexibility: Data Fabric's scalable and flexible nature allows it to grow with the organization, supporting the evolving needs of a Data Mesh. It can accommodate new data sources, formats, and integration requirements, ensuring the data infrastructure remains agile and responsive.

By leveraging a Data Fabric, organizations can create a robust, scalable, and efficient data infrastructure that enables domain teams to work independently while benefiting from shared data resources and insights.

Complementary Paradigms

It’s important to note that Data Mesh and Data Fabric are not competing paradigms; instead, they complement each other. Both concepts address different aspects of data management and integration, and when combined, they can create a comprehensive and effective data infrastructure.

Data Mesh is a paradigm focusing on the organizational and architectural aspects of managing data at scale. It promotes decentralization, domain-oriented architecture, data as a product, and self-serve data platform capabilities. The main goal of Data Mesh is to enable better collaboration, data ownership, and data sharing across different domain teams in large-scale organizations.

Data Fabric, on the other hand, is a technological framework that addresses the challenges of data access, integration, and management across disparate sources. It provides a unified and consistent view of data, making it easier for users to access, analyze, and use data across the organization.

Data Mesh and Data Fabric are complementary concepts that, when combined, can create a robust and efficient data infrastructure that delivers several tangible benefits. Data Fabric enables seamless data access and integration across sources and domains that support the decentralized approach promoted by Data Mesh. In addition, the unified view Data Fabric provides across disparate sources aligns well with the domain-oriented Data Mesh architecture, enabling teams to share and access data from other domains while working independently. Data Fabric simplifies data access, integration, and management while supporting self-serve data platform capabilities emphasized in the Data Mesh paradigm. Finally, Data Fabric can enhance data discoverability, governance, and compliance within a Data Mesh by offering metadata management, cataloging, lineage, and quality features.

Complementary Concepts

In summary, Data Mesh and Data Fabric are complementary concepts that, when combined, can create a robust and efficient data infrastructure. As noted above, Data Mesh focuses on the organizational and architectural aspects, while Data Fabric provides the underlying technological framework for effective data management and integration. By leveraging both paradigms, organizations can build a comprehensive data infrastructure that promotes collaboration, data ownership, and efficient data sharing across domains.

Part Three: Building a Data Fabric or Data Mesh

How to Build a Data Fabric

Building a Data Fabric involves creating a unified and consistent data infrastructure that simplifies data access, integration, and management across disparate data sources. A Data Fabric provides seamless access to data, allowing users to easily explore, analyze, and gain insights from their data. Here are the steps to build a Data Fabric:

• Assess your data landscape: Understand your organization's data landscape, including the various data sources (databases, data lakes, data warehouses, APIs, etc.), data formats, and data storage systems. Identify data silos, data integration challenges, and any gaps in data governance and security.
• Define your Data Fabric goals: Establish the goals for your Data Fabric, such as improving data access, increasing data integration efficiency, enhancing data governance, or enabling real-time analytics. These goals will guide your Data Fabric design and implementation.
• Design your Data Fabric architecture: Create a high-level architecture for your Data Fabric that addresses data ingestion, storage, processing, integration, access, and governance. Your architecture should be flexible, scalable, and adaptable to handle evolving data needs and technologies.
• Choose the right technologies and tools: Evaluate and select the appropriate technologies and tools to build your Data Fabric. This may include data virtualization tools, data integration platforms, data catalogs, and data governance solutions. Consider factors such as ease of use, scalability, performance, and compatibility with your existing data infrastructure.
• Develop a data model: Design a unified data model that captures the structure, relationships, and semantics of your data across all data sources. This data model should serve as the foundation for data access and integration within your Data Fabric.
• Implement data ingestion and integration: Develop processes for ingesting and integrating data from various sources into your Data Fabric. This may involve using ETL (Extract, Transform, Load) processes, data pipelines, or data virtualization techniques to create a unified view of your data.
• Establish data governance and security: Implement data governance and security policies, processes, and controls within your Data Fabric. This includes defining data access controls, data quality rules, data lineage tracking, and data privacy policies.
• Create a data catalog: Develop a data catalog that provides metadata, data lineage, and data quality information for all data assets within your Data Fabric. This will help users discover, understand, and access data more effectively.
• Monitor and maintain your Data Fabric: Continuously monitor the performance, scalability, and data quality of your Data Fabric. Address any issues or bottlenecks and make improvements as needed to optimize your data infrastructure.
• Train and support users: Provide training and support to help users adopt and leverage your Data Fabric effectively. Encourage a data-driven culture and promote collaboration among users to maximize the value of your Data Fabric.

Building a Data Fabric requires a thoughtful approach to architecture, technology selection, data modeling, and governance. By following these steps and continuously iterating and improving your Data Fabric, you can create a unified and consistent data infrastructure that simplifies data access, integration, and management across your organization.

How to Build a Data Mesh

Building a Data Mesh involves rethinking your data architecture, organizational structure, and culture to enable a decentralized, domain-oriented, and self-serve data infrastructure. Implementing a Data Mesh requires following a set of principles and practices. Here's a step-by-step guide to help you build a Data Mesh:

• Identify and define data domains: Identify the various domains within your organization. Domains are typically organized around specific business areas, functions, or product lines. Define the scope and boundaries of each domain, and assign domain experts or teams to manage the associated data products.
• Treat data as a product: Shift the mindset from treating data as a byproduct of business processes to considering it a valuable product that serves internal and external users. Assign product owners or data product managers to each data domain, responsible for the quality, availability, and usability of the domain's data products.
• Establish a self-serve data platform: Develop or adopt a data platform that enables domain teams to discover, access, and use data from other domains independently. This self-serve platform should provide tools and services for data ingestion, storage, processing, and analytics, empowering domain teams to manage their data products effectively.
• Implement domain-oriented data architecture: Design and implement a data architecture that supports the decentralized nature of Data Mesh. Each domain should have its own data storage, processing, and access capabilities while adhering to organizational standards for data security, privacy, and compliance.
• Foster a data-sharing culture: Encourage a culture of data-sharing and collaboration across domains. This includes creating incentives for sharing data, fostering open communication, and providing mechanisms for discovering and accessing data products from other domains.
• Establish data governance and compliance standards: Define and enforce data governance, data quality, and compliance standards across all domains. This includes setting up policies and processes for data lineage, data cataloging, data access control, and data security.
• Adopt data observability and monitoring: Implement data observability and monitoring practices to ensure the health and performance of data products across all domains. This includes tracking data quality, data freshness, and data availability, as well as setting up alerts and notifications for any issues or anomalies.
• Implement a federated data catalog: Create a federated data catalog that allows users to discover, understand, and access data products from various domains. The catalog should include metadata, data lineage information, and data quality metrics to help users make informed decisions about using the data.
• Encourage cross-domain collaboration: Promote collaboration between domain teams through regular meetings, workshops, and knowledge-sharing sessions. This will help teams learn from each other's experiences, share best practices, and identify opportunities for improving the overall data infrastructure.
• Continuously iterate and improve: Building a Data Mesh is an ongoing process that requires continuous iteration and improvement. Monitor the effectiveness of your Data Mesh implementation, gather feedback from domain teams and users, and make necessary adjustments to optimize performance, usability, and collaboration.

By following these steps and embracing the core principles of Data Mesh, you can build a decentralized, domain-oriented, and self-serve data infrastructure that scales effectively and promotes collaboration and data sharing across your organization.

Part Four: What’s in a Name?

Having looked more closely at the components that make up a Data Fabric and Data Mesh, the first thing that may have jumped out is the significant overlap between the two. Here’s a quick recap. Both a Data Fabric and a Data Mesh require the following:

• Data Discoverability
• Data Access
• Self-Service
• Data Governance
• Security

The Data Fabric and Data Mesh: Complementary Approaches

So, maybe a Data Fabric is NOT a competing approach to a Data Mesh. What if a Data Fabric is actually complementary to a Data Mesh – implementing a Data Fabric first accelerates the implementation of a Data Mesh; should it be needed later on?

A Data Fabric actually supports building a Data Mesh because the Fabric provides the underlying data management and integration framework that enables the Data Mesh's core principles to function effectively. To recap what we discussed in Chapter 2, here's how a Data Fabric contributes to building a Data Mesh:

• Decentralized data management: A Data Fabric enables seamless access, integration, and management of data across various sources and domains. This supports the Data Mesh's decentralized approach by allowing domain teams to work independently while still having access to data from other domains as needed.
• Domain-oriented architecture: A Data Fabric's ability to provide a unified and consistent view of data across disparate sources supports the Data Mesh's domain-oriented architecture. It allows each domain team to focus on their specific data products while still having the ability to access and share data with other teams.
• Self-serve data platform: A Data Fabric simplifies data access, integration, and management by abstracting the underlying complexities. This makes it easier for domain teams to access and work with data, promoting the self-serve data platform capabilities emphasized in the Data Mesh paradigm.
• Data discoverability: A Data Fabric can help improve data discoverability through metadata management, data cataloging, and data lineage capabilities. These features are essential for a Data Mesh, as they enable domain teams to find, understand, and use data from other domains effectively.
• Data governance and compliance: A Data Fabric provides a foundation for maintaining data governance and compliance within a Data Mesh. It supports data quality, security, and privacy, ensuring that data usage adheres to organizational policies and regulatory requirements.
• Scalability and flexibility: A Data Fabric's scalable and flexible nature allows it to grow with the organization, supporting the evolving needs of a Data Mesh. It can accommodate new data sources, formats, and integration requirements, ensuring the data infrastructure remains agile and responsive.

In short, a Data Fabric is highly relevant to building a Data Mesh, as it provides the necessary data management and integration capabilities to support the Data Mesh's core principles. Leveraging a Data Fabric enables organizations to create and deploy a robust, scalable, highly efficient data infrastructure, enabling domain teams to work independently while still benefiting from shared data resources and insights.

Creating a comprehensive data infrastructure

Wait…so.. I can have both?

Yes.. but only IF you NEED both.

It’s important to reiterate that Data Mesh and Data Fabric are not competing paradigms but actually complement one another. While each addresses different aspects of data management and integration, combined, they can create a truly comprehensive and effective data infrastructure.

The main goal of Data Mesh is to enable better collaboration, data ownership, and data sharing across different domain teams in large-scale organizations. To achieve those goals, a Data Mesh focuses on the organizational and architectural aspects of managing data at scale, promoting decentralization, domain-oriented architecture, data as a product, and self-serve data platform capabilities.

On the other hand, a Data Fabric provides a unified and consistent view of data, making it easier for users to access, analyze, and use data across the organization. It focuses on addressing the challenges of data access, integration, and management across disparate sources.

When you implement a Data Fabric first, having a Data Mesh becomes instantly easier and a natural add-on. At the end of the day, the Data Fabric and Data Mesh can complement each other in the following ways:

• Decentralized data management: a Data Fabric enables seamless data access and integration across various sources and domains, supporting the decentralized approach promoted by the Data Mesh.
• Domain-oriented architecture: Data Fabric's ability to provide a unified view of data across disparate sources aligns well with Data Mesh's domain-oriented architecture, allowing domain teams to work independently while still having the ability to share and access data from other domains.
• Self-serve data platform: The simplification of data access, integration, and management provided by a Data Fabric supports the self-serve data platform capabilities emphasized in the Data Mesh paradigm.
• Data discoverability and governance: a Data Fabric can enhance data discoverability, governance, and compliance within a Data Mesh by offering metadata management, data cataloging, data lineage, and data quality features.

To conclude, the Data Mesh and Data Fabric are complementary concepts. As individual entities, they offer unique qualities, with the Data Mesh focusing on the organizational and architectural aspects, while the Data Fabric provides the underlying technological framework for effective data management and integration. Combined, they can create a robust and efficient data infrastructure. By leveraging both paradigms, organizations can build a comprehensive data infrastructure that promotes collaboration, data ownership, and efficient data sharing across domains.