Data Mesh - simply explained

Do you want to optimize your business processes with Data Mesh? We are going to show you the possibilities.
Contact
Let’s discuss how we can
support you and solve challenges together.

Is data mesh the right approach for organizing a data platform?

Is a company viable if it does not master the handling of data? The answer is no. Today, no one can imagine a modern company without data businesses. Only those who analyze data and gain insights earlier and more successfully than competitors can hold their own in the market. But in order to successfully use the business with data, a platform for data collection and storage must  be built first. Which data platform is the right one? A new and modern approach for organizing a data platform is the so-called data mesh architecture. In this article, you will learn everything you need to know about data mesh and what benefits this approach offers your company.

Data Mesh and the Basics

Traditional and historically grown database structures are usually complex and slow to update. The Data Mesh concept remedies these shortcomings by moving from a centralized to a decentralized approach.The term data mesh was coined by Zhamak Dehghani, director of technology at the IT consulting company ThoughtWorks, and became popular around 2019. It is a new data architecture approach that helps organizations develop a fourth-generation data platform and make data use more efficient. The new data model aims to create a decentralized, cross-domain data architecture with centralized management and common standards for data integration, as well as a centralized infrastructure with self-service functions. A data mesh architecture is based on three core elements: decentralized data storage, distribution of data, and data management.

Four fundamental principles of Data Mesh

Traditionally, data is converted from various sources and uploaded to a central repository for further analysis. This often leads to errors, largely due to the instability of the data. The availability, reliability, and trustworthiness of the data in this architecture is determined by the central network node, not by the network components that generate and use it. Therefore, multiple sources of information can enter the analysis process. This in turn leads to a lack of trust in the results of the data analysis. The data mesh topology is designed to eliminate these problems. An architecture conforming to the data mesh requirements generally four basic principles.

Domain ownership

The business units within a company are the responsible data owners. They generate data for their own use and make it available to other units.

Principles of Data Mesh

Data as a product

The business units within a company are the responsible data owners. They generate data for their own use and make it available to other units.

Self-service data platform

The business units within a company are the responsible data owners. They generate data for their own use and make it available to other units.

Federated computational governance

The business units within a company are the responsible data owners. They generate data for their own use and make it available to other units.

Data Mesh benefits and required data organization

The main difference between the data mesh architecture and previous generations of data storage platforms is that data is not just transferred exclusively to the data lake. Rather, the data sets are shared among business domains. The rule is that whoever creates and owns the data knows its data best. Thus, data is stored and used by teams within the business divisions in a way that is appropriate for them. Data Mesh thus enables the connection of data stored in silos. Different data consumers within the company are enabled to perform large-scale automated data analysis. The distributed data architecture reduces the load on the network and reduces data access bottlenecks and costs for storage systems. Other goals of decentralized data storage and connection of cross-domain data are: discoverable, interoperable, secure and reliable data products from the individual teams.

However, for such an architecture to work effectively, some conditions for the data must be fulfilled in advance:

The benefits of data management in a data mesh architecture

The advantages of Data Mesh are obvious. Although data lake technology is no longer at the heart of the data mesh architecture, all existing data lake technologies and tools can still be used. Other benefits include:
Data democratization
Simplified implementation of self-service applications when using multiple data sources, and improved data accessibility
Cost efficiency
Scalable and flexible costs through domain-based use of cloud storage
Reduction of technical debt
Cost reduction through less complexity with a decentralized data structure
Interoperability
Simplified API access to data and implementation of tools thanks to data consistency across domains
Data security and compliance
Promoting stronger governance methods through access controls and domain-independent data for sensitive data
These conditions are partly associated with a high effort, so it must be weighed up in the individual case whether the introduction of a data mesh is worthwhile for the company.

Fields of application of Data Mesh

It takes a lot of effort for companies to switch to a data mesh architecture, so it must be weighed up in the individual case whether the introduction of a data mesh is worthwhile for the company. For the transformation process to work, it is important to take all business processes seriously. Only then the principles of a domain-oriented and decentralized architecture of data can be integrated into a company in a way that brings success. Data mesh is becoming increasingly popular. Already several companies in the German market such as Roche, Zalando and Adidas have moved to a data mesh architecture for data storage and use. Data Mesh architecture is especially in demand among companies with high business and sector growth. Good data architecture examples include representatives of large retailers that have either already converted to this architecture or are initiating the transition process. New branches and lines of business are thus quickly incorporated into the data processing and preparation process. However, companies from the insurance and FinTech industries are also notable data architecture examples that have recently decided to introduce data models based on data mesh principles.

Summary

Data Mesh has heralded a paradigm shift in Big Data management. The motto here is: move away from centralized, monolithic data architecture to decentralization of data! The data platform organized according to these principles takes into account the weaknesses of proprietary data storage and of the use of data lake as a panacea. The data mesh architecture makes extensive use of streaming technologies, stores data in the cloud, and combines batch and streaming processing. In this way, companies can analyze data in real time while reducing data infrastructure management costs.Whether this approach is suitable for building a data platform is ultimately something each company must decide for itself. Especially for small companies and start-ups which do not yet have a complex, evolved data structure, this approach may bring more hurdles than benefits.
Want to know more?

Featured posts.

Creation of a data set catalog: for each data set, the requisite meta-information should be available to enable the data to be found quickly.
Each data set is provided with a unique address so that program-controlled access is possible.
Verifying and ensuring that data is valid and up to date.
Describing the semantics and syntax of data to create easily usable data sets.
Establishment of guidelines and standards for efficient data integration across different domains.
Ensuring secure access to the data.