Data Product Oriented Architectures

Design your data domains to represent your business but don’t blindly copy&paste the org-chart

In his talk “Data Domains and a Federated Data Governance Model”, Jan Mark Pleijsant (ABN AMRO) guided us through how he designed the data domains for a large bank. A few take-aways:

1. It’s hard to align 300 data owners spread out over different business units, each with their own operational responsibilities besides being a data owner. It’s even harder to get them to care about data quality, documentation and lineage. That’s why they decided to move to a model with dedicated data domains, with people full-time dedicated to working on the data products for each domain.

2. “If you have to describe your organisation in 10–15 terms, what would they be?” Those terms are a good first shot at finding your data domains. For ABN AMRO, those terms were things like “Customer&Party”, “Employee&Organisation”, “Mortgages” and “Markets”

3. Start with principles to steer and validate design choices: For them, the principles were documented in the following slide:

   
   

   
   

Principles for designing data domains

A dataset ≠ a data asset ≠ a data product

Oh boy, as an industry we can’t align on what data products are. Everyone has their own definition. But if one thing is clear, it’s that it’s not just another name for a dataset or a data asset. In their talk on “Data Products in Industry Practice”, Christoph Gröger and Rainer Metje (Bosch) discussed how they differentiate between them.

Data products according to Bosch

What I like about this structure, is that Bosch puts the concept of a marketplace at the data product level. You don’t go shopping for individual datasets, or even data assets. You shop for data products, which is, by definition interoperable, something with definite business value, and consumed through a specific technical and legal framework.

Obligatory Ikea analogy of a data product and a marketplace :-)

They made an obligatory Ikea analogy. A chair is a data product. It consits out of raw data, but it includes metadata, semantics and templates. You can find data products in the data marketplace (Ikea stores). And you, as a consumer, might have different use cases for them.

Do you even MCP bro?

MCP is hot. We all jumped on genAI through RAG architectures. But with MCP, there is no need to extract data, store them in vector stores and then ask questions. You can just ask the question directly to the source system. That’s exactly what Simon Harrer did with data products. His idea behind this, is:

Enable AI agents to find and access any data product for semantic business context while enforcing data governance policies.

And his architecture looks like this:

Data Product MCP architecture

The demo he showed was really nice. Using Claude on your desktop, you could ask any business question to the MCP server. The server would then retrieve the right metadata and policies from the data marketplace, in this case, the Data Mesh Manager, in order to send the right query to the data platform, if you are allowed to run that query.

The powerful thing about this setup, is that it uses all the context that’s already present in your data marketplace and applies it in an automated way to make the life of your business analysts earlier. Of course, it’s still early prototyping, but this is the kind of “Acceleration with AI” that goes beyond the buzzwords and shows actual promises.

You can read all about it here: https://dataproduct-mcp.com/

Data products in practice with our Data Product Portal

Last but not least, this blog cannot be complete without mentioning our very own Jonny Daenen presenting how we are guiding customers in the Data Product landscape.

Jonny on stage

Jonny made the analogy with early Windows systems, where it was hard to install and manage apps, dealing with drivers and whatnot. Until Apple came along and introduced the concept of a marketplace: A simple interface to discover and install apps.

The analogy with data holds true because we’re still in the early Windows days where it’s very hard to discover and start using data products. Instead, you’re stuck in technical catalog pages, staring at terraform scripts or dealing with python code.

Jonny demoed how we solved this at a client using our own open-source Data Product Portal, which is a marketplace for data products.

Screenshot of the Data Product Portal showing the interaction between data domains and data products

What’s important in any marketplace, is that you get a good view on value, cost and quality of every data product:

• Although, the “value” bit is hard to measure, you should always be able to link a data product to some kind of business value.

• In terms of costs, you can track both cloud and engineering costs.

• Quality finally, is defined in terms of “Readiness”. How well is this data product following the standards you set out in an organisation?

• 
  

Screenshot of the Data Product Portal highlighting value, cost and quality.

Jonny showed more possibilities, like automating the underlying infrastructure to create the right projects, CICD pipelines and permissions, once a data product is being created. We feel this engineering part was overlooked a bit at the conference. Like this is magic that solves itself.

In reality, data platform building is a lot of hard work that goes on behind the scenes. But it’s needed to give the users of your solution a smooth experience.