This informative piece delves into the complexities of effective data governance and explores strategies to overcome common challenges. Readers will discover how to streamline approvals, bridge data gaps, and adopt a proactive shift-left approach to achieve effective data management. …

A Data Mesh implementation (like every significant transformation) should survive for a minimum of 10–15 years, possibly even more. Are you planning for that? During the past months, while Data Mesh is becoming increasingly popular, I have seen many people and companies sharing their architecture to implement the Data Mesh…

When it comes to modelling Data Products people go into panic. There are no clear rules, only some conceptual and not actionable indications. Below we’ll go through some rule of thumb we can follow when modelling data products. Are aggregate data domains a good practice? To distribute the ownership of data modelling to multiple data teams, we…

The race to become a data-driven company is increasingly heated, challenging the data engineering practice foundations. Let’s look at the Data Mesh phenomenon, which proposes a socio-technical approach to decentralize the data ownership to the business domains. The “quick buy-in” that Data Mesh principles obtain from business executives demonstrates that…

I think it’s time to freshen up the world’s perception of Data Engineering. This article has the dual purpose of raising awareness and acting as a manifesto for all the people at Agile Lab along with our past, present, and future customers. In the collective imagination, the Data Engineer is…

We all know that Data Mesh is inspired by DDD principles ( by Eric Evans), transposing them from the operational plane ( microservice architecture ) to the analytical one ( data mesh ). Since one year and a half, we have been helping many big enterprises to adopt the data…

What is Data Mesh? Data mesh is not a technology, but it is a practice…a really complex one. It is a new way to look at data, data engineering, and all the processes involved with that, i.e. Data Governance, Data Quality, auditing, Data Warehousing, and Machine Learning. It’s completely changing…

Who of you has right now the reasonable certainty that all your Spark jobs are performing at their maximum without wasting more computational resources than necessary? If so, what information are you basing it on? Would you be able to quantify the waste of resources in economic terms? …

Data Lake and Data Warehouse in real time and low cost “A data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale” After two years of apparently no innovation in the field of big data technologies, now a new wave…