Automated metadata-based detection of the data product consumers in data mesh

Abstract

Object is distributed data mesh. The article reviews main types of data product consumers. There are a lot of them depending on technology stack, architecture, components, purpose, etc. At the same time, it’s critical to keep inventory of all consumers of data products in distributed data mesh platform as it allows to notify them about changes in data products, collect and keep requirements for the data product during its lifecycle. The aim is to collect list of data products consumers automatically. It allows to reduce manual effort which will decrease operational cost of data mesh support. To achieve the goal a couple of automated methods for detecting data product consumers were analyzed, in particular, API-based method, event-based method, metadata-based method. After deep investigation of mentioned methods, it was discovered that API-based and event-based methods require significant changes of data mesh platform architecture by adding new software components. At the same time, metadata-based method doesn’t have such limitations and provide acceptable results. A couple of approaches of adopting metadata-based method of data product consumers identification were proposed for different technologies and components. Ref. 7, pic. 4

Keywords

• artificial intelligence
• data mesh
• data product
• data lineage