My company uses a common data lake for various teams and organizations to share their datasets. We enforce schemes, and owners are responsible for the correctne

My company uses a common data lake for various teams and organizations to share their datasets. We enforce schemes, and owners are responsible for the correctness and quality of their data, although some consumers usually build some tests in their downstream pipelines to validate data sets from the data lake. It has its problems. How do I know a data set I am consuming is complete or correct? As a consumer, how do I even know what to test for? Although producers are technically on the hook for the quality of their data, it is not clear which data sets in the data lake have what level of support from the owning team. You probably want to get in writing from the producer that it’s safe to consume or take a dependency on their data. Personally, I don’t want to vend a service to provide data sets, especially when they’re in hundreds of gigabytes or terabyte scale (for analytical use cases). I don’t want to support all that infrastructure and have engineers supporting he service logic itself, and so a common abstraction makes sense, at least on paper. One of my recent projects involved using this data lake as an integration point. The specific data lake implementation means that for consumers, the exact time that a dataset (after it is written to the data lake) is available for querying is non deterministic, and that resulted in a lot of friction. What is the size of the logs you want to consume? Are you able to query specific log lines by wildcard or key? How fast do you need the data once it has been produced? Are you willing to support your own infrastructure (hosts, configuration, patching, etc)? If there is a datalake, is it a common/managed service, and so is the cost of supporting your own service worth it? Does your data lake enforce schemes on write? If so, how easy is it to change schemes later on, and how does that impact consumers? How do you identify or track consumers of the data, whether you’re using a data lake or a service? Is there support for versioning?