3 ms·
IMHO, it’s an antipattern in almost every way. Being “common” doesn’t make it “smart”. The preprocessing argument is invalid for a lake, and the upfront “integr
by crsn 7y ago
IMHO, it’s an antipattern in almost every way. Being “common” doesn’t make it “smart”. The preprocessing argument is invalid for a lake, and the upfront “integration” time savings are cannibalized later by maintenance and risk overhead vs. proper service integrations. And everything you mentioned aside, there’s security and DPP risk. Lakes should not be used for collaboration between systems - that’s what lakeshore data marts are for, at worst, or real service-to-service APIs, ideally. (We have a good word for this in German: Datensparsamkeit. See also https://martinfowler.com/bliki/Datensparsamkeit.html https://martinfowler.com/bliki/Datensparsamkeit.html). The people and services regularly using the lake should be data scientists and analytics folks, probably not prod services.
- kyllo 7y agoSecurity and privacy risk are also good things to consider, thanks. Our data lake does have processes and protections in place for this (GDPR tagging and deleting, retention policies, security groups for access control) but just because my streams are compliant doesn't mean downstream consumers are.