5 ms·
> It is important that all data put in the lake should have a clear provenance in place and time. Every data item should have a clear trace to what system it ca
by saosebastiao 11y ago
> It is important that all data put in the lake should have a clear provenance in place and time. Every data item should have a clear trace to what system it came from and when the data was produced. The data lake thus contains a historical record. This might come from feeding Domain Events into the lake, a natural fit with Event Sourced systems. But it could also come from systems doing a regular dump of current state into the lake - an approach that's valuable when the source system doesn't have any temporal capabilities but you want a temporal analysis of its data. A consequence of this is that data put into the lake is immutable, an observation once stated cannot be removed (although it may be refuted later), you should also expect ContradictoryObservations.
> The data lake is schemaless, it's up to the source systems to decide what schema to use and for consumers to work out how to deal with the resulting chaos. Furthermore the source systems are free to change their inflow data schemas at will, and again the consumers have to cope. Obviously we prefer such changes to be as minimally disruptive as possible, but scientists prefer messy data to losing data.
So basically, a data lake is a fancy word for a file system?
- threeseed 11y agoNo. It is a combination of structured and unstructured data. It's just that the data is not in some consolidated or coherent schema that is useful for the business to use. Hadoop and NoSQL systems are critical for this role since often it is extremely time consuming to (a) design the final end state schema and (b) create the ETL processes to populate it. So the idea is to just fill the data lake as quickly as possible and then work out later how to use it. Data lakes are a concept that apply mainly to enterprises so we are talking about big data and complex, multi disciplinary/functional schemas.
- PaulHoule 11y agoA key thing is that an HDFS-like system has compute resources that scale with the storage, so the time to do a "full scan" of the system is constant with respect to the number of nodes.
- deleted 11y ago[deleted]
- slantyyz 11y ago>> So basically, a data lake is a fancy word for a file system? To me, it looks more like an ODS (Operational Data Store) with some data-mart like curation steps before it hits users.
- EwanToo 11y agoI'm not sure about that, to me an ODS has a schema applied to it to make the data fit into a relational database structure. A data lake should be storing the data in it's original form, without transformation.
- slantyyz 11y ago>> ODS has a schema applied to it to make the data fit into a relational database structure. Yes, I get that. I was probably oversimplifying. Practically speaking, the BI projects I've been on for small to medium enterprises (<150M in size, non-tech) still have predominantly relational data sources. In many of those cases, a warehouse is an expensive effort in over-engineering, but a data-lake-like ODS (without transformations in the main data tables) with curated subject-area marts is a more sensible solution. --edit, added "non-tech" to my qualifier
- temuze 11y ago> So basically, a data lake is a fancy word for a file system? Well, when you get down to it, any database is a just a file system with an API :) I think a "data lake" might be a combination of raw files with metadata. For example, raw scrapes on S3 + an entry on Postgres with a link to the file, the date it was scraped and the version of the bot that scraped it.