3 ms·
Please be aware there is also a newer concept, which is ELT. Extract Load Transform. This is the data lake principle: you firstly dump all data verbatim into o
by idiocrat 5y ago
Please be aware there is also a newer concept, which is ELT.
Extract Load Transform.
This is the data lake principle: you firstly dump all data verbatim into one coherent data storage. (data ingestion phase)
The transform will eventually be done by the data scientists (data curation phase).
The data curation is really difficult as it requires in-depth knowledge of the data, removing of slack, creation and lookup of reference data, data normalization for the datawarehouse schema, etc.
The data ingestion is relatively easy task, because the dirty data is just collected from upstream databases without any processing/conversion.
I believe data lakes is kind of "cheating" by IT departments.
- banku_brougham 5y ago1. Great explanation of ELT, for a long time ETL monkey this broke through for me. 2. GDPR/COPPA means you really have to think about your data lake before dumping verbatim into long term storage.
- jpgvm 5y agoUltimately regulations will change over time. I think the most important thing to keep in mind is storing data in a form that is amenable to large scale batch transformation. This way you can respond by kicking off a task to "rewrite the world" to scrub fields or drop records. Bug reason why I encourage stuff like Avro, Parquet or the even just loading to BigQuery.