3 ms·

Not that I know what anything means in "big data lake OLAP database" anymore, but I always thought a data lake implied a lot of hybrid sources/formats/structure

Not that I know what anything means in "big data lake OLAP database" anymore, but I always thought a data lake implied a lot of hybrid sources/formats/structures for the data, but the advocacy here implies that the data is all ingested and reformatted, which to me is a data warehouse. But then again, data lake may simply be what a data warehouse is now called in marketspeak. Also, I stopped paying attention when the treadmill of new frameworks became unbearable to track, is spark now settled as the standard of distributed "processing", as in mapreduce / distributed query / distributed batch / etc? I get that performance can improve by unifying to a file format like parquet, but again that seems like a data warehouse. A data lake should be something over heterogenous sources with "drivers" or "adaptors" IMO, in particular because the restoration of the data inputs stays in the knowledge domain of the source production database maintainers.

MrPowers 4y ago
You're understandably confused by the industry terminology that's ambiguous and morphing over time. Data lakes are typically CSV/JSON/ORC/Avro/Parquet files stored in a storage system (cloud storage like AWS S3 or HDFS). Data lakes are schema on read (the query engine gets the schema when reading the data). A data warehouse is something like Redshift that bundles storage and compute. You have to buy the storage and compute as a single package. Data warehouses are schema on write. The schema is defined when the table is created. And yes, I'd say that Spark is generally considered the "standard" distributed data processing engine these days although there are alternatives.