3 ms·
You're understandably confused by the industry terminology that's ambiguous and morphing over time. Data lakes are typically CSV/JSON/ORC/Avro/Parquet files st
by MrPowers 4y ago
You're understandably confused by the industry terminology that's ambiguous and morphing over time.
Data lakes are typically CSV/JSON/ORC/Avro/Parquet files stored in a storage system (cloud storage like AWS S3 or HDFS). Data lakes are schema on read (the query engine gets the schema when reading the data).
A data warehouse is something like Redshift that bundles storage and compute. You have to buy the storage and compute as a single package. Data warehouses are schema on write. The schema is defined when the table is created.
And yes, I'd say that Spark is generally considered the "standard" distributed data processing engine these days although there are alternatives.