3 ms·
Data Warehouse is usually a relational database designed for large OLAP analysis with features like column-oriented storage, vectorized processing, and distribu
by manigandham 6y ago
Data Warehouse is usually a relational database designed for large OLAP analysis with features like column-oriented storage, vectorized processing, and distributed scale-out architecture. Since it's a database, the focus is on strong schemas and structured data, although all major systems also support JSON datatypes now.
Data Lake is usually object storage or other large storage pool with raw files. These can be different formats like JSON, AVRO, Parquet containing with strong schemas or unstructured data. Processing can be done by engines like Spark, Presto, Drill, etc that support less advanced SQL but more robust access across data files and storage locations. The point is to serve as a general dumping ground or "lake" of all the data and then manage it afterwards (including cleaning and moving important records to a data warehouse).
SQL Server is a single-node OLTP relational database but most database engines are fast enough now that you can do everything you need up to hundreds of millions of rows. Best SQL and feature support with full update capabilities. Some DBs like SQL Server have also added OLAP features like columnstore tables to further delay or eliminate the need for a data warehouse.
- sails 6y agoGreat answer. On Data Lakes: I often use an S3 data lake construct as a staging area for my Snowflake data warehouse.