3 ms·
"Data Lakehouse" is sadly term ruined by AWS. It used to mean "data lake extended to support data warehouse use cases". So something like HDFS or S3 with Delt
by glogla 5y ago
"Data Lakehouse" is sadly term ruined by AWS.
It used to mean "data lake extended to support data warehouse use cases".
So something like HDFS or S3 with Delta (from DBX) or Apache Iceberg storage formats, utilizing Spark or Presto/Trino or something for compute. One unified platform built on scalable big data technologies, that can do transactions, SQL MERGE, smart partitioning and other bells and whistles.
Then AWS decided to unveil "AWS LakeHouse" which meant you have both S3 and Redshift and use both at the same time - lake and warehouse next to each other.
This is not what lakehouse meant until then It is also terrible design - having data in two places means you now have to implement access control, logging, auditing, data access and so on twice. You also have to sync data between the two storages, keep track of what is where and keep track of what is the single source of truth
Truly idiotic design / marketing that could only have come from AWS. But since any larger company has army of "enterprise architects" who went from "nobody was ever fired for recommending IBM" to "nobody was ever fired for recommending Oracle" to "nobody was ever fired for recommending AWS" who will just internally enforce whatever bullshit vendors pushes on them ... it is almost what "lakehouse" means nowadays.
AWS truly is the Oracle of 2020s. Fuck them.
(Rant over, sorry, got carried away)
- atwebb 5y ago> It is also terrible design - having data in two places means you now have to implement access control, logging, auditing, data access and so on twice. I've understood and implemented differently. With Spectrum (or Polybase for SQL Server / Synapse), you can extended into the data lake. Copy over aggregate/curated data or something you need to special use cases on. Leave the structured, columnar data within the cheap storage. You pay per scan but it is cheap (at least to a point). Also, Databricks took the Lakehouse moniker and sprinted with it. AWS was late to the game from what I saw (at least for marketing terminology adoption).
- glogla 5y agoYou can do "lakehouse" just with Redshift, but in AWS pictures, you'll see Glue Jobs, Glue Elastic Views, Sagemaker, Aurora ... it's a huge mess. With ra3 redshift, you pay storage cost of s3 for internal data as well, so unless you use the s3 with something else, I don't see much point in using spectrum. Still, something like Snowflake works much better. They actually seem to have vision and not just "us too!" like AWS.
- atwebb 5y agoYou pay the storage cost in S3 as well, depending on tier Snowflake will not necessarily be a cost savings from compute either. Redshift could really use some elasticity beyond a factor of 2 and some warm resume features.
- glogla 5y agoJust to clarify, I meant Redshift ra3 storage costs as much as s3, so you're not saving much by keeping things in s3 instead of in Redshift. Although I don't know how Redshift compression compares to something like gzipped parquet. Maybe the data ends up taking more space and thus money. Agreed on that elasticity.