4 ms·
Modern data warehouses (Snowflake, BigQuery, and maybe Redshift RA3) have incorporated all the key features of data lakes: - The cost of storage is the same as
by georgewfraser 7y ago
Modern data warehouses (Snowflake, BigQuery, and maybe Redshift RA3) have incorporated all the key features of data lakes:
- The cost of storage is the same as S3.
- Storage and compute can be scaled independently.
- You can store multiple levels of curation in the same system: a normalized schema that reflects the source, alongside a dimensional schema that has been thoroughly ETL’d.
- Compute can be scaled horizontally to basically any level of parallelism you desire.
Given these facts, it is unclear what rationale still exists for data lakes. The only remaining major advantage of a data lake is that you aren’t subject to as much vendor lock-in.
- cjalmeida 7y agoNot being subject to vendor lock-in is huge in itself. You can save plenty of money if you have the scale to move out of S3. That’s important because you can usually trade CPU for storage by storing data in multiple formats, optimized for different access patterns. But mostly, the Hadoop ecosystem is very open. The tools are still maturing and it’s easier to debug open source tools than dealing with the generally poor support in most managed solutions.
- charlieflowers 7y agoCan you clarify what you mean by "if you have the scale to move out of S3"? Why does it take scale to move out of S3? And I thought S3 was cheap, so how would moving out save money?
- bargle0 7y agoAmazon has to make a profit on selling services. You don’t have to make a profit providing services internally. There are certain inelasticities that both of you have to pay for: power, real estate, internet, etc. If you’re big enough, you can do it cheaper than Amazon.
- cjalmeida 7y agoS3 is flexible and scalable, but it is not cheap. I'd be hard pressed to run the numbers now, but at some point it's cheaper to just do storage yourself. But to be fair, you'll go on-premise due to the computing or bandwidth costs first. And you'll likely move data to the same datacenter to avoid expensive transfer costs. I've also had to work in places where you simply could not put your data in the cloud due to regulatory reasons.
- dx034 7y agoS3 isn't cheap if used with other services. Either you use AWS for everything or you pay with bandwidth. It's cheap to get your data in, using it or getting it out isn't cheap at all.
- grakic 7y agoRecently I saw a term "lakehouse" for applying data lake design on data warehouse technology. With a big house, you can have a lake inside.
- vgt 7y agoYes!!!! I wrote some words just on this topic recently!! https://medium.com/@vtereshko/data-warehouse-storage-or-a-data-lake-why-not-both-f0e3d916fc08 https://medium.com/@vtereshko/data-warehouse-storage-or-a-da... (Pm on BigQuery)