5 ms·
As opposed to what setup exactly? What do you propose everyone uses instead of Lakehouses?
by glogla 4y ago
As opposed to what setup exactly? What do you propose everyone uses instead of Lakehouses?
- anonymousDan 4y agoI guess a setup such as hdfs where storage and compute are colocated and not disaggregated. But that also offers similar transactional semantics to lakehouses.
- berkle4455 4y agoDisaggregated storage. You still have separation of compute and storage with the bandwidth and latency benefits of directly attached storage. Object stores are a terribly inefficient way to access and store changes of data.
- nostrebored 4y agoInefficient in what respect? It really depends on your access patterns, performance requirements, and the way data is structured and stored in your object store - someone who thinks it’s often a bad idea
- slt2021 4y agoit's not that it is inefficient, it is that data processing approach has to be different with lakehouses. you can't do OLTP type workload with lakehouses - meaning updating the same value in a row million times per second, because object storage is not supposed to be used in OLTP way. however you can easily do that with relational DB, just UPDATE the value and thats it. underlying RDBMS engine will keep memory buffers updated, will update pages automatically and keep WAL for durability. for lakehouse the proper way is to setup streaming processing and use in-memory cache to do hot data processing, and once data cools down - write to lakehouse table (object storage) like once per batch (like once every few mins). and restart batch from the beginning in case of failure - for durability.
- thinkharderdev 4y agoIt's definitely inefficient w/r/t performance (reading from Object Storage will always be several times slower than reading from disk/SSD) but the point is usually to minimize cost.