4 ms·
Inefficient in what respect? It really depends on your access patterns, performance requirements, and the way data is structured and stored in your object store
by nostrebored 4y ago
Inefficient in what respect? It really depends on your access patterns, performance requirements, and the way data is structured and stored in your object store
- someone who thinks it’s often a bad idea
- slt2021 4y agoit's not that it is inefficient, it is that data processing approach has to be different with lakehouses. you can't do OLTP type workload with lakehouses - meaning updating the same value in a row million times per second, because object storage is not supposed to be used in OLTP way. however you can easily do that with relational DB, just UPDATE the value and thats it. underlying RDBMS engine will keep memory buffers updated, will update pages automatically and keep WAL for durability. for lakehouse the proper way is to setup streaming processing and use in-memory cache to do hot data processing, and once data cools down - write to lakehouse table (object storage) like once per batch (like once every few mins). and restart batch from the beginning in case of failure - for durability.