3 ms·
it's not that it is inefficient, it is that data processing approach has to be different with lakehouses. you can't do OLTP type workload with lakehouses - mea
by slt2021 4y ago
it's not that it is inefficient, it is that data processing approach has to be different with lakehouses.
you can't do OLTP type workload with lakehouses - meaning updating the same value in a row million times per second, because object storage is not supposed to be used in OLTP way.
however you can easily do that with relational DB, just UPDATE the value and thats it. underlying RDBMS engine will keep memory buffers updated, will update pages automatically and keep WAL for durability.
for lakehouse the proper way is to setup streaming processing and use in-memory cache to do hot data processing, and once data cools down - write to lakehouse table (object storage) like once per batch (like once every few mins). and restart batch from the beginning in case of failure - for durability.