4 ms·

Data Lakehouse is where you store your data on object stores and spin up a bunch of instances (cpu/memory) as needed to crunch the data on the timeline you desi

Data Lakehouse is where you store your data on object stores and spin up a bunch of instances (cpu/memory) as needed to crunch the data on the timeline you desire. It’s incredible to me how “solutions” are continually invented that give cloud providers plenty of stages to charge for not only storage but movement and processing of your data too.

anonymousDan 4y ago
Why would you be charged for movement in this case, I thought intra datacenter traffic was free? Or you mean you get charged to update/query the object store?
glogla 4y ago
As opposed to what setup exactly? What do you propose everyone uses instead of Lakehouses?
- anonymousDan 4y ago
  I guess a setup such as hdfs where storage and compute are colocated and not disaggregated. But that also offers similar transactional semantics to lakehouses.
- berkle4455 4y ago
  Disaggregated storage. You still have separation of compute and storage with the bandwidth and latency benefits of directly attached storage. Object stores are a terribly inefficient way to access and store changes of data.
  - nostrebored 4y ago
    Inefficient in what respect? It really depends on your access patterns, performance requirements, and the way data is structured and stored in your object store - someone who thinks it’s often a bad idea
    - slt2021 4y ago
      it's not that it is inefficient, it is that data processing approach has to be different with lakehouses. you can't do OLTP type workload with lakehouses - meaning updating the same value in a row million times per second, because object storage is not supposed to be used in OLTP way. however you can easily do that with relational DB, just UPDATE the value and thats it. underlying RDBMS engine will keep memory buffers updated, will update pages automatically and keep WAL for durability. for lakehouse the proper way is to setup streaming processing and use in-memory cache to do hot data processing, and once data cools down - write to lakehouse table (object storage) like once per batch (like once every few mins). and restart batch from the beginning in case of failure - for durability.
  - thinkharderdev 4y ago
    It's definitely inefficient w/r/t performance (reading from Object Storage will always be several times slower than reading from disk/SSD) but the point is usually to minimize cost.
KptMarchewa 4y ago
No, that's data lake. Data lakehouse is data lake where your objects are wrapped in a "table format" like Iceberg that allows you to query (update, itd) them like they were stored in a traditional data warehouse.
- berkle4455 4y ago
  The data is still stored on object storage. It’s just a more performant format. The root of the problem is using object storage improperly.
  - KptMarchewa 4y ago
    Format is the same - parquet. Only the way metadata is stored is different. >The root of the problem is using object storage improperly. I don't see anything improper for any Parquet-with-metadata format.
  - articsputnik 4y ago
    Data Lakehouse to me is: > A Data Lakehouse open data management architecture that combines the flexibility, cost-efficiency, and scale of Data Lake with the data management and ACID transactions of Data Warehouse with Data Lake Table Formats (Delta Lake, Apache Iceberg & Apache Hudi) that enable Business Intelligence (BI) and Machine Learning (ML) on all data. The initial concept was created by Databricks in the CIDR Paper (https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf) in 2021.

anonymousDan 4y ago

Why would you be charged for movement in this case, I thought intra datacenter traffic was free? Or you mean you get charged to update/query the object store?

glogla 4y ago

As opposed to what setup exactly? What do you propose everyone uses instead of Lakehouses?

anonymousDan 4y ago
I guess a setup such as hdfs where storage and compute are colocated and not disaggregated. But that also offers similar transactional semantics to lakehouses.
berkle4455 4y ago
Disaggregated storage. You still have separation of compute and storage with the bandwidth and latency benefits of directly attached storage. Object stores are a terribly inefficient way to access and store changes of data.
- nostrebored 4y ago
  Inefficient in what respect? It really depends on your access patterns, performance requirements, and the way data is structured and stored in your object store - someone who thinks it’s often a bad idea
  - slt2021 4y ago
    it's not that it is inefficient, it is that data processing approach has to be different with lakehouses. you can't do OLTP type workload with lakehouses - meaning updating the same value in a row million times per second, because object storage is not supposed to be used in OLTP way. however you can easily do that with relational DB, just UPDATE the value and thats it. underlying RDBMS engine will keep memory buffers updated, will update pages automatically and keep WAL for durability. for lakehouse the proper way is to setup streaming processing and use in-memory cache to do hot data processing, and once data cools down - write to lakehouse table (object storage) like once per batch (like once every few mins). and restart batch from the beginning in case of failure - for durability.
- thinkharderdev 4y ago
  It's definitely inefficient w/r/t performance (reading from Object Storage will always be several times slower than reading from disk/SSD) but the point is usually to minimize cost.

KptMarchewa 4y ago

No, that's data lake. Data lakehouse is data lake where your objects are wrapped in a "table format" like Iceberg that allows you to query (update, itd) them like they were stored in a traditional data warehouse.

berkle4455 4y ago
The data is still stored on object storage. It’s just a more performant format. The root of the problem is using object storage improperly.
- KptMarchewa 4y ago
  Format is the same - parquet. Only the way metadata is stored is different. >The root of the problem is using object storage improperly. I don't see anything improper for any Parquet-with-metadata format.
- articsputnik 4y ago
  Data Lakehouse to me is: > A Data Lakehouse open data management architecture that combines the flexibility, cost-efficiency, and scale of Data Lake with the data management and ACID transactions of Data Warehouse with Data Lake Table Formats (Delta Lake, Apache Iceberg & Apache Hudi) that enable Business Intelligence (BI) and Machine Learning (ML) on all data. The initial concept was created by Databricks in the CIDR Paper (https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf) in 2021.