3 ms·

The data lakehouse space is the most exciting evolution of the data warehousing industry imo! If anyone wants a comparison guide from a neutral party, we did on

by swyx 4y ago

The data lakehouse space is the most exciting evolution of the data warehousing industry imo! If anyone wants a comparison guide from a neutral party, we did one last year: https://airbyte.com/blog/data-lake-lakehouse-guide-powered-by-table-formats-delta-lake-iceberg-hudi https://airbyte.com/blog/data-lake-lakehouse-guide-powered-b... I cant paste images here but imo this table comparing the 3 formats is the big takeaway https://assets-global.website-files.com/6064b31ff49a2d31e0493af1/6307243701a9dd1c204660ca_data-lake-file-formats.jpeg https://assets-global.website-files.com/6064b31ff49a2d31e049... (explained inline, we do cite onehouse heavily but we are independent of them)

joeharris76 4y ago
[dead]
- glogla 4y ago
  > Personally, I find Brooklyn Data's post to be more practically useful since they used each format to do real work and show how they perform in practice. First, you should probably add disclaimer that you work for Databricks. Second, they really did not. They came up with a test case of 1) create a table 2) merge into it 3) delete from it 4) delete from it 5) delete from it 6) delete from it (but call it "GDPR request" in chart). This is a completely absurd workload that should give anyone a pause. In real world, Lakehouse tables are either appended to, merged into with periodic compaction, or dropped and recreated (using something like dbt). Running a delete four times is just bizarre, and making a general statement out of it is just suspiciously biased. Of course it is very interesting coincidence that this independent company came up with non-realistic benchmark showing how better Delta is compared to major competitor, that everyone from Databricks is sharing everything.
  - joeharris76 4y ago
    1) create a table (and then query it) - seems very common 2) merge more into it (and then query it) - also seems very common 3) delete (a tiny bit) from it (and then query it) - also seems very common 4) delete (a little more) from it (and then query it) - also seems very common 5) delete (a decent chunk) from it (and then query it) - also seems very common 6) delete (all data for 1 key) from it (aka "GDPR request") - new problem, but common I sense a pattern here. Not sure what makes these tests "absurd" since everyone does them.

by swyx 4y ago

joeharris76 4y ago

[dead]

glogla 4y ago
> Personally, I find Brooklyn Data's post to be more practically useful since they used each format to do real work and show how they perform in practice. First, you should probably add disclaimer that you work for Databricks. Second, they really did not. They came up with a test case of 1) create a table 2) merge into it 3) delete from it 4) delete from it 5) delete from it 6) delete from it (but call it "GDPR request" in chart). This is a completely absurd workload that should give anyone a pause. In real world, Lakehouse tables are either appended to, merged into with periodic compaction, or dropped and recreated (using something like dbt). Running a delete four times is just bizarre, and making a general statement out of it is just suspiciously biased. Of course it is very interesting coincidence that this independent company came up with non-realistic benchmark showing how better Delta is compared to major competitor, that everyone from Databricks is sharing everything.
- joeharris76 4y ago
  1) create a table (and then query it) - seems very common 2) merge more into it (and then query it) - also seems very common 3) delete (a tiny bit) from it (and then query it) - also seems very common 4) delete (a little more) from it (and then query it) - also seems very common 5) delete (a decent chunk) from it (and then query it) - also seems very common 6) delete (all data for 1 key) from it (aka "GDPR request") - new problem, but common I sense a pattern here. Not sure what makes these tests "absurd" since everyone does them.