3 ms·

Neat that you plan to support both Delta Lake and Apache Iceberg I'm curious about HN's position between these two formats? I'm having a hard time deciphering

Neat that you plan to support both Delta Lake and Apache Iceberg I'm curious about HN's position between these two formats? I'm having a hard time deciphering which might be the industry winner (or perhaps they both have a place, no "winner" necessary)

retakeming 2y ago
This is anecdotal, but I feel that we (ParadeDB) have received more requests for Iceberg integration vs. Delta Lake. We were actually hesitant to launch pg_lakehouse without Iceberg support, but pulled the trigger on it because the iceberg-rust crate is still in its early days. We will probably be contributing to iceberg-rust to make it work with pg_lakehouse.
- kiwicopple 2y ago
  > We will probably be contributing to iceberg-rust to make it work with pg_lakehouse That's great news, thanks for your contributions to open source (here, and all the other extensions)
- lukekim 2y ago
  Also anecdotal, but we (Spice AI) see more requests for Iceberg, but in practice more deployments of Delta Lake.
  - FridgeSeal 2y ago
    My theory is that everyone would _prefer_ to use Iceberg, but isn’t as widely supported _yet_, so they’re stuck with Delta in the interim.
    - philippemnoel 2y ago
      I think you're spot on
slap_shot 2y ago
There isn't a winner and there likely won't be one (at least not for a long time). Tabular will likely be acquired by Snowflake and the two industry behemoths now back their own formats, and each will treat their own as a first class citizen.
- philippemnoel 2y ago
  Agreed, this is why we want to support both. Maybe even Apache Hudi down the line. But I hope the industry converges to a main standard rather than Snowflake/Databricks fighting for their own formats. They can differentiate on much more meaningful features
  - jaychia 2y ago
    There’s a lot of interesting work happening in this area (see: XTable). We are building a Python distributed query engine, and share a lot of the same frustrations… in fact until quite recently most of the table formats only had JVM client libraries and so integrating it purely natively with Daft was really difficult. We finally managed to get read integrations across Iceberg/DeltaLake/Hudi recently as all 3 now have Python/Rust-facing APIs. Funny enough, the only non-JVM implementation of Hudi was contributed by the Hudi team and currently still lives in our repo :D (https://github.com/Eventual-Inc/Daft/tree/main/daft/hudi/pyhudi https://github.com/Eventual-Inc/Daft/tree/main/daft/hudi/pyh...) It’s still the case that these libraries still lag behind their JVM counterparts though, so it’s going to be a while before we see full support across the full featureset of each table format. But we’re definitely seeing a large appetite for working with table formats outside of the JVM ecosystem (e.g. in Python and Rust)
    - philippemnoel 2y ago
      Are you using the iceberg-rust crate for Rust? It's a rather young project, have you found it sufficient for your needs (if using)?
      - sammysidhu 2y ago
kcirerick 2y ago
I'm also building in the lakehouse space and anecdotally have seen more excitement around Iceberg over delta lake just because of its completely open source origins. Iceberg has evolved faster and has had more contributions from a more diverse set of contributors than Delta Lake. Not sure if this will change with a Snowflake <> Tabular acquisition but I'd easily bet on Iceberg if current trends continue.
- philippemnoel 2y ago
  We agree. We plan to bring Iceberg support as a first-class citizen as soon as we can, but unfortunately the support in Rust these days is still limited. We and the community are working on it

retakeming 2y ago

This is anecdotal, but I feel that we (ParadeDB) have received more requests for Iceberg integration vs. Delta Lake. We were actually hesitant to launch pg_lakehouse without Iceberg support, but pulled the trigger on it because the iceberg-rust crate is still in its early days. We will probably be contributing to iceberg-rust to make it work with pg_lakehouse.

kiwicopple 2y ago
> We will probably be contributing to iceberg-rust to make it work with pg_lakehouse That's great news, thanks for your contributions to open source (here, and all the other extensions)
lukekim 2y ago
Also anecdotal, but we (Spice AI) see more requests for Iceberg, but in practice more deployments of Delta Lake.
- FridgeSeal 2y ago
  My theory is that everyone would _prefer_ to use Iceberg, but isn’t as widely supported _yet_, so they’re stuck with Delta in the interim.
  - philippemnoel 2y ago
    I think you're spot on

slap_shot 2y ago

There isn't a winner and there likely won't be one (at least not for a long time). Tabular will likely be acquired by Snowflake and the two industry behemoths now back their own formats, and each will treat their own as a first class citizen.

philippemnoel 2y ago
Agreed, this is why we want to support both. Maybe even Apache Hudi down the line. But I hope the industry converges to a main standard rather than Snowflake/Databricks fighting for their own formats. They can differentiate on much more meaningful features
- jaychia 2y ago
  There’s a lot of interesting work happening in this area (see: XTable). We are building a Python distributed query engine, and share a lot of the same frustrations… in fact until quite recently most of the table formats only had JVM client libraries and so integrating it purely natively with Daft was really difficult. We finally managed to get read integrations across Iceberg/DeltaLake/Hudi recently as all 3 now have Python/Rust-facing APIs. Funny enough, the only non-JVM implementation of Hudi was contributed by the Hudi team and currently still lives in our repo :D (https://github.com/Eventual-Inc/Daft/tree/main/daft/hudi/pyhudi https://github.com/Eventual-Inc/Daft/tree/main/daft/hudi/pyh...) It’s still the case that these libraries still lag behind their JVM counterparts though, so it’s going to be a while before we see full support across the full featureset of each table format. But we’re definitely seeing a large appetite for working with table formats outside of the JVM ecosystem (e.g. in Python and Rust)
  - philippemnoel 2y ago
    Are you using the iceberg-rust crate for Rust? It's a rather young project, have you found it sufficient for your needs (if using)?
    - sammysidhu 2y ago

kcirerick 2y ago

I'm also building in the lakehouse space and anecdotally have seen more excitement around Iceberg over delta lake just because of its completely open source origins. Iceberg has evolved faster and has had more contributions from a more diverse set of contributors than Delta Lake. Not sure if this will change with a Snowflake <> Tabular acquisition but I'd easily bet on Iceberg if current trends continue.

philippemnoel 2y ago
We agree. We plan to bring Iceberg support as a first-class citizen as soon as we can, but unfortunately the support in Rust these days is still limited. We and the community are working on it