3 ms·
Apache Hudi 1.0 released with secondary indexes for data lakehouses
- deleted 2y ago[deleted]
- cloud8bits 2y agoAre there good use cases for indexing on data lakes?
- sudha_sakthee 2y agoFaster upserts directly benefit from indices on the write side and the read side benefits from fast lookups.
- deleted 2y ago[deleted]
- sudha_sakthee 2y agoFirst lakehouse system to introduce secondary index!
- redhouse 2y agoReally curious of the performance gain we can get from the secondary index, and the cost of it.
- v5c6 2y agoIt’s comparable to and depends on selectivity of the query, like any database index. On a 10TB tpc-ds web_sales with 1:150 selectivity, we see an impressive 95% gain. If the query fetches out most records for e.g, then gains will be lower
- dunwaldo 2y agoIs this too little, too late while Snowflake and Databricks are marketing Iceberg full steam? Maybe Hudi will hang on a little longer than Delta if it builds new things like this?
- v5c6 2y agoAn open source community cannot out market big vendors. But can certainly out execute and the judicious engineers will continue making choices based on technical evaluations, to keep it going. I’d be very surprised if delta goes away, since iceberg still is not feature complete to replace it. Databricks has somewhat of a confusing position now, which is hurting themselves. It’d be interesting to watch.