4 ms·

Is anyone non-Uber using Hudi in production? Can someone comment on what developing against it is like compared to other warehousing technologies?

• cmollis 6y ago
  We're about to evaluate it since it's incubating now in AWS' EMR. I think if you have existing spark workloads that require update or delete of existing data stored in parquet, then it might be a good choice since it fills in a major gap there. The other choice is Delta Lake which provides similar capability.
  • xyzzy_plugh 6y ago
    I believe Netflix's Iceberg (now also Apache) aims to solve the same problems.
    • hashhar 6y ago
      Iceberg solves more problems than what Hudi does. The biggest win is computed partitions and not having to define a strict partitioning strategy from the start.
    • sub3421 6y ago
      Unlike hudi and delta lake, iceberg does not support updates/deletes yet.
• hashhar 6y ago
  Yes, we are using it in production (somewhat) - for a a few tables. Our workflow with it is: 1. Stream new data into a partitioned S3 bucket available as a table in metastore. 2. Use Hudi to create a copy-on-write compacted version of the table (think SELECT * FROM (SELECT *, row_number() over (partition by uuid order by ts desc) as rnum) WHERE rnum=1). Now people who are interested on the snapshot of the table rather than the event log can query the Hudi table instead of the raw event log tables. This has improved perf for the common use-cases while allowing us to use a common pipeline. Earlier we would have to dump such data into an RDS (Aurora Postgres) with UPSERT but that meant table bloat was an issue and cost was high depending on the date-range we wanted to store compacted.