9 ms·

Apache iceberg the Hadoop of the modern-data-stack?

115 points by samrohn 2y ago

hendiatris 2y ago
This is a huge challenge with Iceberg. I have found that there is substantial bang for your buck in tuning how parquet files are written, particularly in terms of row group size and column-level bloom filters. In addition to that, I make heavy use of the encoding options (dictionary/RLE) while denormalizing data into as few files as possible. This has allowed me to rely on DuckDB for querying terabytes of data at low cost and acceptable performance. What we are lacking now is tooling that gives you insight into how you should configure Iceberg. Does something like this exist? I have been looking for something that would show me the query plan that is developed from Iceberg metadata, but didn’t find anything. It would go a long way to showing where the bottleneck is for queries.
- joking 2y ago
  [flagged]
- jasonjmcghee 2y ago
  Have you written about your parquet strategy anywhere? Or have suggested reading related to the tuning you've done? Super interested.
  - indoordin0saur 2y ago
    Also very interested in the parquet tuning. I have been building my data lake and most optimization I do is just with efficient partitioning.
    - hendiatris 2y ago
      I will write something up when the dust settles, I’m still testing things out. It’s a project where the data is fairly standardized but there is about a petabyte to deal with, so I think it makes sense to make investments in efficiency at the lower level rather than through tons of resources at it. That has meant a custom parser for the input data written in Rust, lots of analysis of the statistics of the data, etc. It has been a different approach to data engineering and one that I hope we see more of. Regarding reading materials, I found this DuckDB post to be especially helpful in realizing how parquet could be better leveraged for efficiency: https://duckdb.org/2024/03/26/42-parquet-a-zip-bomb-for-the-big-data-age.html https://duckdb.org/2024/03/26/42-parquet-a-zip-bomb-for-the-...
    - EdwardDiego 2y ago
      What query engine are you using? Tends to be that an optimal file size for Parquet is about 1GiB, once again, the "many small files" problem of Hadoop remains. Then it's things like, can you organise your data in such a way to take advantage of RLE etc.?
      - indoordin0saur 2y ago
        Either Spark or Redshift (serverless)
- chrsig 2y ago
  how nested is the data in the parquet files?
- EdwardDiego 2y ago
  Parquet tuning has always been like that, ever since it first came out in 2013. I worry with Iceberg that people think it's just a case of "use an Iceberg table in Snowflake" and boom, amazingly fast querying of data in S3!
Gasp0de 2y ago
Does anyone have a good alternative for storing large amounts of very small files that need to be individually queriable? We are dealing with a large amount of sensor readings that we need to be able to query on a per sensor basis and a timespan, and we are dealing with the problem mentioned in the article, that storing millions of small files in S3 is expensive.
- paulsutter 2y ago
  If you want to keep them in S3, consolidate into sorted parquet files. You get random access to row groups, and only the columns you need are read so it’s very efficient. DuckDB can both build and access these files efficiently. You could compact files hourly/nightly/weekly whatever Of course you could also use Aurora for a clean scalable Postgres that can survive zone failures for a simpler solution
  - Gasp0de 2y ago
    The problem is that the initial writing is already so expensive, I guess we'd have to write multiple sensors into the same file instead of having one file per sensor per interval. I'll look into parquet access options, if we could write 10k sensors into one file but still read a single sensor from that file that could work.
    - spothedog1 2y ago
      New S3 Table Buckets [1] do automatic compaction [1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tab...
      - necrobrit 2y ago
        Table buckets are currently quite hard to use for a lot of use cases as they _only_ support primitive types. No nested types. Hopefully this will come at some point. Product looks very cool otherwise.
    - 2y ago
paulsutter 2y ago
Does this feel about 3x too verbose, like it’s generated?
- jasonjmcghee 2y ago
  Idk if it's the verbosity but yes, reads as generated to me. Specifically sounds like ChatGPT's writing.
- mritchie712 2y ago
  100%, might be gpt4.5
alexmorley 2y ago
Most of these issues will be ring true to lots of folk using Iceberg at the moment. But this does not: > Yet, competing table formats like Delta Lake and Hudi mirror this fragmentation. [ ... ] > Just as Spark emerged as the dominant engine in the Hadoop ecosystem, a dominant table format and catalog may appear in the Iceberg era. I think extremely few people are making bets on any other open source table format now - that consolidation has already happened in 2023-2024 (see e.g. Databricks who have their own competing format leaning heavily into iceberg; or adoption from all of the major data warehouse providers).
- twoodfin 2y ago
  Microsoft is right now making a huge bet on Delta by way of their “Microsoft Fabric” initiative (as always with Microsoft: Is it a product? Is it a branding scheme? Yes.) They seem to be the only vendor crazy enough to try to fast-follow Databricks, who is clearly driving the increasingly elaborate and sophisticated Delta ecosystem (check the GitHub traffic…) But Microsoft + Databricks is a lot of momentum for Delta. On the merits of open & simple, I agree, better for everyone if Iceberg wins out—as Iceberg and not as some Frankenstandard mashed together with Delta by the force of 1,000 Databricks engineers.
  - datadrivenangel 2y ago
    The only reason Microsoft is using Delta is to emphasize to CTOs and investors that fabric is as good as databricks, even when that is obviously false to anyone who has smelled the evaporative scent of vaporware before.
    - esafak 2y ago
      Microsoft's gonna Microsoft.
    - twoodfin 2y ago
      Very different business, of course, but Databricks v. Fabric reminds me a lot of Slack v. Teams. Regardless of the relative merits now, I think everyone agrees that a few years ago Slack was clearly superior. Microsoft could have certainly bought Slack instead of pumping probably billions into development, marketing, discounts to destroy them. I think Microsoft could and would consider buying Databricks—$80–100B is a lot, but not record-shattering. If I were them, though, I’d spend a few billion competing as an experiment, first.
alienreborn 2y ago
Better article (imo) on similar topic: https://www.dataengineeringweekly.com/p/is-apache-iceberg-the-new-hadoop https://www.dataengineeringweekly.com/p/is-apache-iceberg-th...
- tomnicholas1 2y ago
  I think the posted article was generated from this one - the structure of the content is so similar.
simlevesque 2y ago
I'm working on an alternative Iceberg client to work better in write heavy use cases. Instead of many smaller files it writes on the same file until it's 1mb in size but it gives it a new name. Then I update the manifest to the new filename and checksum. I keep old files on disk for 60 seconds to allow pending queries. I'm also working on auto compaction, when I have ten 1mb files I compact them, same with ten 10mb files, etc... I feel like this could be a game changer for the ecosystem. It's more cpu and network heavy for writes but the reads are always fast. And the writes are still faster than pyiceberg. I want to hear opinions or how this could never work.
- mritchie712 2y ago
  nice! anywhere we can follow your progress?
  - simlevesque 2y ago
    Not right now sadly I have some work obligations taking my time but I can't wait to share more. I'm using a basic implementation that's not backed by iceberg, just Parquet files in hive partitions that I can query using DuckDB.
- thom 2y ago
  Interesting. My personal feeling is that we're slowly headed to a world where we can have our cake and eat it: fast bulk ingestion, fast OLAP, fast OLTP, low latency, all together in the same datastore. I'm hoping we just get to collapse whole complex data platforms into a single consistent store with great developer experience, and never look back.
  - simlevesque 2y ago
    I think it's possible too and the Iceberg spec allows it but the implementations are not suited for every use case.
  - ndm000 2y ago
    I’ve felt the same way. It’s so inefficient to have two patterns - OLAP and OLTP - both using SQL interfaces but requiring syncing between systems. There are some physical limits at play though. OLAP will always take less processing and disk usage if the data it needs is all right next to each other (columnar storage) where as OLTP’s need for fast writes usually means row based storage is more efficient. I think the solution would be one system that stores data consistently both ways and knows when to use which method for a given query.
mritchie712 2y ago
This is a bit overblown. Is Iceberg "easy" to set up? No. Can you get set up in a week? Yes. If you really need a datalake, spending a week setting it up is not so bad. We have a guide[0] here that will get you started in under an hour. For smaller (e.g. under 10tb) data where you don't need real-time, DuckDB is becoming a really solid option. Here's on setup[1] we've played around with using Arrow Flight. If you don't want to mess with any of this, we[2] spin it all up for you. 0 - https://www.definite.app/blog/cloud-iceberg-duckdb-aws https://www.definite.app/blog/cloud-iceberg-duckdb-aws 1 - https://www.definite.app/blog/duck-takes-flight https://www.definite.app/blog/duck-takes-flight 2 - https://www.definite.app/ https://www.definite.app/
- simlevesque 2y ago
  I think Iceberg can work in real time but the current implementations make it impossible. I have a vision for a way to make it work. I made another comment here. Your blog posts were helpful, I digged a bit in the Duck Takes Flight code in python and rust.
- pid-1 2y ago
  If you're already in AWS, why wouldn't you use AWS Glue Catalog + AWS SDK for pandas + Athena? You can setup a data lake, save data and start doing queries in like 10 minutes with this setup.
  - thedougd 2y ago
    These days you can 'just' create an S3 tables bucket. https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tab...
  - mritchie712 2y ago
    Agreed. A lot of people worry would worry about "vendor lock-in" here, but it's certainly convenient.
  - tsss 2y ago
    Athena is really expensive though and you will often run into a hard limit on the size of your query.
zhousun 2y ago
The only datastack iceberg (or lakehouse) will never replace is OLTP systems, for high-concurrency updates optimistic concurrency control & object store is simply a no go. Iceberg out-of-the-box is "NOT" good at streaming use cases, unlike formats like Hudi or Paimon, the table format does not have the concept of merge/ index. However, the beauty of iceberg is it is very unopinionated, so it is indeed possible to design an engine to stream write to iceberg. As far as I know this is how engines like Upsolver was implemented: 1. Have in-memory buffer to track incoming rows before flushing a version to iceberg (every 10s to a few minutes). 2. Build Indexing structure to write position deletes/ deletion vector instead of equality deletes. 3. The writer will all try to merge small files and optimize the table. And stay tuned, we at https://www.mooncake.dev/ https://www.mooncake.dev/ are working on a solution to mirror a postgres table to iceberg, and keep them always up-to-date.
robertkoss 2y ago
This article is just shameless advertising for Estuary Flow, a company that the author is working for. "Operational Maturity", as if Iceberg, Delta or Hudi are not mature. These are battle-tested frameworks that have been in production for years. The "small files problem" is not really a problem because every framework supports some way of compacting smaller files. Just run a nightly job that compacts the small files and you're good 2 go.
- prpl 2y ago
  AWS can do it for you with S3 tables.
theyinwhy 2y ago
What's a good alternative? Google BigQuery?
datax2 2y ago
"Hadoop’s meteoric rise led many organizations to implement it without understanding its complexities, often resulting in underutilized clusters or over-engineered architectures. Iceberg is walking a similar path." This pain is too real, and too close to home. I've seen this outcome turn the entire business off of consuming their data via hadoop because it turns into a wasteland of delayed deliveries, broken datasets, op's teams who cannot scale, and architects overselling too robust designs. I've tried to scale down hadoop to the business user with visual etl tools like Alteryx, but there again compatibility between Alteryx and hadoop suck via ODBC connectors. I came from an AWS based stack into a poorly leapfrogged data stack and it's hard not to pull my hair out between the business struggling to use it and infra + op's not keeping up. Now these teams want to push to iceburg or big query while ignoring the mountains of tech debt they have created. Don't get me wrong Hadoop isn't a bad idea, its just complex and a time suck, and unless you have time to dedicate to properly deploy these solutions which most business do not, your implementation will suffer, your business will suffer. "While the parallels to Hadoop are striking, we also have the opportunity to avoid its pitfalls." no one in IT learns from their failures unless they are writing the checks, most will flip before they feel the pain.
orthoxerox 2y ago
I think the complexity of Iceberg is overblown. It's just a table format and it's strictly better than the Hive-style /schema/table/partition_key=partition_value/one_of_many_files.parquet It has a lot of knobs to fiddle with (more than Delta Lake, which tries very hard to come up with good defaults), but even if you don't touch any of them, you already end up with tables that are as good as Hive's, except now your writers don't break your readers. This is already a massive boon that lets you escape the rigidity of a timetable schedule for your data pipelines. Anything else you can come up with (switching your table to MOR and rewriting it as a separate step etc) is further improvements.

115 points by samrohn 2y ago

This is a huge challenge with Iceberg. I have found that there is substantial bang for your buck in tuning how parquet files are written, particularly in terms of row group size and column-level bloom filters. In addition to that, I make heavy use of the encoding options (dictionary/RLE) while denormalizing data into as few files as possible. This has allowed me to rely on DuckDB for querying terabytes of data at low cost and acceptable performance. What we are lacking now is tooling that gives you insight into how you should configure Iceberg. Does something like this exist? I have been looking for something that would show me the query plan that is developed from Iceberg metadata, but didn’t find anything. It would go a long way to showing where the bottleneck is for queries.

joking 2y ago
[flagged]
jasonjmcghee 2y ago
Have you written about your parquet strategy anywhere? Or have suggested reading related to the tuning you've done? Super interested.
- indoordin0saur 2y ago
  Also very interested in the parquet tuning. I have been building my data lake and most optimization I do is just with efficient partitioning.
  - hendiatris 2y ago
    I will write something up when the dust settles, I’m still testing things out. It’s a project where the data is fairly standardized but there is about a petabyte to deal with, so I think it makes sense to make investments in efficiency at the lower level rather than through tons of resources at it. That has meant a custom parser for the input data written in Rust, lots of analysis of the statistics of the data, etc. It has been a different approach to data engineering and one that I hope we see more of. Regarding reading materials, I found this DuckDB post to be especially helpful in realizing how parquet could be better leveraged for efficiency: https://duckdb.org/2024/03/26/42-parquet-a-zip-bomb-for-the-big-data-age.html https://duckdb.org/2024/03/26/42-parquet-a-zip-bomb-for-the-...
  - EdwardDiego 2y ago
    What query engine are you using? Tends to be that an optimal file size for Parquet is about 1GiB, once again, the "many small files" problem of Hadoop remains. Then it's things like, can you organise your data in such a way to take advantage of RLE etc.?
    - indoordin0saur 2y ago
      Either Spark or Redshift (serverless)
chrsig 2y ago
how nested is the data in the parquet files?
EdwardDiego 2y ago
Parquet tuning has always been like that, ever since it first came out in 2013. I worry with Iceberg that people think it's just a case of "use an Iceberg table in Snowflake" and boom, amazingly fast querying of data in S3!

Gasp0de 2y ago

Does anyone have a good alternative for storing large amounts of very small files that need to be individually queriable? We are dealing with a large amount of sensor readings that we need to be able to query on a per sensor basis and a timespan, and we are dealing with the problem mentioned in the article, that storing millions of small files in S3 is expensive.

paulsutter 2y ago
If you want to keep them in S3, consolidate into sorted parquet files. You get random access to row groups, and only the columns you need are read so it’s very efficient. DuckDB can both build and access these files efficiently. You could compact files hourly/nightly/weekly whatever Of course you could also use Aurora for a clean scalable Postgres that can survive zone failures for a simpler solution
- Gasp0de 2y ago
  The problem is that the initial writing is already so expensive, I guess we'd have to write multiple sensors into the same file instead of having one file per sensor per interval. I'll look into parquet access options, if we could write 10k sensors into one file but still read a single sensor from that file that could work.
  - spothedog1 2y ago
    New S3 Table Buckets [1] do automatic compaction [1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tab...
    - necrobrit 2y ago
      Table buckets are currently quite hard to use for a lot of use cases as they _only_ support primitive types. No nested types. Hopefully this will come at some point. Product looks very cool otherwise.
  - 2y ago

paulsutter 2y ago

Does this feel about 3x too verbose, like it’s generated?

jasonjmcghee 2y ago
Idk if it's the verbosity but yes, reads as generated to me. Specifically sounds like ChatGPT's writing.
mritchie712 2y ago
100%, might be gpt4.5

alexmorley 2y ago

Most of these issues will be ring true to lots of folk using Iceberg at the moment. But this does not: > Yet, competing table formats like Delta Lake and Hudi mirror this fragmentation. [ ... ] > Just as Spark emerged as the dominant engine in the Hadoop ecosystem, a dominant table format and catalog may appear in the Iceberg era. I think extremely few people are making bets on any other open source table format now - that consolidation has already happened in 2023-2024 (see e.g. Databricks who have their own competing format leaning heavily into iceberg; or adoption from all of the major data warehouse providers).

twoodfin 2y ago
Microsoft is right now making a huge bet on Delta by way of their “Microsoft Fabric” initiative (as always with Microsoft: Is it a product? Is it a branding scheme? Yes.) They seem to be the only vendor crazy enough to try to fast-follow Databricks, who is clearly driving the increasingly elaborate and sophisticated Delta ecosystem (check the GitHub traffic…) But Microsoft + Databricks is a lot of momentum for Delta. On the merits of open & simple, I agree, better for everyone if Iceberg wins out—as Iceberg and not as some Frankenstandard mashed together with Delta by the force of 1,000 Databricks engineers.
- datadrivenangel 2y ago
  The only reason Microsoft is using Delta is to emphasize to CTOs and investors that fabric is as good as databricks, even when that is obviously false to anyone who has smelled the evaporative scent of vaporware before.
  - esafak 2y ago
    Microsoft's gonna Microsoft.
  - twoodfin 2y ago
    Very different business, of course, but Databricks v. Fabric reminds me a lot of Slack v. Teams. Regardless of the relative merits now, I think everyone agrees that a few years ago Slack was clearly superior. Microsoft could have certainly bought Slack instead of pumping probably billions into development, marketing, discounts to destroy them. I think Microsoft could and would consider buying Databricks—$80–100B is a lot, but not record-shattering. If I were them, though, I’d spend a few billion competing as an experiment, first.

alienreborn 2y ago

Better article (imo) on similar topic: https://www.dataengineeringweekly.com/p/is-apache-iceberg-the-new-hadoop https://www.dataengineeringweekly.com/p/is-apache-iceberg-th...

tomnicholas1 2y ago
I think the posted article was generated from this one - the structure of the content is so similar.

simlevesque 2y ago

I'm working on an alternative Iceberg client to work better in write heavy use cases. Instead of many smaller files it writes on the same file until it's 1mb in size but it gives it a new name. Then I update the manifest to the new filename and checksum. I keep old files on disk for 60 seconds to allow pending queries. I'm also working on auto compaction, when I have ten 1mb files I compact them, same with ten 10mb files, etc... I feel like this could be a game changer for the ecosystem. It's more cpu and network heavy for writes but the reads are always fast. And the writes are still faster than pyiceberg. I want to hear opinions or how this could never work.

mritchie712 2y ago
nice! anywhere we can follow your progress?
- simlevesque 2y ago
  Not right now sadly I have some work obligations taking my time but I can't wait to share more. I'm using a basic implementation that's not backed by iceberg, just Parquet files in hive partitions that I can query using DuckDB.
thom 2y ago
Interesting. My personal feeling is that we're slowly headed to a world where we can have our cake and eat it: fast bulk ingestion, fast OLAP, fast OLTP, low latency, all together in the same datastore. I'm hoping we just get to collapse whole complex data platforms into a single consistent store with great developer experience, and never look back.
- simlevesque 2y ago
  I think it's possible too and the Iceberg spec allows it but the implementations are not suited for every use case.
- ndm000 2y ago
  I’ve felt the same way. It’s so inefficient to have two patterns - OLAP and OLTP - both using SQL interfaces but requiring syncing between systems. There are some physical limits at play though. OLAP will always take less processing and disk usage if the data it needs is all right next to each other (columnar storage) where as OLTP’s need for fast writes usually means row based storage is more efficient. I think the solution would be one system that stores data consistently both ways and knows when to use which method for a given query.

mritchie712 2y ago

This is a bit overblown. Is Iceberg "easy" to set up? No. Can you get set up in a week? Yes. If you really need a datalake, spending a week setting it up is not so bad. We have a guide[0] here that will get you started in under an hour. For smaller (e.g. under 10tb) data where you don't need real-time, DuckDB is becoming a really solid option. Here's on setup[1] we've played around with using Arrow Flight. If you don't want to mess with any of this, we[2] spin it all up for you. 0 - https://www.definite.app/blog/cloud-iceberg-duckdb-aws https://www.definite.app/blog/cloud-iceberg-duckdb-aws 1 - https://www.definite.app/blog/duck-takes-flight https://www.definite.app/blog/duck-takes-flight 2 - https://www.definite.app/ https://www.definite.app/

simlevesque 2y ago
I think Iceberg can work in real time but the current implementations make it impossible. I have a vision for a way to make it work. I made another comment here. Your blog posts were helpful, I digged a bit in the Duck Takes Flight code in python and rust.
pid-1 2y ago
If you're already in AWS, why wouldn't you use AWS Glue Catalog + AWS SDK for pandas + Athena? You can setup a data lake, save data and start doing queries in like 10 minutes with this setup.
- thedougd 2y ago
  These days you can 'just' create an S3 tables bucket. https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tab...
- mritchie712 2y ago
  Agreed. A lot of people worry would worry about "vendor lock-in" here, but it's certainly convenient.
- tsss 2y ago
  Athena is really expensive though and you will often run into a hard limit on the size of your query.

zhousun 2y ago

The only datastack iceberg (or lakehouse) will never replace is OLTP systems, for high-concurrency updates optimistic concurrency control & object store is simply a no go. Iceberg out-of-the-box is "NOT" good at streaming use cases, unlike formats like Hudi or Paimon, the table format does not have the concept of merge/ index. However, the beauty of iceberg is it is very unopinionated, so it is indeed possible to design an engine to stream write to iceberg. As far as I know this is how engines like Upsolver was implemented: 1. Have in-memory buffer to track incoming rows before flushing a version to iceberg (every 10s to a few minutes). 2. Build Indexing structure to write position deletes/ deletion vector instead of equality deletes. 3. The writer will all try to merge small files and optimize the table. And stay tuned, we at https://www.mooncake.dev/ https://www.mooncake.dev/ are working on a solution to mirror a postgres table to iceberg, and keep them always up-to-date.

robertkoss 2y ago

This article is just shameless advertising for Estuary Flow, a company that the author is working for. "Operational Maturity", as if Iceberg, Delta or Hudi are not mature. These are battle-tested frameworks that have been in production for years. The "small files problem" is not really a problem because every framework supports some way of compacting smaller files. Just run a nightly job that compacts the small files and you're good 2 go.

prpl 2y ago
AWS can do it for you with S3 tables.

theyinwhy 2y ago

What's a good alternative? Google BigQuery?

datax2 2y ago

"Hadoop’s meteoric rise led many organizations to implement it without understanding its complexities, often resulting in underutilized clusters or over-engineered architectures. Iceberg is walking a similar path." This pain is too real, and too close to home. I've seen this outcome turn the entire business off of consuming their data via hadoop because it turns into a wasteland of delayed deliveries, broken datasets, op's teams who cannot scale, and architects overselling too robust designs. I've tried to scale down hadoop to the business user with visual etl tools like Alteryx, but there again compatibility between Alteryx and hadoop suck via ODBC connectors. I came from an AWS based stack into a poorly leapfrogged data stack and it's hard not to pull my hair out between the business struggling to use it and infra + op's not keeping up. Now these teams want to push to iceburg or big query while ignoring the mountains of tech debt they have created. Don't get me wrong Hadoop isn't a bad idea, its just complex and a time suck, and unless you have time to dedicate to properly deploy these solutions which most business do not, your implementation will suffer, your business will suffer. "While the parallels to Hadoop are striking, we also have the opportunity to avoid its pitfalls." no one in IT learns from their failures unless they are writing the checks, most will flip before they feel the pain.

orthoxerox 2y ago

I think the complexity of Iceberg is overblown. It's just a table format and it's strictly better than the Hive-style /schema/table/partition_key=partition_value/one_of_many_files.parquet It has a lot of knobs to fiddle with (more than Delta Lake, which tries very hard to come up with good defaults), but even if you don't touch any of them, you already end up with tables that are as good as Hive's, except now your writers don't break your readers. This is already a massive boon that lets you escape the rigidity of a timetable schedule for your data pipelines. Anything else you can come up with (switching your table to MOR and rewriting it as a separate step etc) is further improvements.