5 ms·
Preview: Amazon S3 Tables and Lakehouse in DuckDB
- ayhanfuat 2y agoAnybody tried S3 tables? How is your experience? It seems more tempting now that DuckDB supports it.
- Kalanos 2y agoHaven't tried it. S3 Tables sounds like a great idea. However, I am wary. For it to be useful, a suite of AWS services probably needs to integrate with it. These services are all managed by different teams that don't always work well together out of the box and often compete with redundant products. For example, configuring SageMaker Studio to use an EMR cluster for Spark was a multi-day hassle with a lot of custom (insecure?) configuration. How is this different from other existing table offerings? AWS is a mess.
- reedf1 2y agoAs a data engineering dabbler; parquet in S3 is beautiful. So is DuckDB. What an incredible match.
- alexott 2y agoPlain parquet has a lot of problems. That’s why iceberg and delta arise
- timenova 2y agoCan you elaborate what kind of problems does plain parquet have?
- pacbard 2y agoApache Iceberg builds an additional layer on top of Parquet files that let's you do ACID transactions, rollbacks, and schema evolution. A Parquet file is a static file that has the whole data associated with a table. You can't insert, update, delete, etc. It's just it. It works ok if you have small tables, but it becomes unwieldy if you need to do whole-table replacements each time your data changes. Apache Iceberg fixes this problem by adding a metadata layer on top of smaller Parquet files (at a 300,000 ft overview).
- pgwhalen 2y agoI knot you’re not OP, but and while this explanation is good, it doesn’t make sense to frame all this as a “problem” for parquet. It’s just a file format, it isn’t intended to have this sort of scope.
- hobs 2y agoThe problem is that the "parquet is beautiful" is extended all the time to pointless things - pq doesn't support appending updates so let's merge thousands of files together to simulate a real table - totally good and fine.
- inkyoto 2y agoWell… when Parquet came out, it was the first necessary evolutionary step required to solve the lack of the metadata problem in CSV extracts. So, it is CSV++ so to speak, or CSV + metadata + compact data storage in a singular file, but not a database table gone astray to wander the world on its own as a file.
- victor106 2y ago> Apache Iceberg builds an additional layer on top of Parquet files that let's you do ACID transactions, rollbacks, and schema evolution. Delta format also supports this, correct?
- orthoxerox 2y agoCorrect. They have feature parity, basically.
- yodon 2y agoCan someone Eli5 the difference between AWS S3 Tables and AWS SimpleDB?
- nattaylor 2y agoS3 Tables is designed for storing and optimizing tabular data in S3 using Apache Iceberg, offering features like automatic optimization and fast query performance. SimpleDB is a NoSQL database service focused on providing simple indexing and querying capabilities without requiring a schema.
- alex_smart 2y agoThey are so completely different that it would be simpler if you explained what similarities you see between the two.
- isjustintime 2y agoThis is pretty exciting. DuckDB is already proving to be a powerful tool in the industry. Previously there was a strong trend of using simple S3-backed blob storage with Parquet and Athena for querying data lakes. It felt like things have gotten pretty complicated, but as integrations improve and Apache Iceberg gains maturity, I'm seeing a shift toward greater flexibility with less SaaS/tool sprawl in data lakes.
- RobinL 2y agoYes - agree! I actually wrote a blog about this just two days ago: May be of interest to people who: - What to know what DuckDB is and why it's interesting - What's good about it - Why for orgs without huge data, we will hopefully see a lot more of 's3 + duckdb' rather than more complex architectures and services, and hopefully (IMHO) less Spark! https://www.robinlinacre.com/recommend_duckdb/ https://www.robinlinacre.com/recommend_duckdb/ I think most people in data science or data engineering should at least try it to get a sense of what it can do Really for me, the most important thing is it makes it so much easier to design and test complex ETL because you're not constantly having to run queries against Athena/Spark to check they work - you can do it all locally, in CI, set up tests, etc.
- yakshaving_jgt 2y agoFunny, I read TFA and came to the comments to share exactly this recent blog post of yours. Big fan of your work, Robin!
- RobinL 2y agoAh nice - reading that made me feel good! Appreciate the feedback!
- hn1986 2y agofrom the blog: "This is a very interesting new development, making DuckDB potentially a suitable replacement for lakehouse formats such as Iceberg or Delta lake for medium scale data." I don't think we'll ever see this, honestly. excellent podcast episode with Joe Reis - I've also never understood this whole idea of "just use Spark" or you gotta get on Redshift.
- AlecBG 2y agoDoes this support time travel queries? Does it support reading everything from one snapshot to another? (This is missing in Athena) If yes to both, does it respect row level deletes when it does this?
- whinvik 2y agoWhen is write support for iceberg coming?
- dm03514 2y agopfsh who needs to write data??? ;p If you have streaming data as a source, I built a side project to write streaming data to s3 in iceberg format: https://sql-flow.com/docs/tutorials/iceberg-sink https://sql-flow.com/docs/tutorials/iceberg-sink https://github.com/turbolytics/sql-flow https://github.com/turbolytics/sql-flow I realize it's not quite what you asked for but wanted to mention it. I'm surprised at lack of native iceberg write support in these tools. Pyiceberg though was quite easy to use, arrow-based API was very helpful as well.
- whinvik 2y agoThanks. This looks cool. However, my issue is the need to introduce one more tool. I feel that without a single tool to read and write to Iceberg, I would not want to introduce it to our team. Spark is cool and all but it requires quite a bit of effort to properly work. And Spark seems to be the only thing right now that can read and write to Iceberg natively with a SQL like interface.
- jaychia 2y agoCheck out Daft (www.getdaft.io) - we've been working really hard on our Iceberg support. Supports full reads/writes (including partitioned writes) and our SQL support is also coming along quite well! Also no cluster, no JVM. Just `pip install daft` and go. Runs locally (as fast as DuckDB for a lot of workloads; faster, if you have S3 cloud data access) and also runs distributed if you have a Ray cluster you can point it at (Disclaimer: I work on it)
- jamesblonde 2y agoDaft is making great progress with Iceberg - faster than PyIceberg in many ways.
- sys13 2y agoWonder why not Delta Lake instead, since Iceberg will merge with Delta
- dm03514 2y agoI've mentioend this whenever iceberg comes up. It's wild how immature the ecosystem is still. Duckdb itself lacks the ability to write iceberg.... https://duckdb.org/docs/stable/extensions/iceberg/overview.html#limitations https://duckdb.org/docs/stable/extensions/iceberg/overview.h... Apache iceberg go ? Nope https://github.com/apache/iceberg-go?tab=readme-ov-file#readwrite-data-support https://github.com/apache/iceberg-go?tab=readme-ov-file#read... Basically java iceberg is the only mature way to do this, it's not a very accessible ecosystem. For a side project I'm using pyiceberg to sink streaming data to iceberg (using DuckDB as the stream processor): https://sql-flow.com/docs/tutorials/iceberg-sink https://sql-flow.com/docs/tutorials/iceberg-sink It's basically a workaround for DuckDB's lack of native support. I am very happy with the Pyicerbg library as a user, It was very easy and the native Arrow support is a glimpse into the future. Arrow as an interchange format is quite amazing. Just open up the iceberg table and append Arrow dataframes to it! https://github.com/turbolytics/sql-flow https://github.com/turbolytics/sql-flow Arrow is quite spectacular and it's cool to see the industry moving to standardize on it as a dataframe. For example, Clickhouse python also support arrow-based insertion: https://sql-flow.com/docs/tutorials/clickhouse-sink https://sql-flow.com/docs/tutorials/clickhouse-sink This makes the glue code trivial to sink into these different systems as long as arrow is used.
- hn1986 2y agotracking write support here: https://github.com/duckdb/duckdb-iceberg/issues/37 https://github.com/duckdb/duckdb-iceberg/issues/37
- barrenko 2y agoWhat the hell is iceberg now?
- Mortiffer 2y agoI came to the same conclusion and moved on. We had some c# applications reading some python
- ramraj07 2y agoIs there a reliable way to convert an existing Parquet directory to iceberg without moving the data?
- _atyler_ 2y agoThis is a great example of how simplicity often wins in practice. Too many systems overcomplicate storage and retrieval, assuming every use case needs full indexing or ultra-low latency. In reality, for many workloads, treating S3 like a raw table and letting the engine handle the heavy lifting makes a lot of sense. Curious to see how it performs under high concurrency—any benchmarks on that yet?
- TheGuyWhoCodes 2y agoDoes DuckDB just delegate the query to S3 Tables? or does it do anything in-engine with the data files? On thing that's missing in DuckDB is predicate pushdown for iceberg - see https://github.com/duckdb/duckdb-iceberg/issues/2 https://github.com/duckdb/duckdb-iceberg/issues/2 Which puts it way behind the competition, performance wise.
- margorczynski 2y agoLooks like they're going the route of Starrocks? https://www.starrocks.io/ https://www.starrocks.io/ Basically decoupling the file/data storage from the distributed computation layer.
- jamesblonde 2y agoThat is exactly what the Lakehouse is about - decoupling storage (Iceberg, Delta, Hudi) from query engine.
- rubenvanwyk 2y agoWow, DuckDB continues to be the MVP.
- deleted 2y ago[deleted]