4 ms·
A Preview of DuckDB v2.0
- c9cf35860db4 1mo agoThe last year of DuckDB enhancements feel like the shift from in-process execution engine (which it is phenomenal at) to an engine that can serve as the foundation of a cloud data warehouse. I know the founders were reticent about not wanting to build that, but I have a feeling it is in the works.
- jbmsf 1mo agoI've been using MotherDuck as a data warehouse for some time now. No regrets.
- ryguyrg 1mo agoRyan Boyd, co-founder at MotherDuck here. love to hear it- thanks for sharing :-)
- ethagnawl 1mo agoUPDATE: Removed uninformed comment about the relationship between DuckDB and MotherDuck.
- Tomte 1mo agoWait, what? I‘ve always assumed MotherDuck is the company by the main developers of DuckDB? They are unrelated?
- ethagnawl 1mo agoI was wrong. Apologies. > They decided to work together, and DuckLabs would become a co-founder of the new endeavor. https://motherduck.com/about-us/ https://motherduck.com/about-us/
- jeremyjh 1mo agoThey launched ducklake: https://ducklake.select/ https://ducklake.select/
- ethagnawl 1mo agoYeah, that was one of the sources of my confusion, too. Some of the recent DuckDB features and extensions would certainly seem to undercut premium features MotherDuck has layered over DuckDB. Though, on the flip side, since there is a relationship between the orgs it does seem like MotherDuck features are a good way for the DuckDB team to field test (future) features without committing to building extensions or adding them to the database itself (e.g. DuckLake and Quack). It is all a bit curious, though, and makes you wonder what's in store for the future. The DuckDB team has hinted at some "big things" coming soon which I don't think are covered in these release notes.
- ryguyrg 1mo agoRyan Boyd, co-founder at MotherDuck here. Yep, we've worked closely with the team, but are separate companies! We're focused on cloud data warehousing and customer-facing analytics. Let me know if you have any other questions!
- markhalonen 1mo agoWas hoping to see procedural functionality like PL/pgSQL... regardless, an astonishing project overall.
- srameshc 1mo agoI <3 DuckDB. It has become one of my go to tools for storing, data processing , integrations and now even graph. More importantly it's fun to use because it is so portable. Looking forward to v2.
- fragebogen 1mo agoDitto! Very happy with the upcoming async support! Now it'll be a nice little db for serving http traffic as well!
- d3Xt3r 1mo agoWhat advantages does it have over SQLite in your use cases? Can you give any examples?
- briHass 1mo agoNot OP, but for me, the lack of essentially any type system in SQLite makes it a total no-go for storing data long-term or that more than one application needs to access. Date/time being an especially painful footgun in SQLite. I view SQLite as something a single application can use for storing state/settings/misc operational data instead of directly writing files, especially if the data being stored is relational or needs ACID. As soon as the data itself has meaning and structure per se, you're better off with something that can help enforce and describe the data: rich datatypes, foreign keys that aren't optional, etc.
- TheFlyingFish 1mo agoDuckDB is heavily focused on analytical workflows, so it stores data in a compressed columnar format. So "rewrite one row" is more efficient in SQLite, but "sum all the values in this column" is vastly more efficient in DuckDB. That's the core of the difference, to me. But the projects also have a pretty different ethos. SQLite has always prioritized simplicity, often at the expense of features, whereas DuckDB's approach to features seems to be "the more the merrier." E.g. the star feature of this new release, client/server mode. Plenty of people would love to see something similar in SQLite, but it doesn't seem likely.
- est 1mo agoThis is cool What about the runtime size? I care this because I intend to run a stripped WASM version of DuckDB in browser.
- zcw100 1mo agoWhat do you plan on stripping and what's your target? The Emscripten based build is ~10Mb. I have a component build so I'd be interesting on how you'd like to break it up.
- datadrivenangel 1mo agoThey're mostly using optional extensions for this new stuff, so the binaries are still small, like ~20mb?
- jtbaker 1mo agoDuckDB is one of the things I've been most excited about in a long time. Introduced it to projects at 3 companies since 2023, greatly lowering resource requirements and running it in a variety of environments. Just having the ability to do out of core bigger than memory data processing on lower end consumer grade hardware is remarkable. Thanks to the team for everything!
- throwaw12 1mo agoCurious to learn more about how people are using it? Are they downloading parquet files and running analyses locally, or are they connecting to Iceberg-like data lake and leveraging DuckDBs query engine capabilities or have you exposed an interface (REST, UI) to query your data?
- arealaccount 1mo agoWe use DuckDB WASM with parquet to build dashboards in-browser. It's cool to be able to write SQL directly in a browser and not have to rely on REST/Graphql/etc to access the data layer.
- jayct 1mo agocurious if you're using something mostly-out-of-the-box to layer on visualizations for your dashboards? relatively new to duckdb, love it so far, looking at alternatives for downstream visualization. so far just exporting datasets and piping into python scripts.
- mediaman 1mo agoI do something similar and just use echarts. Very happy with it.
- jtbaker 1mo agoFor a schema-first (vs. code first) approach (which I think would be a sweet spot for agent driven dashboarding), I'd suggest looking at https://vega.github.io/vega-lite/ https://vega.github.io/vega-lite/ or https://vega.github.io/vega/ https://vega.github.io/vega/. A little higher level than full D3 but gives you a little higher level approach.
- sv123 1mo agoDuckDB is so cool, game changer when it comes to local data processing.
- hnlb53nrpg 1mo agoSame problem, different day
- anentropic 1mo agoPlease document the new "extensible PEG-based parser" for extension authors
- badatnames 1mo agoLooks like an awesome release, but the smell of AI from that post is horrid. Here is a wild idea: is it really so hard to edit out sentences structured and punctuated like this - it's so painfully obvious and distracts from the content. The effect is real.
- drannex 1mo agoI don't really get an AI smell on this, in fact I see multiple parts that an AI would have corrected - grammatical issues, personal writing quirks, etc. Writing similar to this: is quite a normal way of writing for technical articles - especially when you are trying to make the point clear and well organized.
- ostwilkens 1mo ago"A major version bump is not something we do lightly, and it is not just ceremony" For me, this sentence pops out like a sore thumb
- jeffbee 1mo ago"We reimplemented ICU" U+1F631 FACE SCREAMING IN FEAR
- amluto 1mo agoI’m currently contemplating that MySQL apparently cannot do an INSTANT change of the collation of an unindexed column, even though, AFAICT, it has no effect whatsoever on the on-disk format or any data structure at all except for the metadata saying what the column type is. I do not enjoy dealing with text encodings and collations in databases.
- logancbrown 1mo agoFunny to think one of my favorite software projects this decade is basically "lets make it easy to host your own OLAP database".
- amluto 1mo agoIf I could have a pet feature added to DuckDB, it would be some form of native ordered table. In a database like Clickhouse or any of the dedicated time series DBMSes or log stores, there’s a built-in concept that a table might have an order, and the database will optimize based on the order. But, for databases that are logically just bags of rows (traditional DBMSes and also DuckDB [0]), you either need an index or you need to rely on full table scans or at least scans of big blocks. DuckDB does the latter really well, but I think it would be quite nice for some workflows to have explicit ordering. Also, I bet compression could work a lot better with ordering hints. All that being said, I’m quite excited about DuckDB 2.0. I want to give the improved VARIANT support a try. [0] Documentation on DuckDB’s native format is rather sparse AFAICT. But the DDL has nothing resembling an ordered table.
- ignaciovdk 1mo agoArc does pruning, and make that scan faster. Check it out: https://github.com/Basekick-Labs/arc https://github.com/Basekick-Labs/arc
- tpetry 1mo agoYou‘re not the only one interested in this. But seems its a big change that would have to change many parts within DuckdB: https://github.com/duckdb/duckdb/discussions/8444 https://github.com/duckdb/duckdb/discussions/8444
- amluto 1mo agoNeat! I’m sort of amazing that nothing in that thread mentioned ClickHouse, even just as a product that has this capability and from which one might take syntactical inspiration.
- aleda145 1mo agoExcited about a stable C++ API for extensions! I made a dry run extension a few months ago (https://github.com/aleda145/duckdb-dryrun https://github.com/aleda145/duckdb-dryrun), will be so nice to build it just once and know that it will always work. Also urge anyone to make an extension, the template makes it quite smooth: https://github.com/duckdb/extension-template https://github.com/duckdb/extension-template
- otter-in-a-suit 1mo agoSuper excited about Quack (partially due to the name). I use duckdb for both analytics and runtime, but I do have to serve/handle/manage a giant, multi-GiB duckdb file as effectively a runtime artifact[1]. I'm aware that this isn't the _perfect_ database for this, but the mix of it being fast, having spatial support, sane coding interfaces, great dbt integration, and me being able to do everything between "run a giant several hundred step dbt pipeline" to "query the output of said pipeline" to "read/query a csv on disk" with the exact same tool is just so nice. If I could centrally manage said asset more akin to a traditional database, I'd be very happy. I've partially solved this with separate databases for different steps in the data pipeline(s) and have even experimented with Clickhouse as a complete alternative, but I really like way too many things about duckdb to replace it. [1]: If you care: https://skaldmaps.com/blog/2026/07/zip-codes-are-a-bad-spatial-abstraction/#a-side-note-duckdb- https://skaldmaps.com/blog/2026/07/zip-codes-are-a-bad-spati...
- mediaman 1mo agoI built a platform for some midsize companies in a specific vertical that is basically a data warehouse with some LLM-driven dashboarding and query tools on top. Typical data size 5-150gb. So I built a service layer around duckdb, where each tenant gets their own duckdb. I'm also in the boat of knowing that duckdb is not the perfect solution for this (the classic use case is running it against local data on a laptop), but there's so much I like about it, and it's really nice that each tenant can have total separation with custom schemas and that it's straightforward to pile data into object storage and form your own lake. And now with things like quack the sharp points around concurrency are relaxing, and it feels like the compromises of using it this way are disappearing.
- tccole 1mo agoI feel like familiarity and ease of use and “good enough” beat out the perfect db for the job in many occasions.
- allan_s 1mo agoAs somebody new to this and with a use case very similar to yours , what would have been a more suitable solution for this ? The guy who first built the architecture made the same decision as yours (I.e one local duckdb for each tenant to work as a copy of big query/their data warehouse) and I dont know what the state of the art for this kind of use cases ?
- drannex 1mo agoReally looking forward to that new Async system, especially when reading/querying against thousands of parquet files. This is going to monumentally affect me and my work - I have to query against millions of massive parquet files and the speed has already been rather wonderful, but if those metrics are to be even 100% in range, this is going to make life so much better. DuckDB is seriously an incredible utility.
- dcmerritt 29d ago[flagged]
- giovannibonetti 1mo agoDisappointed, since I was expecting they would rewrite the implementation from C++ to Zig. I bet that would increase the number of positive pull requests they get, since most developers prefer to stay away from C++ nowadays.
- brunoborges 1mo agoHow does DuckDB compares with PostgreSQL / MariaDB ?
- tuvix 1mo agoDuckDB is an in-process (now I guess less so with Quack) OLAP database for analytical workloads. PostgreSQL or MariaDB are OLTP row-oriented databases that are great for application/transaction-focused workloads but are less great when you want to query across a giant amount of data.
- tomjakubowski 1mo agoDuckDB is much like SQLite, but built for OLAP workloads: it's in-process, with a single file format on a disk, and (unlike SQLite) the data is stored in columns for better OLAP performance. Like SQLite, concurrency options are limited compared to client-server databases like Postgres or MariaDB. DuckDB 2.0 will be adding a client-server mode with the Quack protocol which can allow for greater concurrency. But you can also use DuckDB as a multitool to connect to and query all kinds of other data sources from one connection. Being able to pull in data from Postgres, CSVs or parquets on a file system or S3, and JSON returned by a web API, and then query across all of it in one place, can be quite handy for ad-hoc data analysis and exploration.
- thejosh 1mo agoI've been working on a demo database project, and have been really impressed by the UI. So glad they decided to put more effort into it, it has made building a "follow along" tutorial really nice.
- dzonga 1mo agowell done to the duckDB team - one of the features I'm waiting for is real time materialized views.
- d33 1mo agoAre there improvements in how memory_limit works? I often had DuckDB get OOM killed because it went beyond its limit. It's definitely one of the reasons why I usually have an AI tune the environment for my datasets.
- noodlesUK 1mo agoI look forward to DuckDB being the engine that underpins the next gen of analytical data tooling. DuckLake already looks amazing and with the Quack protocol seems like it will be a great natural fit for lots more types of tooling, such as sensor data etc.
- dartharva 1mo agoWould be really cool if they were to add statistical functions too. I'd jump at the chance of getting to use this over pandas
- biophysboy 1mo agoYou might know this already, but you can query pandas/polars/arrow tables directly w/ duckdb and use whatever stats packages you feel like alongside it in the same python script. I feel like they do a decent job sticking to the simpler statistical fans that make sense in sql.
- dartharva 1mo agoSorry, I meant duckdb-cli, not duckdb in Python.
- 1egg0myegg0 1mo agoOh, then you might like these DuckDB community extensions that provide stats functions! https://duckdb.org/community_extensions/extensions/stats_duck https://duckdb.org/community_extensions/extensions/stats_duc... https://duckdb.org/community_extensions/extensions/stochastic https://duckdb.org/community_extensions/extensions/stochasti...
- dartharva 1mo agoWill check them out, thanks!
- swasheck 1mo agowhich statistical functions do you want? i’m curious because i love duckdb and use it for a variety of projects but always want to learn more about how to use tools better.
- dartharva 1mo agoBasic regression and interpolation for starters; perhaps some classification and outlier detection too. Disclaimer: I self-studied applied DuckDB for my job, I don't know if these already exist.
- orthoxerox 1mo agoDuckDB keeps getting better and better. I wonder when something like Apache Gluten will pick it up as a backend.
- formerly_proven 1mo agoIn terms of project trajectory this is also an interesting contrast. DuckDB is "SQLite, but for OLAP". DuckDB 1.0 was in 2024. DuckDB 2.0 (new API, new storage format, new ...) is in 2026. SQLite has been 3.x since 2004.
- valentynkit 1mo ago[dead]
- remywang 1mo agoIf you like DuckDB, please consider funding DB research [1]! [1]: https://news.ycombinator.com/item?id=49336147 https://news.ycombinator.com/item?id=49336147
- cheema33 1mo agoWith some of these changes, it appears to be encroaching on clickhouse territory. Or are they still very different products?
- te_chris 1mo agoWell, I can tell you this is making me actively reconsider dropping duckdb for ch as we go from prototype to prod
- encoderer 1mo agoDefinitely encroaching. Our last product (SaaS observability) uses Clickhouse. Our next product (self-hosted observability) uses DuckDB.
- adhamsalama 1mo agoI was building an OpenTelemetry observability platform in a single executable file (in Golang) and was using ClickHouse as the database, and replaced it with an embedded DuckDB, so now it's truly a single executable file platform. https://github.com/adhamsalama/nabatshy https://github.com/adhamsalama/nabatshy
- encoderer 1mo agoThis looks solid! We are also going for a single-executable (typescript/bun in our case), but we are specifically going after agent monitoring in the first version. We think people shouldn't have to send all of their agent conversations to a vendor like DataDog or Sentry.
- ChillyCapy 1mo agoGreat work! I built a browser tool for querying local Parquet, CSV, JSON, Excel, Arrow, Avro, DBF, and SQLite files with DuckDB-WASM. Most probably after DuckDB v2.0 release I will revamp my tool as well.
- rrgok 1mo agoSometimes I prefer DuckDB query language to MySQL or Postgres. It is not even about performance, it just feels right. I just connect to my DBS from DuckDB just to use its query language. And besides it gives you a single language frontend.
- onnimonni 1mo agoThis! For example I tried to ask PosgreSQL to Add support for QUALIFY keyword since it makes window queries much more simpler to use. Postgres didnt want it. DuckDB maintainers are always thinking about the DX and all nice syntactic sugar.
- hans_castorp 1mo agoThe Postgres devs rejected it, because it wasn't part of the standard back then, and apparently work was going on, to include it into the standard. Now that it's included in the standard, there is work to implement it: https://hackorum.dev/topics/51960#message-1125429 https://hackorum.dev/topics/51960#message-1125429
- rrgok 1mo agoOr the fact I can join between Excel, MySQL and Postgres. That absolutely a killer feature.
- HawtAds 1mo agoHow stable is DuckDB these days? I remember it was very buggy compared to Clickhouse.
- nikolay 1mo agoIt's sad that almost no migration framework supports DuckDB, and overall support is highly limited, but it's a great product I've been using daily for 6 months without encountering a single issue. I hope v2 brings DuckDB more attention and increases third-party support!
- maxman88 1mo ago[flagged]
- barrrrald 1mo agoWe've built our whole platform around DuckDB at Hex. Our product truly could not exist without it.
- therealdrag0 1mo agoHate to bring it up, but 10,000 commits in less than 6 months is a lot. Is AI a major contribute here? Is AI use for accelerated development of a beloved tool like DuckDB enough to quiet lingering doubters?
- nh2 1mo agoIf you merge PRs that have commit mesages like this, it's easy to arrive at 10000 commits in 6 months: rename to NodePointer instead format Revert "format" Revert "rename to NodePointer instead" rename to OptionalNodePtr woops update comment slot renames more renames Source: https://github.com/duckdb/duckdb/pull/23605 https://github.com/duckdb/duckdb/pull/23605 If every Ctrl+S is a commit, it'll go up fast. "woops"!
- Mic92 1mo agoThey seem to squash PRs using the pr description as commit message. So this sounds like a good approach to me.
- oefrha 1mo agoThey don’t. These are all on-tree commits behind a merge commit.
- mastermage 1mo agoi love woops
- luizfelberti 1mo ago> A repository is a name, a URL prefix, and one or more RSA public keys that are trusted to sign the extensions served from it. Is it too late to beg Hannes and Mark to let us have something like minisign instead of RSA? :) Very excited for 2.0, congrats folks!
- dangoodmanUT 1mo agoIt's funny to me that we still don't have incremental materialized views. All of the parts are there (export state, agg_state (forget fn name), finalize). I wonder if they're avoiding an explicit war with clickhouse or something. I do recall they mentioned they want to add this to ducklake. Incremental MVs are ClickHouse's best feature. If DDB adds this, the last moat is distributed query execution.
- ghshephard 1mo agoHave you run into scenarios where a simple view doesn't accomplish what you require? I always feel like views do everything I want - because the speed is so great, the full recompute isn't that big a deal. Maybe it's a bigger deal when you have multiple users/and or more repeated queries against something that's really expensive?
- Pedro_Ribeiro 1mo agoI've had some use cases with low latency requirements and frequent data updates. If you want to serve data on a frontend fast it's often not feasible to recompute everything every time
- hantusk 1mo agoIt is implemented here as DuckDB extension: https://github.com/ila/openivm/ https://github.com/ila/openivm/
- dm03514 1mo ago<3 duckdb run realtime analytics pipeline using a (moderately popular) stream processing engine I built on top of DuckDB. Looking forward to what duckdb provides in terms of perf out of the box! https://github.com/turbolytics/sql-flow https://github.com/turbolytics/sql-flow DuckDB has been a fantastic engine to build on (in python), and processes thousands of events per second, day in an day out, without issue
- andyferris 1mo agoI was curious to see they are advertising OLTP-like transactional processing speed. It would be super convenient to have one DB for OLTP and OLAP purposes! Has anybody here tried using it that way? (though I don't see any way to deal with write skew and other transactional guarantees - no SERIALIZABLE optimisitic concurrency, no SELECT FOR UPDATE pessimistic concurrency, etc)
- tizerluo 1mo ago[flagged]
- gw32 1mo ago> The VARIANT type shipped in DuckDB v1.5, and the way to think about it is JSON on steroids. Basically, imagine if JSON were fast. [...] DuckDB automatically detects the common structure hidden in your semi-structured data and “shreds” it, so it compresses well in storage I am really looking forward to this hitting v2.0. I can't stand uncompressed JSON - so space-inefficient. But heterogenous JSON in parquet files is such a pain because of schema differences causing fields to be silently dropped. Having DuckDB solve this is exactly what I've been looking for.
- raihansaputra 1mo agoi know they're not making any promises but i'm also very excited for it
- dcmerritt 29d ago[dead]
- fenestella 1mo ago[flagged]
- deleted 1mo ago[deleted]
- jinjin2 1mo agoI’m mostly using Exasol these days (the concurrency and smooth scaling to multi-node is just too seductive), but with the introduction of Quack I might take another look at DuckDB. I’ll have to see how well it handles many agents reading and writing to it concurrently.
- ivolimmen 1mo agoI am a crew member of Joy Of Coding (https://joyofcoding.org https://joyofcoding.org) where we invited Hannes to do a talk. He is a great speaker. Seeing this we will need to invite him another time!
- mastermage 1mo agoI love DuckDB genuinely more than sqlite even though they do completely different things but DuckDB has like for me the perfect mix between simplicity, embedded capabilities and expressiveness. (and actual Time and Date Types).
- MILP 1mo agoLove DuckDB. It’s so fast and portable! I mainly use the query engine as part of my ETL process for creating SQLite database. I’d love to pitch it at work, but we’re heavily invested in BigQuery, which makes it a bit difficult.
- rubenfiszel 1mo agoWe have bet early on DuckDB and Ducklake for Windmill and couldn't be happier. The focus on server/client mode is interesting, it opens the way for orchestrators like ours to have "lite" workers/jobs for duckdb that connect to one central bigger beefy nodes and improve the overall efficieny. I'm very curious if benchmark shows that there are performance benefits to do so thanks to co-location and overall less cpu cycles wastes.
- zachwill 1mo agoBig fan of Windmill here! Use it heavily. The way you guys integrate seamlessly with Bun/DuckDB/Postgres/Python is just fantastic. Keep up the great work, Ruben!
- hermitcrab 1mo agoI'm looking for a lightweight client-server database where I can connect 3 or 4 GUI clients to a single database and concurrently edit the database. Low transaction volumes (probably a few edits per minute). Would DuckDb + Quack be suitable?
- piokoch 1mo agoWhy not MySQL/MariaDB? DuckDB and Quack might be new and flashy, but sooner or later you will need things like connection pooling, data retention, backups, replication. In a real setup there is nothing lightweight about DuckDB as it will have to answer the same issues other DB-s do. If you want to go with something less mainstream, why not trying Firebird DB, a very solid solution.
- hermitcrab 1mo agoI hadn't heard of MariaDB. IIRC Firebird is declining in popularity and I really want something that is still going to be supported in 10 years time.
- christoff12 1mo agoI love reading all of the DuckDB love here. It's good software.
- Kvarnek 1mo agoThe single-writer limit was always the awkward part. Quack finally addresses it.
- ChristmasTomer 1mo ago[dead]
- suralind 1mo agoThat feature set is crazy! I absolutely love what you guys did. Thanks for the hard work!
- maxman88 1mo ago[flagged]
- deleted 29d ago[deleted]