13 ms·

DuckDB – An in-process SQL OLAP database management system

161 points by freilanzer 4y ago

samwillis 4y ago
DuckDB is awesome! Last year I was working on something using SQLite, users could perform analytical queries that would scan the entire 3gb db and generate aggregates. It would take at least 45 seconds to do the queries. I did a dump of the db and imported to DuckDB. The same queries now only take 1.5 seconds with exactly the same SQL. Obviously there is a trade off, inserts are slower on DuckDB. But for a low write, analytical read app it's perfect. I tried to use the SQLite connector but struggled to get it working. Need to circle back and have another go. With their SQLite and Postgres connectors, as a Django dev I would love an app that lets you run specific queries on your DB via DuckDB almost transparently. Would be awesome for analytical dashboards.
- mcdonje 4y ago
  Yeah, I see it as the sqlite of the reporting world. Especially with columnar stores and parquet support.
- dunefox 4y ago
  I'd need a MySQL/MariaDB connector for it to be really useful for me. But still, it looks great.
bluehat974 4y ago
DuckCon 2023 - State of the Duck - Mark Raasveldt and Hannes Mühleisen https://youtu.be/rdnPkLSkoyU https://youtu.be/rdnPkLSkoyU
lichtenberger 4y ago
I wonder if we'll in the near future also see JVM based in-memory or hybrid OLAP database systems, which will make use of SIMD instructions and columnar storage layouts with the incubating Vector API. It would be also interesting to see how we can process semi-structured data in a similar way.
d_watt 4y ago
I'm duck-curious. Looking at how it's deployed, as an in process database, how do people actually use this in production? Trying to figure out where I might actually want to think about replacing current databases or analyses with DuckDB. EG if you deployed new code 1. Do you have a stateful machine you're doing an old school "Kill the old process, start the new process" deploy, and there's some duckdb file on disk that is maintained? 2. Or do you back that duckdb file in some sort of shared disk (Eg EBS), and have a rolling deploy where multiple applications access the same DB at the same time? 3. Or is DuckDB is treated as ephemeral, and you're using it to process data on the fly, so persisted state isn't an issue?
- dunefox 4y ago
  Good questions, I'm thinking about evaluating it along with pola.rs as data science tools. Would be interesting to hear from someone using it already.
- romski 4y ago
  Yes using it in production - Stateless and ephemeral. For sure there’s a learning curve.
  - d_watt 4y ago
    I see. Would it be fair to say you treat it almost like Pandas, except that it has a lower memory footprint since data is written to disk instead of memory. IE you use it for on the fly analysis of large frames of data, not like more traditional database/datawarehouse?
    - chrisjc 4y ago
      I'm not sure they're saying that. BTW, your questions are exactly those that I've been ask over the last few months, but also with a lot of focus over the last few days. Still learning as much as I can so the following might not be true. For what it's worth, there's a difference between using duckdb to query a set of files vs loading a bunch of files in to a table. But once the data has been loaded into a table it can be backed up as a duckdb db file. Therefore it might be more performant to preprocess duckdb db files (perhaps a process that works in conjunction with whatever manages your external tables) and load these db files into duckdb as needed (on the fly analysis) instead of loading datafiles into duckdb, transforming and CTAS every time. https://duckdb.org/docs/sql/statements/attach https://duckdb.org/docs/sql/statements/attach Of course all of this might be introducing more latency esp if you're trying to do NRT analytics. I assume you could partition your data into multiple db files similar to how you would probably do it with your data files (managing external tables).
lopkeny12ko 4y ago
I appreciate the clarity on the explicitly unsupported use cases in "When to not use DuckDB." There are so many infrastructure products, especially database products for some reason, where the marketing team takes control of the messaging away from engineers, and push outlandish claims on how their new DB is faster than all the competition, can support any workload, can scale infinitely, etc.
SnowflakeOnIce 4y ago
DuckDB is terrific. I'm bullish on its potential for simplifying many big data pipelines. Particularly, it's plausible that DuckDB + Parquet could be used on a large SMP machine (32+ cores and 128GB+ memory) to deal with data munging for 100s of gigabytes to several terabytes, all from SQL, without dealing with Hadoop, Spark, Ray, etc. I have successfully used DuckDB like above for preparing an ML dataset from about 100GB of input. DuckDB is undergoing rapid development these days. There have been format-breaking changes and bugs that could lose data. I would not yet trust DuckDB for long-term storage or archival purposes. Parquet is a better choice for that.
- RobinL 4y ago
  I'd love to hear any real world experiences of anyone who's tried to run jobs that would usually require a spark cluster on a single machine with loads of cores and memory. How big can you go, and how does speed compare to Spark? (I'm guessing significantly faster from my experience using Duckdb on smaller machines)
  - jakewins 4y ago
    I used it as glue for a monthly job, last week, processing ~1B event records, juggling them a bit and pushing them as 2M parquet segment files to S3. Mixed experience, would definitely not put it in a system that isn’t ok crashing frequently. It segfaults on Alpine (argh, C++!), and force exits the whole NodeJS process when it gets unexpected HTTP responses from S3. In an archive run of 2M pushes it’ll crash the process 4-5 times. Overall still really, really like it, but learned to not trust it
    - tracker1 4y ago
      Node + AWS for data ingress has been pretty painful in my experience (mostly dynamo feeds from large csv (whois database)). In the end, rewrote in C# (core 2) and it was able to complete more reliably. I'm guessing that go and rust would also be better. I like node, really like JS, but I just think that maybe the AWS libraries aren't that great in the space. It would run for 3-5 hours, then just blow up unexpectedly, even with plenty of memory overhead, and not really bandwidth limited, with appropriate retries and slowdown for dynamo rejections. If I never have to write ETL pipelines again, I won't be upset about it.
vgt 4y ago
DuckDB is such a breath of fresh air! We at MotherDuck at working very closely with the DuckDB folks to build a DuckDB-based cloud service. I'm talking to various folks in the industry about the details in 1:1. Feel free to reach out to tino at motherduck.com. (co-founder and head of Produck at MotherDuck)
jengleton 4y ago
Great to see this posted here! DuckDB is an integral part of an in-browser data analytics tool that I've been working on. It compiles to WASM and runs in a web worker. Queries against WASM DuckDB regularly run 10x faster than the original JavaScript implementation!
- thenaturalist 4y ago
  In case you're interested in or willing to share more details about the tool or your architecture I'd be an interested reader.
- hummus_bae 4y ago
  It's amazing to see Rust used so much even in web projects. It's my favorite language and I don't want to use it for web programming anymore. I've used it for too many "real" web apps (I mean webrtc signaling and web socket) to go through the pains of optimization. But it's still fun to work with.
out_of_step 4y ago
DuckDB has an awesome integration with Apache Arrow and very useful implementations in R and Python. Great for easy larger-than-memory analyses https://duckdb.org/2021/12/03/duck-arrow.html https://duckdb.org/2021/12/03/duck-arrow.html
bwanab 4y ago
There's a nice intro to DuckDB for julia developers by Bogumił Kamiński, the creator of DataFrames.jl here: https://juliazoid.com/welcome-to-duckdb-3c4e75f50b97 https://juliazoid.com/welcome-to-duckdb-3c4e75f50b97. Interesting since in some ways, as he points out, it's in direct competition with DataFrames for use cases, but he gives it a very positive treatment and shows how they can work together using advantages of standard SQL along with processing power of DataFrames.
- nickpeterson 4y ago
  When someone gives fair opinions on something that directly competes were their own work, you should take their opinion very seriously. It’s an excellent quality in a person, and shows they’re more focused on the problem than their ego.
  - phillc73 4y ago
    Quite a while ago, when duckdb was just a duckling, I wrote an R package that supported direct manipulation of R dataframes using SQL.[1] duckdb was the engine for this. The approach was never as fast as data.table but did approach the speed of dplyr for more complex queries. Life had other things in store for me and I haven’t touched this library for a while now. At the time there was no Julia connector for duckdb, but now that there is, I’d like to try this approach in that language. [1] https://github.com/phillc73/duckdf https://github.com/phillc73/duckdf
ctc24 4y ago
We're big fans of DuckDb at https://prequel.co https://prequel.co! We use it as part of our own dataframe implementation in Go. The speed is unbeatable and the tool is top notch. There are a few rough edges (it's not quite 1.0 level of stability yet), but the team is super reactive and has fixed bugs we've reported in < 48hrs pretty much every time.
kiratp 4y ago
Its disappointing that C++ was chosen to build something that is going to live in-process. Rust would have been so much safer. All the segfaults you get when running DucDB supports this statement.
- mcdonje 4y ago
  I agree about Rust's memory safety advantage over C++, but I disagree that it's disappointing from a project perspective. Some DB experts made a good DB using a performant language they're comfortable with. You can't make project choices in a vacuum, and you can't assume others can either. People have limited time. The choice they were facing was probably not C++ vs Rust, but C++ vs nothing because they didn't have time to learn a new language before starting their project. Also, their first release was in 2019, so they probably heard of it, but that's around the beginning of its recent spike in popularity. It's starting to be viewed as a good long term option, but back then a lot of people were still wondering if it was a fad. I'm learning Rust, and I'm a big fan, but this is a bad take.
  - kiratp 4y ago
    We built <osmos.io> in Rust and we started in 2019. Our first lines of product code were also my first lines of Rust. It can be done and it’s not that hard. I think the choice of C++ for an in-process DB that is going to be very popular makes the entire industry less secure. If Chrome, one of the largest budget C++ code bases, still has memory bugs then there is no way DuckDB won’t.
intrasight 4y ago
I see no MDX query support. How is this an OLAP database? Or do I misunderstand what it is?
- slt2021 4y ago
  MDX is microsoft proptech, OLAP is more generic term to define analytical processing (in contrast to transaction processing as in CRUD). OLAP cube is microsoft's take on OLAP using 90s technologies for tech stacks from 1990s (Windows Server + SQL Server + SSAS). DuckDB is a modern take on OLAP
  - intrasight 4y ago
    Every OLAP database that I've used supports MDX MDX == OLAP SQL == relational
wasd 4y ago
Our database is heroku postgresql database. What's the best way to get this working with DuckDB? I see there's a postgresql connector but I'm not totally following how to deploy it. Would I just spin up a dyno with the docker image / custom build pack and connect it to the DB?
- chrisjc 4y ago
  Doesn't postgres have a columnar option? If so, you could prob get better performance for your analytical interactions if you switched some tables to columnar. Otherwise check out postgres scanner. https://github.com/duckdblabs/postgres_scanner https://github.com/duckdblabs/postgres_scanner They have a blog entry about it too: https://motherduck.com/blog/duckdb-ecosystem-newsletter-two/ https://motherduck.com/blog/duckdb-ecosystem-newsletter-two/
everybodyknows 4y ago
Anyone tried using the Go driver? It has a few open bugs: https://github.com/marcboeker/go-duckdb/issues https://github.com/marcboeker/go-duckdb/issues And a third-party effort, only: https://duckdb.org/docs/api/overview https://duckdb.org/docs/api/overview
- kagitac 4y ago
  I've found the CGO boundary to be quite slow for large result sets and have taken to just running commands that do SELECT and COPY to files on the system and then read those.
argella 4y ago
Isn’t SQLite adding features for analytic queries that should take the wind out of the sails of duckdb?
- tanin 4y ago
  I doubt sqlite will catch up soon in terms of analytics due to a few below reasons. The SQL dialect is so much lacking that it seems intentional. They are meant to be a transactional database, not an analytics one. Sqlite also takes pride on stability (deployed on a billion android devices). Adding 100+ analytics capabilities e.g. functions is not gonna be easy in terms of maintaining stability. I want to be wrong though because my paid app (superintendent.app) uses Sqlite. Not supporting analytics well is the number one complaint.
- simonw 4y ago
  Not really. There was an academic project around this but I've not seen signs that it will be integrated with SQLite core: https://simonwillison.net/2022/Sep/1/sqlite-duckdb-paper/ https://simonwillison.net/2022/Sep/1/sqlite-duckdb-paper/ The key thing to consider here is trade-offs. Analytical databases tend to be optimized for analytical queries at the expense of fast atomic read-write transactions. SQLite is mainly used in situations where fast atomic read-write transactions are key - that's why it's used in so many mobile phone applications, for example. It's not going to grow analytical-query-at-scale capabilities if that means negatively impacting the stuff it's really good at already.
__mharrison__ 4y ago
Not trying to troll, but assuming proficiency in python, when would someone prefer this to say Pandas (or Polars)? I've written a lot of OLAP queries (wrote a materialization layer for MonetDb and Postgres years ago). I find Pandas so much easier to work with for semi complicated work.
- cdavid 4y ago
  The obvious one is speed/data size: you can handle much larger data with duckdb compared to pandas. Depending on the data, maybe 10x larger, sometimes even more. From an ergonomics perspective, I find pandas much harder to use casually than SQL. When I was an IC and was using it a lot, I was proefficient in it. But now that I code maybe 5 hours / month at work, I can't really do anything non trivial besides basic stuff/pivots. OTOH, I never really forget SQL.
epberry 4y ago
We just switched to DuckDB from Postgres to analyze AWS billing data and wrote up our experience, https://www.vantage.sh/blog/querying-aws-cost-data-duckdb https://www.vantage.sh/blog/querying-aws-cost-data-duckdb Arguably Postgres was never the right tool to use for this analysis but nonetheless I was surprised at how much faster DuckDB was.
- thewisenerd 4y ago
  ha! i'm using duckdb for a similar use-case.. plotting 18-month aggregates take less than 2 seconds.. vendor only provides data for the last 3 months (through the web portal). (i'm "only" working with 30 million rows though)
brunoqc 4y ago
Anyone both tried duckdb and clickhouse-local?
- wenc 4y ago
  I use duckdb and looked into clickhouse-local. The dealbreaker for me was that clickhouse-local supported only a subset of SQL (that most people would probably be ok with, but not sufficient for a lot of complex analytics work). For instance, clickhouse doesn't support lead/lag functions natively (though it does propose workarounds). Here's clickhouse's SQL support: https://clickhouse.com/docs/en/sql-reference/ https://clickhouse.com/docs/en/sql-reference/ Compare this to DuckDB's (on the sidebar): https://duckdb.org/docs/sql/introduction https://duckdb.org/docs/sql/introduction DuckDB's SQL coverage is much more complete and matches my experience with full blown databases like Postgres and Redshift. As well, performance-wise DuckDB is currently still somewhat faster than clickhouse-local [1] but I would say this is a secondary consideration -- as long as either is "fast enough for your purposes" this shouldn't be an issue -- and clickhouse is plenty fast. The primary consideration for me would be the SQL support. That said, if you don't use any complex SQL, clickhouse-local seems like it would be a worthy contender. [1] https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQXRoZW5hIChwYXJ0aXRpb25lZCkiOmZhbHNlLCJBdGhlbmEgKHNpbmdsZSkiOmZhbHNlLCJBdXJvcmEgZm9yIE15U1FMIjpmYWxzZSwiQXVyb3JhIGZvciBQb3N0Z3JlU1FMIjpmYWxzZSwiQnl0ZUhvdXNlIjpmYWxzZSwiQ2l0dXMiOmZhbHNlLCJjbGlja2hvdXNlLWxvY2FsIChwYXJ0aXRpb25lZCkiOnRydWUsImNsaWNraG91c2UtbG9jYWwgKHNpbmdsZSkiOnRydWUsIkNsaWNrSG91c2UiOmZhbHNlLCJDbGlja0hvdXNlICh0dW5lZCkiOmZhbHNlLCJDbGlja0hvdXNlICh6c3RkKSI6ZmFsc2UsIkNsaWNrSG91c2UgQ2xvdWQiOmZhbHNlLCJDcmF0ZURCIjpmYWxzZSwiRGF0YWJlbmQiOmZhbHNlLCJkYXRhZnVzaW9uIjpmYWxzZSwiQXBhY2hlIERvcmlzIjpmYWxzZSwiRHJ1aWQiOmZhbHNlLCJEdWNrREIgKFBhcnF1ZXQpIjp0cnVlLCJEdWNrREIiOnRydWUsIkVsYXN0aWNzZWFyY2giOmZhbHNlLCJFbGFzdGljc2VhcmNoICh0dW5lZCkiOmZhbHNlLCJHcmVlbnBsdW0iOmZhbHNlLCJIZWF2eUFJIjpmYWxzZSwiSHlkcmEiOmZhbHNlLCJJbmZvYnJpZ2h0IjpmYWxzZSwiS2luZXRpY2EiOmZhbHNlLCJNYXJpYURCIENvbHVtblN0b3JlIjpmYWxzZSwiTWFyaWFEQiI6ZmFsc2UsIk1vbmV0REIiOmZhbHNlLCJNb25nb0RCIjpmYWxzZSwiTXlTUUwgKE15SVNBTSkiOmZhbHNlLCJNeVNRTCI6ZmFsc2UsIlBpbm90IjpmYWxzZSwiUG9zdGdyZVNRTCAodHVuZWQpIjpmYWxzZSwiUG9zdGdyZVNRTCI6ZmFsc2UsIlF1ZXN0REIgKHBhcnRpdGlvbmVkKSI6ZmFsc2UsIlF1ZXN0REIiOmZhbHNlLCJSZWRzaGlmdCI6ZmFsc2UsIlNlbGVjdERCIjpmYWxzZSwiU2luZ2xlU3RvcmUiOmZhbHNlLCJTbm93Zmxha2UiOmZhbHNlLCJTUUxpdGUiOmZhbHNlLCJTdGFyUm9ja3MgKHR1bmVkKSI6ZmFsc2UsIlN0YXJSb2NrcyI6ZmFsc2UsIlRpbWVzY2FsZURCIChjb21wcmVzc2lvbikiOmZhbHNlLCJUaW1lc2NhbGVEQiI6ZmFsc2V9LCJ0eXBlIjp7InN0YXRlbGVzcyI6dHJ1ZSwibWFuYWdlZCI6dHJ1ZSwiSmF2YSI6dHJ1ZSwiY29sdW1uLW9yaWVudGVkIjp0cnVlLCJDKysiOnRydWUsIk15U1FMIGNvbXBhdGlibGUiOnRydWUsInJvdy1vcmllbnRlZCI6dHJ1ZSwiQyI6dHJ1ZSwiUG9zdGdyZVNRTCBjb21wYXRpYmxlIjp0cnVlLCJDbGlja0hvdXNlIGRlcml2YXRpdmUiOnRydWUsImVtYmVkZGVkIjp0cnVlLCJSdXN0Ijp0cnVlLCJzZWFyY2giOnRydWUsImRvY3VtZW50Ijp0cnVlLCJ0aW1lLXNlcmllcyI6dHJ1ZX0sIm1hY2hpbmUiOnsic2VydmVybGVzcyI6dHJ1ZSwiMTZhY3UiOnRydWUsIkwiOnRydWUsIk0iOnRydWUsIlMiOnRydWUsIlhTIjp0cnVlLCJjNmEuNHhsYXJnZSwgNTAwZ2IgZ3AyIjp0cnVlLCJjNS40eGxhcmdlLCA1MDBnYiBncDIiOnRydWUsImM2YS5tZXRhbCwgNTAwZ2IgZ3AyIjp0cnVlLCIxNiB0aHJlYWRzIjp0cnVlLCIyMCB0aHJlYWRzIjp0cnVlLCIyNCB0aHJlYWRzIjp0cnVlLCIyOCB0aHJlYWRzIjp0cnVlLCIzMCB0aHJlYWRzIjp0cnVlLCI0OCB0aHJlYWRzIjp0cnVlLCI2MCB0aHJlYWRzIjp0cnVlLCJtNWQuMjR4bGFyZ2UiOnRydWUsImYxNnMgdjIiOnRydWUsImM2YS40eGxhcmdlLCAxNTAwZ2IgZ3AyIjp0cnVlLCJyYTMuMTZ4bGFyZ2UiOnRydWUsInJhMy40eGxhcmdlIjp0cnVlLCJyYTMueGxwbHVzIjp0cnVlLCJTMiI6dHJ1ZSwiUzI0Ijp0cnVlLCIyWEwiOnRydWUsIjNYTCI6dHJ1ZSwiNFhMIjp0cnVlLCJYTCI6dHJ1ZX0sImNsdXN0ZXJfc2l6ZSI6eyIxIjp0cnVlLCIyIjp0cnVlLCI0Ijp0cnVlLCI4Ijp0cnVlLCIxMiI6dHJ1ZSwiMTYiOnRydWUsIjMyIjp0cnVlLCI2NCI6dHJ1ZSwiMTI4Ijp0cnVlLCJzZXJ2ZXJsZXNzIjp0cnVlLCJ1bmRlZmluZWQiOnRydWV9LCJtZXRyaWMiOiJob3QiLCJxdWVyaWVzIjpbdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZV19 https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQXRoZW5hIC...
dang 4y ago
Related—others?: Motherduck Raises $47.5M for DuckDB - https://news.ycombinator.com/item?id=33610218 https://news.ycombinator.com/item?id=33610218 - Nov 2022 (2 comments) Modern Data Stack in a Box with DuckDB - https://news.ycombinator.com/item?id=33191938 https://news.ycombinator.com/item?id=33191938 - Oct 2022 (5 comments) Querying Postgres Tables Directly from DuckDB - https://news.ycombinator.com/item?id=33035803 https://news.ycombinator.com/item?id=33035803 - Sept 2022 (38 comments) Notes on the SQLite DuckDB Paper - https://news.ycombinator.com/item?id=32684424 https://news.ycombinator.com/item?id=32684424 - Sept 2022 (28 comments) Show HN: CSVFiddle – Query CSV files with DuckDB in the browser - https://news.ycombinator.com/item?id=31946039 https://news.ycombinator.com/item?id=31946039 - July 2022 (13 comments) Show HN: Easily Convert WARC (Web Archive) into Parquet, Then Query with DuckDB - https://news.ycombinator.com/item?id=31867179 https://news.ycombinator.com/item?id=31867179 - June 2022 (15 comments) Range joins in DuckDB - https://news.ycombinator.com/item?id=31530639 https://news.ycombinator.com/item?id=31530639 - May 2022 (24 comments) Friendlier SQL with DuckDB - https://news.ycombinator.com/item?id=31355050 https://news.ycombinator.com/item?id=31355050 - May 2022 (133 comments) Fast analysis with DuckDB and Pyarrow - https://news.ycombinator.com/item?id=31217782 https://news.ycombinator.com/item?id=31217782 - April 2022 (53 comments) Directly running DuckDB queries on data stored in SQLite files - https://news.ycombinator.com/item?id=30801575 https://news.ycombinator.com/item?id=30801575 - March 2022 (23 comments) Parallel Grouped Aggregation in DuckDB - https://news.ycombinator.com/item?id=30589250 https://news.ycombinator.com/item?id=30589250 - March 2022 (10 comments) DuckDB quacks Arrow: A zero-copy data integration between Arrow and DuckDB - https://news.ycombinator.com/item?id=29433941 https://news.ycombinator.com/item?id=29433941 - Dec 2021 (13 comments) DuckDB-Wasm: Efficient analytical SQL in the browser - https://news.ycombinator.com/item?id=29039235 https://news.ycombinator.com/item?id=29039235 - Oct 2021 (58 comments) Comparing SQLite, DuckDB and Arrow with UN trade data - https://news.ycombinator.com/item?id=29010103 https://news.ycombinator.com/item?id=29010103 - Oct 2021 (79 comments) DuckDB is the better SQLite with APIs for Java/Python/R and it's got potential - https://news.ycombinator.com/item?id=28692997 https://news.ycombinator.com/item?id=28692997 - Sept 2021 (2 comments) Fastest table sort in the West – Redesigning DuckDB's sort - https://news.ycombinator.com/item?id=28328657 https://news.ycombinator.com/item?id=28328657 - Aug 2021 (27 comments) Querying Parquet with Precision Using DuckDB - https://news.ycombinator.com/item?id=27634840 https://news.ycombinator.com/item?id=27634840 - June 2021 (32 comments) DuckDB now has a Node.js API - https://news.ycombinator.com/item?id=25289574 https://news.ycombinator.com/item?id=25289574 - Dec 2020 (4 comments) DuckDB – An embeddable SQL database like SQLite, but supports Postgres features - https://news.ycombinator.com/item?id=24531085 https://news.ycombinator.com/item?id=24531085 - Sept 2020 (160 comments) DuckDB: SQLite for Analytics - https://news.ycombinator.com/item?id=23287278 https://news.ycombinator.com/item?id=23287278 - May 2020 (67 comments)
sa46 4y ago
Does anyone have experience using client-side DuckDB with WASM? It looks really compelling to ship analytics to the browser by running DuckDB over parquet files.
aynyc 4y ago
I don't get this: * When to not use DuckDB: Multiple concurrent processes reading from a single writable database* So no concurrent reads? Or is it no concurrent reads while writing?
- tkhattra 4y ago
  the latter - you can either have multiple concurrent readers or a single writer. see https://duckdb.org/faq.html#how-does-duckdb-handle-concurrency https://duckdb.org/faq.html#how-does-duckdb-handle-concurren....
  - aynyc 4y ago
    What if I run multiple processes? One is to write, the rest are reads?
linhvn 4y ago
What is the differentiation of DuckDB compared to previous OLAP databases such as Snowflake or Singlestore?

161 points by freilanzer 4y ago

DuckDB is awesome! Last year I was working on something using SQLite, users could perform analytical queries that would scan the entire 3gb db and generate aggregates. It would take at least 45 seconds to do the queries. I did a dump of the db and imported to DuckDB. The same queries now only take 1.5 seconds with exactly the same SQL. Obviously there is a trade off, inserts are slower on DuckDB. But for a low write, analytical read app it's perfect. I tried to use the SQLite connector but struggled to get it working. Need to circle back and have another go. With their SQLite and Postgres connectors, as a Django dev I would love an app that lets you run specific queries on your DB via DuckDB almost transparently. Would be awesome for analytical dashboards.

mcdonje 4y ago
Yeah, I see it as the sqlite of the reporting world. Especially with columnar stores and parquet support.
dunefox 4y ago
I'd need a MySQL/MariaDB connector for it to be really useful for me. But still, it looks great.

bluehat974 4y ago

DuckCon 2023 - State of the Duck - Mark Raasveldt and Hannes Mühleisen https://youtu.be/rdnPkLSkoyU https://youtu.be/rdnPkLSkoyU

lichtenberger 4y ago

I wonder if we'll in the near future also see JVM based in-memory or hybrid OLAP database systems, which will make use of SIMD instructions and columnar storage layouts with the incubating Vector API. It would be also interesting to see how we can process semi-structured data in a similar way.

d_watt 4y ago

I'm duck-curious. Looking at how it's deployed, as an in process database, how do people actually use this in production? Trying to figure out where I might actually want to think about replacing current databases or analyses with DuckDB. EG if you deployed new code 1. Do you have a stateful machine you're doing an old school "Kill the old process, start the new process" deploy, and there's some duckdb file on disk that is maintained? 2. Or do you back that duckdb file in some sort of shared disk (Eg EBS), and have a rolling deploy where multiple applications access the same DB at the same time? 3. Or is DuckDB is treated as ephemeral, and you're using it to process data on the fly, so persisted state isn't an issue?

dunefox 4y ago
Good questions, I'm thinking about evaluating it along with pola.rs as data science tools. Would be interesting to hear from someone using it already.
romski 4y ago
Yes using it in production - Stateless and ephemeral. For sure there’s a learning curve.
- d_watt 4y ago
  I see. Would it be fair to say you treat it almost like Pandas, except that it has a lower memory footprint since data is written to disk instead of memory. IE you use it for on the fly analysis of large frames of data, not like more traditional database/datawarehouse?
  - chrisjc 4y ago
    I'm not sure they're saying that. BTW, your questions are exactly those that I've been ask over the last few months, but also with a lot of focus over the last few days. Still learning as much as I can so the following might not be true. For what it's worth, there's a difference between using duckdb to query a set of files vs loading a bunch of files in to a table. But once the data has been loaded into a table it can be backed up as a duckdb db file. Therefore it might be more performant to preprocess duckdb db files (perhaps a process that works in conjunction with whatever manages your external tables) and load these db files into duckdb as needed (on the fly analysis) instead of loading datafiles into duckdb, transforming and CTAS every time. https://duckdb.org/docs/sql/statements/attach https://duckdb.org/docs/sql/statements/attach Of course all of this might be introducing more latency esp if you're trying to do NRT analytics. I assume you could partition your data into multiple db files similar to how you would probably do it with your data files (managing external tables).

lopkeny12ko 4y ago

I appreciate the clarity on the explicitly unsupported use cases in "When to not use DuckDB." There are so many infrastructure products, especially database products for some reason, where the marketing team takes control of the messaging away from engineers, and push outlandish claims on how their new DB is faster than all the competition, can support any workload, can scale infinitely, etc.

SnowflakeOnIce 4y ago

DuckDB is terrific. I'm bullish on its potential for simplifying many big data pipelines. Particularly, it's plausible that DuckDB + Parquet could be used on a large SMP machine (32+ cores and 128GB+ memory) to deal with data munging for 100s of gigabytes to several terabytes, all from SQL, without dealing with Hadoop, Spark, Ray, etc. I have successfully used DuckDB like above for preparing an ML dataset from about 100GB of input. DuckDB is undergoing rapid development these days. There have been format-breaking changes and bugs that could lose data. I would not yet trust DuckDB for long-term storage or archival purposes. Parquet is a better choice for that.

RobinL 4y ago
I'd love to hear any real world experiences of anyone who's tried to run jobs that would usually require a spark cluster on a single machine with loads of cores and memory. How big can you go, and how does speed compare to Spark? (I'm guessing significantly faster from my experience using Duckdb on smaller machines)
- jakewins 4y ago
  I used it as glue for a monthly job, last week, processing ~1B event records, juggling them a bit and pushing them as 2M parquet segment files to S3. Mixed experience, would definitely not put it in a system that isn’t ok crashing frequently. It segfaults on Alpine (argh, C++!), and force exits the whole NodeJS process when it gets unexpected HTTP responses from S3. In an archive run of 2M pushes it’ll crash the process 4-5 times. Overall still really, really like it, but learned to not trust it
  - tracker1 4y ago
    Node + AWS for data ingress has been pretty painful in my experience (mostly dynamo feeds from large csv (whois database)). In the end, rewrote in C# (core 2) and it was able to complete more reliably. I'm guessing that go and rust would also be better. I like node, really like JS, but I just think that maybe the AWS libraries aren't that great in the space. It would run for 3-5 hours, then just blow up unexpectedly, even with plenty of memory overhead, and not really bandwidth limited, with appropriate retries and slowdown for dynamo rejections. If I never have to write ETL pipelines again, I won't be upset about it.

vgt 4y ago

DuckDB is such a breath of fresh air! We at MotherDuck at working very closely with the DuckDB folks to build a DuckDB-based cloud service. I'm talking to various folks in the industry about the details in 1:1. Feel free to reach out to tino at motherduck.com. (co-founder and head of Produck at MotherDuck)

jengleton 4y ago

Great to see this posted here! DuckDB is an integral part of an in-browser data analytics tool that I've been working on. It compiles to WASM and runs in a web worker. Queries against WASM DuckDB regularly run 10x faster than the original JavaScript implementation!

thenaturalist 4y ago
In case you're interested in or willing to share more details about the tool or your architecture I'd be an interested reader.
hummus_bae 4y ago
It's amazing to see Rust used so much even in web projects. It's my favorite language and I don't want to use it for web programming anymore. I've used it for too many "real" web apps (I mean webrtc signaling and web socket) to go through the pains of optimization. But it's still fun to work with.

out_of_step 4y ago

DuckDB has an awesome integration with Apache Arrow and very useful implementations in R and Python. Great for easy larger-than-memory analyses https://duckdb.org/2021/12/03/duck-arrow.html https://duckdb.org/2021/12/03/duck-arrow.html

bwanab 4y ago

There's a nice intro to DuckDB for julia developers by Bogumił Kamiński, the creator of DataFrames.jl here: https://juliazoid.com/welcome-to-duckdb-3c4e75f50b97 https://juliazoid.com/welcome-to-duckdb-3c4e75f50b97. Interesting since in some ways, as he points out, it's in direct competition with DataFrames for use cases, but he gives it a very positive treatment and shows how they can work together using advantages of standard SQL along with processing power of DataFrames.

nickpeterson 4y ago
When someone gives fair opinions on something that directly competes were their own work, you should take their opinion very seriously. It’s an excellent quality in a person, and shows they’re more focused on the problem than their ego.
- phillc73 4y ago
  Quite a while ago, when duckdb was just a duckling, I wrote an R package that supported direct manipulation of R dataframes using SQL.[1] duckdb was the engine for this. The approach was never as fast as data.table but did approach the speed of dplyr for more complex queries. Life had other things in store for me and I haven’t touched this library for a while now. At the time there was no Julia connector for duckdb, but now that there is, I’d like to try this approach in that language. [1] https://github.com/phillc73/duckdf https://github.com/phillc73/duckdf

ctc24 4y ago

We're big fans of DuckDb at https://prequel.co https://prequel.co! We use it as part of our own dataframe implementation in Go. The speed is unbeatable and the tool is top notch. There are a few rough edges (it's not quite 1.0 level of stability yet), but the team is super reactive and has fixed bugs we've reported in < 48hrs pretty much every time.

kiratp 4y ago

Its disappointing that C++ was chosen to build something that is going to live in-process. Rust would have been so much safer. All the segfaults you get when running DucDB supports this statement.

mcdonje 4y ago
I agree about Rust's memory safety advantage over C++, but I disagree that it's disappointing from a project perspective. Some DB experts made a good DB using a performant language they're comfortable with. You can't make project choices in a vacuum, and you can't assume others can either. People have limited time. The choice they were facing was probably not C++ vs Rust, but C++ vs nothing because they didn't have time to learn a new language before starting their project. Also, their first release was in 2019, so they probably heard of it, but that's around the beginning of its recent spike in popularity. It's starting to be viewed as a good long term option, but back then a lot of people were still wondering if it was a fad. I'm learning Rust, and I'm a big fan, but this is a bad take.
- kiratp 4y ago
  We built <osmos.io> in Rust and we started in 2019. Our first lines of product code were also my first lines of Rust. It can be done and it’s not that hard. I think the choice of C++ for an in-process DB that is going to be very popular makes the entire industry less secure. If Chrome, one of the largest budget C++ code bases, still has memory bugs then there is no way DuckDB won’t.

intrasight 4y ago

I see no MDX query support. How is this an OLAP database? Or do I misunderstand what it is?

slt2021 4y ago
MDX is microsoft proptech, OLAP is more generic term to define analytical processing (in contrast to transaction processing as in CRUD). OLAP cube is microsoft's take on OLAP using 90s technologies for tech stacks from 1990s (Windows Server + SQL Server + SSAS). DuckDB is a modern take on OLAP
- intrasight 4y ago
  Every OLAP database that I've used supports MDX MDX == OLAP SQL == relational

wasd 4y ago

Our database is heroku postgresql database. What's the best way to get this working with DuckDB? I see there's a postgresql connector but I'm not totally following how to deploy it. Would I just spin up a dyno with the docker image / custom build pack and connect it to the DB?

chrisjc 4y ago
Doesn't postgres have a columnar option? If so, you could prob get better performance for your analytical interactions if you switched some tables to columnar. Otherwise check out postgres scanner. https://github.com/duckdblabs/postgres_scanner https://github.com/duckdblabs/postgres_scanner They have a blog entry about it too: https://motherduck.com/blog/duckdb-ecosystem-newsletter-two/ https://motherduck.com/blog/duckdb-ecosystem-newsletter-two/

everybodyknows 4y ago

Anyone tried using the Go driver? It has a few open bugs: https://github.com/marcboeker/go-duckdb/issues https://github.com/marcboeker/go-duckdb/issues And a third-party effort, only: https://duckdb.org/docs/api/overview https://duckdb.org/docs/api/overview

kagitac 4y ago
I've found the CGO boundary to be quite slow for large result sets and have taken to just running commands that do SELECT and COPY to files on the system and then read those.

argella 4y ago

Isn’t SQLite adding features for analytic queries that should take the wind out of the sails of duckdb?

tanin 4y ago
I doubt sqlite will catch up soon in terms of analytics due to a few below reasons. The SQL dialect is so much lacking that it seems intentional. They are meant to be a transactional database, not an analytics one. Sqlite also takes pride on stability (deployed on a billion android devices). Adding 100+ analytics capabilities e.g. functions is not gonna be easy in terms of maintaining stability. I want to be wrong though because my paid app (superintendent.app) uses Sqlite. Not supporting analytics well is the number one complaint.
simonw 4y ago
Not really. There was an academic project around this but I've not seen signs that it will be integrated with SQLite core: https://simonwillison.net/2022/Sep/1/sqlite-duckdb-paper/ https://simonwillison.net/2022/Sep/1/sqlite-duckdb-paper/ The key thing to consider here is trade-offs. Analytical databases tend to be optimized for analytical queries at the expense of fast atomic read-write transactions. SQLite is mainly used in situations where fast atomic read-write transactions are key - that's why it's used in so many mobile phone applications, for example. It's not going to grow analytical-query-at-scale capabilities if that means negatively impacting the stuff it's really good at already.

__mharrison__ 4y ago

Not trying to troll, but assuming proficiency in python, when would someone prefer this to say Pandas (or Polars)? I've written a lot of OLAP queries (wrote a materialization layer for MonetDb and Postgres years ago). I find Pandas so much easier to work with for semi complicated work.

cdavid 4y ago
The obvious one is speed/data size: you can handle much larger data with duckdb compared to pandas. Depending on the data, maybe 10x larger, sometimes even more. From an ergonomics perspective, I find pandas much harder to use casually than SQL. When I was an IC and was using it a lot, I was proefficient in it. But now that I code maybe 5 hours / month at work, I can't really do anything non trivial besides basic stuff/pivots. OTOH, I never really forget SQL.

epberry 4y ago

We just switched to DuckDB from Postgres to analyze AWS billing data and wrote up our experience, https://www.vantage.sh/blog/querying-aws-cost-data-duckdb https://www.vantage.sh/blog/querying-aws-cost-data-duckdb Arguably Postgres was never the right tool to use for this analysis but nonetheless I was surprised at how much faster DuckDB was.

thewisenerd 4y ago
ha! i'm using duckdb for a similar use-case.. plotting 18-month aggregates take less than 2 seconds.. vendor only provides data for the last 3 months (through the web portal). (i'm "only" working with 30 million rows though)

brunoqc 4y ago

Anyone both tried duckdb and clickhouse-local?

wenc 4y ago
I use duckdb and looked into clickhouse-local. The dealbreaker for me was that clickhouse-local supported only a subset of SQL (that most people would probably be ok with, but not sufficient for a lot of complex analytics work). For instance, clickhouse doesn't support lead/lag functions natively (though it does propose workarounds). Here's clickhouse's SQL support: https://clickhouse.com/docs/en/sql-reference/ https://clickhouse.com/docs/en/sql-reference/ Compare this to DuckDB's (on the sidebar): https://duckdb.org/docs/sql/introduction https://duckdb.org/docs/sql/introduction DuckDB's SQL coverage is much more complete and matches my experience with full blown databases like Postgres and Redshift. As well, performance-wise DuckDB is currently still somewhat faster than clickhouse-local [1] but I would say this is a secondary consideration -- as long as either is "fast enough for your purposes" this shouldn't be an issue -- and clickhouse is plenty fast. The primary consideration for me would be the SQL support. That said, if you don't use any complex SQL, clickhouse-local seems like it would be a worthy contender. [1] https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQXRoZW5hIChwYXJ0aXRpb25lZCkiOmZhbHNlLCJBdGhlbmEgKHNpbmdsZSkiOmZhbHNlLCJBdXJvcmEgZm9yIE15U1FMIjpmYWxzZSwiQXVyb3JhIGZvciBQb3N0Z3JlU1FMIjpmYWxzZSwiQnl0ZUhvdXNlIjpmYWxzZSwiQ2l0dXMiOmZhbHNlLCJjbGlja2hvdXNlLWxvY2FsIChwYXJ0aXRpb25lZCkiOnRydWUsImNsaWNraG91c2UtbG9jYWwgKHNpbmdsZSkiOnRydWUsIkNsaWNrSG91c2UiOmZhbHNlLCJDbGlja0hvdXNlICh0dW5lZCkiOmZhbHNlLCJDbGlja0hvdXNlICh6c3RkKSI6ZmFsc2UsIkNsaWNrSG91c2UgQ2xvdWQiOmZhbHNlLCJDcmF0ZURCIjpmYWxzZSwiRGF0YWJlbmQiOmZhbHNlLCJkYXRhZnVzaW9uIjpmYWxzZSwiQXBhY2hlIERvcmlzIjpmYWxzZSwiRHJ1aWQiOmZhbHNlLCJEdWNrREIgKFBhcnF1ZXQpIjp0cnVlLCJEdWNrREIiOnRydWUsIkVsYXN0aWNzZWFyY2giOmZhbHNlLCJFbGFzdGljc2VhcmNoICh0dW5lZCkiOmZhbHNlLCJHcmVlbnBsdW0iOmZhbHNlLCJIZWF2eUFJIjpmYWxzZSwiSHlkcmEiOmZhbHNlLCJJbmZvYnJpZ2h0IjpmYWxzZSwiS2luZXRpY2EiOmZhbHNlLCJNYXJpYURCIENvbHVtblN0b3JlIjpmYWxzZSwiTWFyaWFEQiI6ZmFsc2UsIk1vbmV0REIiOmZhbHNlLCJNb25nb0RCIjpmYWxzZSwiTXlTUUwgKE15SVNBTSkiOmZhbHNlLCJNeVNRTCI6ZmFsc2UsIlBpbm90IjpmYWxzZSwiUG9zdGdyZVNRTCAodHVuZWQpIjpmYWxzZSwiUG9zdGdyZVNRTCI6ZmFsc2UsIlF1ZXN0REIgKHBhcnRpdGlvbmVkKSI6ZmFsc2UsIlF1ZXN0REIiOmZhbHNlLCJSZWRzaGlmdCI6ZmFsc2UsIlNlbGVjdERCIjpmYWxzZSwiU2luZ2xlU3RvcmUiOmZhbHNlLCJTbm93Zmxha2UiOmZhbHNlLCJTUUxpdGUiOmZhbHNlLCJTdGFyUm9ja3MgKHR1bmVkKSI6ZmFsc2UsIlN0YXJSb2NrcyI6ZmFsc2UsIlRpbWVzY2FsZURCIChjb21wcmVzc2lvbikiOmZhbHNlLCJUaW1lc2NhbGVEQiI6ZmFsc2V9LCJ0eXBlIjp7InN0YXRlbGVzcyI6dHJ1ZSwibWFuYWdlZCI6dHJ1ZSwiSmF2YSI6dHJ1ZSwiY29sdW1uLW9yaWVudGVkIjp0cnVlLCJDKysiOnRydWUsIk15U1FMIGNvbXBhdGlibGUiOnRydWUsInJvdy1vcmllbnRlZCI6dHJ1ZSwiQyI6dHJ1ZSwiUG9zdGdyZVNRTCBjb21wYXRpYmxlIjp0cnVlLCJDbGlja0hvdXNlIGRlcml2YXRpdmUiOnRydWUsImVtYmVkZGVkIjp0cnVlLCJSdXN0Ijp0cnVlLCJzZWFyY2giOnRydWUsImRvY3VtZW50Ijp0cnVlLCJ0aW1lLXNlcmllcyI6dHJ1ZX0sIm1hY2hpbmUiOnsic2VydmVybGVzcyI6dHJ1ZSwiMTZhY3UiOnRydWUsIkwiOnRydWUsIk0iOnRydWUsIlMiOnRydWUsIlhTIjp0cnVlLCJjNmEuNHhsYXJnZSwgNTAwZ2IgZ3AyIjp0cnVlLCJjNS40eGxhcmdlLCA1MDBnYiBncDIiOnRydWUsImM2YS5tZXRhbCwgNTAwZ2IgZ3AyIjp0cnVlLCIxNiB0aHJlYWRzIjp0cnVlLCIyMCB0aHJlYWRzIjp0cnVlLCIyNCB0aHJlYWRzIjp0cnVlLCIyOCB0aHJlYWRzIjp0cnVlLCIzMCB0aHJlYWRzIjp0cnVlLCI0OCB0aHJlYWRzIjp0cnVlLCI2MCB0aHJlYWRzIjp0cnVlLCJtNWQuMjR4bGFyZ2UiOnRydWUsImYxNnMgdjIiOnRydWUsImM2YS40eGxhcmdlLCAxNTAwZ2IgZ3AyIjp0cnVlLCJyYTMuMTZ4bGFyZ2UiOnRydWUsInJhMy40eGxhcmdlIjp0cnVlLCJyYTMueGxwbHVzIjp0cnVlLCJTMiI6dHJ1ZSwiUzI0Ijp0cnVlLCIyWEwiOnRydWUsIjNYTCI6dHJ1ZSwiNFhMIjp0cnVlLCJYTCI6dHJ1ZX0sImNsdXN0ZXJfc2l6ZSI6eyIxIjp0cnVlLCIyIjp0cnVlLCI0Ijp0cnVlLCI4Ijp0cnVlLCIxMiI6dHJ1ZSwiMTYiOnRydWUsIjMyIjp0cnVlLCI2NCI6dHJ1ZSwiMTI4Ijp0cnVlLCJzZXJ2ZXJsZXNzIjp0cnVlLCJ1bmRlZmluZWQiOnRydWV9LCJtZXRyaWMiOiJob3QiLCJxdWVyaWVzIjpbdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZSx0cnVlLHRydWUsdHJ1ZV19 https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQXRoZW5hIC...

dang 4y ago

Related—others?: Motherduck Raises $47.5M for DuckDB - https://news.ycombinator.com/item?id=33610218 https://news.ycombinator.com/item?id=33610218 - Nov 2022 (2 comments) Modern Data Stack in a Box with DuckDB - https://news.ycombinator.com/item?id=33191938 https://news.ycombinator.com/item?id=33191938 - Oct 2022 (5 comments) Querying Postgres Tables Directly from DuckDB - https://news.ycombinator.com/item?id=33035803 https://news.ycombinator.com/item?id=33035803 - Sept 2022 (38 comments) Notes on the SQLite DuckDB Paper - https://news.ycombinator.com/item?id=32684424 https://news.ycombinator.com/item?id=32684424 - Sept 2022 (28 comments) Show HN: CSVFiddle – Query CSV files with DuckDB in the browser - https://news.ycombinator.com/item?id=31946039 https://news.ycombinator.com/item?id=31946039 - July 2022 (13 comments) Show HN: Easily Convert WARC (Web Archive) into Parquet, Then Query with DuckDB - https://news.ycombinator.com/item?id=31867179 https://news.ycombinator.com/item?id=31867179 - June 2022 (15 comments) Range joins in DuckDB - https://news.ycombinator.com/item?id=31530639 https://news.ycombinator.com/item?id=31530639 - May 2022 (24 comments) Friendlier SQL with DuckDB - https://news.ycombinator.com/item?id=31355050 https://news.ycombinator.com/item?id=31355050 - May 2022 (133 comments) Fast analysis with DuckDB and Pyarrow - https://news.ycombinator.com/item?id=31217782 https://news.ycombinator.com/item?id=31217782 - April 2022 (53 comments) Directly running DuckDB queries on data stored in SQLite files - https://news.ycombinator.com/item?id=30801575 https://news.ycombinator.com/item?id=30801575 - March 2022 (23 comments) Parallel Grouped Aggregation in DuckDB - https://news.ycombinator.com/item?id=30589250 https://news.ycombinator.com/item?id=30589250 - March 2022 (10 comments) DuckDB quacks Arrow: A zero-copy data integration between Arrow and DuckDB - https://news.ycombinator.com/item?id=29433941 https://news.ycombinator.com/item?id=29433941 - Dec 2021 (13 comments) DuckDB-Wasm: Efficient analytical SQL in the browser - https://news.ycombinator.com/item?id=29039235 https://news.ycombinator.com/item?id=29039235 - Oct 2021 (58 comments) Comparing SQLite, DuckDB and Arrow with UN trade data - https://news.ycombinator.com/item?id=29010103 https://news.ycombinator.com/item?id=29010103 - Oct 2021 (79 comments) DuckDB is the better SQLite with APIs for Java/Python/R and it's got potential - https://news.ycombinator.com/item?id=28692997 https://news.ycombinator.com/item?id=28692997 - Sept 2021 (2 comments) Fastest table sort in the West – Redesigning DuckDB's sort - https://news.ycombinator.com/item?id=28328657 https://news.ycombinator.com/item?id=28328657 - Aug 2021 (27 comments) Querying Parquet with Precision Using DuckDB - https://news.ycombinator.com/item?id=27634840 https://news.ycombinator.com/item?id=27634840 - June 2021 (32 comments) DuckDB now has a Node.js API - https://news.ycombinator.com/item?id=25289574 https://news.ycombinator.com/item?id=25289574 - Dec 2020 (4 comments) DuckDB – An embeddable SQL database like SQLite, but supports Postgres features - https://news.ycombinator.com/item?id=24531085 https://news.ycombinator.com/item?id=24531085 - Sept 2020 (160 comments) DuckDB: SQLite for Analytics - https://news.ycombinator.com/item?id=23287278 https://news.ycombinator.com/item?id=23287278 - May 2020 (67 comments)

sa46 4y ago

Does anyone have experience using client-side DuckDB with WASM? It looks really compelling to ship analytics to the browser by running DuckDB over parquet files.

aynyc 4y ago

I don't get this: * When to not use DuckDB: Multiple concurrent processes reading from a single writable database* So no concurrent reads? Or is it no concurrent reads while writing?

tkhattra 4y ago
the latter - you can either have multiple concurrent readers or a single writer. see https://duckdb.org/faq.html#how-does-duckdb-handle-concurrency https://duckdb.org/faq.html#how-does-duckdb-handle-concurren....
- aynyc 4y ago
  What if I run multiple processes? One is to write, the rest are reads?

linhvn 4y ago

What is the differentiation of DuckDB compared to previous OLAP databases such as Snowflake or Singlestore?