3 ms·
DuckDB does not have any kind of sharding or clustering? It doesn't even have a server (unless you count the HTTP Server Extension)?
by atombender 1y ago
DuckDB does not have any kind of sharding or clustering? It doesn't even have a server (unless you count the HTTP Server Extension)?
- brunohaid 1y agoGood point and was mostly re Typesense (can't edit the comment anymore). But given that duckdb handles "take this n GB parquet file/shard from a random location, load it into memory and be ready in < 1 sec" very well I'd argue it's quite easy to build something that scales horizontally. We use it for both the importer pipeline that processes the 2B row / 200GB compressed GBIF.org parquet dataset and queries like https://www.meso.cloud/plants/pinophyta/cupressales/pinopsida/cupressaceae/sequoiadendron_giganteum https://www.meso.cloud/plants/pinophyta/cupressales/pinopsid... and the sheer amount of functions[1] beyond simple stuff like "how close is a/b to x/y" or is "n within area x" is just a joy to work with. [1] https://duckdb.org/docs/stable/core_extensions/spatial/functions.html https://duckdb.org/docs/stable/core_extensions/spatial/funct...
- steve_adams_86 1y agoDuckdb is so absurdly portable you can solve all kinds of performance and scaling concerns this way. It's really a wonderful, fun piece of software.
- maxboone 1y agoMotherduck is the spin-off from DuckDBthat does that. But there's also: https://duckdb.org/2025/05/27/ducklake.html https://duckdb.org/2025/05/27/ducklake.html You can also attach DuckDB to Apache Flight which will make it work beyond local operation.