3 ms·
DuckDB is incredible for large stats projects using R. R can't really handle large datasets and simply runs out of memory if you load everything. You get around
by Dayshine 6y ago
DuckDB is incredible for large stats projects using R. R can't really handle large datasets and simply runs out of memory if you load everything. You get around this by splitting your data into a bunch of RData files but there's quite a bit of overhead maintaining those, recreating them when you need more variables, etc.
With DuckDB you can just load it once into the database then use dplyr/dbplyr to transparently access data with the queries pushing down to the DB afaik. And it runs anywhere and doesn't need local admin (like most database servers).
(SQLLite can't handle many columns without a custom build)
- Zelazny7 6y agoHave you figured out a way to add columns to DuckDB from R vectors? I'm not sure this is possible yet without using inefficient insert operations. Ideally, I'd like to pull in some columns from the DB, generate a new column vector and store it back in the DB by adding a column to an existing table.