3 ms·
what are some practical uses for this if living in a pure R world? Is this like BigMemory but for data frames? https://cran.r-project.org/web/packages/bigmemo
by stephenl 11y ago
what are some practical uses for this if living in a pure R world?
Is this like BigMemory but for data frames?
https://cran.r-project.org/web/packages/bigmemory/index.html https://cran.r-project.org/web/packages/bigmemory/index.html
Thanks.
- hadley 11y agoIt's often much faster than rds. And in the long long term there will be tools for computing on feather files that don't require loading it into memory. (In the short term I'll add ways to pull in slices of the full dataset)
- alsocasey 11y agoFaster because it isn't (currently) using compression (which rds uses by default) or faster period? Either way, the idea of mixed Python/R pipelines with feather file intermediates input/outputs is pretty sweet. Learn in scikit, save to feather, plot in ggplot2... using Make to tie the pieces together?
- hadley 11y agoIt's usually faster than either compressed or uncompressed RDS - but if you have heavily duplicated data, compressed RDS can be faster than feather (depending on some tradeoff between compression speed and disk speed). Feather will probably gain compression support eventually.