3 ms·
I hadn't used data.table or plyr because the native R functions were giving me good performance even at tens of thousands of rows. But now that I'm doing analy
by minimaxir 13y ago
I hadn't used data.table or plyr because the native R functions were giving me good performance even at tens of thousands of rows.
But now that I'm doing analysis on hundreds of thousands of rows, doing aggregation takes awhile. This article convinced me to give those packages a try. If data.table and plyr aggregate functions are indeed paralellizeable, that's a big deal, especially when implementing bootstrap resampling.
- jasonpbecker 13y agoYou should know that dplyr, plyr's replacement is already fairly stable and worth using. For basic tasks, it's as fast or faster than data.table in my experience, with the caveat that it is more likely to copy for some methods than data.table which is very strict about this.
- minimaxir 13y agoWow, if the benchmarks are true, then dplyr sounds perfect. http://blog.rstudio.org/2014/01/17/introducing-dplyr/ http://blog.rstudio.org/2014/01/17/introducing-dplyr/
- peatmoss 13y agoAnd being able to combine it with an RDBMS means that you can potentially do plyr-ish things on datasets that can't fit in memory. I think the SQL generated by dplyr tries to be smart / efficient too.
- chubot 13y agoI use data.table and like it. It's fast (and I think memory efficient). plyr is IMO so slow to be unusable on just hundreds of thousands rows. dplyr, as mentioned, is the replacement that fixes it, but it was literally first released a couple months ago this year. I have no doubt Hadley will produce something brilliant, but I'm guessing it will undergo a lot of change to get there.