4 ms·
plyr, while having awesome syntax, is really slow for anything beyond even the most modest dataset. Try using it with 50k rows, not to mention true "big data".
by jasonpbecker 14y ago
plyr, while having awesome syntax, is really slow for anything beyond even the most modest dataset. Try using it with 50k rows, not to mention true "big data".
data.table is much faster. It's an extension to data.frames that adds some additional constrains/rules that allows for much faster operations including aggregating, subsetting, and merging data.
Cleaning data, however, does not have a steep learning curve or high difficulty level in R-- it has a steep learning curve and high difficulty level period. Implementing good procedures for data munging is 80% of the job.
- EzGraphs 14y agoYeah - for the small data sets, Ruby or Python is often easier easier to use to whip a data set into a form that can be simply slurped into R as a dataframe.
- mogrim 14y agoHas to be said I cheated a bit on the Coursera course, I used Ruby to clean up some of the data. Seemed a lot easier, and it's a lot more likely I'd use some kind of scripting language (be it Perl, Ruby, Grep, ...) to do a first clean up than head directly to R to do the job.
- aheilbut 14y agoWith pandas / statsmodels / patsy, it's getting easier to stick with python for everything.
- GrumpySimon 14y agoDon't forget Bokeh[1] - a replacement for ggplot2 [1] https://github.com/ContinuumIO/Bokeh https://github.com/ContinuumIO/Bokeh
- banachtarski 14y agoThis is a good point. Usually, I use haskell or something to do large scale transformations prior to a plot or something. However, I still use plyr from time to time to do quick one offs on smaller samples.
- hadley 14y agoWith 50,000 rows plyr should be fine, and I've tested it up to 1,000,000 and it's reasonably. The problem is more that it doesn't scale terribly well in the number of groups.