4 ms·
While data.table does have impressive benchmarks, the rise of dplyr in the years since with its significantly more readable and concise syntax makes life a lot
by minimaxir 10y ago
While data.table does have impressive benchmarks, the rise of dplyr in the years since with its significantly more readable and concise syntax makes life a lot easier for data sets that are not super large. (https://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html https://cran.rstudio.com/web/packages/dplyr/vignettes/introd...)
The vignette linked above describes some compatability between dplyr syntax and data.table, but I admit I have not tested that.
Nowadays, if you are working with multigigabyte datasets, it may be worth looking into Spark/SparkR (https://spark.apache.org/docs/latest/sparkr.html https://spark.apache.org/docs/latest/sparkr.html) for manipulating big data in a scalable manner instead, as there is feature parity with data.table + analytical tools [GLM] as a bonus. (The syntax is still messy, to my annoyance)
- nerdponx 10y agoNowadays I almost always use both data.table and dplyr together in my projects. It's a really expressive combination.
- claytonjy 10y agoDo you use data.table syntax, or do you use dplyr with the tbl_dt class to abstract that away? I was a data.table acolyte for years, but after being forced to learn dplyr I can't imagine going back. I'm hoping if I need update-by-reference in the future I can use tbl_dt but I haven't played with it yet.
- nerdponx 10y agoIf I'm using them together, I'll start off a pipeline with a big DT expression, then pipe it through a few dplyr verbs as a kind of "post processing." I find that some tasks are frustrating with one package but easy with the other, so more frequently I just have some "dplyr lines" and some "data.table lines" in my scripts. And the compatibility between the two has improved a lot, so you don't get so many "invalid selref" warnings and unexpected behavior when you combine them in pipelines
- localhost 10y agoYou can rent an r3.8xlarge instance with 344GB RAM on AWS for $2.66/hr. If that isn't enough, an x1.32xlarge instance with ~2TB RAM is $13.34/hr. Assuming your data is already in AWS, of course. Of course those machines have a crap ton of CPU power (I'm explicitly not saying cores because I'm mystified by what vCPU really means) as well which you will be hard pressed to take advantage of but if it's RAM you need they're super-cheap. If you prefer Azure, a G5 instance with 448GB RAM is available for $9.65/hr. Not a lot of need for Spark/SparkR for the vast majority of data sets given the cheapness of compute these days.
- qaq 10y agovCPU is a hyperthread
- michaelsbradley 10y agoThe CPU power would not be too hard to utilize, if one's task is amenable to fairly coarse-grained parallelization, which is rather easy to implement in R scripts with the help of foreach[1] and doFuture[2]. Such an approach worked to great effect for me recently, when I needed to perform a `zoo::rollapply`[3] across a time series with tens of millions of rows. The speedup when throwing more cores (7, on my laptop) at it is roughly linear. If I ever need to scale up the analysis to hundreds of millions of rows, the 128 vCPUs of an x1 EC2 instance would be well worth the $$/hour. [1] https://cran.r-project.org/web/packages/foreach/index.html https://cran.r-project.org/web/packages/foreach/index.html [2] https://cran.r-project.org/web/packages/doFuture/index.html https://cran.r-project.org/web/packages/doFuture/index.html [3] http://www.rdocumentation.org/topics/1505094 http://www.rdocumentation.org/topics/1505094
- huac 10y agoShameless self promotion: my `slide_apply` function shows 5-10x improvement over `zoo::rollapply` (despite no Rcpp call, somehow...) https://gist.github.com/stillmatic/fadfd3269b900e1fd7ee https://gist.github.com/stillmatic/fadfd3269b900e1fd7ee if your function has a well-defined rolling form, e.g. rolling mean or standard deviation, you should use a pre-optimized function for that. package `catools` has a bunch of useful ones.
- i000 10y agoThe syntax is very concise and cryptic, but what is messy about dt[i,j,k]? data.table transformed the way I do analysis. Honestly probably 75% of my lines have are dt[] calls. It allows me to analyze on 500M row data.sets routinely on a powerful workstation interactively.
- IndianAstronaut 10y agoHow much memory does a 500m row data set use in data.table?
- akg_67 10y agoWhile I like and use dplyr time to time, I wish it could handle POSIXlt datetime. Also, it seems to barf on some dataframes just because the df contained a column format, dplyr doesn't support even if it is not being used in processing.
- huac 10y agojust use lubridate, it has a lot less headache to it. my beef is with the new tibble-ish format of dplyr. the format doesn't support lists as cells, unlike data frames. right now, `tbl_df` is a good in-between, but since dplyr is converging to the tibble/feather format, it's gonna be rough