4 ms·

Also worth plugging the advanced speed of R’s data.table package which continues to trump dplyr to this day. The syntax is also more compact and straightforward

Also worth plugging the advanced speed of R’s data.table package which continues to trump dplyr to this day. The syntax is also more compact and straightforward once you understand how to query data with it.

minimaxir 5y ago
There is an official dplyr extension that leverages data.table: https://dtplyr.tidyverse.org/ https://dtplyr.tidyverse.org/
civilized 5y ago
I don't like it as much as dplyr and I stand behind that. It's too "clever", especially with respect to joins. Everything is fine "once you understand how to use it", even assembly code, but it's not equally expressive or intuitive. So I don't value data.table speed that much, it's my thinking and typing speed that's usually the limiting factor. I would always recommend dplyr over anything else for someone learning how to use tables. I also can't help but point out that data.table has the worst first FAQ answer I've ever seen in software documentation: https://cran.r-project.org/web/packages/data.table/vignettes/datatable-faq.html#j-num https://cran.r-project.org/web/packages/data.table/vignettes.... Just astonishingly bad. I could write an essay about the unique and diverse ways in which this thing is both incredibly poorly organized and deeply user-hostile. But if you truly have a need for speed on large datasets, it may be for you.
- nojito 5y ago
  The FAQ isn't for new data.table users. https://rdatatable.gitlab.io/data.table/articles/datatable-intro.html https://rdatatable.gitlab.io/data.table/articles/datatable-i... Which is why it isn't really linked anywhere else.
  - deleted 5y ago
    [deleted]
    - temp8964 5y ago
      You should stop insulting others and yourself.
      - craftinator 5y ago
        The irony in the parent comment is not nearly subtle enough.
        temp8964 5y ago
        Please just stop.
temp8964 5y ago
The easiest to understand data frame API syntax is SQL: select cols from df where rows match condition group by grouping cols. data.table syntax is just like that. But less verbose. Plus super fast. No reason to not love it.
- civilized 5y ago
  I agree that if that's all you do with data, data.table makes it easy.
vatican_banker 5y ago
In what way data.table trumps dplyr? Genuinely interested in knowing. While data.table is faster than dplyr, data manipulations with data.table are difficult to read/understand/maintain. dplyr also grew into a full-fledge list of libraries to work on data-related projects (the tidyverse). These libraries are _very_ well thought out and enables productivity with minimal learning curve [anecdotal]
- RcrdBrt 5y ago
  Anecdotal data: I found that data.table ingestion speed with fread() trumps absolutely everything else
  - civilized 5y ago
    This observation is pretty widely shared.
- extr 5y ago
  the easiest way to think about it is data.table is for people who are doing a lot of exploratory data analysis every day. If you're doing the same thing over and over, it makes sense to create a DSL specific to that task and optimize the hell out of it. that's basically data.table. dplyr is for everyone else, and it's great and important that it exists, because most people don't want to (and shouldn't need to) learn a DSL to do some basic filtering/sorting/grouping of 100mb of data.
- nojito 5y ago
  The difficulty to read is a misnomer. Dt[rows, columns, groups] Assuming your dplyr code is generally split apply combine, the dt version is shorter and easier to reason around. https://atrebas.github.io/post/2019-03-03-datatable-dplyr/ https://atrebas.github.io/post/2019-03-03-datatable-dplyr/
  - civilized 5y ago
    I disagree. Doing data manipulation one action at a time in a piped sequence is easiest to reason about because the state right before you apply a new operation is always clear. data.table, on the other hand, is a fancy clever gadget with many knobs and buttons you have to turn and press just so to get the desired result. It's only simple if all you do is filter, group by, and summarize. To illustrate, let's look at what you have to do in data.table in order to achieve the equivalent of a grouped filter in dplyr (from the dtplyr translation vignette): dplyr: df %>% group_by(a) %>% filter(b < mean(b)) data.table: DT[DT[, .I[b < mean(b)], by = .(a)]$V1] Compared to the simple, declarative feel of the dplyr, there's a lot of weird stuff going on in the data.table version. You have to put DT inside itself? What is .I? Where did V1 come from? Janky stuff. (And yes I know precisely what is going on in the data.table version, I just think it's ugly and illustrates my point about composability and legibility extremely well.) The reason data.table has all these independent knobs is because it wants you to cram your entire query into a single command, so it can optimize the query more easily and squeeze every drop of performance. NOT because it's more understandable, because it isn't. The best of both worlds -- an optimizable query and one-action-at-a-time syntax -- can be achieved with a lazy system like Apache Spark or dtplyr.
gullywhumper 5y ago
One plus with dplyr is that I can share the code with non-R programmers (and even some non-programmers) and they can follow what is happening pretty easily, while data.table takes some more explanation.

minimaxir 5y ago

There is an official dplyr extension that leverages data.table: https://dtplyr.tidyverse.org/ https://dtplyr.tidyverse.org/

civilized 5y ago

I don't like it as much as dplyr and I stand behind that. It's too "clever", especially with respect to joins. Everything is fine "once you understand how to use it", even assembly code, but it's not equally expressive or intuitive. So I don't value data.table speed that much, it's my thinking and typing speed that's usually the limiting factor. I would always recommend dplyr over anything else for someone learning how to use tables. I also can't help but point out that data.table has the worst first FAQ answer I've ever seen in software documentation: https://cran.r-project.org/web/packages/data.table/vignettes/datatable-faq.html#j-num https://cran.r-project.org/web/packages/data.table/vignettes.... Just astonishingly bad. I could write an essay about the unique and diverse ways in which this thing is both incredibly poorly organized and deeply user-hostile. But if you truly have a need for speed on large datasets, it may be for you.

nojito 5y ago
The FAQ isn't for new data.table users. https://rdatatable.gitlab.io/data.table/articles/datatable-intro.html https://rdatatable.gitlab.io/data.table/articles/datatable-i... Which is why it isn't really linked anywhere else.
- deleted 5y ago
  [deleted]
  - temp8964 5y ago
    You should stop insulting others and yourself.
    - craftinator 5y ago
      The irony in the parent comment is not nearly subtle enough.
      - temp8964 5y ago
        Please just stop.

temp8964 5y ago

The easiest to understand data frame API syntax is SQL: select cols from df where rows match condition group by grouping cols. data.table syntax is just like that. But less verbose. Plus super fast. No reason to not love it.

civilized 5y ago
I agree that if that's all you do with data, data.table makes it easy.

vatican_banker 5y ago

In what way data.table trumps dplyr? Genuinely interested in knowing. While data.table is faster than dplyr, data manipulations with data.table are difficult to read/understand/maintain. dplyr also grew into a full-fledge list of libraries to work on data-related projects (the tidyverse). These libraries are _very_ well thought out and enables productivity with minimal learning curve [anecdotal]

RcrdBrt 5y ago
Anecdotal data: I found that data.table ingestion speed with fread() trumps absolutely everything else
- civilized 5y ago
  This observation is pretty widely shared.
extr 5y ago
the easiest way to think about it is data.table is for people who are doing a lot of exploratory data analysis every day. If you're doing the same thing over and over, it makes sense to create a DSL specific to that task and optimize the hell out of it. that's basically data.table. dplyr is for everyone else, and it's great and important that it exists, because most people don't want to (and shouldn't need to) learn a DSL to do some basic filtering/sorting/grouping of 100mb of data.
nojito 5y ago
The difficulty to read is a misnomer. Dt[rows, columns, groups] Assuming your dplyr code is generally split apply combine, the dt version is shorter and easier to reason around. https://atrebas.github.io/post/2019-03-03-datatable-dplyr/ https://atrebas.github.io/post/2019-03-03-datatable-dplyr/
- civilized 5y ago
  I disagree. Doing data manipulation one action at a time in a piped sequence is easiest to reason about because the state right before you apply a new operation is always clear. data.table, on the other hand, is a fancy clever gadget with many knobs and buttons you have to turn and press just so to get the desired result. It's only simple if all you do is filter, group by, and summarize. To illustrate, let's look at what you have to do in data.table in order to achieve the equivalent of a grouped filter in dplyr (from the dtplyr translation vignette): dplyr: df %>% group_by(a) %>% filter(b < mean(b)) data.table: DT[DT[, .I[b < mean(b)], by = .(a)]$V1] Compared to the simple, declarative feel of the dplyr, there's a lot of weird stuff going on in the data.table version. You have to put DT inside itself? What is .I? Where did V1 come from? Janky stuff. (And yes I know precisely what is going on in the data.table version, I just think it's ugly and illustrates my point about composability and legibility extremely well.) The reason data.table has all these independent knobs is because it wants you to cram your entire query into a single command, so it can optimize the query more easily and squeeze every drop of performance. NOT because it's more understandable, because it isn't. The best of both worlds -- an optimizable query and one-action-at-a-time syntax -- can be achieved with a lazy system like Apache Spark or dtplyr.

gullywhumper 5y ago

One plus with dplyr is that I can share the code with non-R programmers (and even some non-programmers) and they can follow what is happening pretty easily, while data.table takes some more explanation.