3 ms·

I strongly recommend data.table R. Tidyverse is an improvement on base R, no question. Data.table has less intuitive syntax and can be harder to learn, but is l

I strongly recommend data.table R. Tidyverse is an improvement on base R, no question. Data.table has less intuitive syntax and can be harder to learn, but is lightning fast and memory efficient. If you're working with more than 1M rows, you should be using data.table. Here are some benchmarks: https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/

civilized 4y ago
I think it's great people like data.table's performance but I don't think performance is a compelling reason to be prescriptive about data.table. It's a reason data.table may be appealing to some. People can try them and decide for themselves if the performance is worth the trouble. In many cases the performance difference with dplyr is hardly noticeable and IMO readability of dplyr becomes the more meaningful advantage. I have plenty of projects with tables with millions of rows, and I rarely find dplyr to be slow enough to get frustrated about. And even then I'd rather endure, or maybe use dtplyr, or maybe use data.table for a small performance-critical part of the code, than make a wholesale switch.
- nojito 4y ago
  Performance is probably the last reason I use datatable. I just enjoy having to read much less code when compared to tidyverse and SQL. Plus the syntax difference between tidyverse and dt are overstated. https://atrebas.github.io/post/2019-03-03-datatable-dplyr/ https://atrebas.github.io/post/2019-03-03-datatable-dplyr/
  - civilized 4y ago
    I find data.table code as obscure as base R if not more so. Purported brevity (always illustrated only on very simple problems) doesn't compensate me enough for the obscurity. I could get used to it, I'm sure. I could also get used to doing everything in base R, pandas, or Clojure. Doesn't mean it's a good use of my time. I get the sense from how some people blog that basic "filter + group by + summarize" type pipelines on giant datasets are all they really do. data.table may be a great choice for that kind of thing. But I use dplyr as a tool of thought. I can solve complex, novel data processing and analytics problems in dplyr more precisely and efficiently than I can explain in words. Instead of having an idea and then having to think about how to code it, the ideas translate into dplyr in an obvious and automatic way. All that said, data.table's authors have worked much harder to provide advanced functionality, most notably non-equijoins, that it looks like the tidyverse may never catch up to. So if you need that kind of thing a lot, you need data.table or maybe DuckDB.
p10_user 4y ago
What if you are working with 1000s of rows? Still worthwhile?

civilized 4y ago

I think it's great people like data.table's performance but I don't think performance is a compelling reason to be prescriptive about data.table. It's a reason data.table may be appealing to some. People can try them and decide for themselves if the performance is worth the trouble. In many cases the performance difference with dplyr is hardly noticeable and IMO readability of dplyr becomes the more meaningful advantage. I have plenty of projects with tables with millions of rows, and I rarely find dplyr to be slow enough to get frustrated about. And even then I'd rather endure, or maybe use dtplyr, or maybe use data.table for a small performance-critical part of the code, than make a wholesale switch.

nojito 4y ago
Performance is probably the last reason I use datatable. I just enjoy having to read much less code when compared to tidyverse and SQL. Plus the syntax difference between tidyverse and dt are overstated. https://atrebas.github.io/post/2019-03-03-datatable-dplyr/ https://atrebas.github.io/post/2019-03-03-datatable-dplyr/
- civilized 4y ago
  I find data.table code as obscure as base R if not more so. Purported brevity (always illustrated only on very simple problems) doesn't compensate me enough for the obscurity. I could get used to it, I'm sure. I could also get used to doing everything in base R, pandas, or Clojure. Doesn't mean it's a good use of my time. I get the sense from how some people blog that basic "filter + group by + summarize" type pipelines on giant datasets are all they really do. data.table may be a great choice for that kind of thing. But I use dplyr as a tool of thought. I can solve complex, novel data processing and analytics problems in dplyr more precisely and efficiently than I can explain in words. Instead of having an idea and then having to think about how to code it, the ideas translate into dplyr in an obvious and automatic way. All that said, data.table's authors have worked much harder to provide advanced functionality, most notably non-equijoins, that it looks like the tidyverse may never catch up to. So if you need that kind of thing a lot, you need data.table or maybe DuckDB.

p10_user 4y ago

What if you are working with 1000s of rows? Still worthwhile?