3 ms·
In my case your guess is completely correct, as I learned data.table in a financial company analyzing large insurance datasets :) I apologize if I came across
by extr 5y ago
In my case your guess is completely correct, as I learned data.table in a financial company analyzing large insurance datasets :)
I apologize if I came across as a hardliner. Sometimes I feel like data.table is not well advertised for how capable it is, so I will defend the library if given the chance. Surprisingly how many "big data workloads" you can replace with a high memory cloud instance and a simple data.table script. Cheers to using the right tool for the right job.
- civilized 5y agoMy background is (somewhat) similar but for some reason I've been a stick in the mud keeping on with dplyr. Somehow I feel the verbosity helps make sure I get things right. And a lot of my code is not throwaway, I have pipelines I have to maintain and teach others to maintain. (There's a guy on my team replacing some of my uglier code with simple data.table non-equijoins and I can't even argue with him as I mentioned earlier!) I'm glad you're having success with data.table and I totally support you against the forces of evil trying to make us use Spark or whatever is the latest big data nonsense to analyze a few million rows. It's like how we may not agree what project management tool to use but we all agree it's not JIRA :) I may end up switching to data.table after all. I find dplyr easier to reason about for complex production pipelines that need to be precisely "correct", but all the package developers are raising the bar all the time and data.table may be OK for this use case by now. I definitely do feel the pain point of dplyr slowness here and there.