3 ms·
I did a similar analysis several years ago. The two fastest solutions that didn't involve writing lower-level code aren't mentioned: A specialized program targe
by absherwin 14y ago
I did a similar analysis several years ago. The two fastest solutions that didn't involve writing lower-level code aren't mentioned: A specialized program targeted at the insurance industry called Emblem and Stata. If I recall, they ran as fast on a single system as RevoScale does on a cluster. Of course, comparing different benchmarks is fraught with peril.
The SAS performance reported is surprisingly bad and likely is a result of the data processing steps involved. A SAS procedure is much more similar to a program than a function. While SAS wasn't the fastest solution, most of its slowness was due to building the design matrix rather than running the actual regression. There are a number of options for both tuning how this is done and caching this preprocessing. The biggest different would be using SSD's on the machine which seems unlikely given the results.
- dbecker 14y agoStata won't work on data that don't fit in memory. It is fast, but it isn't an option for very large datasets.
- absherwin 14y agoTrue but 150MM records with 70 degrees of freedom would likely be doable with 16GB of RAM and almost certainly with 32 as most of the variables are likely boolean and so can be stored in a single byte which Stata supports unlike R and SAS.