3 ms·
Sometimes it can take 12 or more hours to run the code on the millions of observations. There's also competition from other researchers who use computational re
by WhompingWindows 8y ago
Sometimes it can take 12 or more hours to run the code on the millions of observations. There's also competition from other researchers who use computational resources, which can mean I have to leave something running for hours due to the server being heavily queried. My workflow also doesn't allow easy interruption of the execution, sometimes it has to execute fully incorrectly before I can change an error or parameter.
- semi-extrinsic 8y agoI dunno how large your data set is, but I just set up for work a 16-core Threadripper workstation with 32 GB RAM and 1 TB M.2 SSD for approx. $2500. If it can regularly save you hours or days of waiting, getting something equivalent should be a no-brainer.
- deleted 8y ago[deleted]
- baldfat 8y agoI would then say your using the wrong tool for your problem? I can't imagine 12 hour runs. I would imagine Spark is a better bet or is that not an option? https://spark.apache.org/docs/latest/sparkr.html https://spark.apache.org/docs/latest/sparkr.html