3 ms·
It's also not hard these days to fire up an instance on AWS or other cloud service which will have up to a few terabytes of RAM. Plus, you can always just bite
by gwern 3y ago
It's also not hard these days to fire up an instance on AWS or other cloud service which will have up to a few terabytes of RAM.
Plus, you can always just bite the bullet and do it all out of swap - might not be as bad as you fear, and often can optimize whatever you're doing to be more online. There are countless tricks left over from the old days for doing out-of-core operations efficiently, whether disk or tape.
- bradknowles 3y agoIf you're firing up anything on AWS with that kind of RAM, you want to make sure you understand how much that's going to cost. The per second cost is always going to look really low and you will tend not to worry about it. However, once you do the math, you could find that you're easily and quickly spending more money than it would cost to buy a machine to run locally with that much RAM.
- gwern 3y agoIt's been fine when I've used terabyte instances for dynamic programming stuff similar to OP. You are babysitting it and waiting for a specific answer, not running a permanent service which will hemorrhage $$$ when you forget to turn it off. At dollars an hour, hard to forget...
- bradknowles 3y agoSo long as you actually play close attention to it, that's not a problem. The issue is that many people fail to pay close attention, and that's where the money can really rack up.