33 ms·

I'm suspicious of the author's actual experience. The fact that scientific computing has a different pattern than the typical web app is actually a good thing.

I'm suspicious of the author's actual experience. The fact that scientific computing has a different pattern than the typical web app is actually a good thing. If you can architect large batch jobs to use spot instances, it's 50-80% cheaper. Also this bit: "you can keep your servers at 100% utilization by maintaining a queue of requested jobs" isn't true in practice. The pattern of research is the work normally comes in waves. You'll want to train a new model or run a number of large simulations. And then there will be periods of tweaking and work on other parts. And then more need for a lot of training. Yes, you can always find work to put on a cluster to keep it >90% utilization, but if it can be elastic (and has compute has budget attached to it), it will rise and fall.

lebovic 4y ago
Author here! I worked for the computing infrastructure for a DNA sequencing facility, and I run a computational biology infrastructure company (trytoolchest.com, YC W22). Both are built on AWS, so I do think AWS in scientific computing has its use-cases – mostly in places where you can't saturate a queue or you want a fast cycle time. Spot instances are still pretty expensive for a steady queue (2x of Hetzer monthly costs, for reference), and you still have to pay AWS data transfer egress costs – which are at least 30x more expensive than a colo or on-prem, if you're saturating a 1 Gbps link. This post was born from frustration at AWS for their pricing and offerings after trying to get people to switch to AWS in scientific computing for years :)