5 ms·

Even as a big cloud detractor, I have to disagree with this. A lot of scientific computing doesn't need a persistent data center, since you are running a ton o

Even as a big cloud detractor, I have to disagree with this. A lot of scientific computing doesn't need a persistent data center, since you are running a ton of simulations that only take a week or so, and scientific computing centers at big universities are a big expense that isn't always well-utilized. Also, when they are full, jobs can wait weeks to run. These computing centers have fairly high overhead, too, although some of that is absorbed by the university/nonprofit who runs them. It is entirely possible that this dynamic, where universities pay some of the cost out of your grant overhead, makes these computing centers synthetically cheaper for researchers when they are actually more expensive. One other issue here is that scientific computing really benefits from ultra-low-latency infiniband networks, and the cloud providers offer something more similar to a virtualized RoCE system, which is a lot slower. That means accounting for cloud servers potentially being slower core-for-core.

lebovic 4y ago
Author here. I agree with your points! I use AWS for a computational biology company I'm working on. A lot of scientific computing can spin up and down within a couple hours on AWS and benefits from fast turnaround. Most academic HPCs (by # of clusters) are slower than a mega-cluster on AWS, not well utilized, and have a lot of bureaucratic process. That said, most of scientific computing (by % of total compute) happens in a different context. There's often a physical machine within the organization that's creating data (e.g. a DNA sequencer, particle accelerator, etc), and a well-maintained HPC cluster that analyzes that data. The researchers have already waited months for their data, so another couple weeks in a queue doesn't impact their cycle. For that context, AWS doesn't really make sense. I do think there's room for a cloud provider that's geared towards an HPC use-case, and doesn't have the app-inspired limits (e.g data transfer) like AWS, GCP, and Azure.
- primax 4y ago
  Honestly spot instances sound perfect for those kind of requirements
davidmr 4y ago
This is tangential to your point, but I’ll just mention that Azure has some properly specced out HPC gear: IB, FPGAs, the works. You used to be able to get time on a Cray XC with an Ares interconnect, but I never have occasion to use it, so I don’t know if you still can. They’ve been aggressively hiring top-notch HPC people for a while.
- hpcjoe 4y ago
  That's the Sentinel system. I worked on it when I was at Cray, and we did some covid stuff[1][2] with a researcher at UAH. We accelerated a docking code using some cool tech I created (in Perl, so there!) and some mods my teammates did to the queuing system. The work won some award at SC20[3] (fka Supercomputing conference). I had considered submitting for the Gordon Bell prize, which had been specifically requesting covid work, though I thought the stuff we had done wasn't terribly sexy. We were getting ~250-500x better performance than single CPU runs. Looking back over these, I gotta chuckle, as this (press releases) is pretty much the only time I'm called "Dr.". :D Back to the OPs points, they are right. In most cases, cloud doesn't make sense for traditional HPC workloads. There are some special cases where it does, those tend to be large ephemeral analysis pipelines, as in bioinformatics and related fields. But for hardcore distributed (mostly MPI) code, running for a long time on a set of nodes interconnected with low latency networks, dedicated local nodes are the better economic deal. During my stint at Cray, I was trying (quite hard) to get supercomputers, real classical ones, into cloud providers, or become a supercomputing cloud provider ourselves. The Met Office system is in Azure, is a Cray Shasta, but that was more of a special case. I couldn't get enough support for this. Such is life. I've moved on. Still doing HPC, but more throughput maximized. [1] https://www.uah.edu/science/departments/math/news/14954-uah-boosts-search-for-covid-19-drugs-using-hpe-cray-sentinel-supercomputer https://www.uah.edu/science/departments/math/news/14954-uah-... [2] A whole marketing writeup was done here https://www.hpe.com/us/en/newsroom/journey-to-accelerate-drug-discovery-for-covid-19.html https://www.hpe.com/us/en/newsroom/journey-to-accelerate-dru... . I tried very hard to correct the errors in the writeups. Sadly I wasn't successful. [3] https://baudry-lab.uah.edu/news#h.121c63ayp0k0 https://baudry-lab.uah.edu/news#h.121c63ayp0k0
  - 4y ago
esalman 4y ago
Neither grant agencies nor universities are ready to pay for commercial compute out of grant money. They'd rather have you run analysis on your work laptop/desktop they already provided. Even some of the folks who manage the HPC are unwilling to help researchers (many of whom are not programmers) to use the HPC, lest they mess up and damage the hardware. Source: I work at a tri-institutional collaboration research center in Georgia.

lebovic 4y ago

Author here. I agree with your points! I use AWS for a computational biology company I'm working on. A lot of scientific computing can spin up and down within a couple hours on AWS and benefits from fast turnaround. Most academic HPCs (by # of clusters) are slower than a mega-cluster on AWS, not well utilized, and have a lot of bureaucratic process. That said, most of scientific computing (by % of total compute) happens in a different context. There's often a physical machine within the organization that's creating data (e.g. a DNA sequencer, particle accelerator, etc), and a well-maintained HPC cluster that analyzes that data. The researchers have already waited months for their data, so another couple weeks in a queue doesn't impact their cycle. For that context, AWS doesn't really make sense. I do think there's room for a cloud provider that's geared towards an HPC use-case, and doesn't have the app-inspired limits (e.g data transfer) like AWS, GCP, and Azure.

primax 4y ago
Honestly spot instances sound perfect for those kind of requirements

davidmr 4y ago

This is tangential to your point, but I’ll just mention that Azure has some properly specced out HPC gear: IB, FPGAs, the works. You used to be able to get time on a Cray XC with an Ares interconnect, but I never have occasion to use it, so I don’t know if you still can. They’ve been aggressively hiring top-notch HPC people for a while.

hpcjoe 4y ago
That's the Sentinel system. I worked on it when I was at Cray, and we did some covid stuff[1][2] with a researcher at UAH. We accelerated a docking code using some cool tech I created (in Perl, so there!) and some mods my teammates did to the queuing system. The work won some award at SC20[3] (fka Supercomputing conference). I had considered submitting for the Gordon Bell prize, which had been specifically requesting covid work, though I thought the stuff we had done wasn't terribly sexy. We were getting ~250-500x better performance than single CPU runs. Looking back over these, I gotta chuckle, as this (press releases) is pretty much the only time I'm called "Dr.". :D Back to the OPs points, they are right. In most cases, cloud doesn't make sense for traditional HPC workloads. There are some special cases where it does, those tend to be large ephemeral analysis pipelines, as in bioinformatics and related fields. But for hardcore distributed (mostly MPI) code, running for a long time on a set of nodes interconnected with low latency networks, dedicated local nodes are the better economic deal. During my stint at Cray, I was trying (quite hard) to get supercomputers, real classical ones, into cloud providers, or become a supercomputing cloud provider ourselves. The Met Office system is in Azure, is a Cray Shasta, but that was more of a special case. I couldn't get enough support for this. Such is life. I've moved on. Still doing HPC, but more throughput maximized. [1] https://www.uah.edu/science/departments/math/news/14954-uah-boosts-search-for-covid-19-drugs-using-hpe-cray-sentinel-supercomputer https://www.uah.edu/science/departments/math/news/14954-uah-... [2] A whole marketing writeup was done here https://www.hpe.com/us/en/newsroom/journey-to-accelerate-drug-discovery-for-covid-19.html https://www.hpe.com/us/en/newsroom/journey-to-accelerate-dru... . I tried very hard to correct the errors in the writeups. Sadly I wasn't successful. [3] https://baudry-lab.uah.edu/news#h.121c63ayp0k0 https://baudry-lab.uah.edu/news#h.121c63ayp0k0
- 4y ago

esalman 4y ago

Neither grant agencies nor universities are ready to pay for commercial compute out of grant money. They'd rather have you run analysis on your work laptop/desktop they already provided. Even some of the folks who manage the HPC are unwilling to help researchers (many of whom are not programmers) to use the HPC, lest they mess up and damage the hardware. Source: I work at a tri-institutional collaboration research center in Georgia.