4 ms·

I'm curious -- is there something special about the calculations which requires bespoke supercomputers rather than large cloud installations?

by JackFr 8y ago

evanb 8y ago
Ha, I just replied to your other message asking for an AWS comparison. The answer is yes, the communications has to be substantially better than anything you can order up. LLNL also has national security concerns that are unparalleled by most AWS applications ;)
SEJeff 8y ago
Likely fast interconnects and higher performance (bare metal vs VMs). Cloud instances tend to have a lot less consistent performance due to oversubscription of the physical hardware, so you have high "jitter". High performance computing systems (like this) tend to have a better grasp of the required resources and can push the hardware to the max without oversubscription too much. The biggest difference however, is that the goal for supercomputers such as this is as high of an average usage rate as feasible. The cloud is abysmal for this where it is more for bursting to say 100k cpu core jobs. For systems like this, they'd want the average utilization to be 80%+ all the time. The cost of constant cloud computing like this, even using reserved instances, would be a multiple of the 162 million USD it cost to build this. Also, the IO patterns you'll see for large amounts of data like this (almost certainly many Petabytes) isn't nearly as cost effective as it is to hire a team to build it yourself.
- evanb 8y ago
  Not being in industry, I forget that AWS doesn't have 100% utilization 100% of the time. One of the reasons the lab likes us lattice QCD people is that we always have more computing to do, and are happy to wait in the queue if we get to run at all. So we really help keep the utilization very high. If the machine ever sits empty, that's a waste of money. You're right that the IO tends to be very high performance and high throughput, too.
  - SEJeff 8y ago
    Yup totally understood. We have a much smaller (but still massive) supercomputer for $REAL_JOB where there is always a queue of work to do with embarrassingly parallel jobs or ranks and ranks of MPI work to do. When we add more resources, the users simply can run their work faster, but it never really stops no matter how much hardware we add. As much as people love to hate them, I'd love to see you get IO profiles remotely similar to what you can get with Lustre or Spectrum Scale (gpfs). They're simply in an entirely different ballpark compared to anything in any public cloud.
    - evanb 8y ago
      We're lucky in the sense that the IO for LQCD is small (compared to other scientific applications), in that we're usually only reading or writing gigabytes to terbytes. But also our code uses parallel HDF5 and it's someone else's job to make sure that works well :)
- wnissen 8y ago
  To add to this, the jitter becomes more and more important the larger the scale you are running. The whole calculation (potentially the whole machine) is sitting around waiting for the slowest single task to finish up. Thus you get counterintuitive architectures, like a dedicated core just to handle networking operations. It seems like a bad deal to throw away 6% of your computation, but the alternative is even worse utilization. Highly coupled calculations are a very different beast because they cannot be executed out of sequence.
rbanffy 8y ago
Cloud is great when you have variable usage. These machines are probably driven near 100% all the time. In that scenario, they are probably more cost-effective than cloud infrastructure.
halbritt 8y ago
Supercomputers tend to need very high bisectional bandwidth. With the Clos network style topologies that are commonplace in large data centers today, I'm not sure one couldn't achieve decent results in the public cloud. AWS networking is pretty terrible, but in GCP, I can get 2gbps per core up to 16Gbps for an 8-core instance. For any bare metal deployment, I'm going to be maxed out around 100Gbps which will be close to saturating an x16 PCIe bus. It's hard to find a dual-cpu frequency optimized processor with less than 8 cores and I'm not sure that'd be cost effective. With hyperthreading, that yields 32 usable cores or around 3.125gbps per core. Even still, I wager they'd go for better density. Also, I can get 8 GPUs along with that 8 core/16gbps instance in GCP. Sounds totally doable to me.
- halbritt 8y ago
  My back of the napkin calculation says that I can get 270 petaflops with 2160 n1-highmem-16 each with 8 v100 gpu on preemptible instances costing roughly $13k/hr or about $10m/mo
  - rdtsc 8y ago
    So with $120m/y in less than 2 years you'd exceed the price of the whole thing and also likely get worse interconnect speed and possibly raw computational speed. AWS looks like a bad deal here.
    - karambahh 8y ago
      If you have a constant load, on premise is always cheaper. Scientific computation has, for a large enough organization, a 100% If you have a variable load, cloud infrastructure may make sense if you can easily auto-scale. In my experience, most business real world applications are multi tiered applications with variable loads hence are a good fit for cloud infrastructure. However, attaining the required application flexibility and KPIs for efficient auto scaling is quite hard and require strong functional & technical expertise.
      - 8y ago

I'm curious -- is there something special about the calculations which requires bespoke supercomputers rather than large cloud installations?

by JackFr 8y ago

I'm curious -- is there something special about the calculations which requires bespoke supercomputers rather than large cloud installations?

evanb 8y ago

Ha, I just replied to your other message asking for an AWS comparison. The answer is yes, the communications has to be substantially better than anything you can order up. LLNL also has national security concerns that are unparalleled by most AWS applications ;)

SEJeff 8y ago

Likely fast interconnects and higher performance (bare metal vs VMs). Cloud instances tend to have a lot less consistent performance due to oversubscription of the physical hardware, so you have high "jitter". High performance computing systems (like this) tend to have a better grasp of the required resources and can push the hardware to the max without oversubscription too much. The biggest difference however, is that the goal for supercomputers such as this is as high of an average usage rate as feasible. The cloud is abysmal for this where it is more for bursting to say 100k cpu core jobs. For systems like this, they'd want the average utilization to be 80%+ all the time. The cost of constant cloud computing like this, even using reserved instances, would be a multiple of the 162 million USD it cost to build this. Also, the IO patterns you'll see for large amounts of data like this (almost certainly many Petabytes) isn't nearly as cost effective as it is to hire a team to build it yourself.

evanb 8y ago
Not being in industry, I forget that AWS doesn't have 100% utilization 100% of the time. One of the reasons the lab likes us lattice QCD people is that we always have more computing to do, and are happy to wait in the queue if we get to run at all. So we really help keep the utilization very high. If the machine ever sits empty, that's a waste of money. You're right that the IO tends to be very high performance and high throughput, too.
- SEJeff 8y ago
  Yup totally understood. We have a much smaller (but still massive) supercomputer for $REAL_JOB where there is always a queue of work to do with embarrassingly parallel jobs or ranks and ranks of MPI work to do. When we add more resources, the users simply can run their work faster, but it never really stops no matter how much hardware we add. As much as people love to hate them, I'd love to see you get IO profiles remotely similar to what you can get with Lustre or Spectrum Scale (gpfs). They're simply in an entirely different ballpark compared to anything in any public cloud.
  - evanb 8y ago
    We're lucky in the sense that the IO for LQCD is small (compared to other scientific applications), in that we're usually only reading or writing gigabytes to terbytes. But also our code uses parallel HDF5 and it's someone else's job to make sure that works well :)
wnissen 8y ago
To add to this, the jitter becomes more and more important the larger the scale you are running. The whole calculation (potentially the whole machine) is sitting around waiting for the slowest single task to finish up. Thus you get counterintuitive architectures, like a dedicated core just to handle networking operations. It seems like a bad deal to throw away 6% of your computation, but the alternative is even worse utilization. Highly coupled calculations are a very different beast because they cannot be executed out of sequence.

rbanffy 8y ago

Cloud is great when you have variable usage. These machines are probably driven near 100% all the time. In that scenario, they are probably more cost-effective than cloud infrastructure.

halbritt 8y ago

Supercomputers tend to need very high bisectional bandwidth. With the Clos network style topologies that are commonplace in large data centers today, I'm not sure one couldn't achieve decent results in the public cloud. AWS networking is pretty terrible, but in GCP, I can get 2gbps per core up to 16Gbps for an 8-core instance. For any bare metal deployment, I'm going to be maxed out around 100Gbps which will be close to saturating an x16 PCIe bus. It's hard to find a dual-cpu frequency optimized processor with less than 8 cores and I'm not sure that'd be cost effective. With hyperthreading, that yields 32 usable cores or around 3.125gbps per core. Even still, I wager they'd go for better density. Also, I can get 8 GPUs along with that 8 core/16gbps instance in GCP. Sounds totally doable to me.

halbritt 8y ago
My back of the napkin calculation says that I can get 270 petaflops with 2160 n1-highmem-16 each with 8 v100 gpu on preemptible instances costing roughly $13k/hr or about $10m/mo
- rdtsc 8y ago
  So with $120m/y in less than 2 years you'd exceed the price of the whole thing and also likely get worse interconnect speed and possibly raw computational speed. AWS looks like a bad deal here.
  - karambahh 8y ago
    If you have a constant load, on premise is always cheaper. Scientific computation has, for a large enough organization, a 100% If you have a variable load, cloud infrastructure may make sense if you can easily auto-scale. In my experience, most business real world applications are multi tiered applications with variable loads hence are a good fit for cloud infrastructure. However, attaining the required application flexibility and KPIs for efficient auto scaling is quite hard and require strong functional & technical expertise.
    - 8y ago