4 ms·

Disclosure: I used to work on Google Cloud. Perlmutter seems like an awesome system. But, I think the “ai exaflops” is a “X GPUS times the NVIDIA peak rate”. T

by boulos 5y ago

Disclosure: I used to work on Google Cloud. Perlmutter seems like an awesome system. But, I think the “ai exaflops” is a “X GPUS times the NVIDIA peak rate”. The new sparsity features on A100 are promising, but haven’t been demonstrated to be nearly as awesome in practice (yet). It also all comes down to workloads: large-scale distributed training is a funny workload! It’s not like LINPACK. If you make your model compute intensive enough, then the networking need mostly becomes bandwidth (for which multi-hundred Gbps worth of NICs is handy) but even without it there are lots of ways to max out your compute. Similarly, storage is a serious need for say giant video corpora, but not for things like text! GPT-2 had like a 40 GiB corpus. For those asking about largest-scale cloud GPU runs, there are basically three examples (chronological) - the work OpenAI did on Five (thousands of V100s on GCP) - the IceCube science work [1] on many clouds (51000 GPUs at peak!) - OpenAI’s 10000-V100 cluster they used for GPT-3 The A100s used here are recently released and another step-change in perf per part (and memory). All major providers now offer them, though with different density and networking configurations (GCP went with 16 in a single box, most folks went with 8, some have lots of networking, etc.). What everyone should be asking is: what awesome stuff is NERSC / LBL going to do with Perlmutter? You can’t just rent one for a few hours on GCP or any other provider :). (But, fwiw, most usage will be small slices: this is the sad fate of giant supercomputers!) [1] https://insidehpc.com/2019/11/sdsc-conducts-50000-gpu-cloudburst-experiment-with-wisconsin-icecube-particle-astrophysics-center/ https://insidehpc.com/2019/11/sdsc-conducts-50000-gpu-cloudb...

prpl 5y ago
Perlmutter is a new kind of supercomputer - It’s mini-cloud disguised as a supercomputer smaller DOE projects to funnel their compute to. Cori was kind of that too, but IIRC kind of on accident because the Xeon Phi wasn’t available during delivery time, so the first “phase” was Haswell nodes. I’m not sure what this means for whole-machine reservation but hopefully that’s a thing of the past in any case. Ultimately this is a machine to solve everybody’s needs, not just ML needs (although it must solve those needs very well in any case) I’m not sure Cori will be remembered particularly fondly, but Perlmutter will probably be because it seems like it will be versatile enough to meet everyone’s needs. For those wondering why not cloud - most simulations generally make sense in cloud because there’s nowhere near the data movement/storage involved and anything in particle physics is always embarrassingly parallel. The data isn’t moving there though - it is very cheap to keep the data at a place like NERSC especially with tape in the mix.
- dekhn 5y ago
  One of the reasons I left DOE for industry many years ago was that so much of the funding went to large supercomputers with about 8 users (the folks who had made their code scale to the largest size) while vast majority of people with compute needs went unsupported. Tape is worthless. Tape is where you put data you never want to retrieve again.
  - prpl 5y ago
    I understand most of what you are saying, and it isn’t pleasant in many cases but I think the situation is improving. Yes, all that happens, and most project computing (Office of Science) is usually pushed to NERSC, but it really depends on many factors. NERSC has a lot of users (much more than 8), probably the most out of the DOE complex, since it’s a user facility. Big projects (>$500M) don’t often have this problem and most of them will push for clusters they own at their home lab. Small projects are most likely to have trouble either in procuring resources or effectively using super computing resources. A dirty secret is many users just use MPI for a job queue/coordination so their jobs “scale” when large reservations are required, but IB is not really exercised in those situations. When you realize that, it’s easy to realize most workflows scale (as long as disk is considered), but very annoying from a development perspective. Tape isn’t worthless because you have to have a disaster plan or long term data preservation plan and it’s usually just a part of that, but some labs/projects have historically pushed as part of the integral data management plan for active/warm data and that is indeed worthless.
    - dekhn 5y ago
      I really doubt people are running embarassingly parallel codes on NERSC supercomputers. I was a PI at NERSC and I was told explicitly, repeatedly, I couldn't do that. That's why I moved to Open Science Grid, and eventually Google's compute platform. Nope, tape is just useless. Live but cold hard drives.

by boulos 5y ago

prpl 5y ago

Perlmutter is a new kind of supercomputer - It’s mini-cloud disguised as a supercomputer smaller DOE projects to funnel their compute to. Cori was kind of that too, but IIRC kind of on accident because the Xeon Phi wasn’t available during delivery time, so the first “phase” was Haswell nodes. I’m not sure what this means for whole-machine reservation but hopefully that’s a thing of the past in any case. Ultimately this is a machine to solve everybody’s needs, not just ML needs (although it must solve those needs very well in any case) I’m not sure Cori will be remembered particularly fondly, but Perlmutter will probably be because it seems like it will be versatile enough to meet everyone’s needs. For those wondering why not cloud - most simulations generally make sense in cloud because there’s nowhere near the data movement/storage involved and anything in particle physics is always embarrassingly parallel. The data isn’t moving there though - it is very cheap to keep the data at a place like NERSC especially with tape in the mix.

dekhn 5y ago
One of the reasons I left DOE for industry many years ago was that so much of the funding went to large supercomputers with about 8 users (the folks who had made their code scale to the largest size) while vast majority of people with compute needs went unsupported. Tape is worthless. Tape is where you put data you never want to retrieve again.
- prpl 5y ago
  I understand most of what you are saying, and it isn’t pleasant in many cases but I think the situation is improving. Yes, all that happens, and most project computing (Office of Science) is usually pushed to NERSC, but it really depends on many factors. NERSC has a lot of users (much more than 8), probably the most out of the DOE complex, since it’s a user facility. Big projects (>$500M) don’t often have this problem and most of them will push for clusters they own at their home lab. Small projects are most likely to have trouble either in procuring resources or effectively using super computing resources. A dirty secret is many users just use MPI for a job queue/coordination so their jobs “scale” when large reservations are required, but IB is not really exercised in those situations. When you realize that, it’s easy to realize most workflows scale (as long as disk is considered), but very annoying from a development perspective. Tape isn’t worthless because you have to have a disaster plan or long term data preservation plan and it’s usually just a part of that, but some labs/projects have historically pushed as part of the integral data management plan for active/warm data and that is indeed worthless.
  - dekhn 5y ago
    I really doubt people are running embarassingly parallel codes on NERSC supercomputers. I was a PI at NERSC and I was told explicitly, repeatedly, I couldn't do that. That's why I moved to Open Science Grid, and eventually Google's compute platform. Nope, tape is just useless. Live but cold hard drives.