4 ms·

This is a nothing burger, I don't mean that in a derisive way though. MPI and Spark solve very different problems. The overlap is basically zero and the fact t

This is a nothing burger, I don't mean that in a derisive way though. MPI and Spark solve very different problems. The overlap is basically zero and the fact that MPI is flat while Spark fluctuates shows this. HPC is a small small fraction of the job market, the number of folks involved in HPC is tiny compared to all the click-log analyzing Spark and Hadoop programmers. Both of those systems are bulk parallel systems, with the majority of folks aggregating low information density data. This is not what the MPI clusters are doing modeling weather and calculating subatomic interactions or simulating wind tunnels. > The idea that the people at Google doing large-scale machine learning problems (which involves huge sparse matrices) are oblivious to scale and numerical performance is just delusional. Scale is not latency! Google scales to sizes many orders of magnitudes larger than MPI clusters, but it does not run workloads with the same connectivity needs that MPI workloads need. It doesn't run Super Computers, it runs massively parallel bulk embarrassingly parallel computers. Chapel is a language, MPI is a transport. The author obviously has skills, but they shouldn't be conflated. Chapel can use MPI. Chapel supports OFI MPI uGNI GASNet. This is not unlike saying don't use SCTP use Python! I am not being charitable.

morelandjs 4y ago
What compute environments are used for LLMs like ChatGPT? Do you see this recent frenzy driving demand for HPC engineers?
- hackandthink 4y ago
  "Our biggest jobs run MPI, and all pods within the job are participating in a single MPI communicator." https://openai.com/research/scaling-kubernetes-to-7500-nodes https://openai.com/research/scaling-kubernetes-to-7500-nodes
- cavisne 4y ago
  LLM training (at least the ones we have research details on) sync weights every step so they have very high networking and latency needs. That’s why every cloud vendor competes on interconnect speeds for GPU machines. So they are basically classic supercomputing workloads.
  - cavisne 4y ago
    I don't see a lot of traditional "HPC engineers" in the ml infrastructure space though. I do wonder if in hindsight using a Slurm cluster for scheduling, Lustre for data, MPI for any connectivity thats not covered by NCCL, would have been better than trying to make object storage, grpc, kubernetes, ray etc work.
    - the_svd_doctor 4y ago
      “HPC engineering” and “Optimizing LLM training” are similar. It’s about profiling, performance modeling, finding bottlenecks, rewriting what’s needed, etc. Lots of overlap, and people from more traditional HPC doing it too. Obviously if you’re a 50yo university professor doing airplane simulations you won’t switch to LLMs today though…
- adw 4y ago
  Some mixture of MPI, NCCL, Gloo, and whatever proprietary stuff TPU clusters do. All of these are basically either trad-HPC in style or literally from the supercomputing community. Interconnects tend to be Infiniband or the like, which, again, straight out of big iron.
- osigurdson 4y ago
  They use Kubernetes and MPI https://openai.com/research/scaling-kubernetes-to-7500-nodes https://openai.com/research/scaling-kubernetes-to-7500-nodes
- p_l 4y ago
  CUDA has support for MPI, including MPI done from GPU itself (over nvlink or with support for accessing host infiniband adapter, iirc).
mirker 4y ago
I think one of the most glaring problems with the argument is HPC maintains tons of legacy simulations which may not even have the original authors around. Adding Spark support wouldn’t make this easier.
- sfpotter 4y ago
  Not sure why this was downvoted. This is absolutely true.
dekhn 4y ago
Google absolutely runs workloads that need supercomputer tight scaling. That's why they built TPUs with ICI networks.
- nerpderp82 4y ago
  You got me on a technicality, but nothing of practicality. This isn't exposed to the larger Google, nor to customers. Google itself as an organism doesn't HPC. It is an ad company that runs ad company systems. Why are there no top-k super computers running on Google?
- HyperSane 4y ago
  what is ICI networks?
  - dekhn 4y ago
    https://dl.acm.org/doi/pdf/10.1145/3360307 https://dl.acm.org/doi/pdf/10.1145/3360307 Very fast (hundreds of gigabits per connection) networks that attach TPUs on the same board, as well as between boards. It's wired up point to point, forming a logical grid (technically a 2D or 3D torus with wraparound links).
  - rlupi 4y ago
    ICI is interchip interconnect in TPUv4 pods. You can read more on paper that my collegues in Google platforms recently published https://arxiv.org/abs/2304.01433 https://arxiv.org/abs/2304.01433 https://cloud.google.com/blog/topics/systems/tpu-v4-enables-performance-energy-and-co2e-efficiency-gains https://cloud.google.com/blog/topics/systems/tpu-v4-enables-... I am so happy this paper is finally out :-) I led the SRE work during their NPI. It's a pain when you have to wait 3 years to discuss what you worked on. They are based on OCS, which powers Google datacenter networks https://arxiv.org/abs/2208.10041 https://arxiv.org/abs/2208.10041
- markhahn 4y ago
  ICI is basically nvlink, not like IB.
phkahler 4y ago
>> HPC is a small small fraction of the job market, the number of folks involved in HPC is tiny compared to all the click-log analyzing Spark and Hadoop programmers. You're basically saying HPC = MPI users, which is really dismissive of a whole bunch of other people making use of vast compute resources. I for one can't wait for these masses to convince chip makers that ieee754 floating point really has a lot of crap that we don't need.
bscphil 4y ago
I think a central element of the article is its focus on the needs of genomics researchers. In this field there are many tasks that are highly parallel but not embarrassingly parallel. Ideally you want to scale out to as many processes as possible, but some basic IPC is mandatory. Though not in the HPC field myself, I've been in a position of helping researchers working with an HPC cluster. On this hardware something like 16 cores per node was typical, so you run out of scaling room quickly using traditional threading libraries. And you really want to scale beyond this - jobs could take weeks or months otherwise. This means using MPI because that's all the HPC supported. This was a massive pain, because MPI is complicated in ways that are unfamiliar to many researchers, most of whom have no low-level language experience at all. What the article calls for are HPC techniques that move communication to a lower level API, rather than exposing them to the programmer / end-user. I think that's exactly the right idea for genomics. Disclaimer: I'm neither an expert in HPC nor genomics.

morelandjs 4y ago

What compute environments are used for LLMs like ChatGPT? Do you see this recent frenzy driving demand for HPC engineers?

hackandthink 4y ago
"Our biggest jobs run MPI, and all pods within the job are participating in a single MPI communicator." https://openai.com/research/scaling-kubernetes-to-7500-nodes https://openai.com/research/scaling-kubernetes-to-7500-nodes
cavisne 4y ago
LLM training (at least the ones we have research details on) sync weights every step so they have very high networking and latency needs. That’s why every cloud vendor competes on interconnect speeds for GPU machines. So they are basically classic supercomputing workloads.
- cavisne 4y ago
  I don't see a lot of traditional "HPC engineers" in the ml infrastructure space though. I do wonder if in hindsight using a Slurm cluster for scheduling, Lustre for data, MPI for any connectivity thats not covered by NCCL, would have been better than trying to make object storage, grpc, kubernetes, ray etc work.
  - the_svd_doctor 4y ago
    “HPC engineering” and “Optimizing LLM training” are similar. It’s about profiling, performance modeling, finding bottlenecks, rewriting what’s needed, etc. Lots of overlap, and people from more traditional HPC doing it too. Obviously if you’re a 50yo university professor doing airplane simulations you won’t switch to LLMs today though…
adw 4y ago
Some mixture of MPI, NCCL, Gloo, and whatever proprietary stuff TPU clusters do. All of these are basically either trad-HPC in style or literally from the supercomputing community. Interconnects tend to be Infiniband or the like, which, again, straight out of big iron.
osigurdson 4y ago
They use Kubernetes and MPI https://openai.com/research/scaling-kubernetes-to-7500-nodes https://openai.com/research/scaling-kubernetes-to-7500-nodes
p_l 4y ago
CUDA has support for MPI, including MPI done from GPU itself (over nvlink or with support for accessing host infiniband adapter, iirc).

mirker 4y ago

I think one of the most glaring problems with the argument is HPC maintains tons of legacy simulations which may not even have the original authors around. Adding Spark support wouldn’t make this easier.

sfpotter 4y ago
Not sure why this was downvoted. This is absolutely true.

dekhn 4y ago

Google absolutely runs workloads that need supercomputer tight scaling. That's why they built TPUs with ICI networks.

nerpderp82 4y ago
You got me on a technicality, but nothing of practicality. This isn't exposed to the larger Google, nor to customers. Google itself as an organism doesn't HPC. It is an ad company that runs ad company systems. Why are there no top-k super computers running on Google?
HyperSane 4y ago
what is ICI networks?
- dekhn 4y ago
  https://dl.acm.org/doi/pdf/10.1145/3360307 https://dl.acm.org/doi/pdf/10.1145/3360307 Very fast (hundreds of gigabits per connection) networks that attach TPUs on the same board, as well as between boards. It's wired up point to point, forming a logical grid (technically a 2D or 3D torus with wraparound links).
- rlupi 4y ago
  ICI is interchip interconnect in TPUv4 pods. You can read more on paper that my collegues in Google platforms recently published https://arxiv.org/abs/2304.01433 https://arxiv.org/abs/2304.01433 https://cloud.google.com/blog/topics/systems/tpu-v4-enables-performance-energy-and-co2e-efficiency-gains https://cloud.google.com/blog/topics/systems/tpu-v4-enables-... I am so happy this paper is finally out :-) I led the SRE work during their NPI. It's a pain when you have to wait 3 years to discuss what you worked on. They are based on OCS, which powers Google datacenter networks https://arxiv.org/abs/2208.10041 https://arxiv.org/abs/2208.10041
markhahn 4y ago
ICI is basically nvlink, not like IB.

phkahler 4y ago

>> HPC is a small small fraction of the job market, the number of folks involved in HPC is tiny compared to all the click-log analyzing Spark and Hadoop programmers. You're basically saying HPC = MPI users, which is really dismissive of a whole bunch of other people making use of vast compute resources. I for one can't wait for these masses to convince chip makers that ieee754 floating point really has a lot of crap that we don't need.

bscphil 4y ago

I think a central element of the article is its focus on the needs of genomics researchers. In this field there are many tasks that are highly parallel but not embarrassingly parallel. Ideally you want to scale out to as many processes as possible, but some basic IPC is mandatory. Though not in the HPC field myself, I've been in a position of helping researchers working with an HPC cluster. On this hardware something like 16 cores per node was typical, so you run out of scaling room quickly using traditional threading libraries. And you really want to scale beyond this - jobs could take weeks or months otherwise. This means using MPI because that's all the HPC supported. This was a massive pain, because MPI is complicated in ways that are unfamiliar to many researchers, most of whom have no low-level language experience at all. What the article calls for are HPC techniques that move communication to a lower level API, rather than exposing them to the programmer / end-user. I think that's exactly the right idea for genomics. Disclaimer: I'm neither an expert in HPC nor genomics.