4 ms·

yes, I can't imagine the architecture of a supercomputer is the right one for LLM training. But maybe? If not, spending years to design and build a system for

by bane 3y ago

yes, I can't imagine the architecture of a supercomputer is the right one for LLM training. But maybe? If not, spending years to design and build a system for weather and nuke simulations and ending up doing something that's totally not made for these systems is kind of a mind bender. I can imagine the conversations that led to this: "we need a government owned LLM" "okay what do we need" "lots of compute power" "well we have this new supercomputer just coming online" "not the right kind of compute" "come on, it's top-500!"

awongh 3y ago
How are the optimal architectures for weather simulations and LLM training different?
- bane 3y ago
  tbh, I'm not entirely informed on what the requirements are for LLM training, but I've noticed that nearly all of the teams don't use unified memory (what makes a super computer "super"). I believe I read somewhere openAI uses a K8s cluster[1] and other teams I know seem to work with other similar non-unified memory systems. If there's no advantage, part of what makes supercomputers expensive is this memory interconnect, so wouldn't it just be better to use a huge K8s cluster or something? I'm honestly not sure, and hoping somebody comments here and provides more information as I'm genuinely interested. 1 - https://openai.com/research/scaling-kubernetes-to-7500-nodes https://openai.com/research/scaling-kubernetes-to-7500-nodes OpenAI says their biggest jobs run on MPI so maybe a supercomputer would be better?
  - spott 3y ago
    What is this “unified memory” you speak of? Modern supercomputers are just a bunch of computers with a (very) high speed interconnect. As far as I’m aware, pretty much every multi node job on a supercomputer uses MPI to communicate, and there isn’t any sort of “unified memory” system. It has been a few years since I worked on one of the big supercomputers though, so maybe you have better knowledge than me. I’d love more information.
visarga 3y ago
Probably optimized for fp64 instead of 4 bit floats.

by bane 3y ago

awongh 3y ago

How are the optimal architectures for weather simulations and LLM training different?

bane 3y ago
tbh, I'm not entirely informed on what the requirements are for LLM training, but I've noticed that nearly all of the teams don't use unified memory (what makes a super computer "super"). I believe I read somewhere openAI uses a K8s cluster[1] and other teams I know seem to work with other similar non-unified memory systems. If there's no advantage, part of what makes supercomputers expensive is this memory interconnect, so wouldn't it just be better to use a huge K8s cluster or something? I'm honestly not sure, and hoping somebody comments here and provides more information as I'm genuinely interested. 1 - https://openai.com/research/scaling-kubernetes-to-7500-nodes https://openai.com/research/scaling-kubernetes-to-7500-nodes OpenAI says their biggest jobs run on MPI so maybe a supercomputer would be better?
- spott 3y ago
  What is this “unified memory” you speak of? Modern supercomputers are just a bunch of computers with a (very) high speed interconnect. As far as I’m aware, pretty much every multi node job on a supercomputer uses MPI to communicate, and there isn’t any sort of “unified memory” system. It has been a few years since I worked on one of the big supercomputers though, so maybe you have better knowledge than me. I’d love more information.

visarga 3y ago

Probably optimized for fp64 instead of 4 bit floats.