4 ms·
How are the optimal architectures for weather simulations and LLM training different?
by awongh 3y ago
How are the optimal architectures for weather simulations and LLM training different?
- bane 3y agotbh, I'm not entirely informed on what the requirements are for LLM training, but I've noticed that nearly all of the teams don't use unified memory (what makes a super computer "super"). I believe I read somewhere openAI uses a K8s cluster[1] and other teams I know seem to work with other similar non-unified memory systems. If there's no advantage, part of what makes supercomputers expensive is this memory interconnect, so wouldn't it just be better to use a huge K8s cluster or something? I'm honestly not sure, and hoping somebody comments here and provides more information as I'm genuinely interested. 1 - https://openai.com/research/scaling-kubernetes-to-7500-nodes https://openai.com/research/scaling-kubernetes-to-7500-nodes OpenAI says their biggest jobs run on MPI so maybe a supercomputer would be better?
- spott 3y agoWhat is this “unified memory” you speak of? Modern supercomputers are just a bunch of computers with a (very) high speed interconnect. As far as I’m aware, pretty much every multi node job on a supercomputer uses MPI to communicate, and there isn’t any sort of “unified memory” system. It has been a few years since I worked on one of the big supercomputers though, so maybe you have better knowledge than me. I’d love more information.