3 ms·
tbh, I'm not entirely informed on what the requirements are for LLM training, but I've noticed that nearly all of the teams don't use unified memory (what makes
by bane 3y ago
tbh, I'm not entirely informed on what the requirements are for LLM training, but I've noticed that nearly all of the teams don't use unified memory (what makes a super computer "super"). I believe I read somewhere openAI uses a K8s cluster[1] and other teams I know seem to work with other similar non-unified memory systems. If there's no advantage, part of what makes supercomputers expensive is this memory interconnect, so wouldn't it just be better to use a huge K8s cluster or something?
I'm honestly not sure, and hoping somebody comments here and provides more information as I'm genuinely interested.
1 - https://openai.com/research/scaling-kubernetes-to-7500-nodes https://openai.com/research/scaling-kubernetes-to-7500-nodes
OpenAI says their biggest jobs run on MPI so maybe a supercomputer would be better?
- spott 3y agoWhat is this “unified memory” you speak of? Modern supercomputers are just a bunch of computers with a (very) high speed interconnect. As far as I’m aware, pretty much every multi node job on a supercomputer uses MPI to communicate, and there isn’t any sort of “unified memory” system. It has been a few years since I worked on one of the big supercomputers though, so maybe you have better knowledge than me. I’d love more information.