4 ms·
> “This new technique” Which also has a name that they even use, model sharding. Which is well known to be very slow because you increase the number of IO oper
by godelski 3y ago
> “This new technique”
Which also has a name that they even use, model sharding. Which is well known to be very slow because you increase the number of IO operations to the GPU. I don't even understand the post because they mention sharding is slow and then say they do sharding but as if they don't. Even the comment in their code calls it sharding. Unless I'm missing something, it's pretty fucking deceptive to call this a new technique. Can we stop with the AI hype? The field is exciting enough that we don't need to be selling snakeoil and selling everything as if it is 10x better than it is.
For anyone interested in model sharding you should look into pytorch's gradient checkpointing documentation and/or fairseq.
- benreesman 3y agoTo echo and in no way contradict the parent’s point, there have been versions of this going on forever. Without getting into a bunch of stuff about FDSP or Accelerate or even the NCCL/All-Reduce type ops (which I obviously hope the interested check out) I’ll signal boost my favorite tech talk ever by maybe the coolest CS prof living: https://youtu.be/l5JqUvTdZts?si=9HlNWHTlRbiatHD1 https://youtu.be/l5JqUvTdZts?si=9HlNWHTlRbiatHD1 I’m biased because I’m also Ben, but I’m not “Ben Rekt” who is a musician and charismatic and way too cool to care about SV.
- godelski 3y agoBtw, that link isn't working for me, this is though https://www.youtube.com/watch?v=l5JqUvTdZts https://www.youtube.com/watch?v=l5JqUvTdZts Hogwild is a really cool routine. There's a lot of work going into FDSP, parallelism, and all kinds of things at the low level for optimizing communication, maximizing parallelization, writing cuda kernels, and so on. I think there's a lot of unsung heros of ML working in these areas that people never even see (including researchers). So many people quote Sutton's Bitter Lesson but these are the people that made that landscape even possible. Without these people working at the low level we wouldn't be able to process large models or large datasets. The fame issue is always a bit weird to me. Like I think more people know of Vaswani (AIAYN) or Dosovitskiy (ViT paper) than those that know Kingma, who wrote the Adam optimizer. Both these works use Adam as well as Rombach's LDM (Stable Diffusion). I'm only mentioning Kingma because I'm more aware of that work but am certain there's tons of work that could be better highlighted (actually illustrating this point).
- gpm 3y agoI was playing with this with the llama model was released - and provided I was loading from memory and not disk it was actually only negligbly slower than the model was without sharding (for e.g. the 8gb models that I could load entirely on my GPU). Each layer took a lot more time to evaluate than it took to copy the weights onto the GPU (RTX 2070 super).
- godelski 3y agoYes, if you can fit it in RAM then the loss isn't too bad if you have the memory pinned and are taking advantage of some of Nvidia's optimizations (which should be defaulted to on). Inference usually isn't too bad but training is heavier. Things have also massively improved over the last few years. The efficiency also really increases with reduced floating point types and quantized models.