4 ms·

To echo and in no way contradict the parent’s point, there have been versions of this going on forever. Without getting into a bunch of stuff about FDSP or Acc

To echo and in no way contradict the parent’s point, there have been versions of this going on forever. Without getting into a bunch of stuff about FDSP or Accelerate or even the NCCL/All-Reduce type ops (which I obviously hope the interested check out) I’ll signal boost my favorite tech talk ever by maybe the coolest CS prof living: https://youtu.be/l5JqUvTdZts?si=9HlNWHTlRbiatHD1 https://youtu.be/l5JqUvTdZts?si=9HlNWHTlRbiatHD1 I’m biased because I’m also Ben, but I’m not “Ben Rekt” who is a musician and charismatic and way too cool to care about SV.

godelski 3y ago
Btw, that link isn't working for me, this is though https://www.youtube.com/watch?v=l5JqUvTdZts https://www.youtube.com/watch?v=l5JqUvTdZts Hogwild is a really cool routine. There's a lot of work going into FDSP, parallelism, and all kinds of things at the low level for optimizing communication, maximizing parallelization, writing cuda kernels, and so on. I think there's a lot of unsung heros of ML working in these areas that people never even see (including researchers). So many people quote Sutton's Bitter Lesson but these are the people that made that landscape even possible. Without these people working at the low level we wouldn't be able to process large models or large datasets. The fame issue is always a bit weird to me. Like I think more people know of Vaswani (AIAYN) or Dosovitskiy (ViT paper) than those that know Kingma, who wrote the Adam optimizer. Both these works use Adam as well as Rombach's LDM (Stable Diffusion). I'm only mentioning Kingma because I'm more aware of that work but am certain there's tons of work that could be better highlighted (actually illustrating this point).