3 ms·
Btw, that link isn't working for me, this is though https://www.youtube.com/watch?v=l5JqUvTdZts https://www.youtube.com/watch?v=l5JqUvTdZts Hogwild is a really
by godelski 3y ago
Btw, that link isn't working for me, this is though https://www.youtube.com/watch?v=l5JqUvTdZts https://www.youtube.com/watch?v=l5JqUvTdZts
Hogwild is a really cool routine. There's a lot of work going into FDSP, parallelism, and all kinds of things at the low level for optimizing communication, maximizing parallelization, writing cuda kernels, and so on. I think there's a lot of unsung heros of ML working in these areas that people never even see (including researchers). So many people quote Sutton's Bitter Lesson but these are the people that made that landscape even possible. Without these people working at the low level we wouldn't be able to process large models or large datasets.
The fame issue is always a bit weird to me. Like I think more people know of Vaswani (AIAYN) or Dosovitskiy (ViT paper) than those that know Kingma, who wrote the Adam optimizer. Both these works use Adam as well as Rombach's LDM (Stable Diffusion). I'm only mentioning Kingma because I'm more aware of that work but am certain there's tons of work that could be better highlighted (actually illustrating this point).