3 ms·

The author and you seem to be talking to different audiences, you're talking about ML eng and OP is talking about ML researchers. Researchers absolutely need t

The author and you seem to be talking to different audiences, you're talking about ML eng and OP is talking about ML researchers. Researchers absolutely need to know a lot, not necessarily all the way to topology or w/e but definitely the underlying mathematical principles in order to advance the field (IMO).

sillysaurusx 4y ago
You're right, I wasn't clear. But I'm a full-time ML researcher. In terms of advancing the field, my contributions so far have been modest, but they're there. Some of my favorite ideas were swarm training (https://battle.shawwn.com/swarm-training-v01a.pdf https://battle.shawwn.com/swarm-training-v01a.pdf), stop loss for stabilizing GAN training (https://twitter.com/search?q=from%3Atheshawwn%20stop%20loss&src=typed_query https://twitter.com/search?q=from%3Atheshawwn%20stop%20loss&...), and getting GPT to play chess (https://www.theregister.com/2020/01/10/gpt2_chess/ https://www.theregister.com/2020/01/10/gpt2_chess/) back when that was a shocking idea. And in terms of cited research, https://arxiv.org/abs/2101.00027 https://arxiv.org/abs/2101.00027 has been the most impactful. There's significant overlap between ML research and ML dev. If I can do it without most of that list, it should give people here some hope of joining the field without needing to immerse themselves in theory.
- extasia 4y ago
  Aha! Nice, I'm actually booking up on transformers as part of my doctoral programme and had a recollection about your gpt chess project earlier today!
p1esk 4y ago
I'm an ML researcher, and I'm with sillysaurusx on this. I actually know most of the things on the list, but only because my research is mainly in model compression and computational efficiency. Recently I've been interested in adapting diffusion models to generate music (in raw audio domain), and I'd say only 5 out of 18 bullet points in ML section are relevant - the rest falls from "nice to know" to "irrelevant".

sillysaurusx 4y ago

You're right, I wasn't clear. But I'm a full-time ML researcher. In terms of advancing the field, my contributions so far have been modest, but they're there. Some of my favorite ideas were swarm training (https://battle.shawwn.com/swarm-training-v01a.pdf https://battle.shawwn.com/swarm-training-v01a.pdf), stop loss for stabilizing GAN training (https://twitter.com/search?q=from%3Atheshawwn%20stop%20loss&src=typed_query https://twitter.com/search?q=from%3Atheshawwn%20stop%20loss&...), and getting GPT to play chess (https://www.theregister.com/2020/01/10/gpt2_chess/ https://www.theregister.com/2020/01/10/gpt2_chess/) back when that was a shocking idea. And in terms of cited research, https://arxiv.org/abs/2101.00027 https://arxiv.org/abs/2101.00027 has been the most impactful. There's significant overlap between ML research and ML dev. If I can do it without most of that list, it should give people here some hope of joining the field without needing to immerse themselves in theory.

extasia 4y ago
Aha! Nice, I'm actually booking up on transformers as part of my doctoral programme and had a recollection about your gpt chess project earlier today!

p1esk 4y ago

I'm an ML researcher, and I'm with sillysaurusx on this. I actually know most of the things on the list, but only because my research is mainly in model compression and computational efficiency. Recently I've been interested in adapting diffusion models to generate music (in raw audio domain), and I'd say only 5 out of 18 bullet points in ML section are relevant - the rest falls from "nice to know" to "irrelevant".