4 ms·

Are math skills really? Most aspects of deep learning don't require a deep understanding of mathematics to understand. Backprop, convolution, attention, recurre

Are math skills really? Most aspects of deep learning don't require a deep understanding of mathematics to understand. Backprop, convolution, attention, recurrent networks, skip connections, GANs, RL, GNNs, etc. can all be stood with only simple calculus and linear algebra. I understand that the theoretical motivation for models is often more math-heavy, but I'm skeptical that motivations need always be mathematical in nature.

thecleaner 2y ago
Every MLE who didnt study Math really likes to downplay its importance. Yeah you dont need measure theoretic probability, but you need a grasp of Lin Alg to structure your computations better. Remember the normalization that we do in attention ? That has a math justification. So I guess yeah academics did have a role in building LLMs. I mean computer scientists really do like to pretend like they invented the whole field. Whereas in reality the average OS, compilers, networks class has nothing to do with core ML. But of course are also important and these barbs dont get us anywhere.
- Matthyze 2y ago
  I think you might've taken my point too strongly. Of course math is very useful, and certain contributions are purely mathematical. I just don't think it is as hard of a requirement for innovation as was claimed.
anthk 2y ago
Forget actual CS and proper engineering without discrete math. Also, without Shannon you wouldn't have neither Telecomms nor Computer Science. Heck, Lisp it's just a formalisation and implementation of Lambda Calculus, which began as a paper... from a Mathematician. Also: https://hakmem.org https://hakmem.org Forget any serious reading without Math skills.
grandempire 2y ago
I’m not saying you can’t use these existing techniques without understand all the theory, but you’re not going to be able to find new techniques. For example, how would you know optimizing a convolution kernel is a good idea if you aren’t familiar with linear time invariant systems?
- Matthyze 2y ago
  I think CNNs follow very naturally from the notion of shift/spatial invariance of visual processing. That doesn't require a mathematical understanding.
  - grandempire 2y ago
    Image processing and shift invariance come from DSP.

thecleaner 2y ago

Every MLE who didnt study Math really likes to downplay its importance. Yeah you dont need measure theoretic probability, but you need a grasp of Lin Alg to structure your computations better. Remember the normalization that we do in attention ? That has a math justification. So I guess yeah academics did have a role in building LLMs. I mean computer scientists really do like to pretend like they invented the whole field. Whereas in reality the average OS, compilers, networks class has nothing to do with core ML. But of course are also important and these barbs dont get us anywhere.

Matthyze 2y ago
I think you might've taken my point too strongly. Of course math is very useful, and certain contributions are purely mathematical. I just don't think it is as hard of a requirement for innovation as was claimed.

anthk 2y ago

Forget actual CS and proper engineering without discrete math. Also, without Shannon you wouldn't have neither Telecomms nor Computer Science. Heck, Lisp it's just a formalisation and implementation of Lambda Calculus, which began as a paper... from a Mathematician. Also: https://hakmem.org https://hakmem.org Forget any serious reading without Math skills.

grandempire 2y ago

I’m not saying you can’t use these existing techniques without understand all the theory, but you’re not going to be able to find new techniques. For example, how would you know optimizing a convolution kernel is a good idea if you aren’t familiar with linear time invariant systems?

Matthyze 2y ago
I think CNNs follow very naturally from the notion of shift/spatial invariance of visual processing. That doesn't require a mathematical understanding.
- grandempire 2y ago
  Image processing and shift invariance come from DSP.