4 ms·

The post is not talking about 'traditional ML' but rather only 'DL' (i.e. neural networks). One of the problems I see here is that it seems to me that the math

The post is not talking about 'traditional ML' but rather only 'DL' (i.e. neural networks). One of the problems I see here is that it seems to me that the math education of many CS grads is woefully lacking. Indeed, deep learning math is basically senior-high-school level calculus. Backpropagation is a straightforwards application of the chain rule. There is really no surprising thing or deep insight. The math for deep learning has been set in place for more than fifty years at this point. The only change was the advent of large enough data and computer chips to process it. That being said, I don't think understanding how the `transformers` library works or composes is the same as understanding how 'deep learning' works. That's like saying knowing how to use GCC gives you a solid understanding of compilers. That's not a ding against those whose main experience with DL is using those libraries. I use many libraries doing things I don't fully understand. I'm not a graphics expert, but I use graphics programs every day, and graphics libraries regularly. That's fine. But using those libraries doesn't make you an expert in computer graphics.

godelski 3y ago
> Indeed, deep learning math is basically senior-high-school level calculus. Actually I find this claim problematic and incorrect. Yes, many parts of ML only require multivariate calculus[0] and those are the parts that most people are exposed to. BUT that doesn't mean there isn't a lot of math hiding around in the background. Even understanding something like activation functions take much higher level math as we got to get into topics such as topology, metric theory, and high dimensional statistics. You don't need this understanding to build models (as you also conclude) and most researchers don't understand much of this either tbh. But I think we need to be clear about these distinctions as model evaluation gets insanely complex. Complexity in evaluation only increases as our performance increases and a major problem we face today is we are marginalizing out important information. I'm sure that anyone that hacks around with LLMs or diffusion will understand how leaderboards are often noisy and how picking top models does not guarantee top performance (and why datasets get continually added). It's because evaluation requires more than boiling down performance to a single number. Here, math skills become extremely important. [0] btw, many will not find this even offered at their high school. In fact, this is often upper level math for many STEM undergraduates and even an elective in my Uni's undergrad CS program. So let's not be so belittling. Shame the education systems instead of people lacking opportunities.
wenc 3y ago
I don’t think I agree because it’s like saying accounting is just elementary level addition, subtraction and multiplication. But elementary school kids can’t do accounting. DL requires a certain degree of mathematical maturity to grasp.

godelski 3y ago

> Indeed, deep learning math is basically senior-high-school level calculus. Actually I find this claim problematic and incorrect. Yes, many parts of ML only require multivariate calculus[0] and those are the parts that most people are exposed to. BUT that doesn't mean there isn't a lot of math hiding around in the background. Even understanding something like activation functions take much higher level math as we got to get into topics such as topology, metric theory, and high dimensional statistics. You don't need this understanding to build models (as you also conclude) and most researchers don't understand much of this either tbh. But I think we need to be clear about these distinctions as model evaluation gets insanely complex. Complexity in evaluation only increases as our performance increases and a major problem we face today is we are marginalizing out important information. I'm sure that anyone that hacks around with LLMs or diffusion will understand how leaderboards are often noisy and how picking top models does not guarantee top performance (and why datasets get continually added). It's because evaluation requires more than boiling down performance to a single number. Here, math skills become extremely important. [0] btw, many will not find this even offered at their high school. In fact, this is often upper level math for many STEM undergraduates and even an elective in my Uni's undergrad CS program. So let's not be so belittling. Shame the education systems instead of people lacking opportunities.

wenc 3y ago

I don’t think I agree because it’s like saying accounting is just elementary level addition, subtraction and multiplication. But elementary school kids can’t do accounting. DL requires a certain degree of mathematical maturity to grasp.