3 ms·

For me the hardest part of learning ML was getting over imposter syndrome. It felt like I needed a PhD and hardcore math skills. That’s what made me so hesitant

For me the hardest part of learning ML was getting over imposter syndrome. It felt like I needed a PhD and hardcore math skills. That’s what made me so hesitant in learning it. I thought: there’s already so many people much more smarter and advanced for me. Why even bother? It wasn’t until I was “forced” to learn it to solve a problem I was facing, that I realized ML is just like any other engineering topic - whether it’s devops or data engineering. You just need motivation, some patience and ideally a project/problem that you can solve while learning all this stuff.

arccy 3y ago
Also, like most topics, there are multiple levels of understanding, and you generally don't have to reach the deepest layers to be somewhat productive.
PH95VuimJjqBqy 3y ago
it's not anymore surprising than realizing graphics programming requires you to have a good grasp on linear algebra. the math is how it's done, it's statistics all the way down. If you don't have a good grasp on statistics you're not going to fully grok ML.
Buttons840 3y ago
One difference from other engineering topics is ML brings a hope that it will solve itself. You make a plan, you implement it, you train the model for days, the problem is not yet solved, but if you let it train a little longer maybe it will be?
bluish29 3y ago
> It felt like I needed a PhD and hardcore math skills I don't know if this helps more or make it worse. But I have both and getting the same feelings all the time. But basically you just need good statistics and linear algebra knowledge, and you will be fine (on the math side). > there’s already so many people much more smarter and advanced for me. Why even bother? That's the very definition of imposter syndrome put in a very good way. But you can ask the same for everything, not only ML. kudos on getting through it, though.
- shortrounddev2 3y ago
  99% of the time people say linear algebra is required for something, they mean knowledge basic operations and properties of tensors more than actual "algebra". I found this when doing computer graphics. Is that true as well here?
  - SonOfLilit 3y ago
    A tiny bit more. Once in a while someone does something elegant with eigenvalues, and Transformers represent their "thoughts" in a very LinAlg way (a linear combination of orthonormal basis vectors in a very high dimension) that was very hard for the only person with no LinAlg background in my study group to grok.
  - tomrod 3y ago
    Basic operations and properties of tensors are exactly what is taught in linear algebra, in addition to (in my experience) more accessible proofs. More or less through singular value decomposition and/or least squares. The core algorithms all build on top of each other. The `algebra` part of linear algebra refers to a `field`, but it might as well also be called arithmetic of tensors.
    - shortrounddev2 3y ago
      Right but I mean that people say you need linear algebra (e.g: the whole textbook), when in actuality you might need the first 3 chapters
      - tomrod 3y ago
        I think I didn't come across clearly. Eigenvalues, SVD, etc are important.
  - godelski 3y ago
    > they mean knowledge basic operations and properties of tensors more than actual "algebra". You're actually describing linear algebra. A core topic is system of equations. You might see a 2-Tensor (matrix) like Ax := [[a,b],[c,d]].[x,y] and you could write it as f = ax + by; g = cx + dy. Often dimensions are implicit so it may not look like this, but it is. But that's a big part of what it is about (there's a whole lot more btw). You're absolutely using linear algebra frequently in graphics. Euler angles are a good example, you're just probably not writing them in matrix/tensor form. You will even get a tiny bit of exposure to {field,group} theory/abstract algebra via quaternions. In ML I'd say it is very similar. The typical researcher is going to have about the same math skills as the typical person studying graphics (I actually started my PhD in HPC graphics). But, and this holds for both domains, having a deeper math understanding only helps. It makes things easier to debug, gives you a better understanding of what the systems are doing, and gives you a lot of tools to solve many problems. I wouldn't ever use math as a strong barrier to entry, but I feel many get complacent with their skill level and we have been discouraging this myth that math doesn't help. Without a doubt, it does.
morkalork 3y ago
It helps to differentiate between the foundational research and the application. Are you expecting to advance SOTA and get published? Then yes, you need a PhD. Are you building stuff in the enterprise world? Then no, it is like any other kind of engineering. Not everyone has to be James Clark Maxwell.
balaji1 3y ago
another problem with ML workflow is the difficulty of iteration - takes long to run, etc. - but SWE are super used to iterating quickly
godelski 3y ago
> For me the hardest part of learning ML was getting over imposter syndrome. It felt like I needed a PhD and hardcore math skills ABD (all but PhD dissertation) here with strong math skills. I get the imposter syndrome, but let me absolutely assure you that the community at large does not have strong math skills. I routinely talk to people doing diffusion research that don't know what covariance is or pdf. People from top ranked schools, with high paper counts and high citation counts. Expertise is often more narrow than it appears. That's okay, as long as we're honest about it. Don't get me wrong, I wish there was more math involved and efforts were more serious. But they aren't. The space is very noisy and little is being done to clean it up (there are some, and I do appreciate those efforts). I'll add that there's one thing more that you need besides motivation and patience: perseverance. ML systems are hard to debug and difficult to evaluate (maybe not for papers, but absolutely for systems that work in the real world). It's okay to not get things perfectly and it is totally okay to not have a model with decent generalization, but context is always important and part of the debugging process is trying to trace these down (which is difficult because you need to do more abstract versions of what is analogous to the silly or random inputs being passed to code). Detecting overfitting is often quite hard and honestly sometimes it is even desirable (GPT being overfit makes it great for information retrieval!). Also, something I tell my students when I teach ML: you don't need math to train good models, but you do need math to know why your models are wrong. So I highly encourage math, but don't let that stop you from getting started. You can also just have a math heavy person on your team and get many benefits that way.
- tdiff 3y ago
  Pdf is a file format
  - yread 3y ago
    It's also something to do with density, probability and function ;)
  - naitgacem 3y ago
    I had a probability class and we definitely made that joke more often than we should :P

arccy 3y ago

Also, like most topics, there are multiple levels of understanding, and you generally don't have to reach the deepest layers to be somewhat productive.

PH95VuimJjqBqy 3y ago

it's not anymore surprising than realizing graphics programming requires you to have a good grasp on linear algebra. the math is how it's done, it's statistics all the way down. If you don't have a good grasp on statistics you're not going to fully grok ML.

Buttons840 3y ago

One difference from other engineering topics is ML brings a hope that it will solve itself. You make a plan, you implement it, you train the model for days, the problem is not yet solved, but if you let it train a little longer maybe it will be?

bluish29 3y ago

> It felt like I needed a PhD and hardcore math skills I don't know if this helps more or make it worse. But I have both and getting the same feelings all the time. But basically you just need good statistics and linear algebra knowledge, and you will be fine (on the math side). > there’s already so many people much more smarter and advanced for me. Why even bother? That's the very definition of imposter syndrome put in a very good way. But you can ask the same for everything, not only ML. kudos on getting through it, though.

shortrounddev2 3y ago
99% of the time people say linear algebra is required for something, they mean knowledge basic operations and properties of tensors more than actual "algebra". I found this when doing computer graphics. Is that true as well here?
- SonOfLilit 3y ago
  A tiny bit more. Once in a while someone does something elegant with eigenvalues, and Transformers represent their "thoughts" in a very LinAlg way (a linear combination of orthonormal basis vectors in a very high dimension) that was very hard for the only person with no LinAlg background in my study group to grok.
- tomrod 3y ago
  Basic operations and properties of tensors are exactly what is taught in linear algebra, in addition to (in my experience) more accessible proofs. More or less through singular value decomposition and/or least squares. The core algorithms all build on top of each other. The `algebra` part of linear algebra refers to a `field`, but it might as well also be called arithmetic of tensors.
  - shortrounddev2 3y ago
    Right but I mean that people say you need linear algebra (e.g: the whole textbook), when in actuality you might need the first 3 chapters
    - tomrod 3y ago
      I think I didn't come across clearly. Eigenvalues, SVD, etc are important.
- godelski 3y ago
  > they mean knowledge basic operations and properties of tensors more than actual "algebra". You're actually describing linear algebra. A core topic is system of equations. You might see a 2-Tensor (matrix) like Ax := [[a,b],[c,d]].[x,y] and you could write it as f = ax + by; g = cx + dy. Often dimensions are implicit so it may not look like this, but it is. But that's a big part of what it is about (there's a whole lot more btw). You're absolutely using linear algebra frequently in graphics. Euler angles are a good example, you're just probably not writing them in matrix/tensor form. You will even get a tiny bit of exposure to {field,group} theory/abstract algebra via quaternions. In ML I'd say it is very similar. The typical researcher is going to have about the same math skills as the typical person studying graphics (I actually started my PhD in HPC graphics). But, and this holds for both domains, having a deeper math understanding only helps. It makes things easier to debug, gives you a better understanding of what the systems are doing, and gives you a lot of tools to solve many problems. I wouldn't ever use math as a strong barrier to entry, but I feel many get complacent with their skill level and we have been discouraging this myth that math doesn't help. Without a doubt, it does.

morkalork 3y ago

It helps to differentiate between the foundational research and the application. Are you expecting to advance SOTA and get published? Then yes, you need a PhD. Are you building stuff in the enterprise world? Then no, it is like any other kind of engineering. Not everyone has to be James Clark Maxwell.

balaji1 3y ago

another problem with ML workflow is the difficulty of iteration - takes long to run, etc. - but SWE are super used to iterating quickly

godelski 3y ago

> For me the hardest part of learning ML was getting over imposter syndrome. It felt like I needed a PhD and hardcore math skills ABD (all but PhD dissertation) here with strong math skills. I get the imposter syndrome, but let me absolutely assure you that the community at large does not have strong math skills. I routinely talk to people doing diffusion research that don't know what covariance is or pdf. People from top ranked schools, with high paper counts and high citation counts. Expertise is often more narrow than it appears. That's okay, as long as we're honest about it. Don't get me wrong, I wish there was more math involved and efforts were more serious. But they aren't. The space is very noisy and little is being done to clean it up (there are some, and I do appreciate those efforts). I'll add that there's one thing more that you need besides motivation and patience: perseverance. ML systems are hard to debug and difficult to evaluate (maybe not for papers, but absolutely for systems that work in the real world). It's okay to not get things perfectly and it is totally okay to not have a model with decent generalization, but context is always important and part of the debugging process is trying to trace these down (which is difficult because you need to do more abstract versions of what is analogous to the silly or random inputs being passed to code). Detecting overfitting is often quite hard and honestly sometimes it is even desirable (GPT being overfit makes it great for information retrieval!). Also, something I tell my students when I teach ML: you don't need math to train good models, but you do need math to know why your models are wrong. So I highly encourage math, but don't let that stop you from getting started. You can also just have a math heavy person on your team and get many benefits that way.

tdiff 3y ago
Pdf is a file format
- yread 3y ago
  It's also something to do with density, probability and function ;)
- naitgacem 3y ago
  I had a probability class and we definitely made that joke more often than we should :P