4 ms·

ML is so much more than just neural networks. I would start by taking a free university level course in statistics. Then I would continue with the basics: SVM,

ML is so much more than just neural networks. I would start by taking a free university level course in statistics. Then I would continue with the basics: SVM, linear regression, naive Bayes, gradient boosting, neural nets etc. I would not only train and fine tune them, but I would also build simple ones myself instead of just using libraries. Then I would continue to what you said, participate in Kaggle competitions, try to solve real world problems. I think that understanding the field from bottom up is priceless. Many people fine tune and train models, but they don't understand how that model works, nor do they know if the model they've chosen is the best fit for the problem they are trying to solve. It's a rather long path if you really want to get good at it. Like in music: you can learn to play a tune by ear or you can learn to have a good, deep and thorough understanding of music.

wanderingmind 3y ago
This type of bottom up approach is terrible idea for a fast moving area like ML. Ultimately to get a job and make money in the area, you need to solve customer problems, starting with libraries and fine tuning is what needs to happen first. However you should try to learn fundamentals as you go and when you get stuck. It will take a very long time if you have a full time job or a student studying something else and doesn't help to get a job
- tsurba 3y ago
  Getting a job to make money and solve customer problems as fast as possible was not a stated goal by the OP. Besides, you are also wrong, having good fundamentals in the maths will help you pick up new methods much faster as they pop up. And especially if you want to come up with new methods (as in research), there are no shortcuts.
  - camhart 3y ago
    They made a reasonable assumption. Abstraction helps you to be productive. It's certainly good to understand everything all the way down to the actual physics behind computing, but its not necessary especially at the start. I don't have to know exactly how logic gates work to program in JS and make a difference. I assume the same applies to ML. Motivation is what prevents most people from learning difficult to learn skills. Finding ways to produce value early on can help keep you motivated.
    - DeathArrow 3y ago
      >I don't have to know exactly how logic gates work to program in JS and make a difference. I don't regard time spent in University to learn how logical gates work and many other useful things, as a loss of time. And, as a web developer/architect (after many other industries I worked in), I typically make more money than peers who don't. And knowing how things work, helped me immensely in my career. Not everything is solvable by looking on Stack Overflow. It depends on what professional level you are content with. Learning was not only motivated by money - for me- I was genuinely curious about computers and software since I was a kid.
      - janalsncm 3y ago
        > I don't regard time spent in University to learn how logical gates work and many other useful things, as a loss of time. It’s unquestionably a loss of time, the only question is whether it’s an optimal use of time. This depends on your goals. For most software engineers looking to use ML, starting with existing frameworks and knowledge and drilling down as necessary is the most prudent method. We all have to make sacrifices in what we learn. Even for yourself, the topics you chose to learn about implicitly left out other parallel topics you didn’t learn about. And you also didn’t learn everything from the least abstracted, most fundamental level. We need to choose the appropriate level of abstraction for the problem at hand, which will depend on each person’s goals.
    - ekianjo 3y ago
      > I don't have to know exactly how logic gates work to program in JS and make a difference. Thats a stupid analogy to the above discussion. A better analogy would be trying to program in JS without understanding for loops and basic programmation concepts. Thats typically what I observe with younger folks jumping into neural nets directly. They have a very shallow understanding of anything and survive with youtube tutorials.
      - PeterisP 3y ago
        Most of what was listed above aren't the fundamentals which would help to properly understand what's happening with neural networks but rather completely different branches of machine learning that have little in common with neural networks. If they learn SVM, naive Bayes, and gradient boosting, then their knowledge will definitely be broader but just as shallow; using your analogy, it's not like trying to program in JS without understanding for loops but rather like trying to program in JS without understanding C, COBOL and Haskell. I'm all for learning the fundamentals properly - but those fundamentals are going to be completely things, things like core principles of statistics (limitations of correlation, confounders, bias/variance, etc), the relevant parts of calculus and linear algebra that matter for understanding optimization, the best practices for management of data, experiments and measurement to not cheat yourself, etc - not the checklist of many different, parallel methods of machine learning like decision trees or reinforcement learning, which are both useful and interesting, but not related or required to properly apply e.g. transformer-based large language models for your task.
- DeathArrow 3y ago
  The better your skillset, the highly sought after you are and harder to replace. And that translates in more money, if money are your sole motivation.
- OJFord 3y ago
  OP did not state that getting an ML job was a goal. I for one would be interested in learning more foundational stuff; I have no interest (though perhaps that process would change this!) in a particularly ML job, and certainly not in learning how to point and click and run other people's work from a YouTube video with a scream-face thumbnail. For those who feel similarly, I asked about it recently, maybe something you like the look of: https://news.ycombinator.com/item?id=38320244 https://news.ycombinator.com/item?id=38320244 I also came across (doesn't seem it was mentioned there) Understanding Deep Learning (Simon Prince) which looks like it might be good: https://mitpress.mit.edu/9780262048644/understanding-deep-learning/ https://mitpress.mit.edu/9780262048644/understanding-deep-le...
- CityOfThrowaway 3y ago
  Strong agree
- ekianjo 3y ago
  > his type of bottom up approach is terrible idea for a fast moving area like ML. Of course not. When you learn this way you will realize there are a myriad of problems that can be solved with more simple algorithms. Trying to make every problem fit to neural networks is pure cargo culting.
  - digging 3y ago
    That's true, but the goal is to learn to use neural networks, not to solve problems efficiently, isn't it?
mickael-kerjean 3y ago
> SVM, linear regression, naive Bayes When I studied ML in 2012, the very first course started with naive Bayes and went one from there. A decade after being away, I see a lot of people around me starting with neural nets to train a model that naive Bayes would be plenty enough for and never heard about naive Bayes. Is that only my experience?
- yCombLinks 3y ago
  Any data that is small enough to quickly iterate on for learning is small enough to use a simpler approach. The point is learning the techniques.
- owlbite 3y ago
  As with any super-hyped technology, people want to use the latest and greatest to solve it, even when traditional methods are cheaper, easier, more reliable and more accurate. See also: crypto.
- PeterisP 3y ago
  It's useful to do learning on small toy problems for ease of debugging and speed of training, so if you want to learn how to apply a powerful technique, you're pretty much inevitably going to start learning it on something for which it's absolutely ineffective overkill. E.g. a common start for learning neural nets is a task like XOR which can be solved with literally a single machine instruction instead of training a ML model for it. But also there are many tasks for which naive Bayes works, but a NN solution can be much more accurate if you're okay with it being also much more compute-intensive. E.g. things like sentiment analysis or simple spam filters are often used as a demonstration of naive Bayes, but you can do much better with more powerful models.
amanda99 3y ago
The risk with this is that you spend 10 hours learning statistics then get demotivated and never do the other 190 hours to get to the good stuff. Then quickly you forget the 10 hours of stats you learnt too as it's irrelevant and you don't use it. For me, playing with things and doing cool & fun stuff is always the way to get deeper into something.
- lolinder 3y ago
  > Then quickly you forget the 10 hours of stats you learnt too as it's irrelevant and you don't use it. Stats is one of a very small number of college courses that I took where I came away thinking "this should be mandatory for all voting adults". I use that stats course way more often than I use even algebra, just to be a functional adult in a world where bad statistics are used day in and day out to manipulate and deceive people into buying things or voting for someone. So, I have to disagree: not only is a basic foundation in stats essential for understanding ML, it's something everyone really should have under their belt anyway to live in the modern world without turning into someone else's pawn.
  - fivestones 3y ago
    Where did you do your stats course? Who taught it? What was the textbook? Asking because it sounds like your experience taking stats was a lot better than mine and I wonder if I could learn some more if I studied it again in a way more similar to what you did.

wanderingmind 3y ago

This type of bottom up approach is terrible idea for a fast moving area like ML. Ultimately to get a job and make money in the area, you need to solve customer problems, starting with libraries and fine tuning is what needs to happen first. However you should try to learn fundamentals as you go and when you get stuck. It will take a very long time if you have a full time job or a student studying something else and doesn't help to get a job

tsurba 3y ago
Getting a job to make money and solve customer problems as fast as possible was not a stated goal by the OP. Besides, you are also wrong, having good fundamentals in the maths will help you pick up new methods much faster as they pop up. And especially if you want to come up with new methods (as in research), there are no shortcuts.
- camhart 3y ago
  They made a reasonable assumption. Abstraction helps you to be productive. It's certainly good to understand everything all the way down to the actual physics behind computing, but its not necessary especially at the start. I don't have to know exactly how logic gates work to program in JS and make a difference. I assume the same applies to ML. Motivation is what prevents most people from learning difficult to learn skills. Finding ways to produce value early on can help keep you motivated.
  - DeathArrow 3y ago
    >I don't have to know exactly how logic gates work to program in JS and make a difference. I don't regard time spent in University to learn how logical gates work and many other useful things, as a loss of time. And, as a web developer/architect (after many other industries I worked in), I typically make more money than peers who don't. And knowing how things work, helped me immensely in my career. Not everything is solvable by looking on Stack Overflow. It depends on what professional level you are content with. Learning was not only motivated by money - for me- I was genuinely curious about computers and software since I was a kid.
    - janalsncm 3y ago
      > I don't regard time spent in University to learn how logical gates work and many other useful things, as a loss of time. It’s unquestionably a loss of time, the only question is whether it’s an optimal use of time. This depends on your goals. For most software engineers looking to use ML, starting with existing frameworks and knowledge and drilling down as necessary is the most prudent method. We all have to make sacrifices in what we learn. Even for yourself, the topics you chose to learn about implicitly left out other parallel topics you didn’t learn about. And you also didn’t learn everything from the least abstracted, most fundamental level. We need to choose the appropriate level of abstraction for the problem at hand, which will depend on each person’s goals.
  - ekianjo 3y ago
    > I don't have to know exactly how logic gates work to program in JS and make a difference. Thats a stupid analogy to the above discussion. A better analogy would be trying to program in JS without understanding for loops and basic programmation concepts. Thats typically what I observe with younger folks jumping into neural nets directly. They have a very shallow understanding of anything and survive with youtube tutorials.
    - PeterisP 3y ago
      Most of what was listed above aren't the fundamentals which would help to properly understand what's happening with neural networks but rather completely different branches of machine learning that have little in common with neural networks. If they learn SVM, naive Bayes, and gradient boosting, then their knowledge will definitely be broader but just as shallow; using your analogy, it's not like trying to program in JS without understanding for loops but rather like trying to program in JS without understanding C, COBOL and Haskell. I'm all for learning the fundamentals properly - but those fundamentals are going to be completely things, things like core principles of statistics (limitations of correlation, confounders, bias/variance, etc), the relevant parts of calculus and linear algebra that matter for understanding optimization, the best practices for management of data, experiments and measurement to not cheat yourself, etc - not the checklist of many different, parallel methods of machine learning like decision trees or reinforcement learning, which are both useful and interesting, but not related or required to properly apply e.g. transformer-based large language models for your task.
DeathArrow 3y ago
The better your skillset, the highly sought after you are and harder to replace. And that translates in more money, if money are your sole motivation.
OJFord 3y ago
OP did not state that getting an ML job was a goal. I for one would be interested in learning more foundational stuff; I have no interest (though perhaps that process would change this!) in a particularly ML job, and certainly not in learning how to point and click and run other people's work from a YouTube video with a scream-face thumbnail. For those who feel similarly, I asked about it recently, maybe something you like the look of: https://news.ycombinator.com/item?id=38320244 https://news.ycombinator.com/item?id=38320244 I also came across (doesn't seem it was mentioned there) Understanding Deep Learning (Simon Prince) which looks like it might be good: https://mitpress.mit.edu/9780262048644/understanding-deep-learning/ https://mitpress.mit.edu/9780262048644/understanding-deep-le...
CityOfThrowaway 3y ago
Strong agree
ekianjo 3y ago
> his type of bottom up approach is terrible idea for a fast moving area like ML. Of course not. When you learn this way you will realize there are a myriad of problems that can be solved with more simple algorithms. Trying to make every problem fit to neural networks is pure cargo culting.
- digging 3y ago
  That's true, but the goal is to learn to use neural networks, not to solve problems efficiently, isn't it?

mickael-kerjean 3y ago

> SVM, linear regression, naive Bayes When I studied ML in 2012, the very first course started with naive Bayes and went one from there. A decade after being away, I see a lot of people around me starting with neural nets to train a model that naive Bayes would be plenty enough for and never heard about naive Bayes. Is that only my experience?

yCombLinks 3y ago
Any data that is small enough to quickly iterate on for learning is small enough to use a simpler approach. The point is learning the techniques.
owlbite 3y ago
As with any super-hyped technology, people want to use the latest and greatest to solve it, even when traditional methods are cheaper, easier, more reliable and more accurate. See also: crypto.
PeterisP 3y ago
It's useful to do learning on small toy problems for ease of debugging and speed of training, so if you want to learn how to apply a powerful technique, you're pretty much inevitably going to start learning it on something for which it's absolutely ineffective overkill. E.g. a common start for learning neural nets is a task like XOR which can be solved with literally a single machine instruction instead of training a ML model for it. But also there are many tasks for which naive Bayes works, but a NN solution can be much more accurate if you're okay with it being also much more compute-intensive. E.g. things like sentiment analysis or simple spam filters are often used as a demonstration of naive Bayes, but you can do much better with more powerful models.

amanda99 3y ago

The risk with this is that you spend 10 hours learning statistics then get demotivated and never do the other 190 hours to get to the good stuff. Then quickly you forget the 10 hours of stats you learnt too as it's irrelevant and you don't use it. For me, playing with things and doing cool & fun stuff is always the way to get deeper into something.

lolinder 3y ago
> Then quickly you forget the 10 hours of stats you learnt too as it's irrelevant and you don't use it. Stats is one of a very small number of college courses that I took where I came away thinking "this should be mandatory for all voting adults". I use that stats course way more often than I use even algebra, just to be a functional adult in a world where bad statistics are used day in and day out to manipulate and deceive people into buying things or voting for someone. So, I have to disagree: not only is a basic foundation in stats essential for understanding ML, it's something everyone really should have under their belt anyway to live in the modern world without turning into someone else's pawn.
- fivestones 3y ago
  Where did you do your stats course? Who taught it? What was the textbook? Asking because it sounds like your experience taking stats was a lot better than mine and I wonder if I could learn some more if I studied it again in a way more similar to what you did.