3 ms·

I got my BSc in computer science and PhD in machine learning, and ended up working in a top FAANG AI research lab. In the hindsight both when doing research fo

I got my BSc in computer science and PhD in machine learning, and ended up working in a top FAANG AI research lab. In the hindsight both when doing research for my PhD and also when working as an engineer I felt the most useful courses from undergrad were linear algebra, algorithms, calculus, operating systems, and statistics in that order. I ended up filling the gaps in my math education later by reading textbooks and taking online courses. IMO an undergrad program should focus on very fundamental theory. If I was in charge of designing CS programs I would quadruple the amount of credits required in math and specifically in linear algebra. You would be surprised how handy and applicable linear algbera is in ML, CV, robotics, computer graphics, finance, etc. etc. Calculus is also important but to a lesser degree. It's a waste of time to teach TensorFlow or teach the trendiest neural network architecture at school. The knowledge becomes irrelevant in a few years, and it's fairly easy to pick it up by reading docs/papers if you know the fundamentals.

throwawaygh 6y ago
> It's a waste of time to teach TensorFlow or teach the trendiest neural network architecture at school. The knowledge becomes irrelevant in a few years, and it's fairly easy to pick it up by reading docs/papers if you know the fundamentals. Well, kind of. You teach one or two instances of such things as a case study in how to learn a framework. Usually Software Engineering courses are the best place to do this. The point is, your ML course should probably not be spending any time on things like pytorch. A sophomore level engineering course should have already taught students how to go through the process of learning a new framework.
ZephyrBlu 6y ago
Which areas of Linear Algebra did you find particularly useful?
- maxov 6y ago
  This answer is probably biased by what I’ve needed to use recently, but I think spectral methods are the most generally useful. E.g. spectra of symmetric matrices, SVD, Courant-Fischer theorem. You may not need this in all of, or even most of, practical ML, but knowing these things are prerequisites in my mind for understanding PCA, CCA, LDA/QDA, multivariate Gaussians (which are foundational in probabilistic interpretations of ML), and covariance. A good understanding of inner products also conditions you to understanding kernels better. You need something a bit different for SVMs. The linear algebra there is basically the geometry of planes and half spaces. Also for optimization you need some different things, but those are typically taught under the moniker of convex analysis, not linear algebra. In specific, my approach to many multivariate estimation problems starts with “take the SVD, are there any properties you can use afterwards?”
- codelord 6y ago
  It's hard to pinpoint a few topics. I'd just suggest to go through an introductory course like Gilbert Strang's intro to linear algebra to get a general understanding and build on your knowledge as needed.

throwawaygh 6y ago

> It's a waste of time to teach TensorFlow or teach the trendiest neural network architecture at school. The knowledge becomes irrelevant in a few years, and it's fairly easy to pick it up by reading docs/papers if you know the fundamentals. Well, kind of. You teach one or two instances of such things as a case study in how to learn a framework. Usually Software Engineering courses are the best place to do this. The point is, your ML course should probably not be spending any time on things like pytorch. A sophomore level engineering course should have already taught students how to go through the process of learning a new framework.

ZephyrBlu 6y ago

Which areas of Linear Algebra did you find particularly useful?

maxov 6y ago
This answer is probably biased by what I’ve needed to use recently, but I think spectral methods are the most generally useful. E.g. spectra of symmetric matrices, SVD, Courant-Fischer theorem. You may not need this in all of, or even most of, practical ML, but knowing these things are prerequisites in my mind for understanding PCA, CCA, LDA/QDA, multivariate Gaussians (which are foundational in probabilistic interpretations of ML), and covariance. A good understanding of inner products also conditions you to understanding kernels better. You need something a bit different for SVMs. The linear algebra there is basically the geometry of planes and half spaces. Also for optimization you need some different things, but those are typically taught under the moniker of convex analysis, not linear algebra. In specific, my approach to many multivariate estimation problems starts with “take the SVD, are there any properties you can use afterwards?”
codelord 6y ago
It's hard to pinpoint a few topics. I'd just suggest to go through an introductory course like Gilbert Strang's intro to linear algebra to get a general understanding and build on your knowledge as needed.