7 ms·

The Gap: Where Machine Learning Education Falls Short

56 points by andreyk 6y ago

Exuma 6y ago
So how about actually giving suggestions of courses or resources that fix this, instead of just saying what's broken and then printing your name.
- lukeplato 6y ago
  He proposes changing the syllabus of advanced/graduate-level courses to skip reviewing linear classification and backprop and make that a pre-requisite.
- marketingPro 6y ago
  Keep reading the article
bonoboTP 6y ago
This complaint comes up time and time again. "Universities should prepare students better for jobs!" "Teach more real life skills!" "Turn CS uni into trade school!" But a computer science university program is not about scikit-learn or TensorFlow! It's about long-lasting principles, underlying mathematics, mental models and ways of thinking. None of my computer science lectures were about how to apply that particular part of CS knowledge in some hot new Python library. It's expected that there will be some amount of time required to adjust to a company's software setup. That's not a big hurdle usually. I'm not saying it should be only theory, though. University courses often have accompanying assignments or projects. Depending on the country in question, they often offer more hands-on, practical courses ("lab courses") as well, where you do actually go through the steps of making the theory work in real life. I had such courses where we played with microcontrollers and FPGAs to understand CPU instructions, assembly and low-level C concepts (but even there the goal wasn't to learn exactly the thing that you will use on the job. Most CS graduates will never need to program FPGAs in their day job.). But sure, there is a place for even more data engineering training, but I don't think it's computer science university programs. Where do people like network engineers learn how to configure Cisco routers and use whatever config software they use? Where do sysadmins learn Bash, Unix, backup management etc? Not at university courses. Wherever they learn those skills, that's where data cleaning, parallelization engineering etc aspects of machine learning should be taught as well.
- jasim 6y ago
  I've oscillated between these two positions for a few years now, when in truth neither positions are really in conflict. When we complain about universities not preparing students better for jobs, what we really mean is that universities are not doing the bare minimum that they should be doing - in case of CS, students should at least know how to program well, and be well versed in the practicalities of computing. That does not exclude learning the fundamentals (which is often denigrated as "theory"). It is just that students often have neither the theory nor the practice, and at a minimum, we're asking, they should know the practice so they can at least be useful in their jobs.
  - marketingPro 6y ago
    The fact that there is somehow a notable size of computer science grads that don't know how to program is the most major red flag. Are students cheating? Is curriculum group based? Is the content not hard enough? If people need to do coding interviews, I see no reason why similar can't be done in college at a 200 and 300 level checkpoint. Programming/logic is easy if you understand. It doesn't need to be directly tested often.
    - bonoboTP 6y ago
      There's lots of hard CS content you can learn and take exams in without writing code. Logic design, complexity theory and automata, graph theory theorem proofs, linear algebra, complex analysis, coding theory (compression, encryption, ...), algorithms, proofs about data structures, operating system theory (scheduling algorithms, deadlocks, race conditions, virtual memory), database theory, etc.
    - zdragnar 6y ago
      When I was at a major US University in thr early 2000's, anything higher than C wasn't taught because the field changes faster than a 4 year degree would make sense- modern languages were seen as the domain of 2 year tech schools. I believe that has since changed, but I am not sure to what extent.
    - 6y ago
codelord 6y ago
I got my BSc in computer science and PhD in machine learning, and ended up working in a top FAANG AI research lab. In the hindsight both when doing research for my PhD and also when working as an engineer I felt the most useful courses from undergrad were linear algebra, algorithms, calculus, operating systems, and statistics in that order. I ended up filling the gaps in my math education later by reading textbooks and taking online courses. IMO an undergrad program should focus on very fundamental theory. If I was in charge of designing CS programs I would quadruple the amount of credits required in math and specifically in linear algebra. You would be surprised how handy and applicable linear algbera is in ML, CV, robotics, computer graphics, finance, etc. etc. Calculus is also important but to a lesser degree. It's a waste of time to teach TensorFlow or teach the trendiest neural network architecture at school. The knowledge becomes irrelevant in a few years, and it's fairly easy to pick it up by reading docs/papers if you know the fundamentals.
- throwawaygh 6y ago
  > It's a waste of time to teach TensorFlow or teach the trendiest neural network architecture at school. The knowledge becomes irrelevant in a few years, and it's fairly easy to pick it up by reading docs/papers if you know the fundamentals. Well, kind of. You teach one or two instances of such things as a case study in how to learn a framework. Usually Software Engineering courses are the best place to do this. The point is, your ML course should probably not be spending any time on things like pytorch. A sophomore level engineering course should have already taught students how to go through the process of learning a new framework.
- ZephyrBlu 6y ago
  Which areas of Linear Algebra did you find particularly useful?
  - maxov 6y ago
    This answer is probably biased by what I’ve needed to use recently, but I think spectral methods are the most generally useful. E.g. spectra of symmetric matrices, SVD, Courant-Fischer theorem. You may not need this in all of, or even most of, practical ML, but knowing these things are prerequisites in my mind for understanding PCA, CCA, LDA/QDA, multivariate Gaussians (which are foundational in probabilistic interpretations of ML), and covariance. A good understanding of inner products also conditions you to understanding kernels better. You need something a bit different for SVMs. The linear algebra there is basically the geometry of planes and half spaces. Also for optimization you need some different things, but those are typically taught under the moniker of convex analysis, not linear algebra. In specific, my approach to many multivariate estimation problems starts with “take the SVD, are there any properties you can use afterwards?”
glitchc 6y ago
“Ah I see, your college degree is in Arts.” “Wait, what do you mean you can’t sing, or act, or draw? What do they teach you in school??!?!?”
_RPL5_ 6y ago
The author makes a point that I relate to. I've been on the receiving end of a couple 'Statistical Learning 101' courses. These courses go roughly as he describes it in his blog post: they first teach you how to multiply two matrices, then launch into linear and logistic regression, then classification via clustering, then decision trees and SVMs, then CNNs and deep learning. Along the way, they do a lecture or two on reinforcement learning and HMMs. In the end, I ended up with a thin smear of half-baked knowledge in my head, where I stop understanding the math once we are half-way through the material. So how to you achieve this level of deep/intuitive understanding: > Without understanding the mathematical underpinnings of key models and techniques in full detail, students aren’t able to quickly choose the right models for certain scenarios. Does anyone have a good study plan with MOOCs and so on? If you have any practical advice, I would appreciate it!
- orange3xchicken 6y ago
  I mean it really depends how deep you want to go. Like you point out that the classes you took are 101 courses. These are really just "tasting" courses. I'm sure if you decided to take more advanced/grad numbered courses, or unnumbered "topics" courses, you would have a better idea of what's going on. In general, "having a deep understanding of models & techniques in full detail" is not well-defined. For example, analysis of linear regression is often offered as a full year-long sequence for graduate students in math/stats depts. Is this necessary for doing linear regression in practice? Not really, but who cares - it's interesting stuff in its own right. Most people just need just enough understanding to finish a job. In general, the precise medium that you use to study something isn't that important as long as it works for you, but there is a good reason that there are longstanding classic textbooks that people swear by in most fields of mathematics. I do strongly feel that in the context of any mathematical subject, that there are few substitutes to the grind - doing proofs and solving problems on your own. Okay, but just to have at least one link in my post, I want to share this guy MathematicalMonk who used to make really great videos on ml-related stuff: https://www.youtube.com/channel/UCcAtD_VYwcYwVbTdvArsm7w https://www.youtube.com/channel/UCcAtD_VYwcYwVbTdvArsm7w
  - _RPL5_ 6y ago
shekharshan 6y ago
The author highlights not teaching enough mathematical theory behind the various techniques. I have tried Andrew Ng's course on Coursera. It uses Octave from what I remember. After a point the lack of mathematical background started to show up. I have always wondered where can I find course that teaches both the mathematical background as well as the hands-on programming in a balanced way.

56 points by andreyk 6y ago

So how about actually giving suggestions of courses or resources that fix this, instead of just saying what's broken and then printing your name.

lukeplato 6y ago
He proposes changing the syllabus of advanced/graduate-level courses to skip reviewing linear classification and backprop and make that a pre-requisite.
marketingPro 6y ago
Keep reading the article

bonoboTP 6y ago

This complaint comes up time and time again. "Universities should prepare students better for jobs!" "Teach more real life skills!" "Turn CS uni into trade school!" But a computer science university program is not about scikit-learn or TensorFlow! It's about long-lasting principles, underlying mathematics, mental models and ways of thinking. None of my computer science lectures were about how to apply that particular part of CS knowledge in some hot new Python library. It's expected that there will be some amount of time required to adjust to a company's software setup. That's not a big hurdle usually. I'm not saying it should be only theory, though. University courses often have accompanying assignments or projects. Depending on the country in question, they often offer more hands-on, practical courses ("lab courses") as well, where you do actually go through the steps of making the theory work in real life. I had such courses where we played with microcontrollers and FPGAs to understand CPU instructions, assembly and low-level C concepts (but even there the goal wasn't to learn exactly the thing that you will use on the job. Most CS graduates will never need to program FPGAs in their day job.). But sure, there is a place for even more data engineering training, but I don't think it's computer science university programs. Where do people like network engineers learn how to configure Cisco routers and use whatever config software they use? Where do sysadmins learn Bash, Unix, backup management etc? Not at university courses. Wherever they learn those skills, that's where data cleaning, parallelization engineering etc aspects of machine learning should be taught as well.

jasim 6y ago
I've oscillated between these two positions for a few years now, when in truth neither positions are really in conflict. When we complain about universities not preparing students better for jobs, what we really mean is that universities are not doing the bare minimum that they should be doing - in case of CS, students should at least know how to program well, and be well versed in the practicalities of computing. That does not exclude learning the fundamentals (which is often denigrated as "theory"). It is just that students often have neither the theory nor the practice, and at a minimum, we're asking, they should know the practice so they can at least be useful in their jobs.
- marketingPro 6y ago
  The fact that there is somehow a notable size of computer science grads that don't know how to program is the most major red flag. Are students cheating? Is curriculum group based? Is the content not hard enough? If people need to do coding interviews, I see no reason why similar can't be done in college at a 200 and 300 level checkpoint. Programming/logic is easy if you understand. It doesn't need to be directly tested often.
  - bonoboTP 6y ago
    There's lots of hard CS content you can learn and take exams in without writing code. Logic design, complexity theory and automata, graph theory theorem proofs, linear algebra, complex analysis, coding theory (compression, encryption, ...), algorithms, proofs about data structures, operating system theory (scheduling algorithms, deadlocks, race conditions, virtual memory), database theory, etc.
  - zdragnar 6y ago
    When I was at a major US University in thr early 2000's, anything higher than C wasn't taught because the field changes faster than a 4 year degree would make sense- modern languages were seen as the domain of 2 year tech schools. I believe that has since changed, but I am not sure to what extent.
  - 6y ago

codelord 6y ago

I got my BSc in computer science and PhD in machine learning, and ended up working in a top FAANG AI research lab. In the hindsight both when doing research for my PhD and also when working as an engineer I felt the most useful courses from undergrad were linear algebra, algorithms, calculus, operating systems, and statistics in that order. I ended up filling the gaps in my math education later by reading textbooks and taking online courses. IMO an undergrad program should focus on very fundamental theory. If I was in charge of designing CS programs I would quadruple the amount of credits required in math and specifically in linear algebra. You would be surprised how handy and applicable linear algbera is in ML, CV, robotics, computer graphics, finance, etc. etc. Calculus is also important but to a lesser degree. It's a waste of time to teach TensorFlow or teach the trendiest neural network architecture at school. The knowledge becomes irrelevant in a few years, and it's fairly easy to pick it up by reading docs/papers if you know the fundamentals.

throwawaygh 6y ago
> It's a waste of time to teach TensorFlow or teach the trendiest neural network architecture at school. The knowledge becomes irrelevant in a few years, and it's fairly easy to pick it up by reading docs/papers if you know the fundamentals. Well, kind of. You teach one or two instances of such things as a case study in how to learn a framework. Usually Software Engineering courses are the best place to do this. The point is, your ML course should probably not be spending any time on things like pytorch. A sophomore level engineering course should have already taught students how to go through the process of learning a new framework.
ZephyrBlu 6y ago
Which areas of Linear Algebra did you find particularly useful?
- maxov 6y ago
  This answer is probably biased by what I’ve needed to use recently, but I think spectral methods are the most generally useful. E.g. spectra of symmetric matrices, SVD, Courant-Fischer theorem. You may not need this in all of, or even most of, practical ML, but knowing these things are prerequisites in my mind for understanding PCA, CCA, LDA/QDA, multivariate Gaussians (which are foundational in probabilistic interpretations of ML), and covariance. A good understanding of inner products also conditions you to understanding kernels better. You need something a bit different for SVMs. The linear algebra there is basically the geometry of planes and half spaces. Also for optimization you need some different things, but those are typically taught under the moniker of convex analysis, not linear algebra. In specific, my approach to many multivariate estimation problems starts with “take the SVD, are there any properties you can use afterwards?”

glitchc 6y ago

“Ah I see, your college degree is in Arts.” “Wait, what do you mean you can’t sing, or act, or draw? What do they teach you in school??!?!?”

_RPL5_ 6y ago

The author makes a point that I relate to. I've been on the receiving end of a couple 'Statistical Learning 101' courses. These courses go roughly as he describes it in his blog post: they first teach you how to multiply two matrices, then launch into linear and logistic regression, then classification via clustering, then decision trees and SVMs, then CNNs and deep learning. Along the way, they do a lecture or two on reinforcement learning and HMMs. In the end, I ended up with a thin smear of half-baked knowledge in my head, where I stop understanding the math once we are half-way through the material. So how to you achieve this level of deep/intuitive understanding: > Without understanding the mathematical underpinnings of key models and techniques in full detail, students aren’t able to quickly choose the right models for certain scenarios. Does anyone have a good study plan with MOOCs and so on? If you have any practical advice, I would appreciate it!

orange3xchicken 6y ago
I mean it really depends how deep you want to go. Like you point out that the classes you took are 101 courses. These are really just "tasting" courses. I'm sure if you decided to take more advanced/grad numbered courses, or unnumbered "topics" courses, you would have a better idea of what's going on. In general, "having a deep understanding of models & techniques in full detail" is not well-defined. For example, analysis of linear regression is often offered as a full year-long sequence for graduate students in math/stats depts. Is this necessary for doing linear regression in practice? Not really, but who cares - it's interesting stuff in its own right. Most people just need just enough understanding to finish a job. In general, the precise medium that you use to study something isn't that important as long as it works for you, but there is a good reason that there are longstanding classic textbooks that people swear by in most fields of mathematics. I do strongly feel that in the context of any mathematical subject, that there are few substitutes to the grind - doing proofs and solving problems on your own. Okay, but just to have at least one link in my post, I want to share this guy MathematicalMonk who used to make really great videos on ml-related stuff: https://www.youtube.com/channel/UCcAtD_VYwcYwVbTdvArsm7w https://www.youtube.com/channel/UCcAtD_VYwcYwVbTdvArsm7w
- _RPL5_ 6y ago

shekharshan 6y ago

The author highlights not teaching enough mathematical theory behind the various techniques. I have tried Andrew Ng's course on Coursera. It uses Octave from what I remember. After a point the lack of mathematical background started to show up. I have always wondered where can I find course that teaches both the mathematical background as well as the hands-on programming in a balanced way.