11 ms·

"The widespread misimpression that data + neural networks is a universal formula has real consequences: [...] in what students choose to study, and in what univ

"The widespread misimpression that data + neural networks is a universal formula has real consequences: [...] in what students choose to study, and in what universities choose to teach." I think this is a problem that deserves more attention. A large portion of CS students nowadays choose to focus on Machine Learning, many brilliant CS students decide to get a PhD in ML in hope of contributing to a field that will continue to develop at a pace similar to the progress we have seen in the last 7 years. This view, I think, is not justified at all. The challenges that AI faces such as incorporating a world model into the techniques that work well will be significantly harder than what most researchers openly admit, probably to the detriment of other CS research areas.

didibus 7y ago
I've also noticed a trend of college graduates who in interview struggle with general software engineering practices and more fundamental coding skills and CS knowledge, but have knowledge and proficiency of ML. If you're a CS BA or MS, and don't plan to do an ML PhD, I'd suggest asking yourself if you want a data science job or a software engineering one. If the latter, remember to focus more on that. Surface knowledge of ML is a bonus, but the rest is more important.
- aidos 7y ago
  That’s definitely something I’ve seen too. Most of the cvs I get through, especially from younger devs, talk about ML. I’ve got my work cut out for me keeping my existing codebase understandably without deliberately introducing opacity!
- weego 7y ago
  My partner has ditched crosstraining into data science and ML because of how ridiculously 'ductape and string' the entire sub-indistry is. Tooling that is pre-2000 quality if it even exists, people with math skills yet only the most rudimentary ability in best practice around data storage and maintenance, coding, domain knowledge and understanding of how much bias they're introducing into results. Her view is its just a free for all of people who have little to no need to justify their output because everyone is waiting and hoping for the magic to happen.
  - deleted 7y ago
    [deleted]
  - itronitron 7y ago
    Larger organizations (by size and longevity) tend to have staff that predate the data science and ML/AI hype train and have therefore worked out more robust tooling. Unfortunately the popularity of ML is driving a lot of adoption of trendy but not necessarily best practices.
    - mattlutze 7y ago
      There was a lot of this sort of thing with the web application boom as well. Lots and lots of decrying how JS libraries and frameworks were ruining the software industry. I've not following that very much in a decade though. Did that whole area end up maturing?
- VBprogrammer 7y ago
  I've noticed this myself but wasn't sure if it was just me. Almost every CV we get has a ML slant but the candidates struggle to write a simple SQL query. From a pure numbers perspective 95% of jobs in our industry have nothing to do with any reasonable definition of machine learning. Even those applications of ML have a large amount of traditional CS going into acquiring, reformatting and storing large datasets.
currymj 7y ago
i wouldn't underestimate the power of good tools here. All the software libraries for ML are very easy to get started with, and make it very easy to prototype cool things. It seems like in other applied areas it's a lot more work to get less results.
- skywhopper 7y ago
  I think that's exactly the problem with ML. You can get interesting looking results fast with very little effort. But then, getting from an impressive demo to something actually useful in the real world is much harder, and will lead you down endless rabbit-holes as you try to improve your results, but things only get worse, not better...
  - randcraw 7y ago
    After 5 years of growth in data mining at my giant pharma, this is exactly what I'm seeing. Most ML projects remain toys while the number of them that advance into something useful can be counted on one hand. (Of course it's a bit hard to assess the impact of a revolution like ML (esp DL) when your company already has hundreds of statisticians who have been employing similar data/experiment analysis techniques for decades, thereby diluting the signal of how disruptive novel forms of ML are within the enterprise.)
autokad 7y ago
> " The challenges that AI faces" Isn't this a good thing? I would think that If it were easy we wouldn't need graduate degrees
account73466 7y ago
Deep learning still at least doubles every two years, it can be estimated by https://scholar.google.com/citations?user=kukA0LcAAAAJ https://scholar.google.com/citations?user=kukA0LcAAAAJ .
- amelius 7y ago
  But how do you estimate the quality of those publications? Is knowledge about DL exploding, or is it saturating?
  - account73466 7y ago
    This person is the most cited scientist in machine learning (when measured by # citations per year) and is about to become the most cited scientist in the whole world. He has a lab of about 100 people and most of his works are at least good. I just provided an estimated. Once you see his citations count has saturated (probably in 5-10 years), you may guess that the hype is over.
    - xamuel 7y ago
      Sounds like someone reeeeeallly good at nudging envelopes and writing grant proposals. Will be mostly forgotten a couple hundred years from now. Academic playwrights thought Shakespeare was a joke in his day.
      - account73466 7y ago
        "a couple hundred years from now" by whom? He contributed to AI, will not be forgotten by AI. Edit: if you don't get what I mean by "not be forgotten", he will be the most cited scientist so why AI will exclude all his papers from some of its training sets (assuming that at least some stage of AI's development it will learn how humans do research)? Sounds unlikely.
        xamuel 7y ago
        By human beings. And he did not contribute to AI capable of contemplating human research like a human does, because no such AI exists yet. When it does exist (if ever), it will surely not consider this person or his hundred lab workers to have contributed to AI. It will consider them to have contributed to applied statistics. Edit: I realized these comments might be very mean to you (assuming you're the lab director with the hundred workers). I should disclaim that what I say is something I assume is true about most people who gloat about their citation counts, but it's certainly not true about all such people, and might not be true about you. Myself, I'm a bit resentful because I'm so bad at the whole academic game, so that probably makes me quite biased.
high_derivative 7y ago
This is a pet peeve of mine as well. A lot of undergrads are not really looking around any more. Universities offer their first data science courses in year one. Undergrads mapping out their paths towards becoming research scientists with dollar signs in their eyes. I think this is not wise career bet at all. There will always be spots for a select few 'pure ML' grads, but at the post phd level the really good jobs are getting pretty thin unless you come from the right adviser + right school + right publication track. On the other hand, if you add a secondary useful skill, like having a really good understanding of distributed systems, or networking, or databases, or embedded systems, or being 'just' a solid overall software engineer, you (i) have something to fall back on if ML dreams do not work out, and (ii) have an edge by being just the right fit for specific teams, while also still competing in the general pool. I think there will be a rude awakening on this in 3-4 years for many.
- codebolt 7y ago
  Another tactic for an undergrad looking for a leg up in their career might be to do a double major of CS + some other engineering field or finance. You can really outperform your peers and become a unique asset in a given industry by having an expert understanding of the domain paired with expertise in software development.
  - tenaciousDaniel 7y ago
    This is true. My first job as a dev was in fintech, but I know literally nothing about finance, nor do I care to. I was not a very good developer in that arena, because you really need domain knowledge to be truly effective in some fields.
  - asdfman123 7y ago
    I majored in geophysics and it doesn't really help me as a software engineer in Houston (although I haven't tried to really exploit it for the most money, because I felt guilty about working in oil and left). What's killer though is having a master's in a field and programming experience.
nmca 7y ago
There are certainly challenges, but "incorporating a world model" has been going well recently: "Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model" https://arxiv.org/abs/1911.08265 https://arxiv.org/abs/1911.08265
- drongoking 7y ago
  Not exactly. Incorporating a world or domain model usually means taking pre-existing declarative knowledge in some form (e.g. from a semantic net) and using it to aid learning. To quote the article, "Model-based reinforcement learning aims to address this issue by first learning a model of the environment’s dynamics, and then planning with respect to the learned model." So they've sped up learning from examples by learning a model first, but it's still learning from examples.
  - nmca 7y ago
    That seems a fairly specific meaning of the term that may not be in wider use, see eg: https://arxiv.org/abs/1803.10122 https://arxiv.org/abs/1803.10122
dxbydt 7y ago
> large portion of CS students nowadays choose to focus on Machine Learning, many brilliant CS students decide to get a PhD in ML These claims are massively exaggerated. Students don't "choose", "decide" etc. In the US, lets be generous & say there are 50 top universities with good CS/ML depts. After students beg plead & submit their GRE scores & recos & transcripts etc, each dept chooses on average 20 students for incoming PhD cohort. Of them easily 20% will wash out for sure. So atmost 800 top students graduating with a PhD in ML each year. Let me caveat by saying there are nowhere close to 50 top univs offering ML PhDs or having an intake of 20 per dept. So overall, the number is probably close to 200-300 students for the whole USA, not 800. So all this handwringing for what some 200 kids will do ? I don't care if they do a PhD in basketweaving, in the grand scheme of things, 200 is not even a drop in bucket.
- IC4RUS 7y ago
  I agree that it might not be a large portion of all CS students, but it does appear that many of the incoming CS PhDs have chosen to go into ML (of my class, maybe 1/3), which serious distracts from interest in other subfields. It's actually a joke in my department that all of the new students are pursuing ML, while almost no one is pursuing theoretical CS or other less popular subfields. I'm also not sure about why the exact number matters - 200 kids matters a lot when future professors are drawn from the pool of students who have successfully completed a PhD.
  - randcraw 7y ago
    I think the problem applies to non-PhDs students as well. I'm seeing a lot of interest from recent non-PhD grads in subjects other than CS wanting to steer their career toward data mining. My concern is that this re-focusing will probably lead them down paths that may leave them undistinguished relative to peers who stay within the more stable but less shiny domains where they're better prepared to succeed. I saw the same thing happen in the late 1990s as everybody and his dog got into web design while it was hot. Few of those folks are doing that now, nor did that skill translate well into other roles since the skill set isn't fundamental to other careers. I'm not sure ML is any different, especially deep learning, since few companies have anywhere near the necessary amount of data to successfully play that game and win.
buboard 7y ago
Cpus have hit their limits and not growing. Software has stagnated and recycles concepts from 50 years ago, and not growing. What's left to study? Either world-scale clouds or the growing field of NNs. Out of the two i 'd pick the 2nd cause it's either more fun or more unpredictable. Students are making a rational choice
- IC4RUS 7y ago
  Even if those fields completely stagnated, there's great progress to be made in high performance computing, security, bioinformatics, and human computer interaction (just to name a few).
thewarrior 7y ago
It may not be ideal for fundamental research but demand for ML practitioners is only likely to increase over time. ML is pure alchemy for a business operating at scale. It’s as if a coal plant could turn its waste emissions into profit. You have this “data exhaust” from all the activity happening within your system that can be used to optimize your system Atleast a few percentage points beyond what is other wise possible. A team of 5 ML engineers can improve an ad targeting system by 5 % and if the company is google that’s billions. ML creates feedback loops of improvements in product that improve usage that lead to more data which further strengthens the moat a business has. It totally makes sense to jump on this train. It won’t solve AI but will make a lot of people wealthy optimizing systems.
- YeGoblynQueenne 7y ago
  >> It may not be ideal for fundamental research but demand for ML practitioners is only likely to increase over time. If research stagnates, application willl also stagnate. For the industry money to keep pouring in there has to be growth and for there to be growth there has to be progress- scientific progress. So if something is "not ideal for fundamental research" it is also "not ideal" for business.
- heavenlyblue 7y ago
  >> A team of 5 ML engineers can improve an ad targeting system by 5 % and if the company is google that’s billions. Not going to pick up on your arbitrary 5% constant here, but please elaborate how these ML engineers are any different from anyone with a quantitative background?
  - bumby 7y ago
    I think that's the key here. It's not ML magic that's driving success, it data literacy. Steve Levitt (of Freakanomics fame) said his advice to students would be to ensure they have base knowledge in understanding data analytics regardless of their domain. ML is just the sexy subset that gets all the attention
    - 7y ago
pradn 7y ago
It's also a problem that students are focusing so much on neural-network-based approaches, neglecting other techniques. There's what, like, 10 places in the world where you'll have enough compute to be able to train cutting-edge neural networks? Real world problems are just as well solved with other techniques like XG-Boost, which wins pretty often in Kaggle, for example.
- hnaccy 7y ago
  >There's what, like, 10 places in the world where you'll have enough compute to be able to train cutting-edge neural networks You can just use a smaller amount of compute with transfer learning style stuff. I can probably fine-tune a transformer model with my pocket money and beat any NLP solution from two years ago.
cdavid 7y ago
Most of the ML useful in industry is getting commoditized at a fast pace, and not enough people understand that. The focus on modeling for applying ML in a business environment is as incongruous as people believing programming is graph algo and compiler techniques. It sometimes is, but rarely. It is much easier to learn the basics of ML if you have good software engineering skills than the opposite in my experience. The one thing that needs time learning is experimental design and quantitative analysis, and this is rarely taught well at university before PhD.
1_over_n 7y ago
This narriative has also taken over the VC world.

didibus 7y ago

I've also noticed a trend of college graduates who in interview struggle with general software engineering practices and more fundamental coding skills and CS knowledge, but have knowledge and proficiency of ML. If you're a CS BA or MS, and don't plan to do an ML PhD, I'd suggest asking yourself if you want a data science job or a software engineering one. If the latter, remember to focus more on that. Surface knowledge of ML is a bonus, but the rest is more important.

aidos 7y ago
That’s definitely something I’ve seen too. Most of the cvs I get through, especially from younger devs, talk about ML. I’ve got my work cut out for me keeping my existing codebase understandably without deliberately introducing opacity!
weego 7y ago
My partner has ditched crosstraining into data science and ML because of how ridiculously 'ductape and string' the entire sub-indistry is. Tooling that is pre-2000 quality if it even exists, people with math skills yet only the most rudimentary ability in best practice around data storage and maintenance, coding, domain knowledge and understanding of how much bias they're introducing into results. Her view is its just a free for all of people who have little to no need to justify their output because everyone is waiting and hoping for the magic to happen.
- deleted 7y ago
  [deleted]
- itronitron 7y ago
  Larger organizations (by size and longevity) tend to have staff that predate the data science and ML/AI hype train and have therefore worked out more robust tooling. Unfortunately the popularity of ML is driving a lot of adoption of trendy but not necessarily best practices.
  - mattlutze 7y ago
    There was a lot of this sort of thing with the web application boom as well. Lots and lots of decrying how JS libraries and frameworks were ruining the software industry. I've not following that very much in a decade though. Did that whole area end up maturing?
VBprogrammer 7y ago
I've noticed this myself but wasn't sure if it was just me. Almost every CV we get has a ML slant but the candidates struggle to write a simple SQL query. From a pure numbers perspective 95% of jobs in our industry have nothing to do with any reasonable definition of machine learning. Even those applications of ML have a large amount of traditional CS going into acquiring, reformatting and storing large datasets.

currymj 7y ago

i wouldn't underestimate the power of good tools here. All the software libraries for ML are very easy to get started with, and make it very easy to prototype cool things. It seems like in other applied areas it's a lot more work to get less results.

skywhopper 7y ago
I think that's exactly the problem with ML. You can get interesting looking results fast with very little effort. But then, getting from an impressive demo to something actually useful in the real world is much harder, and will lead you down endless rabbit-holes as you try to improve your results, but things only get worse, not better...
- randcraw 7y ago
  After 5 years of growth in data mining at my giant pharma, this is exactly what I'm seeing. Most ML projects remain toys while the number of them that advance into something useful can be counted on one hand. (Of course it's a bit hard to assess the impact of a revolution like ML (esp DL) when your company already has hundreds of statisticians who have been employing similar data/experiment analysis techniques for decades, thereby diluting the signal of how disruptive novel forms of ML are within the enterprise.)

autokad 7y ago

> " The challenges that AI faces" Isn't this a good thing? I would think that If it were easy we wouldn't need graduate degrees

account73466 7y ago

Deep learning still at least doubles every two years, it can be estimated by https://scholar.google.com/citations?user=kukA0LcAAAAJ https://scholar.google.com/citations?user=kukA0LcAAAAJ .

amelius 7y ago
But how do you estimate the quality of those publications? Is knowledge about DL exploding, or is it saturating?
- account73466 7y ago
  This person is the most cited scientist in machine learning (when measured by # citations per year) and is about to become the most cited scientist in the whole world. He has a lab of about 100 people and most of his works are at least good. I just provided an estimated. Once you see his citations count has saturated (probably in 5-10 years), you may guess that the hype is over.
  - xamuel 7y ago
    Sounds like someone reeeeeallly good at nudging envelopes and writing grant proposals. Will be mostly forgotten a couple hundred years from now. Academic playwrights thought Shakespeare was a joke in his day.
    - account73466 7y ago
      "a couple hundred years from now" by whom? He contributed to AI, will not be forgotten by AI. Edit: if you don't get what I mean by "not be forgotten", he will be the most cited scientist so why AI will exclude all his papers from some of its training sets (assuming that at least some stage of AI's development it will learn how humans do research)? Sounds unlikely.
      - xamuel 7y ago
        By human beings. And he did not contribute to AI capable of contemplating human research like a human does, because no such AI exists yet. When it does exist (if ever), it will surely not consider this person or his hundred lab workers to have contributed to AI. It will consider them to have contributed to applied statistics. Edit: I realized these comments might be very mean to you (assuming you're the lab director with the hundred workers). I should disclaim that what I say is something I assume is true about most people who gloat about their citation counts, but it's certainly not true about all such people, and might not be true about you. Myself, I'm a bit resentful because I'm so bad at the whole academic game, so that probably makes me quite biased.

high_derivative 7y ago

This is a pet peeve of mine as well. A lot of undergrads are not really looking around any more. Universities offer their first data science courses in year one. Undergrads mapping out their paths towards becoming research scientists with dollar signs in their eyes. I think this is not wise career bet at all. There will always be spots for a select few 'pure ML' grads, but at the post phd level the really good jobs are getting pretty thin unless you come from the right adviser + right school + right publication track. On the other hand, if you add a secondary useful skill, like having a really good understanding of distributed systems, or networking, or databases, or embedded systems, or being 'just' a solid overall software engineer, you (i) have something to fall back on if ML dreams do not work out, and (ii) have an edge by being just the right fit for specific teams, while also still competing in the general pool. I think there will be a rude awakening on this in 3-4 years for many.

codebolt 7y ago
Another tactic for an undergrad looking for a leg up in their career might be to do a double major of CS + some other engineering field or finance. You can really outperform your peers and become a unique asset in a given industry by having an expert understanding of the domain paired with expertise in software development.
- tenaciousDaniel 7y ago
  This is true. My first job as a dev was in fintech, but I know literally nothing about finance, nor do I care to. I was not a very good developer in that arena, because you really need domain knowledge to be truly effective in some fields.
- asdfman123 7y ago
  I majored in geophysics and it doesn't really help me as a software engineer in Houston (although I haven't tried to really exploit it for the most money, because I felt guilty about working in oil and left). What's killer though is having a master's in a field and programming experience.

nmca 7y ago

There are certainly challenges, but "incorporating a world model" has been going well recently: "Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model" https://arxiv.org/abs/1911.08265 https://arxiv.org/abs/1911.08265

drongoking 7y ago
Not exactly. Incorporating a world or domain model usually means taking pre-existing declarative knowledge in some form (e.g. from a semantic net) and using it to aid learning. To quote the article, "Model-based reinforcement learning aims to address this issue by first learning a model of the environment’s dynamics, and then planning with respect to the learned model." So they've sped up learning from examples by learning a model first, but it's still learning from examples.
- nmca 7y ago
  That seems a fairly specific meaning of the term that may not be in wider use, see eg: https://arxiv.org/abs/1803.10122 https://arxiv.org/abs/1803.10122

dxbydt 7y ago

> large portion of CS students nowadays choose to focus on Machine Learning, many brilliant CS students decide to get a PhD in ML These claims are massively exaggerated. Students don't "choose", "decide" etc. In the US, lets be generous & say there are 50 top universities with good CS/ML depts. After students beg plead & submit their GRE scores & recos & transcripts etc, each dept chooses on average 20 students for incoming PhD cohort. Of them easily 20% will wash out for sure. So atmost 800 top students graduating with a PhD in ML each year. Let me caveat by saying there are nowhere close to 50 top univs offering ML PhDs or having an intake of 20 per dept. So overall, the number is probably close to 200-300 students for the whole USA, not 800. So all this handwringing for what some 200 kids will do ? I don't care if they do a PhD in basketweaving, in the grand scheme of things, 200 is not even a drop in bucket.

IC4RUS 7y ago
I agree that it might not be a large portion of all CS students, but it does appear that many of the incoming CS PhDs have chosen to go into ML (of my class, maybe 1/3), which serious distracts from interest in other subfields. It's actually a joke in my department that all of the new students are pursuing ML, while almost no one is pursuing theoretical CS or other less popular subfields. I'm also not sure about why the exact number matters - 200 kids matters a lot when future professors are drawn from the pool of students who have successfully completed a PhD.
- randcraw 7y ago
  I think the problem applies to non-PhDs students as well. I'm seeing a lot of interest from recent non-PhD grads in subjects other than CS wanting to steer their career toward data mining. My concern is that this re-focusing will probably lead them down paths that may leave them undistinguished relative to peers who stay within the more stable but less shiny domains where they're better prepared to succeed. I saw the same thing happen in the late 1990s as everybody and his dog got into web design while it was hot. Few of those folks are doing that now, nor did that skill translate well into other roles since the skill set isn't fundamental to other careers. I'm not sure ML is any different, especially deep learning, since few companies have anywhere near the necessary amount of data to successfully play that game and win.

buboard 7y ago

Cpus have hit their limits and not growing. Software has stagnated and recycles concepts from 50 years ago, and not growing. What's left to study? Either world-scale clouds or the growing field of NNs. Out of the two i 'd pick the 2nd cause it's either more fun or more unpredictable. Students are making a rational choice

IC4RUS 7y ago
Even if those fields completely stagnated, there's great progress to be made in high performance computing, security, bioinformatics, and human computer interaction (just to name a few).

thewarrior 7y ago

It may not be ideal for fundamental research but demand for ML practitioners is only likely to increase over time. ML is pure alchemy for a business operating at scale. It’s as if a coal plant could turn its waste emissions into profit. You have this “data exhaust” from all the activity happening within your system that can be used to optimize your system Atleast a few percentage points beyond what is other wise possible. A team of 5 ML engineers can improve an ad targeting system by 5 % and if the company is google that’s billions. ML creates feedback loops of improvements in product that improve usage that lead to more data which further strengthens the moat a business has. It totally makes sense to jump on this train. It won’t solve AI but will make a lot of people wealthy optimizing systems.

YeGoblynQueenne 7y ago
>> It may not be ideal for fundamental research but demand for ML practitioners is only likely to increase over time. If research stagnates, application willl also stagnate. For the industry money to keep pouring in there has to be growth and for there to be growth there has to be progress- scientific progress. So if something is "not ideal for fundamental research" it is also "not ideal" for business.
heavenlyblue 7y ago
>> A team of 5 ML engineers can improve an ad targeting system by 5 % and if the company is google that’s billions. Not going to pick up on your arbitrary 5% constant here, but please elaborate how these ML engineers are any different from anyone with a quantitative background?
- bumby 7y ago
  I think that's the key here. It's not ML magic that's driving success, it data literacy. Steve Levitt (of Freakanomics fame) said his advice to students would be to ensure they have base knowledge in understanding data analytics regardless of their domain. ML is just the sexy subset that gets all the attention
  - 7y ago

pradn 7y ago

It's also a problem that students are focusing so much on neural-network-based approaches, neglecting other techniques. There's what, like, 10 places in the world where you'll have enough compute to be able to train cutting-edge neural networks? Real world problems are just as well solved with other techniques like XG-Boost, which wins pretty often in Kaggle, for example.

hnaccy 7y ago
>There's what, like, 10 places in the world where you'll have enough compute to be able to train cutting-edge neural networks You can just use a smaller amount of compute with transfer learning style stuff. I can probably fine-tune a transformer model with my pocket money and beat any NLP solution from two years ago.

cdavid 7y ago

Most of the ML useful in industry is getting commoditized at a fast pace, and not enough people understand that. The focus on modeling for applying ML in a business environment is as incongruous as people believing programming is graph algo and compiler techniques. It sometimes is, but rarely. It is much easier to learn the basics of ML if you have good software engineering skills than the opposite in my experience. The one thing that needs time learning is experimental design and quantitative analysis, and this is rarely taught well at university before PhD.

1_over_n 7y ago

This narriative has also taken over the VC world.