5 ms·

> but many were skeptical as to the longevity of the performance DeepMind was able to achieve For a non-biologist, on what is this skepticism based? Just pure

> but many were skeptical as to the longevity of the performance DeepMind was able to achieve For a non-biologist, on what is this skepticism based? Just purely based on following ML news it looks like the trend for ML solutions has been that they've overtaken expert-systems once they've gained a solid foodhold in a field. Maybe this is some perception bias. Are there any cases where ML performed decently but then hit a ceiling while expert systems kept improving?

whimsicalism 6y ago
ML is a super overloaded term. There are definitely cases where machine learned statistical solutions do not perform as well as the systems tuned by the experts, but if you can define the task well and get the data for a deep solution, usually those will overtake.
- penagwin 6y ago
  This. I believe technically just linear regression could be considered "machine learning".
  - misnome 6y ago
    I've seen people at bio conferences actively calling linear regression machine learning.
    - diab0lic 6y ago
      This is likely because linear regression meets most widely accepted definitions of machine learning. [0][1] It is simple and very effective when learning in linear space. [0] https://en.wikipedia.org/wiki/Machine_learning https://en.wikipedia.org/wiki/Machine_learning [1] https://www.cs.cmu.edu/~tom/mlbook.html https://www.cs.cmu.edu/~tom/mlbook.html
    - b3kart 6y ago
      Sorry, I don’t get it. Are you saying fitting a linear regression model to data and making predictions somehow isn’t machine learning? I am confused.
ramraj07 6y ago
It's because for many researchers ML is just to take a standard keras or scikitlearn model shove their data in and get some table or number out, and see if that solves their problem. If that's your only ML experience then I suppose this is how sceptical you'd be of ML in general. It looks like DeepMind invented a completely new method for this round that's not just an extension of their previous work, showing how much you can gain if you don't shoebox yourself into just trying to improve existing methods. That all the scientists were highly skeptical about the scope of ML (and these are computer scientists to begin with mind you) just shows how little they knew of what they did know of what a computer or a program can possibly do, which is a bit appalling to be honest.
- timr 6y ago
  "It looks like DeepMind invented a completely new method for this round that's not just an extension of their previous work, showing how much you can gain if you don't shoebox yourself into just trying to improve existing methods. That all the scientists were highly skeptical about the scope of ML (and these are computer scientists to begin with mind you) just shows how little they knew of what they did know of what a computer or a program can possibly do, which is a bit appalling to be honest." My PhD (now over a decade ago...yikes) was in applying much simpler ML methods to these kinds of problems (I started in protein folding, finished in protein / nucleic acid recognition, but my real interest was always protein design). Even back then, it was clear that ML methods had a lot more potential for structural biology (pun unintended) than for which they were being given credit. But it was hard to get interest from a research community that cared little about non-physical solutions. No matter how well you did, people would dismiss it as a "black box solution", and that pretty much limited your impact. Some of this is understandable: even today, it's not at all clear that a custom-built ML model for protein folding is of much use to anyone -- particularly a model that doesn't consider all of the atoms in the protein. The traditional justification for research in this area is that if you could develop a sufficiently general model of protein physics, it would also allow you to do all sorts of other stuff that is much more interesting: rational protein design, drug binding, etc. The alphafold model is not really useful for any of this, so in a way, it's kind of like the weinermobile of science: cool and impressive when done well ("hey! a giant hot dog on wheels!"), but not really useful outside of the niche for which it was designed. So it's hard to blame researchers in this field -- who generally have to chase funding and justify their existence -- from pursuing the application of deep learning to this one, narrow problem domain. Obviously there will now be a wave of follow-on research, and it's impossible to know what methods this will spawn. Maybe this will revolutionize computational structural biology, maybe not. But I think it's a little unfair to demonize the entire field. Protein folding just traditionally hasn't been a very useful or interesting area, and like all "pure science", it leads to a lot of small-stakes, tribal thinking amongst the few players who can afford to compete. This is right out of Thomas Kuhn: a newcomer sweeps into a field, glances at the work of the past, then bashes it over the head, dismissively.
garmaine 6y ago
> Are there any cases where ML performed decently but then hit a ceiling while expert systems kept improving? Yes, this describes entire history of AI including several boom-bust cycles. In particular the 80's come to mind. Yes the practitioners think that there's no technical barriers stopping them from eating the world, but that's exactly what people thought about other so-called revolutionary advances. Although to be pedantic, "expert systems" is the technology behind AI boom of the 80's. At the time people were saying expert systems can't be as good as existing algorithms (including what we would now call "machine learning" techniques), then suddenly the expert systems were better and there was rampant speculation real AI was around the corner. Then they plateaued. We appear to be at the tail end of the maximum hype part of the boom-bust cycle. Thinking that the rapid gains being made by the current deep learning approaches will soon hit a wall is a reasonable outside-view prediction to make: nearly every time we've had a similarly transformative technology in the AI space and elsewhere, hitting the wall is exactly what happened. The onus would be on practitioners to show that this time really is different.
- sdenton4 6y ago
  I think the disconnect this time around is in productionization. We're getting breakthroughs in a wide range of problems, and translating those gains in the problem space into 'real' stable, practical solutions we can use in the world is the remaining gap, and often takes years of additional effort. It's still really expensive to launch this stuff, and often requires domain expertise that the ML research team doesn't have. We're seeing a lot of this pattern: ML Researcher shows up, says 'hey gimme your hardest problem in a nice parseable format' and then knocks a solution out of the park. The ML researcher then goes to the next field of study, leaving (say) the doctors or whatever to try to bridge the gap between the nice competition data and actual medical records. It also turns out that there's a host of closely related but different problems that ALSO need to be solved for the competition problem to really be useful. I don't think this means that the ML has failed, though; it's probably similar to the situation for accounting software circa 1980: everything was on paper, so using a computerized system was more trouble than it was worth. But today the situation in accounting has completely flipped. Apply N+1 years of consistent effort improving data ecosystems, and the ML might be a lot easier to use on generic real world problems.

whimsicalism 6y ago

ML is a super overloaded term. There are definitely cases where machine learned statistical solutions do not perform as well as the systems tuned by the experts, but if you can define the task well and get the data for a deep solution, usually those will overtake.

penagwin 6y ago
This. I believe technically just linear regression could be considered "machine learning".
- misnome 6y ago
  I've seen people at bio conferences actively calling linear regression machine learning.
  - diab0lic 6y ago
    This is likely because linear regression meets most widely accepted definitions of machine learning. [0][1] It is simple and very effective when learning in linear space. [0] https://en.wikipedia.org/wiki/Machine_learning https://en.wikipedia.org/wiki/Machine_learning [1] https://www.cs.cmu.edu/~tom/mlbook.html https://www.cs.cmu.edu/~tom/mlbook.html
  - b3kart 6y ago
    Sorry, I don’t get it. Are you saying fitting a linear regression model to data and making predictions somehow isn’t machine learning? I am confused.

ramraj07 6y ago

It's because for many researchers ML is just to take a standard keras or scikitlearn model shove their data in and get some table or number out, and see if that solves their problem. If that's your only ML experience then I suppose this is how sceptical you'd be of ML in general. It looks like DeepMind invented a completely new method for this round that's not just an extension of their previous work, showing how much you can gain if you don't shoebox yourself into just trying to improve existing methods. That all the scientists were highly skeptical about the scope of ML (and these are computer scientists to begin with mind you) just shows how little they knew of what they did know of what a computer or a program can possibly do, which is a bit appalling to be honest.

timr 6y ago
"It looks like DeepMind invented a completely new method for this round that's not just an extension of their previous work, showing how much you can gain if you don't shoebox yourself into just trying to improve existing methods. That all the scientists were highly skeptical about the scope of ML (and these are computer scientists to begin with mind you) just shows how little they knew of what they did know of what a computer or a program can possibly do, which is a bit appalling to be honest." My PhD (now over a decade ago...yikes) was in applying much simpler ML methods to these kinds of problems (I started in protein folding, finished in protein / nucleic acid recognition, but my real interest was always protein design). Even back then, it was clear that ML methods had a lot more potential for structural biology (pun unintended) than for which they were being given credit. But it was hard to get interest from a research community that cared little about non-physical solutions. No matter how well you did, people would dismiss it as a "black box solution", and that pretty much limited your impact. Some of this is understandable: even today, it's not at all clear that a custom-built ML model for protein folding is of much use to anyone -- particularly a model that doesn't consider all of the atoms in the protein. The traditional justification for research in this area is that if you could develop a sufficiently general model of protein physics, it would also allow you to do all sorts of other stuff that is much more interesting: rational protein design, drug binding, etc. The alphafold model is not really useful for any of this, so in a way, it's kind of like the weinermobile of science: cool and impressive when done well ("hey! a giant hot dog on wheels!"), but not really useful outside of the niche for which it was designed. So it's hard to blame researchers in this field -- who generally have to chase funding and justify their existence -- from pursuing the application of deep learning to this one, narrow problem domain. Obviously there will now be a wave of follow-on research, and it's impossible to know what methods this will spawn. Maybe this will revolutionize computational structural biology, maybe not. But I think it's a little unfair to demonize the entire field. Protein folding just traditionally hasn't been a very useful or interesting area, and like all "pure science", it leads to a lot of small-stakes, tribal thinking amongst the few players who can afford to compete. This is right out of Thomas Kuhn: a newcomer sweeps into a field, glances at the work of the past, then bashes it over the head, dismissively.

garmaine 6y ago

> Are there any cases where ML performed decently but then hit a ceiling while expert systems kept improving? Yes, this describes entire history of AI including several boom-bust cycles. In particular the 80's come to mind. Yes the practitioners think that there's no technical barriers stopping them from eating the world, but that's exactly what people thought about other so-called revolutionary advances. Although to be pedantic, "expert systems" is the technology behind AI boom of the 80's. At the time people were saying expert systems can't be as good as existing algorithms (including what we would now call "machine learning" techniques), then suddenly the expert systems were better and there was rampant speculation real AI was around the corner. Then they plateaued. We appear to be at the tail end of the maximum hype part of the boom-bust cycle. Thinking that the rapid gains being made by the current deep learning approaches will soon hit a wall is a reasonable outside-view prediction to make: nearly every time we've had a similarly transformative technology in the AI space and elsewhere, hitting the wall is exactly what happened. The onus would be on practitioners to show that this time really is different.

sdenton4 6y ago
I think the disconnect this time around is in productionization. We're getting breakthroughs in a wide range of problems, and translating those gains in the problem space into 'real' stable, practical solutions we can use in the world is the remaining gap, and often takes years of additional effort. It's still really expensive to launch this stuff, and often requires domain expertise that the ML research team doesn't have. We're seeing a lot of this pattern: ML Researcher shows up, says 'hey gimme your hardest problem in a nice parseable format' and then knocks a solution out of the park. The ML researcher then goes to the next field of study, leaving (say) the doctors or whatever to try to bridge the gap between the nice competition data and actual medical records. It also turns out that there's a host of closely related but different problems that ALSO need to be solved for the competition problem to really be useful. I don't think this means that the ML has failed, though; it's probably similar to the situation for accounting software circa 1980: everything was on paper, so using a computerized system was more trouble than it was worth. But today the situation in accounting has completely flipped. Apply N+1 years of consistent effort improving data ecosystems, and the ML might be a lot easier to use on generic real world problems.