4 ms·

See [0] for a critical review of (the popularization of) this paper. As usual, there’s something real here but it’s way overhyped because everyone involved bene

See [0] for a critical review of (the popularization of) this paper. As usual, there’s something real here but it’s way overhyped because everyone involved benefits from over-playing the role of deep learning here. [0:https://arxiv.org/abs/2112.04324 https://arxiv.org/abs/2112.04324] Edit: the critical review isn’t about the technical content of the paper per se, but rather the hyped news around it.

version_five 5y ago
There seems to be a lot of animosity towards deep learning, to the point that it comes off as jealousy - I mean read the other long thread in this discussion. I don't mean to say the criticisms in the article you linked are not valid, just that there are a lot of people ready to jump on any failure or exaggeration related to ML, giving it far more scrutiny than most research disciplines. It gets more money than most as well so maybe that's ok. And I know it gets overhyped in business contexts and media. But there are research preprints. That said, my observations about this story was that one one hand, everyone talks about how AI/ML will change everything and has lots of applications. But this article strikes my as clearly as a story of a solution in search of a problem, that had to go deep into a branch of theoretical math to find a place where ML could make a demonstrable impact. I know this application is not the only thing ML is good at (for context I own a machine learning consulting company and am bullish on the business applications of ML). But when you look at deepmind and the money that's gone into it, between the often cited "data centre cooling optimization" and this, they are not exactly having a massive impact, even as they do a lot of cool stuff.
- Jensson 5y ago
  > There seems to be a lot of animosity towards deep learning, to the point that it comes off as jealousy This happens every time people post research here. Any article about new battery tech will have people saying "What about all the other new battery techs that never went anywhere? This is just a prototype, it wont revolutionize the industry", and they are right. The difference is that when people say those things in a thread about ML, then AI optimists who thinks that these things are evidence that the singularity is near starts to argue that no, this time is truly different! And they do that every single time, AI optimists have believed that computers will take over any day now for 50 years. If those AI optimists didn't post so much then those threads wouldn't grow.
- vanusa 5y ago
  That "other thread" wasn't throwing hate at deep learning -- not by any stretch. Orr even talking about the failures (or even exaggerations) of ML -- compared to what ML as such claims to be able to do. ML/DL are both perfectly fine for what they do, and have of course made great strides in recent years. The critique was the tendency to continually pass these strides off as "AI" or something asymptotically close to it. Either on the part of sleazy companies pushing self-driving cars like they're just around the corner. Or supposedly reputable science journals that really ought to know better by now.
  - deleted 5y ago
    [deleted]
- psyklic 5y ago
  > [This article] had to go deep into a branch of theoretical math to find a place where ML could make a demonstrable impact I have no opinion on this paper, but that is generally how cutting-edge research works. First, choose an impactful unsolved problem you have the highest chance of solving. Then, use the new learnings to ascend higher mountains.
  - version_five 5y ago
    I think you miss my point. This is not the first thing AI/ML has been demonstrated on. There are lots of powerful theoretical techniques that have done really well on research problems. But ML had been challenged to actually break into large scale practical utility. I think there is an argument that a lot of the mainstream work is solutions looking for problems, and my point is they found a problem finally, and it's a pretty obscure one. I see this more as a victory for finding something ML is good at, rather than finding it can help with a common or even specific real world problem. I dont like to repeat myself, but I am an AI bull, I believe there is lots of real value that we haven't got out of it yet. But problems like this don't help build a case for how useful it's supposed to be, given the investment deep mind has received and the fact that they and others are searching hard for practical applications. I don't mean to criticize the research in any way,I'm talking about commercial utility.
    - psyklic 5y ago
      As far as assisting with pure math problems, it is in its infancy. Yet ML in general has had numerous practical successes that most people use daily (speech recognition, reverse image search, recommendation systems, etc.). Yesterday, Yann LeCun wrote on his Facebook page: "I didn't know that Facebook, and now Meta, would be built around AI. If you extirpated deep learning out of Meta today, the company would literally crumble." [Source: https://www.facebook.com/yann.lecun/posts/10158029031237143?comment_id=10158029040167143&reply_comment_id=10158029456812143 https://www.facebook.com/yann.lecun/posts/10158029031237143?...]
- hispaman 5y ago
  There is an element of jealousy/envy, I think. Many here want to be a Data Scientist, and many here want to work at Google (well, Deepmind), and seeing them make progress faster than they ever could (working on a big team with access to google-level resources) is really upsetting. The most upsetting thing is the idea that their names will probably be cemented into history, even if in a small way, and in the future people might discover them in textbooks or whatever. For a lot of people this makes the envy even worse. So the natural desire is downplay the discovery, because this does two things: one it makes you appear smarter than them, and two it might just prevent them from actually achieving something great, which would take away your envy. I can get where these people are coming from, but it's so silly to say "look at these idiots, they didn't even consider that their research has limitations!" and think it makes you look like you're genuinely concerned about the limits of the research and not just an envious person picking away at small mistakes because you can't do the hard work.
- bglazer 5y ago
  > "they are not exactly having a massive impact, even as they do a lot of cool stuff" AlphaFold has changed the course of structural biology research and some descendant of that technology will absolutely be used to design drugs that many people take. The academic research community would eventually have built models with the same performance as AlphaFold2, probably within the decade. Nonetheless, DeepMind was first, and way ahead of everyone else. I don't understand how the goalposts have moved so far that solving protein folding only counts as a cool trick or novelty.
  - klipt 5y ago
    > I don't understand how the goalposts have moved so far that solving protein folding only counts as a cool trick or novelty. It's the AI paradox. Before you solve a problem, like playing superhuman chess, people say "that's hard, you'd need real Artificial Intelligence to do that!" After you solve the problem, they see the solution and say "wait, that's just engineering, not real AI". Happens every time.
    - linspace 5y ago
      There are two interpretations: in one you put a metric and when you find people is cheating you move the metric. In the other interpretation you really don't have any idea what you want and when people give you concrete results you change your mind (this rings so close to any job...)
    - MayeulC 5y ago
      That's true of every np-complete (or similar class) problem: it's easy to verify that a solution work, but hard to find it in the space of possible solutions. Reportedly, quantum computers can help us with some problems of that kind.
- karmakaze 5y ago
  My world timeline basically split into before and after when AlphaGo won game 3. What we imagined was mechanically possible and not possible was greatly mistaken. I remember walking around in a light-headed daze, watching people go about their daily business not realizing that the world had changed. Everything since then is still great to read about and I do get surprised from time to time, especially about the diversity of applications rather than more strength in a given area. At the same time, I'm not shocked by any of it. I don't even think discovering basic material/cells or microorganisms on a nearby planet or moon would surprise me as much today as AlphaGo did. We don't have confirmation, but it's not entirely unexpected--'How' is that mystery rather than imagining its possibility. Discovering new areas of application however obscure or deep in any field is what I find more amazing than any practical immediate application of ML-after all the G in AGI is general so we should be hitting all the bases.
  - echelon 5y ago
    > I don't even think discovering basic material/cells or microorganisms on a nearby planet or moon would surprise me as much today as AlphaGo did. AlphaGo was a given. If we don't nuke ourselves, we'll eventually develop AGI (or boosted BCI cybernetics). It's priced into the timeline. Discovering alien biology tells us about our place in the universe. It gives us a sense of the distribution of life and what else could be out there. If what we find is similar to us, it points to some kind of panspermia or global convergent evolution. Maybe other life will be shockingly different with wildly new means of representing information. Nothing like our nucleic acids. Who knows? I put my bets on us developing AGI before meeting our cosmic neighbors, sadly. The Fermi paradox is strong.
    - karmakaze 5y ago
      Before AlphaGo's achievement, it was still expected to occur, but the expectation was in about 20 years with much more developed tech. The shock and surprise was that using larger scale of combinations of current tech could achieve the feat. The qualitative change in worldview is that 'dumb' machines can do intelligent things. AlphaZero was a tour-de-force of how dumbness can excel. In hindsight we can say that these things happening in our lifetime with tech of the day isn't surprising, but if following closely as-it-happened, it was a big surprise to everyone including the AlphaGo team. I don't think anyone knew if there was any limit where efficiency/effectiveness of the systems being built would top out in the complexity it could tackle.
- d0mine 5y ago
  The comment reminds me gpt-generated content: some plausible individual phrases but no sense as a whole (the beginning contradicts the end)
- cyber_kinetist 5y ago
  My perspective is that, ML/DL is just another tool in the shed, with its own advantages and disadvantages. It excels when you have massive amounts of data to begin with, but fails catastrophically otherwise. And encoding domain specific knowledge about your data is the most important trick of creating good models and loss functions, so deep learning won't "automatically solve your problem" unless you're doing something cookie-cutter like image classification. The verdict is that people in every scientific/engineering field should learn a bit of ML/DL (the underlying theory in it is not that hard), think a bit about the benefits and costs of it, tuck it in their toolbelt as one of their various tools, and move on. There's no harm in learning about the new trendy stuff, but keep in mind that there's no point in banging even the most high-quality hammer at a screw.
  - dave_sullivan 5y ago
    I really disagree. > It excels when you have massive amounts of data to begin with, but fails catastrophically otherwise. Then what is zero shot learning and does this not represent a major advance? Sure, you need lots of data, but then it learns things not even in the distribution. That's pretty cool. > And encoding domain specific knowledge about your data is the most important trick of creating good models and loss functions This is true of all machine learning in general, but I don't think it's more true of DL specifically. > so deep learning won't "automatically solve your problem" unless you're doing something cookie-cutter like image classification. Remember a few years ago when image classification was hard? Let's take a moment to appreciate how many things deep learning made easier in the last decade. Now it automatically solves it, with more and more models pretrained for tasks released on huggingfaces every day. > tuck it in their toolbelt as one of their various tools, and move on. Disagree, if I'm training any classifier, I'm starting that work in pytorch (after data wrangling of course). Deep learning changed the entire field. Learn it. Use it.

version_five 5y ago

There seems to be a lot of animosity towards deep learning, to the point that it comes off as jealousy - I mean read the other long thread in this discussion. I don't mean to say the criticisms in the article you linked are not valid, just that there are a lot of people ready to jump on any failure or exaggeration related to ML, giving it far more scrutiny than most research disciplines. It gets more money than most as well so maybe that's ok. And I know it gets overhyped in business contexts and media. But there are research preprints. That said, my observations about this story was that one one hand, everyone talks about how AI/ML will change everything and has lots of applications. But this article strikes my as clearly as a story of a solution in search of a problem, that had to go deep into a branch of theoretical math to find a place where ML could make a demonstrable impact. I know this application is not the only thing ML is good at (for context I own a machine learning consulting company and am bullish on the business applications of ML). But when you look at deepmind and the money that's gone into it, between the often cited "data centre cooling optimization" and this, they are not exactly having a massive impact, even as they do a lot of cool stuff.

Jensson 5y ago
> There seems to be a lot of animosity towards deep learning, to the point that it comes off as jealousy This happens every time people post research here. Any article about new battery tech will have people saying "What about all the other new battery techs that never went anywhere? This is just a prototype, it wont revolutionize the industry", and they are right. The difference is that when people say those things in a thread about ML, then AI optimists who thinks that these things are evidence that the singularity is near starts to argue that no, this time is truly different! And they do that every single time, AI optimists have believed that computers will take over any day now for 50 years. If those AI optimists didn't post so much then those threads wouldn't grow.
vanusa 5y ago
That "other thread" wasn't throwing hate at deep learning -- not by any stretch. Orr even talking about the failures (or even exaggerations) of ML -- compared to what ML as such claims to be able to do. ML/DL are both perfectly fine for what they do, and have of course made great strides in recent years. The critique was the tendency to continually pass these strides off as "AI" or something asymptotically close to it. Either on the part of sleazy companies pushing self-driving cars like they're just around the corner. Or supposedly reputable science journals that really ought to know better by now.
- deleted 5y ago
  [deleted]
psyklic 5y ago
> [This article] had to go deep into a branch of theoretical math to find a place where ML could make a demonstrable impact I have no opinion on this paper, but that is generally how cutting-edge research works. First, choose an impactful unsolved problem you have the highest chance of solving. Then, use the new learnings to ascend higher mountains.
- version_five 5y ago
  I think you miss my point. This is not the first thing AI/ML has been demonstrated on. There are lots of powerful theoretical techniques that have done really well on research problems. But ML had been challenged to actually break into large scale practical utility. I think there is an argument that a lot of the mainstream work is solutions looking for problems, and my point is they found a problem finally, and it's a pretty obscure one. I see this more as a victory for finding something ML is good at, rather than finding it can help with a common or even specific real world problem. I dont like to repeat myself, but I am an AI bull, I believe there is lots of real value that we haven't got out of it yet. But problems like this don't help build a case for how useful it's supposed to be, given the investment deep mind has received and the fact that they and others are searching hard for practical applications. I don't mean to criticize the research in any way,I'm talking about commercial utility.
  - psyklic 5y ago
    As far as assisting with pure math problems, it is in its infancy. Yet ML in general has had numerous practical successes that most people use daily (speech recognition, reverse image search, recommendation systems, etc.). Yesterday, Yann LeCun wrote on his Facebook page: "I didn't know that Facebook, and now Meta, would be built around AI. If you extirpated deep learning out of Meta today, the company would literally crumble." [Source: https://www.facebook.com/yann.lecun/posts/10158029031237143?comment_id=10158029040167143&reply_comment_id=10158029456812143 https://www.facebook.com/yann.lecun/posts/10158029031237143?...]
hispaman 5y ago
There is an element of jealousy/envy, I think. Many here want to be a Data Scientist, and many here want to work at Google (well, Deepmind), and seeing them make progress faster than they ever could (working on a big team with access to google-level resources) is really upsetting. The most upsetting thing is the idea that their names will probably be cemented into history, even if in a small way, and in the future people might discover them in textbooks or whatever. For a lot of people this makes the envy even worse. So the natural desire is downplay the discovery, because this does two things: one it makes you appear smarter than them, and two it might just prevent them from actually achieving something great, which would take away your envy. I can get where these people are coming from, but it's so silly to say "look at these idiots, they didn't even consider that their research has limitations!" and think it makes you look like you're genuinely concerned about the limits of the research and not just an envious person picking away at small mistakes because you can't do the hard work.
bglazer 5y ago
> "they are not exactly having a massive impact, even as they do a lot of cool stuff" AlphaFold has changed the course of structural biology research and some descendant of that technology will absolutely be used to design drugs that many people take. The academic research community would eventually have built models with the same performance as AlphaFold2, probably within the decade. Nonetheless, DeepMind was first, and way ahead of everyone else. I don't understand how the goalposts have moved so far that solving protein folding only counts as a cool trick or novelty.
- klipt 5y ago
  > I don't understand how the goalposts have moved so far that solving protein folding only counts as a cool trick or novelty. It's the AI paradox. Before you solve a problem, like playing superhuman chess, people say "that's hard, you'd need real Artificial Intelligence to do that!" After you solve the problem, they see the solution and say "wait, that's just engineering, not real AI". Happens every time.
  - linspace 5y ago
    There are two interpretations: in one you put a metric and when you find people is cheating you move the metric. In the other interpretation you really don't have any idea what you want and when people give you concrete results you change your mind (this rings so close to any job...)
  - MayeulC 5y ago
    That's true of every np-complete (or similar class) problem: it's easy to verify that a solution work, but hard to find it in the space of possible solutions. Reportedly, quantum computers can help us with some problems of that kind.
karmakaze 5y ago
My world timeline basically split into before and after when AlphaGo won game 3. What we imagined was mechanically possible and not possible was greatly mistaken. I remember walking around in a light-headed daze, watching people go about their daily business not realizing that the world had changed. Everything since then is still great to read about and I do get surprised from time to time, especially about the diversity of applications rather than more strength in a given area. At the same time, I'm not shocked by any of it. I don't even think discovering basic material/cells or microorganisms on a nearby planet or moon would surprise me as much today as AlphaGo did. We don't have confirmation, but it's not entirely unexpected--'How' is that mystery rather than imagining its possibility. Discovering new areas of application however obscure or deep in any field is what I find more amazing than any practical immediate application of ML-after all the G in AGI is general so we should be hitting all the bases.
- echelon 5y ago
  > I don't even think discovering basic material/cells or microorganisms on a nearby planet or moon would surprise me as much today as AlphaGo did. AlphaGo was a given. If we don't nuke ourselves, we'll eventually develop AGI (or boosted BCI cybernetics). It's priced into the timeline. Discovering alien biology tells us about our place in the universe. It gives us a sense of the distribution of life and what else could be out there. If what we find is similar to us, it points to some kind of panspermia or global convergent evolution. Maybe other life will be shockingly different with wildly new means of representing information. Nothing like our nucleic acids. Who knows? I put my bets on us developing AGI before meeting our cosmic neighbors, sadly. The Fermi paradox is strong.
  - karmakaze 5y ago
    Before AlphaGo's achievement, it was still expected to occur, but the expectation was in about 20 years with much more developed tech. The shock and surprise was that using larger scale of combinations of current tech could achieve the feat. The qualitative change in worldview is that 'dumb' machines can do intelligent things. AlphaZero was a tour-de-force of how dumbness can excel. In hindsight we can say that these things happening in our lifetime with tech of the day isn't surprising, but if following closely as-it-happened, it was a big surprise to everyone including the AlphaGo team. I don't think anyone knew if there was any limit where efficiency/effectiveness of the systems being built would top out in the complexity it could tackle.
d0mine 5y ago
The comment reminds me gpt-generated content: some plausible individual phrases but no sense as a whole (the beginning contradicts the end)
cyber_kinetist 5y ago
My perspective is that, ML/DL is just another tool in the shed, with its own advantages and disadvantages. It excels when you have massive amounts of data to begin with, but fails catastrophically otherwise. And encoding domain specific knowledge about your data is the most important trick of creating good models and loss functions, so deep learning won't "automatically solve your problem" unless you're doing something cookie-cutter like image classification. The verdict is that people in every scientific/engineering field should learn a bit of ML/DL (the underlying theory in it is not that hard), think a bit about the benefits and costs of it, tuck it in their toolbelt as one of their various tools, and move on. There's no harm in learning about the new trendy stuff, but keep in mind that there's no point in banging even the most high-quality hammer at a screw.
- dave_sullivan 5y ago
  I really disagree. > It excels when you have massive amounts of data to begin with, but fails catastrophically otherwise. Then what is zero shot learning and does this not represent a major advance? Sure, you need lots of data, but then it learns things not even in the distribution. That's pretty cool. > And encoding domain specific knowledge about your data is the most important trick of creating good models and loss functions This is true of all machine learning in general, but I don't think it's more true of DL specifically. > so deep learning won't "automatically solve your problem" unless you're doing something cookie-cutter like image classification. Remember a few years ago when image classification was hard? Let's take a moment to appreciate how many things deep learning made easier in the last decade. Now it automatically solves it, with more and more models pretrained for tasks released on huggingfaces every day. > tuck it in their toolbelt as one of their various tools, and move on. Disagree, if I'm training any classifier, I'm starting that work in pytorch (after data wrangling of course). Deep learning changed the entire field. Learn it. Use it.