5 ms·

> Are there any cases where ML performed decently but then hit a ceiling while expert systems kept improving? Yes, this describes entire history of AI includin

> Are there any cases where ML performed decently but then hit a ceiling while expert systems kept improving? Yes, this describes entire history of AI including several boom-bust cycles. In particular the 80's come to mind. Yes the practitioners think that there's no technical barriers stopping them from eating the world, but that's exactly what people thought about other so-called revolutionary advances. Although to be pedantic, "expert systems" is the technology behind AI boom of the 80's. At the time people were saying expert systems can't be as good as existing algorithms (including what we would now call "machine learning" techniques), then suddenly the expert systems were better and there was rampant speculation real AI was around the corner. Then they plateaued. We appear to be at the tail end of the maximum hype part of the boom-bust cycle. Thinking that the rapid gains being made by the current deep learning approaches will soon hit a wall is a reasonable outside-view prediction to make: nearly every time we've had a similarly transformative technology in the AI space and elsewhere, hitting the wall is exactly what happened. The onus would be on practitioners to show that this time really is different.

sdenton4 6y ago
I think the disconnect this time around is in productionization. We're getting breakthroughs in a wide range of problems, and translating those gains in the problem space into 'real' stable, practical solutions we can use in the world is the remaining gap, and often takes years of additional effort. It's still really expensive to launch this stuff, and often requires domain expertise that the ML research team doesn't have. We're seeing a lot of this pattern: ML Researcher shows up, says 'hey gimme your hardest problem in a nice parseable format' and then knocks a solution out of the park. The ML researcher then goes to the next field of study, leaving (say) the doctors or whatever to try to bridge the gap between the nice competition data and actual medical records. It also turns out that there's a host of closely related but different problems that ALSO need to be solved for the competition problem to really be useful. I don't think this means that the ML has failed, though; it's probably similar to the situation for accounting software circa 1980: everything was on paper, so using a computerized system was more trouble than it was worth. But today the situation in accounting has completely flipped. Apply N+1 years of consistent effort improving data ecosystems, and the ML might be a lot easier to use on generic real world problems.
- garmaine 6y ago
  Next time you fly through a busy airport, think about the system which assigns planes to gates in realtime based on a large number of variable factors in order to maximize utilization and minimize waits. This is an expert system design in the 80's and which allowed a huge increase in the number of planes handled per day at the busiest airports. Or when you drive your car, think about the lights-out factory that built-it, using robotics technologies developed in the 80's and 90's, and the freeways which largely operate without choke points again due to expert system models used by city planners. These advances were just as revolutionary before, and people were just as excited about AI technologies eating the world. Still, it largely didn't happen. To continue the example of robotics, we don't have an equivalent of the Jetson's home robot Rosey. We can make a robot assemble a $50,000 car, but we can't get it to fold the laundry. These rapid successes you see aren't literally "any problem from any field" -- it's specific problems chosen specifically for their likely ease in solving using current methods. DeepMind didn't decide to take on protein folding at random; they looked around and picked a problem that they thought they could solve. Don't expect them to have as much success on every problem they put their minds to. No, machine learning is not trivially solving the hardest problems in every field. Not even close. In biomedicine, for example, protein folding is probably one of the easiest challenges. It's a hard problem, yes, but it's self-contained: given an amino acid sequence, predict the structure. Unlike, say, predicting the metabolism of a drug applied to a living system, which requires understanding an extremely dense network of existing metabolic pathways and their interdependencies on local cell function. There's no magic ML pixie dust that can make that hard problem go away.
  - sdenton4 6y ago
    Well, we can agree that world peace is off the table! Beyond that, let's notice that expert systems did indeed change how airports and freeways work: They improved the areas where they solved problems. Deployment happened. What we're seeing now is new classes of previously unsolvable problems falling. Deployment in medicine is known to be particularly hard, but not impossible. My read on the situation is that there have been a number of ML applications in the current round that have been kinda-successful 'in vitro' and failed in deployment. That doesn't mean that all deployments will fail. Furthermore... Neil Lawrence points out that in most cases we change the world to fit new technologies. For example, mechanized tomato pickers suck, so we develop a more machine-resistant tomato. Cars break easily on dirt roads, so we pave half the planet. ML/AI somehow flips people's expectations of how technology works, and expect the algorithms to adapt to the world. This is almost certainly wrong. "it's specific problems chosen specifically for their likely ease in solving using current methods. DeepMind didn't decide to take on protein folding at random; they looked around and picked a problem that they thought they could solve." I'm actually not sure this is at all true. Protein folding is a long-standing grand challenge on which no current methods were working. My guess is that it was initially chosen for potential impact, and chased with more resources after some initial success.
deeviant 6y ago
> We appear to be at the tail end of the maximum hype part of the boom-bust cycle. Thinking that the rapid gains being made by the current deep learning approaches will soon hit a wall is a reasonable outside-view prediction to make: nearly every time we've had a similarly transformative technology in the AI space and elsewhere, hitting the wall is exactly what happened. The onus would be on practitioners to show that this time really is different. What a take. Neural networks just took a huge bite out of protein folding and your take is: This just in, the Deep Learning boom is about to go bust! Asinine.
- garmaine 6y ago
  It's not asinine to have realistic expectations and not give in to hyperbolic claims.

sdenton4 6y ago

I think the disconnect this time around is in productionization. We're getting breakthroughs in a wide range of problems, and translating those gains in the problem space into 'real' stable, practical solutions we can use in the world is the remaining gap, and often takes years of additional effort. It's still really expensive to launch this stuff, and often requires domain expertise that the ML research team doesn't have. We're seeing a lot of this pattern: ML Researcher shows up, says 'hey gimme your hardest problem in a nice parseable format' and then knocks a solution out of the park. The ML researcher then goes to the next field of study, leaving (say) the doctors or whatever to try to bridge the gap between the nice competition data and actual medical records. It also turns out that there's a host of closely related but different problems that ALSO need to be solved for the competition problem to really be useful. I don't think this means that the ML has failed, though; it's probably similar to the situation for accounting software circa 1980: everything was on paper, so using a computerized system was more trouble than it was worth. But today the situation in accounting has completely flipped. Apply N+1 years of consistent effort improving data ecosystems, and the ML might be a lot easier to use on generic real world problems.

garmaine 6y ago
Next time you fly through a busy airport, think about the system which assigns planes to gates in realtime based on a large number of variable factors in order to maximize utilization and minimize waits. This is an expert system design in the 80's and which allowed a huge increase in the number of planes handled per day at the busiest airports. Or when you drive your car, think about the lights-out factory that built-it, using robotics technologies developed in the 80's and 90's, and the freeways which largely operate without choke points again due to expert system models used by city planners. These advances were just as revolutionary before, and people were just as excited about AI technologies eating the world. Still, it largely didn't happen. To continue the example of robotics, we don't have an equivalent of the Jetson's home robot Rosey. We can make a robot assemble a $50,000 car, but we can't get it to fold the laundry. These rapid successes you see aren't literally "any problem from any field" -- it's specific problems chosen specifically for their likely ease in solving using current methods. DeepMind didn't decide to take on protein folding at random; they looked around and picked a problem that they thought they could solve. Don't expect them to have as much success on every problem they put their minds to. No, machine learning is not trivially solving the hardest problems in every field. Not even close. In biomedicine, for example, protein folding is probably one of the easiest challenges. It's a hard problem, yes, but it's self-contained: given an amino acid sequence, predict the structure. Unlike, say, predicting the metabolism of a drug applied to a living system, which requires understanding an extremely dense network of existing metabolic pathways and their interdependencies on local cell function. There's no magic ML pixie dust that can make that hard problem go away.
- sdenton4 6y ago
  Well, we can agree that world peace is off the table! Beyond that, let's notice that expert systems did indeed change how airports and freeways work: They improved the areas where they solved problems. Deployment happened. What we're seeing now is new classes of previously unsolvable problems falling. Deployment in medicine is known to be particularly hard, but not impossible. My read on the situation is that there have been a number of ML applications in the current round that have been kinda-successful 'in vitro' and failed in deployment. That doesn't mean that all deployments will fail. Furthermore... Neil Lawrence points out that in most cases we change the world to fit new technologies. For example, mechanized tomato pickers suck, so we develop a more machine-resistant tomato. Cars break easily on dirt roads, so we pave half the planet. ML/AI somehow flips people's expectations of how technology works, and expect the algorithms to adapt to the world. This is almost certainly wrong. "it's specific problems chosen specifically for their likely ease in solving using current methods. DeepMind didn't decide to take on protein folding at random; they looked around and picked a problem that they thought they could solve." I'm actually not sure this is at all true. Protein folding is a long-standing grand challenge on which no current methods were working. My guess is that it was initially chosen for potential impact, and chased with more resources after some initial success.

deeviant 6y ago

> We appear to be at the tail end of the maximum hype part of the boom-bust cycle. Thinking that the rapid gains being made by the current deep learning approaches will soon hit a wall is a reasonable outside-view prediction to make: nearly every time we've had a similarly transformative technology in the AI space and elsewhere, hitting the wall is exactly what happened. The onus would be on practitioners to show that this time really is different. What a take. Neural networks just took a huge bite out of protein folding and your take is: This just in, the Deep Learning boom is about to go bust! Asinine.

garmaine 6y ago
It's not asinine to have realistic expectations and not give in to hyperbolic claims.