12 ms·

Deep reinforcement learning is a waste of time (2019)

95 points by headalgorithm 5y ago

vsskanth 5y ago
As of 2021, has anything improved ? I'm not sure if this was before or after alphazero
- sdenton4 5y ago
  The article is about the failure of reinforcement learning to make it out of games and into any kind of real world task, so I think alpha go doesn't really change the argument.
  - nuclearnice3 5y ago
    Maybe AlphaFold does? https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology https://deepmind.com/blog/article/alphafold-a-solution-to-a-... Notably it's still an entry in a common task framework contest, not a piece of software for the lab, as far as i know.
    - xiphias2 5y ago
      AlphaFold is from DeepMind, but it uses modern neural networks, but not reinforcement learning. DeepMind is not just simply wasting money, they are doing important AI research in other areas as well.
- throwawaygh 5y ago
  One thing that's changed is: > So now we have the top machine learning research institutes, DeepMind and OpenAI, still spending the majority of their time and resources on Deep RL DeepMind has diversified at least some since 2019, and I'm fairly confident that OpenAI is spending more resources on huge transformer models than on RL these days. Which is really the only thing that has changed, since even in 2019 there are at least a dozen world-class institutions doing AI/ML research aimed at addressing issues raised in this blog post (and others). The blog post is accurate about OpenAI/Deepmind c. 2019, but is wrong about the overall composition of research effort in the field c. 2019. Outside of two small and very new labs, most ML research wasn't focused on RL, and most RL research wasn't focused on DRL as a silver bullet. Sort of of the west coast SV version of only paying attention to work out of MIT and Stanford and therefore missing most of the interesting things happening in the world.
- xiphias2 5y ago
  Most of the improvements in real world have been by improving data representation for perception (improvements in transformers, self supervised learning), so so far the article seems right.
- timkam 5y ago
  I think when looking at least at large scale applications in the context of games (and not just super-expensive showcases), like in the Stockfish chess engine, we see that it's not primarily about depth, it's about architecture design. Reference: start here https://stockfishchess.org/blog/2021/stockfish-14/ https://stockfishchess.org/blog/2021/stockfish-14/ and go down the rabbit hole... In the broadly useful domain of recommender systems (which typically make use of some type of RL-like feedback loop, but can be implemented using simple clustering approaches), at least in 2019, neural network-based approaches didn't seem to fair too well, either: https://arxiv.org/pdf/1907.06902.pdf https://arxiv.org/pdf/1907.06902.pdf (arXiv pre-print, but this is an award-winning paper). Since then, it seems that researchers are moving away from getting deeper and deeper (the low-hanging fruit), and try to be more creative instead: new architectures, combining symbolic (logic-based) and sub-symbolic (ML-based) AI, etc.
roenxi 5y ago
There is that lovely quote floating around that change is slower than people expect in the short term but faster in the long term. It is far too early to write of plain ol' deep reinforcement learning as a failure. It hasn't yet been 5 years since AlphaGo really shocked the unwary, and that was rather cutting edge at in 2016.
- QuesnayJr 5y ago
  I first saw the quote in something by Robert Cringely. (The Internet claims that Cringely attributed it to Amara, but I couldn't find the original source. Maybe "Accidental Empires"?)
  - AlexCoventry 5y ago
    I saw a similar quote from Asimov or Heinlein or the like, in the introduction to a science-fiction anthology in the eighties.
    - stevesimmons 5y ago
      Maybe quoting Roy Amara, a 1960s Stanford Computer scientist. He said "we overestimate the impact of technology in the short-term and underestimate the effect in the long run" [0]. I dimly remember similar sentiments in Arthur C. Clarke's Profiles of the Future (1662) [1], where he talked about "Hazards of Prophecy", where predictions suffered from either "failure of imagination" (predicted too little change) or "failure of nerve" (could/should have foreseen far reaching change, but chickened out and wrote down a watered-down version). As ACC said it "The failure of nerve seems to be the more common; it occurs when even given all the relevant facts the would-be prophet cannot see that they point to an inescapable conclusion." [0] https://fs.blog/2019/05/gates-law/ https://fs.blog/2019/05/gates-law/ [1] https://fabiusmaximus.com/2017/12/26/arthur-c-clarke-about-predicting-technology/ https://fabiusmaximus.com/2017/12/26/arthur-c-clarke-about-p...
- beefman 5y ago
  I think short- and long-term predictions are wrong in different ways. Short-term predictions often fail to come true, leading to the diagnosis of slowness. Long-term predictions often fail to even come false, because the world has changed so much the prediction is no longer applicable. In these cases, change was faster than expected, but not in a way that settles the original point (often pertaining to progress or some other value judgement).
dannyz 5y ago
I find these kind of articles just perplexing. Research is incremental, tiny steps pushing the boundaries of knowledge. DeepMind has done things that were thought to be decades away using deep reinforcement learning. These research advancements may or may not end up being important for AGI in the future, but that's just what research is.
- freeone3000 5y ago
  It's refuting the premise that supervised RL becomes less supervised because you put the feedback in a handcrafted function and use a neural network. Deep RL in its current state should be grouped with Supervised RL, in other words (which is why I personally think that imitation learning is a great way forward, in contrast with the author). The issue is the amount of interactive tweaking and lack of a natural reward function that prevents DeepRL from being unsupervised.
  - rich_sasha 5y ago
    AlphaZero is not supervised, in the sense that it learned from known correct actions (earlier versions of AlphGo did learn from online games). So although it needed human supervision, sure, it didn’t need us to provide correct answers. The authors point does somewhat stand that you don’t have the problem of reward engineering in board games so they are a dead end from that point of view - they skirt around the core problem instead of tackling it.
    - freeone3000 5y ago
      AlphaZero only works on video games. If you remove its ability to judge progress by game score, which is a reward function (though not the one used, due to delayed reward issues), then it's not capable of finding its feedback. It only works in constructed environments where the environment provides the reward function implicitly. Maybe we can video-game-ify laundry folding sufficiently? I'm doubtful.
- shadowlight 5y ago
  I don't agree with this article but it is not perplexing at all. Dead ends exist. The universe is highly, highly limited and everything eventually has a dead end. The question is, are we there yet? For certain things yes, for other things no. But to assume there is never a dead end and that everything can be overcome through incremental development and research is patently a false assumption. There are many examples of dead ends within research and development. Thus in short his proposal is likely wrong, but it is not a perplexing proposal. Nor is his proposal guaranteed to be wrong and there is a possibility he may be right. For example Elon predicted self driving will be a finished problem in a year. Guess what? I actually sort of dislike this whole "perplexing" attitude that some people have. It's like yeah his opinion seems wrong or his opinion is not the norm, but there's no need to treat it as if it's "perplexing." It's like you observing animal behavior in a lab and your so "perplexed" on how someone can have a differing opinion. People can have differing opinions and sometimes these opinions can be right and overturn an existing paradigm. Instead of saying you find someone perplexing or strange, just say you disagree. It's more civil and it respects the underdogs of the past who fought against overwhelming odds to change entire schools of thought and bring our knowledge closer to answering the ultimate question. So perplexing how some people are so rude nowadays. See what I did there?
xvilka 5y ago
At this point it's clear that "AI" was overhyped (as usual), and we are heading into one more "winter". AGI is still probably centuries away.
- aikinai 5y ago
  Is that clear? I see all sorts of new and useful implementations of AI coming out everyday.
- jstx1 5y ago
  You’re implying that it’s either AGI or nothing and I can’t agree. There are plenty of applications and use cases for ML out there. Should that be called “AI”? I don’t care too much personally. Is it overhyped? Sure. But there is enough useful stuff there that I don’t expect another winter or AGI any time soon.
  - throwawaygh 5y ago
    I think AI is over-hyped now in exactly the same way that the dot com bubble over-estimated the impact of the internet at exactly the same time that most people were under-estimating the impact of the internet. "In 2020 everyone will laugh at the idiots investing in the dot coms, and also three of the five largest companies in the world will be internet companies founded after 1990" would've seemed like a contradiction. Both sides were wrong in the late 90s. The techno-hippies talking about AGI are insane, and so were the techno-hippies who imagined the internet would connect the world and solve all our problems. The MBAs are selling bullshit as always and a lot of it'll go bust. But there's a lot of value in the past decade of advancements in ML and a lot more to come, with a lot of chaff in the wheat. The future will be shaped by ML more than the average person thinks, and those changes will be more prosaic than the hypsters would have us imagine, and those changes will be huge in ways that people kind of see now but take on an unexpected shape. I think in 2050 we'll be nowhere close to fully autonomous robotics, but also the combined forces of the USA military will in aggregate have a robotic land/air/sea fighting force that's 1) larger than most of the world's militaries and 2) capable of toppling a nation-state like Iraq or Afghanistan completely autonomously (ie, without any remote control). I think in 2050 we won't have robotaxis, but every warehouse and port in the developed world will be a nearly lights-out operation. I think in 2050 we'll still have wait staff and baristas but most non-sitdown food prep establishments will have at most one employee. I think in 2050 chatbots will still be useless for replacing callcenter work but also video games will have incredibly immersive social environments that are at least as stimulating as real social interactions. And so on.
thunderbird120 5y ago
There has been a lot of discussion recently in certain AI research communities recently about whether or not the entire idea of reinforcement learning is even necessary. Self supervised sequence models seem to represent the best path forward for general purpose problem solving agents because we can essentially just keep improving them by increasing parameter counts. There was a previous HN discussion about the idea here https://news.ycombinator.com/item?id=27659526 https://news.ycombinator.com/item?id=27659526
- thomasahle 5y ago
  Those models require a _lot_ of training data to exist. And I have never seen supervised sequence model achieve super human performance at anything, the way AlphaGo did.
MathYouF 5y ago
Makes me wonder if anyone has looked into using genetic algorithms combined with RL where the genetics determine the reward function. This seems to be how humans have evolved. Ultimately, all living animals are here based on only one reward function, the ability to have had an uninterrupted chain of reproduction. Our nervous system provides stimuli and our brains chemicals provide positive or negative rewards (pain or pleasure) that optimize us taking actions that result in having an uninterrupted chain of reproduction (it's why sex feels good and putting your hand on a stove feels bad). Presumably, both the reward function within our brain, as well as the signal it interprets (nervous system) evolved to find a more optimal combination of inputs and reward scalars for each input to maximize for this singular goal (reproduction). Maybe we need to frame RL goals in much more simple terms, and allow genetic algorithms to evolve their own inputs and reward functions on their own. RL is one of my weakest fields of knowledge in the AI field, so I'm sure some of this has been tried before, I'm curious how much and what the results have been.
- polishdude20 5y ago
  I think the stove and sex examples are on the right track but these qualities are also what every animal experiences. Well... judging by the face of a dog when he's humping your leg, I'm sure it feels good for him. Anyways, I think there's another ingredient that's missing that we humans uniquely have. I think that ingredient is the fear of death. The knowledge that of all our intellect and powers as a human, we will inevitably die. Its better summed up by terror management theory I believe.
  - MathYouF 5y ago
    Dogs have achieved an unbroken chain of survival dating back as far as your ancestors have, so they've achieved the same survival goals as you. They've managed to do so without our intellectual abilities, which goes to prove that our goals of making RL algorithms become "smart" is malformed since high level complex and abstract reasoning skills apparently aren't a necessarily prerequisite to survival, at least in our earth environment.
    - 5y ago
bruce343434 5y ago
Note to author: TFA has a couple of spelling mistakes
max_ 5y ago
AI people don't have a formal/scientific structure of intelligence. Noam Chomsky thinks that deep learning is more of engineering than science [0] He says what we need is something like a Turing machine but for intelligence. [0]: https://youtu.be/ndwIZPBs8Y4 https://youtu.be/ndwIZPBs8Y4
- sdenton4 5y ago
  Why is Chomsky worth listening to on ML/AI? It's birds and airplanes: airplanes don't have feathers, but they solve different problems. Aerodynamics is a science, actual airplanes are engineered. But aerodynamics as a science evolved more slowly than actual flying airplanes; that's where we're at with ML.
  - max_ 5y ago
    I find his arguments valid by merit
api 5y ago
Reward function design and overfitting or “cheating” (optimizing to some incidental thing instead of the real problem) is a major reason genetic and evolutionary AI approaches never took off. I don’t think anyone ever figured out how to make reward function design easier or make any kind of unsupervised evolutionary learning work. I played with EC a lot in college and these systems were almost comically good at the “cheating” part of overfitting. I watched evolving programs do things like learn the scheduling behavior of the OS kernel (because the reward function was threaded) or the disk timing differences resulting from where different parts of the sample set were stored on the drive. They could guess the answer by inferring load time.
abeppu 5y ago
I wish the author had picked a title like "Single-task reinforcement learning is a waste" or "Reward function engineering is a waste". At the very end of his essay he mentions some directions: homeostasis, Friston's free-energy principle, and predictive processing. I agree that all of these are very interesting. A few steps less ambitious is recent work on reinforcement learning to reach desired outcomes without specifying reward functions. All of these seem to require more focus on learning the model of dynamics in the relevant domain (what happens when I take action A from state S?) and less on value / policy learning. Ok great, but why can't that happen in the context of "deep reinforcement learning"? For complex environments with partial observability, don't we probably want something like states represented in an embedding? As we improve our model, is there anything broken about the approach of optimizing a differentiable function? "There are likely better approaches to deep RL, and they include ..." seems like a better framing than "deep RL is a waste of time."
sdenton4 5y ago
The new hotness this month just says 'screw it; treat RL as a supervised sequence problem with causal transformers.' https://arxiv.org/abs/2106.01345 https://arxiv.org/abs/2106.01345 'We consider the following shift in paradigm: instead of training a policy through conventional RL algorithms like temporal difference (TD) learning [6], we will train transformer models on collected experience using a sequence modeling objective. This will allow us to bypass the need for bootstrapping for long term credit assignment – thereby avoiding one of the “deadly triad” [6] known to destabilize RL. It also avoids the need for discounting future rewards, as typically done in TD learning, which can induce undesirable short-sighted behaviors. Additionally, we can make use of existing transformer frameworks widely used in language and vision that are easy to scale, utilizing a large body of work studying stable training of transformer models.'
- mountainriver 5y ago
  Transformer all the things
  - sillysaurusx 5y ago
    Yuck. I hate how expensive transformers are. You can see it clearly in transformers vs gans; GANs can generate a frame in about 20ms, whereas it takes seconds or more to make a frame with transformers. It’s not even clear that it’s necessarily better quality. That said, I doubt we’ll be able to make an RL GAN, so maybe this is the best way. Though now I wonder how well an RL GAN might work…
    - sdenton4 5y ago
      Yeah, I work on on-device audio, so definitely agree on the expense problem. In audio we've now got a few different approaches that work really well for sequence modeling, and we're constantly finding cool ways to make inference run faster. Reframing reinforcement learning as sequence modeling /should/ make it possible to reuse (m)any of the approaches we use for audio, including GANs. Generative audio is nicely analogous to RL problems : There's complex state and interesting predictive distributions, which shift subtly over time, and you need to combine short-term good behavior (good individual samples) in a reasonable way to get good long-term behavior (matching melspectra).
PaulHoule 5y ago
Deep-learned convolutional nets work wonders for visual recognition. Visual recognition via "the old AI" looked impossible by 1980, but today it looks easy. (I see the visual segmentation models for self-driving cars from the "autonomous systems lab" in the next building over and think... It would be so easy to make something that honks for cyclists.) Text analysis, reinforcement learning, etc. seem to be areas where deep learning might very well reach a plateau. In the case of images the meaning is not changed by a random "hot pixel", but changing one letter in a sentence can reverse the meaning of a sentence, changing one piece position in a chess game is the difference between a win or loss. These "binary" situations aren't a good match with the assumptions of continuity, differentiablity and such that neural networks depend on.
- throwawaygh 5y ago
  > Text analysis, reinforcement learning, etc. seem to be areas where deep learning might very well reach a plateau... changing one piece position in a chess game is the difference between a win or loss. These "binary" situations aren't a good match with the assumptions of continuity, differentiablity and such that neural networks depend on. Deep learning can't do board games like chess and go?! Wait... are you using pretty darn subtle dry sarcasm to argue that deep learning won't reach a plateau?
  - PaulHoule 5y ago
    I think that markov chain monte carlo is pretty cool even with lightweight playouts. (pick a random move) The neural net by itself is a "half-baked" chess or go player, it needs the MCMC to be a strong player. (MCMC plus lightweight playouts can beat me at chess if not at go.) Same with text-analysis, code generation and such. If you can build a hybrid system where the neural net comes up with half-baked answers that can be corrected by a system which is capable of comprehending things like "well-formed" and "valid" then you could be cooking with gas. What I am seeing though is that people aren't "beginning with the end in mind" the way the Wright Brothers did with flying, rather they are throwing stuff at the wall and seeing what sticks.
vsskanth 5y ago
Has there been any work on "growth" based RL models ? Like how a human baby starts off with a small brain and little capability to move around, but slowly starts to roll over, gets head control, crawl and eventually walk, grasp and develop fine motor control as their brains and physical abilities grow together. The current method looks to me like starting from scratch with a fully capable human with a huge untrained brain instead of progressively expanding their actuators and control plane
- mountainriver 5y ago
  Yeah there is a field that does incremental learning and “lifelong learning” which employ these tactics. It’s successful on some domains
- procgen 5y ago
  I recommend Peter Hiesinger's 'The Self-Assembling Brain' if you're interested in a neurobiologist's perspective on this problem and its importance to AI.
Digitalis33 5y ago
I prefer to listen to people who actually put AI/ML models into production, instead of random programmer / AI dilettante types.
- cesaref 5y ago
  You'll only hear one side of the coin. I would also want to hear from people who understand AI/ML and have decided not to use it in an application domain. That's probably the most valuable info, knowing when not to use a tool.
  - wizzwizz4 5y ago
    I understand ML reasonably well, and have decided to use classical AI instead, because I want real-time performance on low-end hardware (and I also want to be able to predict and extend the operating parameters). It's a lot harder, though, because I'll have to understand the problems at a fundamental (mathematical) level better than I currently do, and all the time I can hear a little voice saying “a neural network could do this in half an hour of work and a week of training”.
  - deleted 5y ago
    [deleted]
- soVeryTired 5y ago
  But this is part of the author's point: where are all the production reinforcement learning models?
hervature 5y ago
To me, this entire piece reads like goal-post moving and lacks understanding of the field and was incredibly premature. > I am confused as to why they continue to spend so much time on building Deep RL systems that beat games with clearly defined rules and point systems. I think everyone gets it now, with enough time, money, and computers almost brute forcing every single possible action, Deep RL can beat almost any game. Right, before AlphaGo, superhuman Go was decades away. Then it became obvious and therefore easy. He brings up a laundry folding robot almost as if discrete games can be trivially applied to robotics. If you think about it for one second, Shogi, the game that AlphaZero solves with the largest action space has 10^5 actions. Consider a robot with 3 joints that can move in any dimension motors. Discretizing the continuous action space into 10 buckets yields an action space of 10^9. Something AlphaZero will never solve. In that sense, the entire analogy is a straw man and everyone in the field knows. This doesn't even touch the fact that 10 buckets would be useless for fine motor control. Since they published this, there have been great advancements in learning action embeddings for robotic tasks. With a couple of expert examples, an algorithm can learn to open a door effectively. That's not that far from folding laundry. Personally, my entire PhD thesis attempts to solves the exact deficiencies they raise. Simply put, the work goes on. Instead of saying that the field is useless, try to understand and contribute to it. At the very least, give technical reasons that it is a dead end path.
- lysozyme 5y ago
  That’s a really interesting way of looking at the difficulty of the problem that’s being solved. I’m curious, how do you arrive at the number 10^9?
  - hervature 5y ago
    You have 3 joints with 3 dimensions. So 9 variables you can control. If each variable has 10 choices, and you need to make a choice for each, that is 10^9 possible combinations.
    - whimsicalism 5y ago
      "3 joints" -> What are the 3 dimensions that joints move along? I naively would have thought one or two (ie. how much spin on one axis and how much spin on the other axis) just by naively looking at my elbow for a second. For an arm of fixed length, polar coordinates have 2 dof.
toisanji 5y ago
how do you think the article stands with a paper just recently publishing "Reward is enough" by Richard Sutton (father of Reinforcement Learning)
dang 5y ago
Discussed a bit at the time: Deep Reinforcement Learning is a waste of time - https://news.ycombinator.com/item?id=21916775 https://news.ycombinator.com/item?id=21916775 - Dec 2019 (3 comments)
phonebucket 5y ago
I have to disagree with a good chunk of this article. The article completely misses the main advantage (to me) of reinforcement learning: Reinforcement learning allows you to optimise on non-differentiable outcomes. I can't differentiate real life, but I want to optimise a process within real life. This feels tantalisingly close to AGI. If I can figure out a reward function, I can use reinforcement learning. Yes, this requires a reward function to be defined. Yes, this is a challenge to AGI. But to say that the big labs are not aware that this is a challenge to AGI is unfair. DeepMind is actively investigating open ended learning: https://deepmind.com/research/publications/open-ended-learning-symmetric-zero-sum-games https://deepmind.com/research/publications/open-ended-learni.... Just because the labs haven't tackled all the questions doesn't mean that they're not busy tackling difficult questions.
- soVeryTired 5y ago
  Can you clarify what you mean by 'non-differentiable outcomes'?
  - ArnoVW 5y ago
    My guess: a function that you can evaluate but that you can not differentiate. Differentiate being "know the slope". Differentiable functions are great because you can run gradient descent on them in a very optimized way. Example: if your objective is to have a very high value, search in the direction that has a mounting slope. Though maybe I'm missing something 'cos it seems to me you can run gradient descend on non-differentiable functions. It just requires more evaluations.
    - LeegleechN 5y ago
      Neural network models have millions of parameters, which means you are trying to optimize in spaces with millions of dimensions. If you can't use differentiation you get hit really hard with the curse of dimensionality.
    - jeeeb 5y ago
      > Though maybe I'm missing something 'cos it seems to me you can run gradient descend on non-differentiable functions Isn't that exactly the point though? If you don't have an analytical solution for the gradient of the loss (reward) wrt the parameters - yes - you could brute force a numerical solution but as the number of parameters grows that quickly becomes infeasible. Approaches such as RL and GA provide a more intelligent way to search the parameter space.
Zamaamiro 5y ago
It’s not useless; it just doesn’t get us closer to AGI. That doesn’t mean it’s useless.
dqpb 5y ago
Put your money where your mouth is. Train a homeostatic surprise minimization model to beat OpenAI’s PPO at Dota 2. The beautiful thing about benchmarks is that you can easily persuade everyone just by beating them.

95 points by headalgorithm 5y ago

vsskanth 5y ago

As of 2021, has anything improved ? I'm not sure if this was before or after alphazero

sdenton4 5y ago
The article is about the failure of reinforcement learning to make it out of games and into any kind of real world task, so I think alpha go doesn't really change the argument.
- nuclearnice3 5y ago
  Maybe AlphaFold does? https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology https://deepmind.com/blog/article/alphafold-a-solution-to-a-... Notably it's still an entry in a common task framework contest, not a piece of software for the lab, as far as i know.
  - xiphias2 5y ago
    AlphaFold is from DeepMind, but it uses modern neural networks, but not reinforcement learning. DeepMind is not just simply wasting money, they are doing important AI research in other areas as well.
throwawaygh 5y ago
One thing that's changed is: > So now we have the top machine learning research institutes, DeepMind and OpenAI, still spending the majority of their time and resources on Deep RL DeepMind has diversified at least some since 2019, and I'm fairly confident that OpenAI is spending more resources on huge transformer models than on RL these days. Which is really the only thing that has changed, since even in 2019 there are at least a dozen world-class institutions doing AI/ML research aimed at addressing issues raised in this blog post (and others). The blog post is accurate about OpenAI/Deepmind c. 2019, but is wrong about the overall composition of research effort in the field c. 2019. Outside of two small and very new labs, most ML research wasn't focused on RL, and most RL research wasn't focused on DRL as a silver bullet. Sort of of the west coast SV version of only paying attention to work out of MIT and Stanford and therefore missing most of the interesting things happening in the world.
xiphias2 5y ago
Most of the improvements in real world have been by improving data representation for perception (improvements in transformers, self supervised learning), so so far the article seems right.
timkam 5y ago
I think when looking at least at large scale applications in the context of games (and not just super-expensive showcases), like in the Stockfish chess engine, we see that it's not primarily about depth, it's about architecture design. Reference: start here https://stockfishchess.org/blog/2021/stockfish-14/ https://stockfishchess.org/blog/2021/stockfish-14/ and go down the rabbit hole... In the broadly useful domain of recommender systems (which typically make use of some type of RL-like feedback loop, but can be implemented using simple clustering approaches), at least in 2019, neural network-based approaches didn't seem to fair too well, either: https://arxiv.org/pdf/1907.06902.pdf https://arxiv.org/pdf/1907.06902.pdf (arXiv pre-print, but this is an award-winning paper). Since then, it seems that researchers are moving away from getting deeper and deeper (the low-hanging fruit), and try to be more creative instead: new architectures, combining symbolic (logic-based) and sub-symbolic (ML-based) AI, etc.

roenxi 5y ago

There is that lovely quote floating around that change is slower than people expect in the short term but faster in the long term. It is far too early to write of plain ol' deep reinforcement learning as a failure. It hasn't yet been 5 years since AlphaGo really shocked the unwary, and that was rather cutting edge at in 2016.

QuesnayJr 5y ago
I first saw the quote in something by Robert Cringely. (The Internet claims that Cringely attributed it to Amara, but I couldn't find the original source. Maybe "Accidental Empires"?)
- AlexCoventry 5y ago
  I saw a similar quote from Asimov or Heinlein or the like, in the introduction to a science-fiction anthology in the eighties.
  - stevesimmons 5y ago
    Maybe quoting Roy Amara, a 1960s Stanford Computer scientist. He said "we overestimate the impact of technology in the short-term and underestimate the effect in the long run" [0]. I dimly remember similar sentiments in Arthur C. Clarke's Profiles of the Future (1662) [1], where he talked about "Hazards of Prophecy", where predictions suffered from either "failure of imagination" (predicted too little change) or "failure of nerve" (could/should have foreseen far reaching change, but chickened out and wrote down a watered-down version). As ACC said it "The failure of nerve seems to be the more common; it occurs when even given all the relevant facts the would-be prophet cannot see that they point to an inescapable conclusion." [0] https://fs.blog/2019/05/gates-law/ https://fs.blog/2019/05/gates-law/ [1] https://fabiusmaximus.com/2017/12/26/arthur-c-clarke-about-predicting-technology/ https://fabiusmaximus.com/2017/12/26/arthur-c-clarke-about-p...
beefman 5y ago
I think short- and long-term predictions are wrong in different ways. Short-term predictions often fail to come true, leading to the diagnosis of slowness. Long-term predictions often fail to even come false, because the world has changed so much the prediction is no longer applicable. In these cases, change was faster than expected, but not in a way that settles the original point (often pertaining to progress or some other value judgement).

dannyz 5y ago

I find these kind of articles just perplexing. Research is incremental, tiny steps pushing the boundaries of knowledge. DeepMind has done things that were thought to be decades away using deep reinforcement learning. These research advancements may or may not end up being important for AGI in the future, but that's just what research is.

freeone3000 5y ago
It's refuting the premise that supervised RL becomes less supervised because you put the feedback in a handcrafted function and use a neural network. Deep RL in its current state should be grouped with Supervised RL, in other words (which is why I personally think that imitation learning is a great way forward, in contrast with the author). The issue is the amount of interactive tweaking and lack of a natural reward function that prevents DeepRL from being unsupervised.
- rich_sasha 5y ago
  AlphaZero is not supervised, in the sense that it learned from known correct actions (earlier versions of AlphGo did learn from online games). So although it needed human supervision, sure, it didn’t need us to provide correct answers. The authors point does somewhat stand that you don’t have the problem of reward engineering in board games so they are a dead end from that point of view - they skirt around the core problem instead of tackling it.
  - freeone3000 5y ago
    AlphaZero only works on video games. If you remove its ability to judge progress by game score, which is a reward function (though not the one used, due to delayed reward issues), then it's not capable of finding its feedback. It only works in constructed environments where the environment provides the reward function implicitly. Maybe we can video-game-ify laundry folding sufficiently? I'm doubtful.
shadowlight 5y ago
I don't agree with this article but it is not perplexing at all. Dead ends exist. The universe is highly, highly limited and everything eventually has a dead end. The question is, are we there yet? For certain things yes, for other things no. But to assume there is never a dead end and that everything can be overcome through incremental development and research is patently a false assumption. There are many examples of dead ends within research and development. Thus in short his proposal is likely wrong, but it is not a perplexing proposal. Nor is his proposal guaranteed to be wrong and there is a possibility he may be right. For example Elon predicted self driving will be a finished problem in a year. Guess what? I actually sort of dislike this whole "perplexing" attitude that some people have. It's like yeah his opinion seems wrong or his opinion is not the norm, but there's no need to treat it as if it's "perplexing." It's like you observing animal behavior in a lab and your so "perplexed" on how someone can have a differing opinion. People can have differing opinions and sometimes these opinions can be right and overturn an existing paradigm. Instead of saying you find someone perplexing or strange, just say you disagree. It's more civil and it respects the underdogs of the past who fought against overwhelming odds to change entire schools of thought and bring our knowledge closer to answering the ultimate question. So perplexing how some people are so rude nowadays. See what I did there?

xvilka 5y ago

At this point it's clear that "AI" was overhyped (as usual), and we are heading into one more "winter". AGI is still probably centuries away.

aikinai 5y ago
Is that clear? I see all sorts of new and useful implementations of AI coming out everyday.
jstx1 5y ago
You’re implying that it’s either AGI or nothing and I can’t agree. There are plenty of applications and use cases for ML out there. Should that be called “AI”? I don’t care too much personally. Is it overhyped? Sure. But there is enough useful stuff there that I don’t expect another winter or AGI any time soon.
- throwawaygh 5y ago
  I think AI is over-hyped now in exactly the same way that the dot com bubble over-estimated the impact of the internet at exactly the same time that most people were under-estimating the impact of the internet. "In 2020 everyone will laugh at the idiots investing in the dot coms, and also three of the five largest companies in the world will be internet companies founded after 1990" would've seemed like a contradiction. Both sides were wrong in the late 90s. The techno-hippies talking about AGI are insane, and so were the techno-hippies who imagined the internet would connect the world and solve all our problems. The MBAs are selling bullshit as always and a lot of it'll go bust. But there's a lot of value in the past decade of advancements in ML and a lot more to come, with a lot of chaff in the wheat. The future will be shaped by ML more than the average person thinks, and those changes will be more prosaic than the hypsters would have us imagine, and those changes will be huge in ways that people kind of see now but take on an unexpected shape. I think in 2050 we'll be nowhere close to fully autonomous robotics, but also the combined forces of the USA military will in aggregate have a robotic land/air/sea fighting force that's 1) larger than most of the world's militaries and 2) capable of toppling a nation-state like Iraq or Afghanistan completely autonomously (ie, without any remote control). I think in 2050 we won't have robotaxis, but every warehouse and port in the developed world will be a nearly lights-out operation. I think in 2050 we'll still have wait staff and baristas but most non-sitdown food prep establishments will have at most one employee. I think in 2050 chatbots will still be useless for replacing callcenter work but also video games will have incredibly immersive social environments that are at least as stimulating as real social interactions. And so on.

thunderbird120 5y ago

There has been a lot of discussion recently in certain AI research communities recently about whether or not the entire idea of reinforcement learning is even necessary. Self supervised sequence models seem to represent the best path forward for general purpose problem solving agents because we can essentially just keep improving them by increasing parameter counts. There was a previous HN discussion about the idea here https://news.ycombinator.com/item?id=27659526 https://news.ycombinator.com/item?id=27659526

thomasahle 5y ago
Those models require a _lot_ of training data to exist. And I have never seen supervised sequence model achieve super human performance at anything, the way AlphaGo did.

MathYouF 5y ago

Makes me wonder if anyone has looked into using genetic algorithms combined with RL where the genetics determine the reward function. This seems to be how humans have evolved. Ultimately, all living animals are here based on only one reward function, the ability to have had an uninterrupted chain of reproduction. Our nervous system provides stimuli and our brains chemicals provide positive or negative rewards (pain or pleasure) that optimize us taking actions that result in having an uninterrupted chain of reproduction (it's why sex feels good and putting your hand on a stove feels bad). Presumably, both the reward function within our brain, as well as the signal it interprets (nervous system) evolved to find a more optimal combination of inputs and reward scalars for each input to maximize for this singular goal (reproduction). Maybe we need to frame RL goals in much more simple terms, and allow genetic algorithms to evolve their own inputs and reward functions on their own. RL is one of my weakest fields of knowledge in the AI field, so I'm sure some of this has been tried before, I'm curious how much and what the results have been.

polishdude20 5y ago
I think the stove and sex examples are on the right track but these qualities are also what every animal experiences. Well... judging by the face of a dog when he's humping your leg, I'm sure it feels good for him. Anyways, I think there's another ingredient that's missing that we humans uniquely have. I think that ingredient is the fear of death. The knowledge that of all our intellect and powers as a human, we will inevitably die. Its better summed up by terror management theory I believe.
- MathYouF 5y ago
  Dogs have achieved an unbroken chain of survival dating back as far as your ancestors have, so they've achieved the same survival goals as you. They've managed to do so without our intellectual abilities, which goes to prove that our goals of making RL algorithms become "smart" is malformed since high level complex and abstract reasoning skills apparently aren't a necessarily prerequisite to survival, at least in our earth environment.
  - 5y ago

bruce343434 5y ago

Note to author: TFA has a couple of spelling mistakes

max_ 5y ago

AI people don't have a formal/scientific structure of intelligence. Noam Chomsky thinks that deep learning is more of engineering than science [0] He says what we need is something like a Turing machine but for intelligence. [0]: https://youtu.be/ndwIZPBs8Y4 https://youtu.be/ndwIZPBs8Y4

sdenton4 5y ago
Why is Chomsky worth listening to on ML/AI? It's birds and airplanes: airplanes don't have feathers, but they solve different problems. Aerodynamics is a science, actual airplanes are engineered. But aerodynamics as a science evolved more slowly than actual flying airplanes; that's where we're at with ML.
- max_ 5y ago
  I find his arguments valid by merit

api 5y ago

Reward function design and overfitting or “cheating” (optimizing to some incidental thing instead of the real problem) is a major reason genetic and evolutionary AI approaches never took off. I don’t think anyone ever figured out how to make reward function design easier or make any kind of unsupervised evolutionary learning work. I played with EC a lot in college and these systems were almost comically good at the “cheating” part of overfitting. I watched evolving programs do things like learn the scheduling behavior of the OS kernel (because the reward function was threaded) or the disk timing differences resulting from where different parts of the sample set were stored on the drive. They could guess the answer by inferring load time.

abeppu 5y ago

I wish the author had picked a title like "Single-task reinforcement learning is a waste" or "Reward function engineering is a waste". At the very end of his essay he mentions some directions: homeostasis, Friston's free-energy principle, and predictive processing. I agree that all of these are very interesting. A few steps less ambitious is recent work on reinforcement learning to reach desired outcomes without specifying reward functions. All of these seem to require more focus on learning the model of dynamics in the relevant domain (what happens when I take action A from state S?) and less on value / policy learning. Ok great, but why can't that happen in the context of "deep reinforcement learning"? For complex environments with partial observability, don't we probably want something like states represented in an embedding? As we improve our model, is there anything broken about the approach of optimizing a differentiable function? "There are likely better approaches to deep RL, and they include ..." seems like a better framing than "deep RL is a waste of time."

sdenton4 5y ago

The new hotness this month just says 'screw it; treat RL as a supervised sequence problem with causal transformers.' https://arxiv.org/abs/2106.01345 https://arxiv.org/abs/2106.01345 'We consider the following shift in paradigm: instead of training a policy through conventional RL algorithms like temporal difference (TD) learning [6], we will train transformer models on collected experience using a sequence modeling objective. This will allow us to bypass the need for bootstrapping for long term credit assignment – thereby avoiding one of the “deadly triad” [6] known to destabilize RL. It also avoids the need for discounting future rewards, as typically done in TD learning, which can induce undesirable short-sighted behaviors. Additionally, we can make use of existing transformer frameworks widely used in language and vision that are easy to scale, utilizing a large body of work studying stable training of transformer models.'

mountainriver 5y ago
Transformer all the things
- sillysaurusx 5y ago
  Yuck. I hate how expensive transformers are. You can see it clearly in transformers vs gans; GANs can generate a frame in about 20ms, whereas it takes seconds or more to make a frame with transformers. It’s not even clear that it’s necessarily better quality. That said, I doubt we’ll be able to make an RL GAN, so maybe this is the best way. Though now I wonder how well an RL GAN might work…
  - sdenton4 5y ago
    Yeah, I work on on-device audio, so definitely agree on the expense problem. In audio we've now got a few different approaches that work really well for sequence modeling, and we're constantly finding cool ways to make inference run faster. Reframing reinforcement learning as sequence modeling /should/ make it possible to reuse (m)any of the approaches we use for audio, including GANs. Generative audio is nicely analogous to RL problems : There's complex state and interesting predictive distributions, which shift subtly over time, and you need to combine short-term good behavior (good individual samples) in a reasonable way to get good long-term behavior (matching melspectra).

PaulHoule 5y ago

Deep-learned convolutional nets work wonders for visual recognition. Visual recognition via "the old AI" looked impossible by 1980, but today it looks easy. (I see the visual segmentation models for self-driving cars from the "autonomous systems lab" in the next building over and think... It would be so easy to make something that honks for cyclists.) Text analysis, reinforcement learning, etc. seem to be areas where deep learning might very well reach a plateau. In the case of images the meaning is not changed by a random "hot pixel", but changing one letter in a sentence can reverse the meaning of a sentence, changing one piece position in a chess game is the difference between a win or loss. These "binary" situations aren't a good match with the assumptions of continuity, differentiablity and such that neural networks depend on.

throwawaygh 5y ago
> Text analysis, reinforcement learning, etc. seem to be areas where deep learning might very well reach a plateau... changing one piece position in a chess game is the difference between a win or loss. These "binary" situations aren't a good match with the assumptions of continuity, differentiablity and such that neural networks depend on. Deep learning can't do board games like chess and go?! Wait... are you using pretty darn subtle dry sarcasm to argue that deep learning won't reach a plateau?
- PaulHoule 5y ago
  I think that markov chain monte carlo is pretty cool even with lightweight playouts. (pick a random move) The neural net by itself is a "half-baked" chess or go player, it needs the MCMC to be a strong player. (MCMC plus lightweight playouts can beat me at chess if not at go.) Same with text-analysis, code generation and such. If you can build a hybrid system where the neural net comes up with half-baked answers that can be corrected by a system which is capable of comprehending things like "well-formed" and "valid" then you could be cooking with gas. What I am seeing though is that people aren't "beginning with the end in mind" the way the Wright Brothers did with flying, rather they are throwing stuff at the wall and seeing what sticks.

vsskanth 5y ago

Has there been any work on "growth" based RL models ? Like how a human baby starts off with a small brain and little capability to move around, but slowly starts to roll over, gets head control, crawl and eventually walk, grasp and develop fine motor control as their brains and physical abilities grow together. The current method looks to me like starting from scratch with a fully capable human with a huge untrained brain instead of progressively expanding their actuators and control plane

mountainriver 5y ago
Yeah there is a field that does incremental learning and “lifelong learning” which employ these tactics. It’s successful on some domains
procgen 5y ago
I recommend Peter Hiesinger's 'The Self-Assembling Brain' if you're interested in a neurobiologist's perspective on this problem and its importance to AI.

Digitalis33 5y ago

I prefer to listen to people who actually put AI/ML models into production, instead of random programmer / AI dilettante types.

cesaref 5y ago
You'll only hear one side of the coin. I would also want to hear from people who understand AI/ML and have decided not to use it in an application domain. That's probably the most valuable info, knowing when not to use a tool.
- wizzwizz4 5y ago
  I understand ML reasonably well, and have decided to use classical AI instead, because I want real-time performance on low-end hardware (and I also want to be able to predict and extend the operating parameters). It's a lot harder, though, because I'll have to understand the problems at a fundamental (mathematical) level better than I currently do, and all the time I can hear a little voice saying “a neural network could do this in half an hour of work and a week of training”.
- deleted 5y ago
  [deleted]
soVeryTired 5y ago
But this is part of the author's point: where are all the production reinforcement learning models?

hervature 5y ago

To me, this entire piece reads like goal-post moving and lacks understanding of the field and was incredibly premature. > I am confused as to why they continue to spend so much time on building Deep RL systems that beat games with clearly defined rules and point systems. I think everyone gets it now, with enough time, money, and computers almost brute forcing every single possible action, Deep RL can beat almost any game. Right, before AlphaGo, superhuman Go was decades away. Then it became obvious and therefore easy. He brings up a laundry folding robot almost as if discrete games can be trivially applied to robotics. If you think about it for one second, Shogi, the game that AlphaZero solves with the largest action space has 10^5 actions. Consider a robot with 3 joints that can move in any dimension motors. Discretizing the continuous action space into 10 buckets yields an action space of 10^9. Something AlphaZero will never solve. In that sense, the entire analogy is a straw man and everyone in the field knows. This doesn't even touch the fact that 10 buckets would be useless for fine motor control. Since they published this, there have been great advancements in learning action embeddings for robotic tasks. With a couple of expert examples, an algorithm can learn to open a door effectively. That's not that far from folding laundry. Personally, my entire PhD thesis attempts to solves the exact deficiencies they raise. Simply put, the work goes on. Instead of saying that the field is useless, try to understand and contribute to it. At the very least, give technical reasons that it is a dead end path.

lysozyme 5y ago
That’s a really interesting way of looking at the difficulty of the problem that’s being solved. I’m curious, how do you arrive at the number 10^9?
- hervature 5y ago
  You have 3 joints with 3 dimensions. So 9 variables you can control. If each variable has 10 choices, and you need to make a choice for each, that is 10^9 possible combinations.
  - whimsicalism 5y ago
    "3 joints" -> What are the 3 dimensions that joints move along? I naively would have thought one or two (ie. how much spin on one axis and how much spin on the other axis) just by naively looking at my elbow for a second. For an arm of fixed length, polar coordinates have 2 dof.

toisanji 5y ago

how do you think the article stands with a paper just recently publishing "Reward is enough" by Richard Sutton (father of Reinforcement Learning)

dang 5y ago

Discussed a bit at the time: Deep Reinforcement Learning is a waste of time - https://news.ycombinator.com/item?id=21916775 https://news.ycombinator.com/item?id=21916775 - Dec 2019 (3 comments)

phonebucket 5y ago

I have to disagree with a good chunk of this article. The article completely misses the main advantage (to me) of reinforcement learning: Reinforcement learning allows you to optimise on non-differentiable outcomes. I can't differentiate real life, but I want to optimise a process within real life. This feels tantalisingly close to AGI. If I can figure out a reward function, I can use reinforcement learning. Yes, this requires a reward function to be defined. Yes, this is a challenge to AGI. But to say that the big labs are not aware that this is a challenge to AGI is unfair. DeepMind is actively investigating open ended learning: https://deepmind.com/research/publications/open-ended-learning-symmetric-zero-sum-games https://deepmind.com/research/publications/open-ended-learni.... Just because the labs haven't tackled all the questions doesn't mean that they're not busy tackling difficult questions.

soVeryTired 5y ago
Can you clarify what you mean by 'non-differentiable outcomes'?
- ArnoVW 5y ago
  My guess: a function that you can evaluate but that you can not differentiate. Differentiate being "know the slope". Differentiable functions are great because you can run gradient descent on them in a very optimized way. Example: if your objective is to have a very high value, search in the direction that has a mounting slope. Though maybe I'm missing something 'cos it seems to me you can run gradient descend on non-differentiable functions. It just requires more evaluations.
  - LeegleechN 5y ago
    Neural network models have millions of parameters, which means you are trying to optimize in spaces with millions of dimensions. If you can't use differentiation you get hit really hard with the curse of dimensionality.
  - jeeeb 5y ago
    > Though maybe I'm missing something 'cos it seems to me you can run gradient descend on non-differentiable functions Isn't that exactly the point though? If you don't have an analytical solution for the gradient of the loss (reward) wrt the parameters - yes - you could brute force a numerical solution but as the number of parameters grows that quickly becomes infeasible. Approaches such as RL and GA provide a more intelligent way to search the parameter space.

Zamaamiro 5y ago

It’s not useless; it just doesn’t get us closer to AGI. That doesn’t mean it’s useless.

dqpb 5y ago

Put your money where your mouth is. Train a homeostatic surprise minimization model to beat OpenAI’s PPO at Dota 2. The beautiful thing about benchmarks is that you can easily persuade everyone just by beating them.