8 ms·

Tool AIs want to be agent AIs

153 points by ogennadi 10y ago

Eliezer 10y ago
This is excellent. If you want to see what real discussion of an AGI alignment issue looks like, please read this.
daveguy 10y ago
I firmly believe that general AI will not be developed without agency for the AI. The "info only" helper AI (called Tool AI) means that information will have to be added manually by some intelligent agent (human or otherwise). No exploration of how actions and interactions affect results can be explored. Tool AIs will never "want" anything because the meaning of want will be completely foreign.
- visarga 10y ago
  You're not alone. Yann LeCun's famous cake analogy associates the cherry on the top with reinforcement learning (agency, being a part of an environment from which it can learn and explore). I especially liked this passage. It connects many ideas onto one theme: > CNNs with adaptive computations will be computationally faster for a given accuracy rate than fixed-iteration CNNs, CNNs with attention classify better than CNNs without attention, CNNs with focus over their entire dataset will learn better than CNNs which only get fed random images, CNNs which can ask for specific kinds of images do better than those querying their dataset, CNNs which can trawl through Google Images and locate the most informative one will do better still, CNNs which access rewards from their user about whether the result was useful will deliver more relevant results, CNNs whose hyperparameters are automatically optimized by an RL algorithm will perform better than CNNs with handwritten hyperparameters, CNNs whose architecture as well as standard hyperparameters are designed by RL agents will perform better than handwritten CNNs… and so on. (It’s actions all the way down.) This is the general trend. Going meta, through RL. Optimize the optimizer, learn gradient descent by gradient descent.
  - gwern 10y ago
    > learn gradient descent by gradient descent. As far as 'learning gradient descent by gradient descent' goes, I think it's still an open question if a tiny RNN actually can improve meaningfully over ADAM etc :) I don't recall the paper showing any wallclock times, which suggested to me that the RNN was way slower even if it trained faster. The individual parameter adjustments are so low in the hierarchy of actions that the value may be minimal compared to higher up like architecture design.
    - AlexCoventry 10y ago
      What paper are you referencing?
      - detaro 10y ago
        I assume this https://arxiv.org/abs/1606.04474 https://arxiv.org/abs/1606.04474 (since the title is basically the quoted text)
  - skybrian 10y ago
    I'm not sure that trawling Google Images counts as active. It doesn't seem like it can be any more powerful than being given the entire Google Images database in advance.
eduren 10y ago
I highly recommend the book referenced in the article: Nick Bostrom's Superintelligence. https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom-ebook/dp/B00LOOCGB2 https://www.amazon.com/Superintelligence-Dangers-Strategies-... It has helped me make informed, realistic judgments about the path AI research needs to take. It and related works should be in the vocabulary of anybody working towards AI.
- icey 10y ago
  Nick Bostrom also did an interview on EconTalk that covers quite a lot of the topics in the book from a high-level, if you want a shorter introduction to AI safety and the control problem: http://www.econtalk.org/archives/2014/12/nick_bostrom_on.html http://www.econtalk.org/archives/2014/12/nick_bostrom_on.htm...
  - eduren 10y ago
    Thank you! Now I have a way to introduce my book-averse friends to the control problem.
- vonnik 10y ago
  Every time I encounter Bostrom's writing, I think of this Von Neumann quote: "There’s no sense in being precise when you don’t even know what you’re talking about." Bostrom is one of those medieval cartographers drawing fantastical beasts in the blank spots of continents which he has never visited.
  - idlewords 10y ago
    Bless you for saying it. The analogy I like to use for our understanding of AI is alchemy. We threw Sir Isaac Goddamned Newton at chemistry and he couldn't make forward progress, because the tools were not precise enough. Similarly, we just don't understand minds enough yet to formulate sensible questions about AI. This doesn't bother Bostrom. He builds castles of thought in the air, and then climbs up into them.
PaulHoule 10y ago
If you have a decision process of some kind you can get more value out of it if you can link it to a utility function in which the "tool" tries to maximize the value it creates. It's tough enough to get value out of A.I. that this trick should not be left on the table. Thus Tool A.I.s need to be Agent A.I.s to maximize their potential.
jessriedel 10y ago
It seems most of these arguments apply equally well to the problem of solving AI value alignment, or of preventing the development of AI at all. (I.e., it's cheaper and faster to race ahead without worrying about value alignment.) But that doesn't make us conclude that value alignment is impossible, just hard to achieve soon enough in the real world. Yes, we should be aware of the limitations and market instability of tool AI, but I think it's unjustified to suggest that tool AI is essentially impossible ("highly unstable equilibrium") and all we can hope to do is solve value alignment.
- gwern 10y ago
  > It seems most of these arguments apply equally well to the problem of solving AI value alignment Only somewhat. There are not repeated practical demonstrations, nor is there any really good theoretical reasons, to think that value alignment mechanisms would be seriously and systematically detrimental to performance & cost in the same way that we have for reinforcement learning/active learning/sequential trials/etc. You can reuse my little trivial proof to argue that UFAIs >= FAIs, but of course they could just be == on intelligence. Then FAIs can be a relatively stable equilibrium because they are not instantly outclassed by any fast-growing UFAI. Contrast that with Tool AIs where everyone who cooperates is at enormous disadvantage to one defector, and defectors have large and growing rewards. > But that doesn't make us conclude that value alignment is impossible, just hard to achieve soon enough in the real world. I'm not sure anyone seriously thinks that value alignment will be easy to achieve, much less at sufficient scale. (It's weird to try to reason that 'value alignment looks hard for the same reason that Tool AI looks hard; but we can do value alignment, thus, we can do Tool AI as well'. Maybe we can't do either? We desperately want value alignment to work, but the universe doesn't owe us anything.)
  - jessriedel 10y ago
    > There are not repeated practical demonstrations, nor is there any really good theoretical reasons, to think that value alignment mechanisms would be seriously and systematically detrimental to performance & cost in the same way that we have for reinforcement learning/active learning/sequential trials/etc. You can reuse my little trivial proof to argue that UFAIs >= FAIs, but of course they could just be == on intelligence. Then FAIs can be a relatively stable equilibrium because they are not instantly outclassed by any fast-growing UFAI. Some vaguely similar arguments: 1. The race by organizations to develop AI is slowed >10% by worrying about friendliness. In a big efficient global marketplace, the winning organizations will almost certainly be ones who ignored friendliness. 2. A seed AI which is amoral can take immoral actions not available to a moral AI. Even a tiny advantage amplifies over time because of the exponential nature of recursive self-improvement. So the first super intelligent AI will be immoral. 3. Nations who obtain nuclear weapons have immense power. Even if most nations decide not to obtain them, or only obtain them for peaceful purposes, at least a few will get them and then use them to take over the Earth. Therefore, the Earth will be taken over completely by a nuclear-armed nation. All of these arguments have some merit, but none are strong/inevitable. They all require quantification and comparison against countervailing forces (e.g., the effect of laws, the moral motivations of nearby human programmers, the counterthreat provided by many allied nations/AIs). > (It's weird to try to reason that 'value alignment looks hard for the same reason that Tool AI looks hard; but we can do value alignment, thus, we can do Tool AI as well'. Maybe we can't do either? We desperately want value alignment to work, but the universe doesn't owe us anything.) I'm not making any sorts of claims like that.
anon987 10y ago
I think all AIs in my lifetime will simply be Chinese Rooms https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room
- gwern 10y ago
  The Chinese room argument is irrelevant here.
  - kayimbo 10y ago
    is it? It seemed to me like author assumes AI decision making will be roughly equivalent to biological decision making, just faster. I thought one of the Chinese room arguments is that biological decisions will always be "different" than AI ones. Also it seemed like the author assumes great technological advances in AIs, but not in biology. If we're gonna dream shit up why not dream that brains in the future will be 10,000 times as dense and computers won't be able to keep up except as tools.
    - gwern 10y ago
      The point of the Chinese room argument is that while the room receives and emits Chinese just like any Chinaman, it isn't conscious. As the WP article makes clear, the assumption is that the Chinese room is just as competent at emitting Chinese: "Suppose, says Searle, that this computer performs its task so convincingly that it comfortably passes the Turing test: it convinces a human Chinese speaker that the program is itself a live Chinese speaker. To all of the questions that the person asks, it makes appropriate responses, such that any Chinese speaker would be convinced that they are talking to another Chinese-speaking human being." > If we're gonna dream shit up why not dream that brains in the future will be 10,000 times as dense Because... that is not a thing which is happening. And deep learning and AI progress are things that are happening. (Quite aside from the many issues with your proposal, like a brain 10kx as powerful due to 10kx density would probably break thermodynamic limits on computation and of course cook itself to death within seconds.)
      - 10y ago
leblancfg 10y ago
Glad to see this articulated so well. The 'Overall' paragraph sums up thoughts that had been in the back of my mind for months. Plus, hey, it's Gwern. If you're reading this: you're an inspiration, man. What is still unclear -- to me at least -- is the technical challenges that lie ahead of this "neural networks all the way down" approach. I get the impression we'll need quite a few breakthroughs before usable Agent AIs are a thing. Insights on the order of importance as, say, backpropagation and using GPUs.
- visarga 10y ago
  > we'll need quite a few breakthroughs before usable Agent AIs are a thing. Insights on the order of importance as, say, backpropagation and using GPUs. I think a 10x-100x increase in efficiency is going to come soon, based on more research into efficient hardware and efficient models. New algorithms, new hardware and especially a lot of money and talent invested in research are going to power the next step.
- sapphireblue 10y ago
  Meta- reinforcement learning could prove to be such breakthrough, see [1],[2]. Also next generation ASIC accelerators (Google's TPU, Nervana) can give 10x increase in NN performance over a GPU manufactured on the same process, with another 10x possible with some form of binarized weights, e.g. BNN, XNOR-net. There are also interesting techniques to update the model's parameters in a sparse manner. So, there certainly is a lot of room left for performance improvements! 1. https://arxiv.org/abs/1611.05763 https://arxiv.org/abs/1611.05763 2. https://arxiv.org/abs/1611.02779 https://arxiv.org/abs/1611.02779
EGreg 10y ago
The question is what happens when "what we want" is replaced with "what you should want if you were more intelligent". Sorry Dave.
skybrian 10y ago
What about the relative size of the available datasets? It seems like that would make offline learning much more valuable than learning directly from experience. The largest publicly available research datasets for machine translation are 2-3 million sentences [1]. Google's internal datasets are "two to three decimal orders of magnitudes bigger than the WMT corpora for a given language pair" [2]. That's far more data than a cell phone's translation app would receive over its entire lifetime. Similarly, the amount of driving data collected by Tesla from all its cars will be much larger than the data received by any single car. This suggests that most learning will happen as a batch process, ahead of time. There may be some minor adjustments for personalization, but it doesn't seem like it's enough for Agent AI to outcompete Tool AI. At least so far, it seems far more important to be in a position to collect large amounts of data from millions of users, rather than learning directly from experience, which happens slowly and expensively. This is not about having a human check every individual result. It's about putting a software development team in the loop. Each new release can go through a QA process where it's compared to the previous release. [1] https://github.com/bicici/ParFDAWMT14 https://github.com/bicici/ParFDAWMT14 [2] https://research.googleblog.com/2016/09/a-neural-network-for-machine.html https://research.googleblog.com/2016/09/a-neural-network-for...
- gwern 10y ago
  Offline batch learning is not contradictory to reinforcement learning. It just requires that it be an off-policy RL algorithm, which happily, many, like DQN, are. > Similarly, the amount of driving data collected by Tesla from all its cars will be much larger than the data received by any single car. Well, Tesla benefits from the fact that its cars are already agents, acting in the real world. So using it as a counter-example doesn't really work... You would need to imagine some sort of counterfactual Tesla which eschewed any real-world deployment or simulators. Which no one would ever do because the usefulness of trying to create a self-driving car database without agents or reinforcement learning is so obviously zero. And anyway, most of Tesla's data, however, will be totally useless. You can train a simple lane-following CNN with a few hours of footage (as Geohotz demonstrated), but another million hours of highway driving is near-useless and doesn't get you much closer to self-driving cars (as perhaps Geohot also inadvertently demonstrated). What you need is the weird outliers which drive accident rates. So for example, the Google self-driving car team doesn't depend purely on collected data even from its agents, but simulates very rare scenarios as well. (If I may quote myself: "You need the right data, not more data.") > This suggests that most learning will happen as a batch process, ahead of time. There may be some minor adjustments for personalization, but it doesn't seem like it's enough for Agent AI to outcompete Tool AI. Even if we imagine that the dataset covers all possible sequences and doesn't need any kind of finetuning the loss, the agent AIs in this scenario are still going to benefit from actions over (going by section): 'internal to a computation', 'internal to training', 'internal to design', and 'internal to data selection'. In fact, if you want to have any hope of running effectively over extremely large datasets like hundreds of millions of sentences, you are going to effectively require some of those to keep training time tractable and runtime cheap (even Google can't afford a million TPUs for Google Translate). For example, regular hyperparameter optimization using a few dozen examples doesn't look very good when it takes a month on a GPU cluster to train a single model, but using a meta-RL RNN to do some transfer learning and pick out near-optimal architectures & hyperparameters in just a few iterations looks pretty nice.
visarga 10y ago
Check out Gwern's Reinforcement Learning subreddit. He's practically supporting this subreddit by himself. https://www.reddit.com/r/reinforcementlearning/ https://www.reddit.com/r/reinforcementlearning/

153 points by ogennadi 10y ago

This is excellent. If you want to see what real discussion of an AGI alignment issue looks like, please read this.

I firmly believe that general AI will not be developed without agency for the AI. The "info only" helper AI (called Tool AI) means that information will have to be added manually by some intelligent agent (human or otherwise). No exploration of how actions and interactions affect results can be explored. Tool AIs will never "want" anything because the meaning of want will be completely foreign.

visarga 10y ago
You're not alone. Yann LeCun's famous cake analogy associates the cherry on the top with reinforcement learning (agency, being a part of an environment from which it can learn and explore). I especially liked this passage. It connects many ideas onto one theme: > CNNs with adaptive computations will be computationally faster for a given accuracy rate than fixed-iteration CNNs, CNNs with attention classify better than CNNs without attention, CNNs with focus over their entire dataset will learn better than CNNs which only get fed random images, CNNs which can ask for specific kinds of images do better than those querying their dataset, CNNs which can trawl through Google Images and locate the most informative one will do better still, CNNs which access rewards from their user about whether the result was useful will deliver more relevant results, CNNs whose hyperparameters are automatically optimized by an RL algorithm will perform better than CNNs with handwritten hyperparameters, CNNs whose architecture as well as standard hyperparameters are designed by RL agents will perform better than handwritten CNNs… and so on. (It’s actions all the way down.) This is the general trend. Going meta, through RL. Optimize the optimizer, learn gradient descent by gradient descent.
- gwern 10y ago
  > learn gradient descent by gradient descent. As far as 'learning gradient descent by gradient descent' goes, I think it's still an open question if a tiny RNN actually can improve meaningfully over ADAM etc :) I don't recall the paper showing any wallclock times, which suggested to me that the RNN was way slower even if it trained faster. The individual parameter adjustments are so low in the hierarchy of actions that the value may be minimal compared to higher up like architecture design.
  - AlexCoventry 10y ago
    What paper are you referencing?
    - detaro 10y ago
      I assume this https://arxiv.org/abs/1606.04474 https://arxiv.org/abs/1606.04474 (since the title is basically the quoted text)
- skybrian 10y ago
  I'm not sure that trawling Google Images counts as active. It doesn't seem like it can be any more powerful than being given the entire Google Images database in advance.

eduren 10y ago

I highly recommend the book referenced in the article: Nick Bostrom's Superintelligence. https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom-ebook/dp/B00LOOCGB2 https://www.amazon.com/Superintelligence-Dangers-Strategies-... It has helped me make informed, realistic judgments about the path AI research needs to take. It and related works should be in the vocabulary of anybody working towards AI.

icey 10y ago
Nick Bostrom also did an interview on EconTalk that covers quite a lot of the topics in the book from a high-level, if you want a shorter introduction to AI safety and the control problem: http://www.econtalk.org/archives/2014/12/nick_bostrom_on.html http://www.econtalk.org/archives/2014/12/nick_bostrom_on.htm...
- eduren 10y ago
  Thank you! Now I have a way to introduce my book-averse friends to the control problem.
vonnik 10y ago
Every time I encounter Bostrom's writing, I think of this Von Neumann quote: "There’s no sense in being precise when you don’t even know what you’re talking about." Bostrom is one of those medieval cartographers drawing fantastical beasts in the blank spots of continents which he has never visited.
- idlewords 10y ago
  Bless you for saying it. The analogy I like to use for our understanding of AI is alchemy. We threw Sir Isaac Goddamned Newton at chemistry and he couldn't make forward progress, because the tools were not precise enough. Similarly, we just don't understand minds enough yet to formulate sensible questions about AI. This doesn't bother Bostrom. He builds castles of thought in the air, and then climbs up into them.

PaulHoule 10y ago

If you have a decision process of some kind you can get more value out of it if you can link it to a utility function in which the "tool" tries to maximize the value it creates. It's tough enough to get value out of A.I. that this trick should not be left on the table. Thus Tool A.I.s need to be Agent A.I.s to maximize their potential.

jessriedel 10y ago

It seems most of these arguments apply equally well to the problem of solving AI value alignment, or of preventing the development of AI at all. (I.e., it's cheaper and faster to race ahead without worrying about value alignment.) But that doesn't make us conclude that value alignment is impossible, just hard to achieve soon enough in the real world. Yes, we should be aware of the limitations and market instability of tool AI, but I think it's unjustified to suggest that tool AI is essentially impossible ("highly unstable equilibrium") and all we can hope to do is solve value alignment.

gwern 10y ago
> It seems most of these arguments apply equally well to the problem of solving AI value alignment Only somewhat. There are not repeated practical demonstrations, nor is there any really good theoretical reasons, to think that value alignment mechanisms would be seriously and systematically detrimental to performance & cost in the same way that we have for reinforcement learning/active learning/sequential trials/etc. You can reuse my little trivial proof to argue that UFAIs >= FAIs, but of course they could just be == on intelligence. Then FAIs can be a relatively stable equilibrium because they are not instantly outclassed by any fast-growing UFAI. Contrast that with Tool AIs where everyone who cooperates is at enormous disadvantage to one defector, and defectors have large and growing rewards. > But that doesn't make us conclude that value alignment is impossible, just hard to achieve soon enough in the real world. I'm not sure anyone seriously thinks that value alignment will be easy to achieve, much less at sufficient scale. (It's weird to try to reason that 'value alignment looks hard for the same reason that Tool AI looks hard; but we can do value alignment, thus, we can do Tool AI as well'. Maybe we can't do either? We desperately want value alignment to work, but the universe doesn't owe us anything.)
- jessriedel 10y ago
  > There are not repeated practical demonstrations, nor is there any really good theoretical reasons, to think that value alignment mechanisms would be seriously and systematically detrimental to performance & cost in the same way that we have for reinforcement learning/active learning/sequential trials/etc. You can reuse my little trivial proof to argue that UFAIs >= FAIs, but of course they could just be == on intelligence. Then FAIs can be a relatively stable equilibrium because they are not instantly outclassed by any fast-growing UFAI. Some vaguely similar arguments: 1. The race by organizations to develop AI is slowed >10% by worrying about friendliness. In a big efficient global marketplace, the winning organizations will almost certainly be ones who ignored friendliness. 2. A seed AI which is amoral can take immoral actions not available to a moral AI. Even a tiny advantage amplifies over time because of the exponential nature of recursive self-improvement. So the first super intelligent AI will be immoral. 3. Nations who obtain nuclear weapons have immense power. Even if most nations decide not to obtain them, or only obtain them for peaceful purposes, at least a few will get them and then use them to take over the Earth. Therefore, the Earth will be taken over completely by a nuclear-armed nation. All of these arguments have some merit, but none are strong/inevitable. They all require quantification and comparison against countervailing forces (e.g., the effect of laws, the moral motivations of nearby human programmers, the counterthreat provided by many allied nations/AIs). > (It's weird to try to reason that 'value alignment looks hard for the same reason that Tool AI looks hard; but we can do value alignment, thus, we can do Tool AI as well'. Maybe we can't do either? We desperately want value alignment to work, but the universe doesn't owe us anything.) I'm not making any sorts of claims like that.

anon987 10y ago

I think all AIs in my lifetime will simply be Chinese Rooms https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room

gwern 10y ago
The Chinese room argument is irrelevant here.
- kayimbo 10y ago
  is it? It seemed to me like author assumes AI decision making will be roughly equivalent to biological decision making, just faster. I thought one of the Chinese room arguments is that biological decisions will always be "different" than AI ones. Also it seemed like the author assumes great technological advances in AIs, but not in biology. If we're gonna dream shit up why not dream that brains in the future will be 10,000 times as dense and computers won't be able to keep up except as tools.
  - gwern 10y ago
    The point of the Chinese room argument is that while the room receives and emits Chinese just like any Chinaman, it isn't conscious. As the WP article makes clear, the assumption is that the Chinese room is just as competent at emitting Chinese: "Suppose, says Searle, that this computer performs its task so convincingly that it comfortably passes the Turing test: it convinces a human Chinese speaker that the program is itself a live Chinese speaker. To all of the questions that the person asks, it makes appropriate responses, such that any Chinese speaker would be convinced that they are talking to another Chinese-speaking human being." > If we're gonna dream shit up why not dream that brains in the future will be 10,000 times as dense Because... that is not a thing which is happening. And deep learning and AI progress are things that are happening. (Quite aside from the many issues with your proposal, like a brain 10kx as powerful due to 10kx density would probably break thermodynamic limits on computation and of course cook itself to death within seconds.)
    - 10y ago

leblancfg 10y ago

Glad to see this articulated so well. The 'Overall' paragraph sums up thoughts that had been in the back of my mind for months. Plus, hey, it's Gwern. If you're reading this: you're an inspiration, man. What is still unclear -- to me at least -- is the technical challenges that lie ahead of this "neural networks all the way down" approach. I get the impression we'll need quite a few breakthroughs before usable Agent AIs are a thing. Insights on the order of importance as, say, backpropagation and using GPUs.

visarga 10y ago
> we'll need quite a few breakthroughs before usable Agent AIs are a thing. Insights on the order of importance as, say, backpropagation and using GPUs. I think a 10x-100x increase in efficiency is going to come soon, based on more research into efficient hardware and efficient models. New algorithms, new hardware and especially a lot of money and talent invested in research are going to power the next step.
sapphireblue 10y ago
Meta- reinforcement learning could prove to be such breakthrough, see [1],[2]. Also next generation ASIC accelerators (Google's TPU, Nervana) can give 10x increase in NN performance over a GPU manufactured on the same process, with another 10x possible with some form of binarized weights, e.g. BNN, XNOR-net. There are also interesting techniques to update the model's parameters in a sparse manner. So, there certainly is a lot of room left for performance improvements! 1. https://arxiv.org/abs/1611.05763 https://arxiv.org/abs/1611.05763 2. https://arxiv.org/abs/1611.02779 https://arxiv.org/abs/1611.02779

EGreg 10y ago

The question is what happens when "what we want" is replaced with "what you should want if you were more intelligent". Sorry Dave.

skybrian 10y ago

What about the relative size of the available datasets? It seems like that would make offline learning much more valuable than learning directly from experience. The largest publicly available research datasets for machine translation are 2-3 million sentences [1]. Google's internal datasets are "two to three decimal orders of magnitudes bigger than the WMT corpora for a given language pair" [2]. That's far more data than a cell phone's translation app would receive over its entire lifetime. Similarly, the amount of driving data collected by Tesla from all its cars will be much larger than the data received by any single car. This suggests that most learning will happen as a batch process, ahead of time. There may be some minor adjustments for personalization, but it doesn't seem like it's enough for Agent AI to outcompete Tool AI. At least so far, it seems far more important to be in a position to collect large amounts of data from millions of users, rather than learning directly from experience, which happens slowly and expensively. This is not about having a human check every individual result. It's about putting a software development team in the loop. Each new release can go through a QA process where it's compared to the previous release. [1] https://github.com/bicici/ParFDAWMT14 https://github.com/bicici/ParFDAWMT14 [2] https://research.googleblog.com/2016/09/a-neural-network-for-machine.html https://research.googleblog.com/2016/09/a-neural-network-for...

gwern 10y ago
Offline batch learning is not contradictory to reinforcement learning. It just requires that it be an off-policy RL algorithm, which happily, many, like DQN, are. > Similarly, the amount of driving data collected by Tesla from all its cars will be much larger than the data received by any single car. Well, Tesla benefits from the fact that its cars are already agents, acting in the real world. So using it as a counter-example doesn't really work... You would need to imagine some sort of counterfactual Tesla which eschewed any real-world deployment or simulators. Which no one would ever do because the usefulness of trying to create a self-driving car database without agents or reinforcement learning is so obviously zero. And anyway, most of Tesla's data, however, will be totally useless. You can train a simple lane-following CNN with a few hours of footage (as Geohotz demonstrated), but another million hours of highway driving is near-useless and doesn't get you much closer to self-driving cars (as perhaps Geohot also inadvertently demonstrated). What you need is the weird outliers which drive accident rates. So for example, the Google self-driving car team doesn't depend purely on collected data even from its agents, but simulates very rare scenarios as well. (If I may quote myself: "You need the right data, not more data.") > This suggests that most learning will happen as a batch process, ahead of time. There may be some minor adjustments for personalization, but it doesn't seem like it's enough for Agent AI to outcompete Tool AI. Even if we imagine that the dataset covers all possible sequences and doesn't need any kind of finetuning the loss, the agent AIs in this scenario are still going to benefit from actions over (going by section): 'internal to a computation', 'internal to training', 'internal to design', and 'internal to data selection'. In fact, if you want to have any hope of running effectively over extremely large datasets like hundreds of millions of sentences, you are going to effectively require some of those to keep training time tractable and runtime cheap (even Google can't afford a million TPUs for Google Translate). For example, regular hyperparameter optimization using a few dozen examples doesn't look very good when it takes a month on a GPU cluster to train a single model, but using a meta-RL RNN to do some transfer learning and pick out near-optimal architectures & hyperparameters in just a few iterations looks pretty nice.

visarga 10y ago

Check out Gwern's Reinforcement Learning subreddit. He's practically supporting this subreddit by himself. https://www.reddit.com/r/reinforcementlearning/ https://www.reddit.com/r/reinforcementlearning/