6 ms·

Not surprising. However, I am not buying the claim that the algorithm is going to transform many other domains: " The system could help optimize power grids,

by strin 9y ago

Not surprising. However, I am not buying the claim that the algorithm is going to transform many other domains: " The system could help optimize power grids, he says, or streamline shipping routes, or refine scientific research." by Wired https://www.wired.com/2017/05/googles-alphago-levels-board-games-power-grids/ https://www.wired.com/2017/05/googles-alphago-levels-board-g... The main concern is data efficiency. AlphaGo essentially baked good movies into value and policy network by playing millions of times. In reality, unless you have a really good simulator, deep reinforcement learning is almost never applicable.

deleted 9y ago
[deleted]
Aron 9y ago
'data efficiency' -- that's a good candidate for the next buzz imo. People talk a lot about how massive amounts of data regressed over giant networks of dumb functions happens to perform well on many tests, but they ignore the fact that the performance of the system is sick. It fails if you tweak the input by small amounts that would never confuse a human. You shouldn't NEED that much data. You need better priors. Current ML systems rely on arbitrary coincidences to succeed at multiple choice tests. I did that in courses I didn't go to class. They are primitive and childish and won't scale with pure data/processing. We are still missing something.
- strin 9y ago
  Prior is in some sense the fruit of data. Humans develop prior by accumulating experiences. Laws of physics are discovered through experimentation and introspection over many data points. So what we really need is a life-long learner - a machine learning algorithm that could extract knowledge from many tasks and store that in its long-term memory. The fact that we don't have an architecture for long-term memory is the main road blocker towards this goal.
londons_explore 9y ago
There are a lot of problems for which you effectively have limitless data. For example, optimizing the shape of an aerofoil, wing, wind turbine, ship, pump, etc. The "real" data is the result of computationally costly fluid dynamics simulations. The network uses as much of those as it likes to modify the design to be better. There are also a lot of problems where there is nearly limitless data (ie. > 10^12 bytes of data). Examples would be MRI/CT scans, all internet video, all TV channels, etc.
- yakult 9y ago
  >There are a lot of problems for which you effectively have limitless data. >For example, optimizing the shape of an aerofoil, wing, wind turbine, ship, pump, etc. The "real" data is the result of computationally costly fluid dynamics simulations. I'm doing something similar at the moment. The data is not limitless. It's in fact quite expensive. In most cases you still need to generate the large number of simulations in your training set, and the simulations are computationally expensive to do. If you're doing it on a metered platform you can tie the cost of the data set directly to the number of cpu hours need to generate it, and from there to a dollar value.
BoiledCabbage 9y ago
You can view there being 5 dimensions/concerns with large scaling factors containing issues needing solving to be generally applicable. 1. How to search 2. How to resolve breadth concerns 3. How to resolve depth concerns 4. How to resolve large training requirements 5. How to form and resolve recursive goals With AlphaGo, it looks like they may have solved 3 1. Search. Monte Carlo will likely be used from here on out. Much more efficient than full brute force search (chess style). 2. Breadth - Policy network. Reduced branching factor from 250 to between 5 and 10. 3. Depth - Value network reduced depth from roughly 75 to < 20 Next they're working on applying reinforcement learning to their two NNs. Ie seeing if they can optimize how well they can do and how quickly they can learn. Maybe there are large gains there. If so, the final task would be finding a game or situation that required nesting of sub-goals like everyday life does. They've made some impressive gains. > In reality, unless you have a really good simulator, deep reinforcement learning is almost never applicable. I could see Deep Mind applying this to something like catching a ball in the near future. Just to show it's generality. It's not that different of a problem space than learning to play breakout.
- mrob 9y ago
  Chess AI does not use brute force search. It evaluates board positions with heuristics (eg. pieces captured, blocked pawns, possibility of castling, etc.), and uses the estimated chance of winning to prune the search tree with techniques such as alpha-beta pruning[0]. Both board evaluation and tree pruning are more difficult in Go. In Chess, capturing material is very important, but capturing stones in Go is often useless because the stones are in a bad position ("dead"). You can't decide who's winning just by counting captures. It's very difficult to hand code a board evaluation function, but AlphaGo's neural network solved the problem. However, the neural network alone, with no tree search, was only enough to reach strong amateur level. Unlike Chess, where moves usually have consequences in the near future, a move in Go can be important tens or even hundreds of moves later. Alpha-beta pruning doesn't handle very distant consequences well, but Monte Carlo tree search does much better. Combined with the neural network it was enough to beat humans. [0] https://en.wikipedia.org/wiki/Alpha%E2%80%93beta_pruning https://en.wikipedia.org/wiki/Alpha%E2%80%93beta_pruning
throwaway010317 9y ago
DeepMind claims that AlphaGo has already paid for itself when it was used to cut Google’s data center cooling costs, so it is not infeasible there may be many other domains which may benefit.
- skybrian 9y ago
  Where did you read that? Are you sure they weren't talking about machine learning in general?
  - gwern 9y ago
    It has been cited in a number articles quoting Demis Hassabis; Hassabis repeated the claim in his recent talk on AG (specifically, that deep RL related to AlphaGo has reduced Google datacenter cooling costs by 40% and is saving Google hundreds of millions of dollars a year) and added in the additional interesting detail that Google has changed its datacenter design plans to add in additional sensors & controls for the deep RL agents. They haven't gone into detail as to how exactly this is done, but my guess is that they use their historical data series on cooling/electricity consumption to bootstrap and then it learns online normally with policy gradients (since cooling and temperature control seem like something with continuous actions rather than a few discrete ones, so an actor-critic rather than DQN).
    - dontreact 9y ago
      If you look more closely at the details here (beyond just deepminds blog post) you'll see that 1. It has not yet been deployed to all data centers. They just turned the system off for a period of time and looked at how much extra money it cost to run the data center like that, then extrapolated. 2. There was already a Google engineer back in 2012 who applied neural networks to this problem and saw huge gains (There is a blog post about this somewhere). In the deepmind blog post, they don't compare to this system, do for all we know it could just be a small refinement of that system. It is actually not clear whether deep RL was actually used or not from the deepminds blog post: it is only kind of implied.
WilliamDhalgren 9y ago
> AlphaGo essentially baked good movies into value and policy network by playing millions of times. I don't think that's a very good description of how AlphaGo was trained at all; you're essentially saying it merely overfits the training set, yet it clearly generalizes rather well to unseen board situations and still evaluates them sucessfully. No machine learning system would be found usefull if all it could do is merely memorize the training data. Re the use of deep reinforcement learning, well for one the role of reinforcement learning in the first version of AlphaGo, the one described in the Nature paper was rather limited, and a small part of its training; it just made a ~3d KGS policy network into a ~5d KGS bot, and used to generate a training sample for the value net. If we had enough recorded human games to train the value net directly, that'd be an unnecessary step anyhow. And you could create such a training set w/o reinforcement learning since there are pure monte carlo bots stronger than 5d KGS - but that'd be far more computationally expensive. But its still not really true that there aren't obvious applications of deep reinforcement learning - indeed robotics is one promising application, and that seems rather relevant. this paper initially demonstrated an impressive improvement in manipulative tasks, and you can prob follow its numerous citations for newer stuff: http://arxiv.org/abs/1504.00702 http://arxiv.org/abs/1504.00702 I do agree that this exact architecture in AlphaGo prob doesn't have applications beyond just teaching us how to play go better; it seems too specialized. I believe they mean it in just the vaguest possible sense; that the kind of deep algorithms demonstrating incredible performance in AlphaGo have diverse applications; but this should not come as a surprise to anyone even loosely following what people have done with deep learning in the past couple of years anyhow.
- arcanus 9y ago
  I'm a huge proponent of alpha Go and I think it is a revolutionary leap. The key I think is, > yet it clearly generalizes rather well to unseen board situations and still evaluates them sucessfully I'm not sure this has been proven to be meaningful in a general sense, as you seem to also imply. Extrapolation can be a tricky answer subtle business. What about unusual board sizes, for which no training data exists? Or if you changed a rule? I'm sure deepmind would say the adversarial approach would work for these cases, but I'm not sure it would. Would be very interesting to see if humans could 'learn' a new state more quickly than the algorithm. That might provide a hint that the algorithm is 'just' fitting the data well (with appropriate baked in regularization, of course). Or if it can more generally 'learn' given system rules.
awkwardtortoise 9y ago
> However, I am not buying the claim that the algorithm is going to transform many other domains: It isn't an "algorithm" as you think it is. It's not like deep blue or chess AIs specifically built for chess. DeepMind is "general" and a step towards a somewhat general AI ( though we are nowhere close to general AI ). Nobody taught Deepmind to play Go. It taught itself how to play Go. > In reality, unless you have a really good simulator, deep reinforcement learning is almost never applicable. That's all learning is...

by strin 9y ago

deleted 9y ago

[deleted]

Aron 9y ago

'data efficiency' -- that's a good candidate for the next buzz imo. People talk a lot about how massive amounts of data regressed over giant networks of dumb functions happens to perform well on many tests, but they ignore the fact that the performance of the system is sick. It fails if you tweak the input by small amounts that would never confuse a human. You shouldn't NEED that much data. You need better priors. Current ML systems rely on arbitrary coincidences to succeed at multiple choice tests. I did that in courses I didn't go to class. They are primitive and childish and won't scale with pure data/processing. We are still missing something.

strin 9y ago
Prior is in some sense the fruit of data. Humans develop prior by accumulating experiences. Laws of physics are discovered through experimentation and introspection over many data points. So what we really need is a life-long learner - a machine learning algorithm that could extract knowledge from many tasks and store that in its long-term memory. The fact that we don't have an architecture for long-term memory is the main road blocker towards this goal.

londons_explore 9y ago

There are a lot of problems for which you effectively have limitless data. For example, optimizing the shape of an aerofoil, wing, wind turbine, ship, pump, etc. The "real" data is the result of computationally costly fluid dynamics simulations. The network uses as much of those as it likes to modify the design to be better. There are also a lot of problems where there is nearly limitless data (ie. > 10^12 bytes of data). Examples would be MRI/CT scans, all internet video, all TV channels, etc.

yakult 9y ago
>There are a lot of problems for which you effectively have limitless data. >For example, optimizing the shape of an aerofoil, wing, wind turbine, ship, pump, etc. The "real" data is the result of computationally costly fluid dynamics simulations. I'm doing something similar at the moment. The data is not limitless. It's in fact quite expensive. In most cases you still need to generate the large number of simulations in your training set, and the simulations are computationally expensive to do. If you're doing it on a metered platform you can tie the cost of the data set directly to the number of cpu hours need to generate it, and from there to a dollar value.

BoiledCabbage 9y ago

You can view there being 5 dimensions/concerns with large scaling factors containing issues needing solving to be generally applicable. 1. How to search 2. How to resolve breadth concerns 3. How to resolve depth concerns 4. How to resolve large training requirements 5. How to form and resolve recursive goals With AlphaGo, it looks like they may have solved 3 1. Search. Monte Carlo will likely be used from here on out. Much more efficient than full brute force search (chess style). 2. Breadth - Policy network. Reduced branching factor from 250 to between 5 and 10. 3. Depth - Value network reduced depth from roughly 75 to < 20 Next they're working on applying reinforcement learning to their two NNs. Ie seeing if they can optimize how well they can do and how quickly they can learn. Maybe there are large gains there. If so, the final task would be finding a game or situation that required nesting of sub-goals like everyday life does. They've made some impressive gains. > In reality, unless you have a really good simulator, deep reinforcement learning is almost never applicable. I could see Deep Mind applying this to something like catching a ball in the near future. Just to show it's generality. It's not that different of a problem space than learning to play breakout.

mrob 9y ago
Chess AI does not use brute force search. It evaluates board positions with heuristics (eg. pieces captured, blocked pawns, possibility of castling, etc.), and uses the estimated chance of winning to prune the search tree with techniques such as alpha-beta pruning[0]. Both board evaluation and tree pruning are more difficult in Go. In Chess, capturing material is very important, but capturing stones in Go is often useless because the stones are in a bad position ("dead"). You can't decide who's winning just by counting captures. It's very difficult to hand code a board evaluation function, but AlphaGo's neural network solved the problem. However, the neural network alone, with no tree search, was only enough to reach strong amateur level. Unlike Chess, where moves usually have consequences in the near future, a move in Go can be important tens or even hundreds of moves later. Alpha-beta pruning doesn't handle very distant consequences well, but Monte Carlo tree search does much better. Combined with the neural network it was enough to beat humans. [0] https://en.wikipedia.org/wiki/Alpha%E2%80%93beta_pruning https://en.wikipedia.org/wiki/Alpha%E2%80%93beta_pruning

throwaway010317 9y ago

DeepMind claims that AlphaGo has already paid for itself when it was used to cut Google’s data center cooling costs, so it is not infeasible there may be many other domains which may benefit.

skybrian 9y ago
Where did you read that? Are you sure they weren't talking about machine learning in general?
- gwern 9y ago
  It has been cited in a number articles quoting Demis Hassabis; Hassabis repeated the claim in his recent talk on AG (specifically, that deep RL related to AlphaGo has reduced Google datacenter cooling costs by 40% and is saving Google hundreds of millions of dollars a year) and added in the additional interesting detail that Google has changed its datacenter design plans to add in additional sensors & controls for the deep RL agents. They haven't gone into detail as to how exactly this is done, but my guess is that they use their historical data series on cooling/electricity consumption to bootstrap and then it learns online normally with policy gradients (since cooling and temperature control seem like something with continuous actions rather than a few discrete ones, so an actor-critic rather than DQN).
  - dontreact 9y ago
    If you look more closely at the details here (beyond just deepminds blog post) you'll see that 1. It has not yet been deployed to all data centers. They just turned the system off for a period of time and looked at how much extra money it cost to run the data center like that, then extrapolated. 2. There was already a Google engineer back in 2012 who applied neural networks to this problem and saw huge gains (There is a blog post about this somewhere). In the deepmind blog post, they don't compare to this system, do for all we know it could just be a small refinement of that system. It is actually not clear whether deep RL was actually used or not from the deepminds blog post: it is only kind of implied.

WilliamDhalgren 9y ago

> AlphaGo essentially baked good movies into value and policy network by playing millions of times. I don't think that's a very good description of how AlphaGo was trained at all; you're essentially saying it merely overfits the training set, yet it clearly generalizes rather well to unseen board situations and still evaluates them sucessfully. No machine learning system would be found usefull if all it could do is merely memorize the training data. Re the use of deep reinforcement learning, well for one the role of reinforcement learning in the first version of AlphaGo, the one described in the Nature paper was rather limited, and a small part of its training; it just made a ~3d KGS policy network into a ~5d KGS bot, and used to generate a training sample for the value net. If we had enough recorded human games to train the value net directly, that'd be an unnecessary step anyhow. And you could create such a training set w/o reinforcement learning since there are pure monte carlo bots stronger than 5d KGS - but that'd be far more computationally expensive. But its still not really true that there aren't obvious applications of deep reinforcement learning - indeed robotics is one promising application, and that seems rather relevant. this paper initially demonstrated an impressive improvement in manipulative tasks, and you can prob follow its numerous citations for newer stuff: http://arxiv.org/abs/1504.00702 http://arxiv.org/abs/1504.00702 I do agree that this exact architecture in AlphaGo prob doesn't have applications beyond just teaching us how to play go better; it seems too specialized. I believe they mean it in just the vaguest possible sense; that the kind of deep algorithms demonstrating incredible performance in AlphaGo have diverse applications; but this should not come as a surprise to anyone even loosely following what people have done with deep learning in the past couple of years anyhow.

arcanus 9y ago
I'm a huge proponent of alpha Go and I think it is a revolutionary leap. The key I think is, > yet it clearly generalizes rather well to unseen board situations and still evaluates them sucessfully I'm not sure this has been proven to be meaningful in a general sense, as you seem to also imply. Extrapolation can be a tricky answer subtle business. What about unusual board sizes, for which no training data exists? Or if you changed a rule? I'm sure deepmind would say the adversarial approach would work for these cases, but I'm not sure it would. Would be very interesting to see if humans could 'learn' a new state more quickly than the algorithm. That might provide a hint that the algorithm is 'just' fitting the data well (with appropriate baked in regularization, of course). Or if it can more generally 'learn' given system rules.

awkwardtortoise 9y ago

> However, I am not buying the claim that the algorithm is going to transform many other domains: It isn't an "algorithm" as you think it is. It's not like deep blue or chess AIs specifically built for chess. DeepMind is "general" and a step towards a somewhat general AI ( though we are nowhere close to general AI ). Nobody taught Deepmind to play Go. It taught itself how to play Go. > In reality, unless you have a really good simulator, deep reinforcement learning is almost never applicable. That's all learning is...