4 ms·

What about reinforcement learning? It seems neglected to me. There are only one or two good reinforcement learning resources I'm aware of compared to the hundre

What about reinforcement learning? It seems neglected to me. There are only one or two good reinforcement learning resources I'm aware of compared to the hundreds of books and blogs on supervised learning. Most machine learning books will mention supervised, unsupervised, and reinforcement learning in the first chapter, then never mention reinforcement learning again. This is my experience at least, and I've read a few introductory machine learning books. Reinforcement learning can drive you car and mow your lawn. Supervised learning can tell you which of two pictures is a cat. Granted, supervised learning techniques can have an important place in the broader framework of reinforcement learning. Reinforcement learning is one of the harder problems.

api 11y ago
The hard thing here is reinforcing the right thing. I did a bunch of work on genetic programming and RL and it worked... uhh... too well. Once it succeeded in learning, near as I could tell, my hard disk geometry. There was a subtle timing difference between data sets and this seemed related to where they were on disk. The literature is loaded with anecdotes like that, some even more hilarious. But you're right. Nature clearly does learn this way, at least for many things. Maybe the way forward is to combine it with supervised and unsupervised learning and let different forms of learning work together.
- gwern 11y ago
  You should write that up. I love examples of GAs and other optimization algorithms going haywire.
  - api 11y ago
    I think it's been too long. I don't even have the exact code anymore. One of my faves was a GA that evolved circuits on FPGAs that would evolve wonderful, sometimes almost magical results... that only worked on the exact FPGA the evolver used to test phenotypes. Apparently it ultra-fitted to exact little quirks of that particular physical piece of silicon. They were never able to figure out exactly what those quirks were. It was a total mystery and the circuits were bizarre. The solution was to use a big pool of different FPGAs and mix them randomly. I also heard once of a case where the use of a poor random function was problematic. Evidently Mr. Darwin is able to reverse engineer the simple three or four line K&R rand() function. After that I started actually using cryptographic PRNGs. We came up with the term "herding shoggoths" for trying to get evolutionary systems to evolve the right thing.
- rayuela 11y ago
  This is what Deepmind seems to dedicate most of their efforts on and what their new AlphaGo models is a great example of (NN + tree search Reinforcement Learning). Pedro Domingo's thesis in his new book is the potential for the progress to be made with the combination of distinct schools within the field of ML
- devonkim 11y ago
  This kind of unintended learning is something that all machine learning practitioners should be careful with. The example of the satellite photo analysis algorithm that turned out to have learned to classify difference between night and day photos comes to mind. This means that your particular reinforcement learning example needs something similar to regularization of cost functions or cross validation for checking for over or under fitting to correct these latency effects. It may be cheaper and more effective to use SSDs instead but the magnitude of the latency effects on your case are not clear either to determine if your algorithm would benefit substantially from a modern SSD.
  - api 11y ago
    The fix was to (1) take steps to make sure the image was entirely pre-loaded into RAM before the test was given, and (2) randomize the latency of the load. That worked and I actually got some real results, which unfortunately were decent but not as impressive as the cheating was. :) When it cheated the thing got 100% right of course. I also learned that rapid convergence on incredible accuracy is suspect.
eli_gottlieb 11y ago
Reinforcement learning involves low-information objective signals. Hence, it sounds "universal" but doesn't actually work very well. https://drive.google.com/file/d/0BxKBnD5y2M8NbWN6XzM5UXkwNDA/view?pli=1 https://drive.google.com/file/d/0BxKBnD5y2M8NbWN6XzM5UXkwNDA...

api 11y ago

The hard thing here is reinforcing the right thing. I did a bunch of work on genetic programming and RL and it worked... uhh... too well. Once it succeeded in learning, near as I could tell, my hard disk geometry. There was a subtle timing difference between data sets and this seemed related to where they were on disk. The literature is loaded with anecdotes like that, some even more hilarious. But you're right. Nature clearly does learn this way, at least for many things. Maybe the way forward is to combine it with supervised and unsupervised learning and let different forms of learning work together.

gwern 11y ago
You should write that up. I love examples of GAs and other optimization algorithms going haywire.
- api 11y ago
  I think it's been too long. I don't even have the exact code anymore. One of my faves was a GA that evolved circuits on FPGAs that would evolve wonderful, sometimes almost magical results... that only worked on the exact FPGA the evolver used to test phenotypes. Apparently it ultra-fitted to exact little quirks of that particular physical piece of silicon. They were never able to figure out exactly what those quirks were. It was a total mystery and the circuits were bizarre. The solution was to use a big pool of different FPGAs and mix them randomly. I also heard once of a case where the use of a poor random function was problematic. Evidently Mr. Darwin is able to reverse engineer the simple three or four line K&R rand() function. After that I started actually using cryptographic PRNGs. We came up with the term "herding shoggoths" for trying to get evolutionary systems to evolve the right thing.
rayuela 11y ago
This is what Deepmind seems to dedicate most of their efforts on and what their new AlphaGo models is a great example of (NN + tree search Reinforcement Learning). Pedro Domingo's thesis in his new book is the potential for the progress to be made with the combination of distinct schools within the field of ML
devonkim 11y ago
This kind of unintended learning is something that all machine learning practitioners should be careful with. The example of the satellite photo analysis algorithm that turned out to have learned to classify difference between night and day photos comes to mind. This means that your particular reinforcement learning example needs something similar to regularization of cost functions or cross validation for checking for over or under fitting to correct these latency effects. It may be cheaper and more effective to use SSDs instead but the magnitude of the latency effects on your case are not clear either to determine if your algorithm would benefit substantially from a modern SSD.
- api 11y ago
  The fix was to (1) take steps to make sure the image was entirely pre-loaded into RAM before the test was given, and (2) randomize the latency of the load. That worked and I actually got some real results, which unfortunately were decent but not as impressive as the cheating was. :) When it cheated the thing got 100% right of course. I also learned that rapid convergence on incredible accuracy is suspect.

eli_gottlieb 11y ago

Reinforcement learning involves low-information objective signals. Hence, it sounds "universal" but doesn't actually work very well. https://drive.google.com/file/d/0BxKBnD5y2M8NbWN6XzM5UXkwNDA/view?pli=1 https://drive.google.com/file/d/0BxKBnD5y2M8NbWN6XzM5UXkwNDA...