3 ms·

I've noticed all the top performing AI reinforcement algorithms i hear about know next to nothing about the initial rules. And not only do they perform as well

by bko 6y ago

I've noticed all the top performing AI reinforcement algorithms i hear about know next to nothing about the initial rules. And not only do they perform as well as more supervised methods, but much better The one exception is self driving. I listened to the Lex Fridman interview with ceo of waymo recently and he made a case for the controlled environment (e.g. separate detection from decision making and planning) and pushed back against the end to end approach that doesn't make any preconceived assumptions about the environment. As an example he takes red lights. They're clearly human engineered signals, so it makes sense to have a module that can explicitly determine the signal as opposed to learning the behavior But that's true about other games as well and end to end methods still outperform. Which makes me ask, is end to end learning an inevitability for self driving as well or is this the one domain special due to complexity or other aspects?

philipkglass 6y ago
A machine controlling a real car gathers feedback no faster than real time and cannot afford to learn the meanings of street signs from the consequences of ignoring them. A machine can learn the rules of Atari games from scratch by playing them orders of magnitude faster than real time and treating "death" as one signal among many. In order for a machine to learn driving the same way it learns Atari games, it seems that it would need an extremely high fidelity virtual environment to learn in. The high fidelity requirement would necessitate a lot of up-front investment in trying to get the simulations right. You might spend a whole career just trying to build a drivable Virtual Philadelphia as challenging as the real thing. The details would also make it much more expensive to run training sessions at high multiples of real time. Given those factors, I'm not surprised that self-driving vehicle experiments just use real environments and don't try to learn the fundamental rules from scratch. But it's an interesting point that these choices may make it harder for agents to keep improving.
- bko 6y ago
  There's a middle ground too which alpha go leveraged which bootstrapped learning from actual human gameplay predicting user actions. That's what comma.ai does and AlphaGo still performed significantly better than explicit rules or higher level abstractions. That and a mix of simulations might yield better results
- ars 6y ago
  If you could make a virtual environment that was that good, you would have already solved the self driving problem. I wonder if Google would be willing to pay people to add cameras to their cars to collect real world data in far larger scale.
  - gaudat 6y ago
    That might be happening now already. I think Google sourced their dataset from dashcam videos uploaded to YouTubr?
  - samatman 6y ago
    I do believe that's the basis of Tesla's play in the space: a bet that enough cameras collecting real-world data can beat out a dedicated from-scratch self-driving system in deployment. Whether this will work remains very much an open question.
- martamorena284 6y ago
  Really not true. We have all the tools available... Just model a city in Unreal Engine 4 (GTA 6 anyone?). Your sensors, like LIDAR could do ray-queries, cameras could render views, etc. It's not 100% real, but probably real enough to learn the basics. Photorealistic graphics should be sufficient for AI to learn real-world interaction. In the end our whole world might be just that, a simulation. So I see zero reason why we couldn't train a self-driving car AI in one. And the coolest thing is that this could also be used as a basis for a AAA video game, and these tend to make billions as well these days, so it's a win win for everyone. AI companies with funding should invest heavily into virtual reality and gaming, because they will need to perfect this to train their models.
  - mckirk 6y ago
    The problem is that these simulations are far from perfect, and AIs are great at exploiting paths-of-least-resistance. So you'd end up with an AI that could drive through your virtual city flawlessly by overfitting on some cues that you wouldn't even notice, but are reliable enough because of the necessarily lower complexity of the simulation. It wouldn't translate to the real world; unless maybe you add enough noise to the simulation to prevent the AI from using too simple cues, but at that point, it's questionable how much insight the AI could still distill from the simulation.
- Shorel 6y ago
  In that case, I would like to see what the AI can learn in Euro Truck Simulator 2.
grandmczeb 6y ago
Minor tangent, but I think you’re referring to this video[1] with Dmitri Dolgov. He’s the CTO of Waymo, not CEO. [1] https://youtu.be/P6prRXkI5HM https://youtu.be/P6prRXkI5HM

by bko 6y ago

philipkglass 6y ago

A machine controlling a real car gathers feedback no faster than real time and cannot afford to learn the meanings of street signs from the consequences of ignoring them. A machine can learn the rules of Atari games from scratch by playing them orders of magnitude faster than real time and treating "death" as one signal among many. In order for a machine to learn driving the same way it learns Atari games, it seems that it would need an extremely high fidelity virtual environment to learn in. The high fidelity requirement would necessitate a lot of up-front investment in trying to get the simulations right. You might spend a whole career just trying to build a drivable Virtual Philadelphia as challenging as the real thing. The details would also make it much more expensive to run training sessions at high multiples of real time. Given those factors, I'm not surprised that self-driving vehicle experiments just use real environments and don't try to learn the fundamental rules from scratch. But it's an interesting point that these choices may make it harder for agents to keep improving.

bko 6y ago
There's a middle ground too which alpha go leveraged which bootstrapped learning from actual human gameplay predicting user actions. That's what comma.ai does and AlphaGo still performed significantly better than explicit rules or higher level abstractions. That and a mix of simulations might yield better results
ars 6y ago
If you could make a virtual environment that was that good, you would have already solved the self driving problem. I wonder if Google would be willing to pay people to add cameras to their cars to collect real world data in far larger scale.
- gaudat 6y ago
  That might be happening now already. I think Google sourced their dataset from dashcam videos uploaded to YouTubr?
- samatman 6y ago
  I do believe that's the basis of Tesla's play in the space: a bet that enough cameras collecting real-world data can beat out a dedicated from-scratch self-driving system in deployment. Whether this will work remains very much an open question.
martamorena284 6y ago
Really not true. We have all the tools available... Just model a city in Unreal Engine 4 (GTA 6 anyone?). Your sensors, like LIDAR could do ray-queries, cameras could render views, etc. It's not 100% real, but probably real enough to learn the basics. Photorealistic graphics should be sufficient for AI to learn real-world interaction. In the end our whole world might be just that, a simulation. So I see zero reason why we couldn't train a self-driving car AI in one. And the coolest thing is that this could also be used as a basis for a AAA video game, and these tend to make billions as well these days, so it's a win win for everyone. AI companies with funding should invest heavily into virtual reality and gaming, because they will need to perfect this to train their models.
- mckirk 6y ago
  The problem is that these simulations are far from perfect, and AIs are great at exploiting paths-of-least-resistance. So you'd end up with an AI that could drive through your virtual city flawlessly by overfitting on some cues that you wouldn't even notice, but are reliable enough because of the necessarily lower complexity of the simulation. It wouldn't translate to the real world; unless maybe you add enough noise to the simulation to prevent the AI from using too simple cues, but at that point, it's questionable how much insight the AI could still distill from the simulation.
Shorel 6y ago
In that case, I would like to see what the AI can learn in Euro Truck Simulator 2.

grandmczeb 6y ago

Minor tangent, but I think you’re referring to this video[1] with Dmitri Dolgov. He’s the CTO of Waymo, not CEO. [1] https://youtu.be/P6prRXkI5HM https://youtu.be/P6prRXkI5HM