4 ms·
"A single neural net policy operating directly from a camera image, trained in simulation with large scale RL, can overcome imprecise sensing and actuation to o
by Animats 3y ago
"A single neural net policy operating directly from a camera image, trained in simulation with large scale RL, can overcome imprecise sensing and actuation to output highly precise control behavior end-to-end."
So that's what it takes. That's so much simpler than the way Boston Dynamics does it, working out all the dynamics in simulation first. It's amazing to see this done from vision to actuators in one net. It's now much less of a mystery how animals managed to evolve the ability to do that.
(I had a go at this problem years ago. I was trying to work out the dynamics from first principles, knowing it would be somewhat off. Then use some self-tuning (a predecessor to machine learning) to fine-tune the thing. Got as far as running up and downhill in 2D in the early 1990s. About one hour of compute for one second of motion.)
- ShamelessC 3y agoDoesn’t the use of RL here imply some level of simulation?
- gwern 3y agoYes, but the key is the RL simulation can be simple & unrealistic since it's used to train offline, not to plan & control live: a control systems approach like the Boston Dynamics requires excellent physics modeling because that's the only way it can do planning or adjust for errors while controlling the actual robot in real time, while the NN is trained to cover many possible physics during development, and at runtime in an actual robot, it just shrugs and adapts to whatever crazy scenario it finds itself in.
- teraflop 3y agoOf course, the details of how to actually implement something like this are way more complex than "just throw everything into a big neural net with images as the inputs and actuators as the output". You need to provide the right kind of guidance in order to learn a usable policy in any reasonable amount of time. A very recent development (which this work builds on) is the idea of "online adaptation". It essentially involves doing the training in two stages: 1. You add a variety of dynamically-varying environmental effects to your simulator, by randomly altering parameters such as ground friction, payload weight distribution, motor effectiveness, and so on. You give the motion controller perfect knowledge of these parameters at all times, and let it learn how to move in response to them. 2. Then, you remove the oracle that tells the controller about the current environmental parameters, and replace it with another neural network that is trained to estimate (a latent representation of) those parameters, based on a very short window of data about the robot's own motor commands and the actual motion that resulted from it. All of this can be done in simulation, many times faster than real-time. But when you transfer the system to a real robot, it adapts to its environment using the estimated parameters, without any of the networks needing to be re-trained. This ends up making it pretty robust to difficult terrain and perturbations. It also has the benefit of papering over subtle differences that arise between the simulated and real-world dynamics. This paper adds a lot of additional refinements to the same basic idea. In the first stage, the system is given perfect knowledge of its surrounding terrain and the locations of some preselected waypoints, and learns to follow them. The second stage replaces those inputs with estimates derived from an RGB+depth camera.
- jupp0r 3y agoI guess this is why evolution is taking unreasonable amounts of time.
- ChatGTP 3y agoAssuming time has no end, what would be a reasonable amount of time ? Let’s say you could evolve at 10000x the speed ? So what ? Where do you need to be ?
- i_am_a_peasant 3y agoWell at the very least you need to evolve at a rate which can keep up with environmental changes.
- pyrale 3y ago> Where do you need to be ? Where is debatable, but you definitely need to be there by end of Q3 or we miss P&E targets.
- taylodl 3y agoOut of the solar system by the time our Sun dies
- ChatGTP 3y ago7/8 billion years...
- taylodl 3y agoEarth's ability to support lifeforms such as human beings is expected to end in the next 500 million to 2 billion years. Granted, that's several orders of magnitude longer than we've been around, but still - the sun will still be shining when humans go extinct, assuming we don't find another home planet and manage to get ourselves to it. It will be billions of years afterward that the sun will actually die.
- andrepd 3y agoThat second paragraph: holy mother of exaggerated claims!
- michaelt 3y ago> "A single neural net policy operating directly from a camera image, trained in simulation with large scale RL, [...] > So that's what it takes. That's so much simpler than the way Boston Dynamics does it, working out all the dynamics in simulation first. I mean, it sounds like they have had to work out all the dynamics, and simulate them. Admittedly they don't have to do it on-robot in real time any more.