9 ms·

Sometimes it's hard to separate signal from noise when you're not part of a field and just hearing about projects/papers, so I wanted to quickly pitch in to say

Sometimes it's hard to separate signal from noise when you're not part of a field and just hearing about projects/papers, so I wanted to quickly pitch in to say that this is a legitimately ground-breaking approach and line of work that you can expect to hear much more about in the future. It's probably the most exciting robotics/manipulation project I'm currently aware of. What's exciting here is that the entire system is trained end-to-end (including the vision component). In other words, it's heading towards agents/robots that consist entirely of a single neural net and that's it; There is no software stack at all - it's just a GPU running a neural net "code base", from perception to actuators. In this respect this work is similar to the Atari game-playing agent that has to learn to see while also learning to play the game. Except this setting is quite a lot more difficult in some respects; In particular, the actions in the Deepmind Atari paper are few and discrete, while here the robot is an actual physical system with a very large-dimensional and continuous action space (joint torques). Also, if you're new to the field you might think "why is the robot so slow?", while someone in the field is thinking "holy crap how can it be so fast?"

zo1 11y ago
Your comment regarding the training of the NN to perceive and action the joint actuators reminded me of an old software project that's remotely related: http://www.framsticks.com/ http://www.framsticks.com/ It's an ambitious project that wanted to "Evolve" agents made of sticks + joints that ran in a 3d world with physics. The joints had signal inputs + activation functions that are similar to AN activation functions.
Intermernet 11y ago
Thanks for confirming what my untrained SNR filters had flagged (seemingly somewhat correctly) as an incredible advance. The video was reminiscent of watching a child learn. This technique seems to be the best bet of all the machine learning techniques to be solvable by Moore's law. If it currently takes about three hours to learn to do these simple tasks with no previous spatial data for the objects, then as the article states "In the next five to 10 years, we may see significant advances in robot learning capabilities through this line of work."
- rasz_pl 11y ago
  >video was reminiscent of watching a child learn Yes! this story and yesterdays RNN one with this comment https://news.ycombinator.com/item?id=9584988 https://news.ycombinator.com/item?id=9584988 "It's like a child learning to talk" make me believe we are very close to some kind of breakthrough using "boring" non-magical methods.
mbq 11y ago
The core problem with the "just a neural net" idea is how to tell a robot what it is supposed to do; the whole AI currently relays on some sort of external harness either training the model or pushing the control algorithm into a desired operation. You may implement something very thin like artificial pain or hunger, but the odds that it will push the system out of a metastable stupor are likely negligible.
jacquesm 11y ago
Any ideas on how the interaction shown here is directed, in other words what defines the 'goals' and what motivates the robot to perform the task?
- nl 11y ago
  I don't know enough robotics to follow the paper[1] entirely, but this seems the relevant part for your question: We evaluated our method by training policies for hanging a coat hanger on a clothes rack, inserting a block into a shape sorting cube, fitting the claw of a toy hammer under a nail with various grasps, and screwing on a bottle cap. The cost function for these tasks encourages low distance between three points on the end-effector and corresponding target points, low torques, and, for the bottle task, spinning the wrist. The equations for these cost functions follow prior work.[23] The reference 23 points at S. Levine, N. Wagener, and P. Abbeel. Learning contactrich manipulation skills with guided policy search. In International Conference on Robotics and Automation(ICRA), 2015. I haven't read that paper yet. Interestingly, they initialise the visual learning model using the ImageNet images. Was it 3 years ago that was considered a pretty much intractable problem, and now the fact a CNN can work on it well enough to be useful isn't even worth a complete sentence. [1] http://arxiv.org/pdf/1504.00702v1.pdf http://arxiv.org/pdf/1504.00702v1.pdf
tormeh 11y ago
Neuroscience suggests that the brain has many fixed-purpose computational units, some neural nets and others not. While doing everything in a single network is indeed impressive, I am concerned about whether this is the most efficient approach. EDIT: Downvoter, care to explain the reason for downvoting?
- spott 11y ago
  >some neural nets and others not. Source for the brain having computational units that aren't neural nets? I'd love to read more on this.
  - tormeh 11y ago
    "In vertebrates, inter-aural time differences are known to be calculated in the superior olivary nucleus of the brainstem. According to Jeffress,[1] this calculation relies on delay lines: neurons in the superior olive which accept innervation from each ear with different connecting axon lengths." http://en.wikipedia.org/wiki/Sound_localization http://en.wikipedia.org/wiki/Sound_localization
    - blennon 11y ago
      How is that not a neural network? This system is basically exploiting the property of action potential propagation down the axon to capture relative timing differences. As an aside, there are a number of useful properties seen in biological neural networks that aren't yet incorporated into multi-layer perceptrons. E.g. short-term plasticity, axonal delays, spiking neurons, etc. I expect that some of these will find their way into the MLP formulation when we can figure out an effective mathematical way of doing so.
      - tormeh 11y ago
        I mean that it's not a neural network in the computational sense. Sorry about the terminology issue.
- rndn 11y ago
plq 11y ago
Here's the same guy (PR2) in action 7 years ago folding towels at 50x speed: https://www.youtube.com/watch?v=gy5g33S0Gzo https://www.youtube.com/watch?v=gy5g33S0Gzo And I doubt it was trained. It seems to be following a predefined procedure instead of learning by itself. It seems to have come a pretty long way if you ask me!
- sdavidmiller 11y ago
  As one of the ex-leads on the PR2 laundry folding project, I can confirm that Pieter's group has come up with a framework that totally blows everything we used to do out of the water.
murbard2 11y ago
Do you see a plausible architecture for long term planning? Putting money in my 401(k) every month to get it a few decades from now isn't something that's accessible to Q-learning, direct policy search or forward simulations. How do we learn hierarchical / semantic representations of our own actions?
- blennon 11y ago
  I think a similar problem needs to be solved in the language generation neural networks. While generated sentences often are formed syntactically correct, they don't make a lot of sense -- there's no over arching thought that it is trying to express. This is even more apparent when you read the subsequent sentences generated. It's clear there's no conceptual goal it's trying to convey.
  - murbard2 11y ago
    In principle, adding more depth to the network can take care of that. If you are keeping track of higher and higher level abstractions, you can remember what you were saying. But you can't remember everything, so ideally you'd also have an attention model that's capable of looking back at what you've written, and make edits. When I balance parentheses, I don't maintain a counter of how many parentheses are open, I look back at what I've written and count them.
cs702 11y ago
Andrej -- thank you.[1] What struck me the most is this number: 92,000. That's the total number of parameters in the neural net guiding the robot. In other words, the robot learned to do this with a tiny toy neural net! -- [1] Thank you not just for this helpful explanatory comment, but also for all the friendly explanatory lectures, presentations, blog posts and open-source code you have shared online over the past few years. You deserve recognition for it.
curiously 11y ago
is the robot possible with the RNN you wrote about earlier? Would love to see youtube channel describing these deep learning /machine learning / AI subjects.
joelthelion 11y ago
Thanks a lot for your comment. Are there any prospects for integrating neural nets with more traditional code? For all their power, neural nets share the limitations of humans: they learned behaviors are approximate and not perfect. So for example if your deep learning robot wanted to play chess, it would be nice to have him switch to using a chess engine instead of learning to play chess. Or you might want to hardcode a few very efficient moves into your otherwise autonomous industrial robot. Is that possible?

zo1 11y ago

Your comment regarding the training of the NN to perceive and action the joint actuators reminded me of an old software project that's remotely related: http://www.framsticks.com/ http://www.framsticks.com/ It's an ambitious project that wanted to "Evolve" agents made of sticks + joints that ran in a 3d world with physics. The joints had signal inputs + activation functions that are similar to AN activation functions.

Intermernet 11y ago

Thanks for confirming what my untrained SNR filters had flagged (seemingly somewhat correctly) as an incredible advance. The video was reminiscent of watching a child learn. This technique seems to be the best bet of all the machine learning techniques to be solvable by Moore's law. If it currently takes about three hours to learn to do these simple tasks with no previous spatial data for the objects, then as the article states "In the next five to 10 years, we may see significant advances in robot learning capabilities through this line of work."

rasz_pl 11y ago
>video was reminiscent of watching a child learn Yes! this story and yesterdays RNN one with this comment https://news.ycombinator.com/item?id=9584988 https://news.ycombinator.com/item?id=9584988 "It's like a child learning to talk" make me believe we are very close to some kind of breakthrough using "boring" non-magical methods.

mbq 11y ago

The core problem with the "just a neural net" idea is how to tell a robot what it is supposed to do; the whole AI currently relays on some sort of external harness either training the model or pushing the control algorithm into a desired operation. You may implement something very thin like artificial pain or hunger, but the odds that it will push the system out of a metastable stupor are likely negligible.

jacquesm 11y ago

Any ideas on how the interaction shown here is directed, in other words what defines the 'goals' and what motivates the robot to perform the task?

nl 11y ago
I don't know enough robotics to follow the paper[1] entirely, but this seems the relevant part for your question: We evaluated our method by training policies for hanging a coat hanger on a clothes rack, inserting a block into a shape sorting cube, fitting the claw of a toy hammer under a nail with various grasps, and screwing on a bottle cap. The cost function for these tasks encourages low distance between three points on the end-effector and corresponding target points, low torques, and, for the bottle task, spinning the wrist. The equations for these cost functions follow prior work.[23] The reference 23 points at S. Levine, N. Wagener, and P. Abbeel. Learning contactrich manipulation skills with guided policy search. In International Conference on Robotics and Automation(ICRA), 2015. I haven't read that paper yet. Interestingly, they initialise the visual learning model using the ImageNet images. Was it 3 years ago that was considered a pretty much intractable problem, and now the fact a CNN can work on it well enough to be useful isn't even worth a complete sentence. [1] http://arxiv.org/pdf/1504.00702v1.pdf http://arxiv.org/pdf/1504.00702v1.pdf

tormeh 11y ago

Neuroscience suggests that the brain has many fixed-purpose computational units, some neural nets and others not. While doing everything in a single network is indeed impressive, I am concerned about whether this is the most efficient approach. EDIT: Downvoter, care to explain the reason for downvoting?

spott 11y ago
>some neural nets and others not. Source for the brain having computational units that aren't neural nets? I'd love to read more on this.
- tormeh 11y ago
  "In vertebrates, inter-aural time differences are known to be calculated in the superior olivary nucleus of the brainstem. According to Jeffress,[1] this calculation relies on delay lines: neurons in the superior olive which accept innervation from each ear with different connecting axon lengths." http://en.wikipedia.org/wiki/Sound_localization http://en.wikipedia.org/wiki/Sound_localization
  - blennon 11y ago
    How is that not a neural network? This system is basically exploiting the property of action potential propagation down the axon to capture relative timing differences. As an aside, there are a number of useful properties seen in biological neural networks that aren't yet incorporated into multi-layer perceptrons. E.g. short-term plasticity, axonal delays, spiking neurons, etc. I expect that some of these will find their way into the MLP formulation when we can figure out an effective mathematical way of doing so.
    - tormeh 11y ago
      I mean that it's not a neural network in the computational sense. Sorry about the terminology issue.
rndn 11y ago

plq 11y ago

Here's the same guy (PR2) in action 7 years ago folding towels at 50x speed: https://www.youtube.com/watch?v=gy5g33S0Gzo https://www.youtube.com/watch?v=gy5g33S0Gzo And I doubt it was trained. It seems to be following a predefined procedure instead of learning by itself. It seems to have come a pretty long way if you ask me!

sdavidmiller 11y ago
As one of the ex-leads on the PR2 laundry folding project, I can confirm that Pieter's group has come up with a framework that totally blows everything we used to do out of the water.

murbard2 11y ago

Do you see a plausible architecture for long term planning? Putting money in my 401(k) every month to get it a few decades from now isn't something that's accessible to Q-learning, direct policy search or forward simulations. How do we learn hierarchical / semantic representations of our own actions?

blennon 11y ago
I think a similar problem needs to be solved in the language generation neural networks. While generated sentences often are formed syntactically correct, they don't make a lot of sense -- there's no over arching thought that it is trying to express. This is even more apparent when you read the subsequent sentences generated. It's clear there's no conceptual goal it's trying to convey.
- murbard2 11y ago
  In principle, adding more depth to the network can take care of that. If you are keeping track of higher and higher level abstractions, you can remember what you were saying. But you can't remember everything, so ideally you'd also have an attention model that's capable of looking back at what you've written, and make edits. When I balance parentheses, I don't maintain a counter of how many parentheses are open, I look back at what I've written and count them.

cs702 11y ago

Andrej -- thank you.[1] What struck me the most is this number: 92,000. That's the total number of parameters in the neural net guiding the robot. In other words, the robot learned to do this with a tiny toy neural net! -- [1] Thank you not just for this helpful explanatory comment, but also for all the friendly explanatory lectures, presentations, blog posts and open-source code you have shared online over the past few years. You deserve recognition for it.

curiously 11y ago

is the robot possible with the RNN you wrote about earlier? Would love to see youtube channel describing these deep learning /machine learning / AI subjects.

joelthelion 11y ago

Thanks a lot for your comment. Are there any prospects for integrating neural nets with more traditional code? For all their power, neural nets share the limitations of humans: they learned behaviors are approximate and not perfect. So for example if your deep learning robot wanted to play chess, it would be nice to have him switch to using a chess engine instead of learning to play chess. Or you might want to hardcode a few very efficient moves into your otherwise autonomous industrial robot. Is that possible?