6 ms·

To me, this entire piece reads like goal-post moving and lacks understanding of the field and was incredibly premature. > I am confused as to why they continue

To me, this entire piece reads like goal-post moving and lacks understanding of the field and was incredibly premature. > I am confused as to why they continue to spend so much time on building Deep RL systems that beat games with clearly defined rules and point systems. I think everyone gets it now, with enough time, money, and computers almost brute forcing every single possible action, Deep RL can beat almost any game. Right, before AlphaGo, superhuman Go was decades away. Then it became obvious and therefore easy. He brings up a laundry folding robot almost as if discrete games can be trivially applied to robotics. If you think about it for one second, Shogi, the game that AlphaZero solves with the largest action space has 10^5 actions. Consider a robot with 3 joints that can move in any dimension motors. Discretizing the continuous action space into 10 buckets yields an action space of 10^9. Something AlphaZero will never solve. In that sense, the entire analogy is a straw man and everyone in the field knows. This doesn't even touch the fact that 10 buckets would be useless for fine motor control. Since they published this, there have been great advancements in learning action embeddings for robotic tasks. With a couple of expert examples, an algorithm can learn to open a door effectively. That's not that far from folding laundry. Personally, my entire PhD thesis attempts to solves the exact deficiencies they raise. Simply put, the work goes on. Instead of saying that the field is useless, try to understand and contribute to it. At the very least, give technical reasons that it is a dead end path.

lysozyme 5y ago
That’s a really interesting way of looking at the difficulty of the problem that’s being solved. I’m curious, how do you arrive at the number 10^9?
- hervature 5y ago
  You have 3 joints with 3 dimensions. So 9 variables you can control. If each variable has 10 choices, and you need to make a choice for each, that is 10^9 possible combinations.
  - whimsicalism 5y ago
    "3 joints" -> What are the 3 dimensions that joints move along? I naively would have thought one or two (ie. how much spin on one axis and how much spin on the other axis) just by naively looking at my elbow for a second. For an arm of fixed length, polar coordinates have 2 dof.
    - hervature 5y ago
      Yes, there are multiple types of joints. In this case, you could think of a ball joint like a shoulder or hip. They move "up-down" and "left-right" but also rotate.
- phunehehe0 5y ago
  I presume it's 3 dimensions for each joint, so 10^3 * 10^3 * 10^3
  - liuzhy71 5y ago
    I think the problem is exaggerated. Even with three ball joints, the action space is not that large since there are constraints on the velocity of the joints. They have to move gradually. So the actual action space is a lot smaller. A lot RL problem has similar continuity constraints, cuz in real world we are dealing with time series signals. I am not a expert in the RL domain yet, so I open myself to any opinion.
- throwawaysea 5y ago
  The GP specified a robot with three joints in his illustrative example. Each joint has three axes (physical dimensions) along which it can move. Each movement has an arbitrary scale of movement of 0-10, since the GP discretized continuous space into 10 buckets arbitrarily for sake of illustration. So 10 choices for each of 3 joints for each of 3 dimensions. That comes out to 10 choices for Joint 1 Dimension 1 x 10 choices for Joint 1 Dimension 2 x … 10 choices for Joint 3 Dimension 3 which comes out to 10^9.
YeGoblynQueenne 5y ago
>> With a couple of expert examples, an algorithm can learn to open a door effectively. "A door". How about "any door"? I'm curious- do you have a link to a demonstration or something similar?
- hervature 5y ago
  The algorithm is called LASER: https://arxiv.org/pdf/2103.15793v2.pdf https://arxiv.org/pdf/2103.15793v2.pdf Again, this is goal-post moving. Just because we are not there yet, it does not mean it will not happen. We just solved it for one door, now we need to generalize.
  - YeGoblynQueenne 5y ago
    Why do you have to start your comment by accusing me of moving the goal posts? I don't even like ball games. Why is it so hard to have a decent conversation on the internet? It's a serious concern and I stand by it. It's perfectly possible -it's even easy- to make algorithms, systems, programs, what have you, that can solve one problem, perform one task, and do nothing else. Hell, you can train a dog to perform one task. Sit boy! Boy sits. And that's all it knows to do. 80% of AI research output, ever since Dartmouth, has been like that. 99% of AI research output in the last ten years is like that. My entire phylum of AI research (I'm a PhD student studying Inductive Logic Programming and even knowing nothing about that, the name should give you an idea) is roundly dismissed today as a "failure" because it couldn't get from "we can solve an instance of this problem" to "we can solve any instance of this problem" quickly enough. Researchers promised it was just a matter of "generalising" like you say, tuning and tuning until everything worked in the real world as well as it worked in blocks worlds, but the sources of funding didn't listen and then we had a winter [1]. Why is what you're doing any different when you look at it from the present? We haven't seen the future yet. In the present, you have a robot that can open one door. In the future, you have a lot of doors that wait for robots to open them. Actually, looking at your paper, you don't even have a robot that can open one door. You have a simulated robot opening a simulated door. Talk about moving the goal posts! Have you seen the Pythons' silly walk sketch? "It's not particularly silly, is it?" "Yes, I think with government backing I can make it very silly" [2]. That's all you got there so far. Edit: Sorry, it's probably not your paper. But it's as far from folding laundry as you can go. I don't even see the simulated robot folding simulated laundry anywhere yet and even "simulated cloth" is a joke - the simulation itself (of the behaviour of cloth) is not even close to the real world. Simulating a robot opening a simulated door might get your paper published, but simulating a robot folding simulated shirts won't even get you a simulated paper. ______________ [1] https://www.youtube.com/watch?v=yReDbeY7ZMU https://www.youtube.com/watch?v=yReDbeY7ZMU [2] https://youtu.be/eCLp7zodUiI?t=83 https://youtu.be/eCLp7zodUiI?t=83

lysozyme 5y ago

That’s a really interesting way of looking at the difficulty of the problem that’s being solved. I’m curious, how do you arrive at the number 10^9?

hervature 5y ago
You have 3 joints with 3 dimensions. So 9 variables you can control. If each variable has 10 choices, and you need to make a choice for each, that is 10^9 possible combinations.
- whimsicalism 5y ago
  "3 joints" -> What are the 3 dimensions that joints move along? I naively would have thought one or two (ie. how much spin on one axis and how much spin on the other axis) just by naively looking at my elbow for a second. For an arm of fixed length, polar coordinates have 2 dof.
  - hervature 5y ago
    Yes, there are multiple types of joints. In this case, you could think of a ball joint like a shoulder or hip. They move "up-down" and "left-right" but also rotate.
phunehehe0 5y ago
I presume it's 3 dimensions for each joint, so 10^3 * 10^3 * 10^3
- liuzhy71 5y ago
  I think the problem is exaggerated. Even with three ball joints, the action space is not that large since there are constraints on the velocity of the joints. They have to move gradually. So the actual action space is a lot smaller. A lot RL problem has similar continuity constraints, cuz in real world we are dealing with time series signals. I am not a expert in the RL domain yet, so I open myself to any opinion.
throwawaysea 5y ago
The GP specified a robot with three joints in his illustrative example. Each joint has three axes (physical dimensions) along which it can move. Each movement has an arbitrary scale of movement of 0-10, since the GP discretized continuous space into 10 buckets arbitrarily for sake of illustration. So 10 choices for each of 3 joints for each of 3 dimensions. That comes out to 10 choices for Joint 1 Dimension 1 x 10 choices for Joint 1 Dimension 2 x … 10 choices for Joint 3 Dimension 3 which comes out to 10^9.

YeGoblynQueenne 5y ago

>> With a couple of expert examples, an algorithm can learn to open a door effectively. "A door". How about "any door"? I'm curious- do you have a link to a demonstration or something similar?

hervature 5y ago
The algorithm is called LASER: https://arxiv.org/pdf/2103.15793v2.pdf https://arxiv.org/pdf/2103.15793v2.pdf Again, this is goal-post moving. Just because we are not there yet, it does not mean it will not happen. We just solved it for one door, now we need to generalize.
- YeGoblynQueenne 5y ago
  Why do you have to start your comment by accusing me of moving the goal posts? I don't even like ball games. Why is it so hard to have a decent conversation on the internet? It's a serious concern and I stand by it. It's perfectly possible -it's even easy- to make algorithms, systems, programs, what have you, that can solve one problem, perform one task, and do nothing else. Hell, you can train a dog to perform one task. Sit boy! Boy sits. And that's all it knows to do. 80% of AI research output, ever since Dartmouth, has been like that. 99% of AI research output in the last ten years is like that. My entire phylum of AI research (I'm a PhD student studying Inductive Logic Programming and even knowing nothing about that, the name should give you an idea) is roundly dismissed today as a "failure" because it couldn't get from "we can solve an instance of this problem" to "we can solve any instance of this problem" quickly enough. Researchers promised it was just a matter of "generalising" like you say, tuning and tuning until everything worked in the real world as well as it worked in blocks worlds, but the sources of funding didn't listen and then we had a winter [1]. Why is what you're doing any different when you look at it from the present? We haven't seen the future yet. In the present, you have a robot that can open one door. In the future, you have a lot of doors that wait for robots to open them. Actually, looking at your paper, you don't even have a robot that can open one door. You have a simulated robot opening a simulated door. Talk about moving the goal posts! Have you seen the Pythons' silly walk sketch? "It's not particularly silly, is it?" "Yes, I think with government backing I can make it very silly" [2]. That's all you got there so far. Edit: Sorry, it's probably not your paper. But it's as far from folding laundry as you can go. I don't even see the simulated robot folding simulated laundry anywhere yet and even "simulated cloth" is a joke - the simulation itself (of the behaviour of cloth) is not even close to the real world. Simulating a robot opening a simulated door might get your paper published, but simulating a robot folding simulated shirts won't even get you a simulated paper. ______________ [1] https://www.youtube.com/watch?v=yReDbeY7ZMU https://www.youtube.com/watch?v=yReDbeY7ZMU [2] https://youtu.be/eCLp7zodUiI?t=83 https://youtu.be/eCLp7zodUiI?t=83