4 ms·

Selecting one of a set of pre-trained actions by voice is cool but not exactly ground-breaking. Using GPT-4V to describe a scene is also pretty simple. The most

Selecting one of a set of pre-trained actions by voice is cool but not exactly ground-breaking. Using GPT-4V to describe a scene is also pretty simple. The most impressive things here to me are the speed of picking up the trash and the fluid passing of objects between hands. It's unclear how general these movement policies are though. The way that guy is standing perfectly still makes me think that it would fail if everything wasn't set up just so. I'd like to see demos with more variation. I don't want to be too negative here though. I think it's a great demo and I can't wait to see more.

fragmede 3y ago
The Ok-robot demo shows that the technology for it to be fairly general is there, though no idea if figure one is using their technology or not. Simply being able to command a robot instead of moving a turtle with gcode is nothing short of astounding to those who aren’t deeply involved and tracking the sota progress in this area. https://ok-robot.github.io/ https://ok-robot.github.io/
Animats 3y ago
> Selecting one of a set of pre-trained actions by voice is cool but not exactly ground-breaking. Yes. Compare "Put That There" (1979).[1] > The way that guy is standing perfectly still makes me think that it would fail if everything wasn't set up just so. I'd like to see demos with more variation. Yes. Unstructured manipulation is hard. Structured robotic manipulation is pretty standard. Picking isolated objects is a solved problem. Here's a robot recycling sorting system, "Max-AI".[2] That's been in use for years. San Francisco recycling uses those robots. So do many other cities. (That's from "Bulk Handling Systems", a company which does exactly what their name says. Recycling and trash come in bulk, and their machines handle it. Shakers, magnets, screens, and vision-based air sorters do 95% of the sorting. The robots only handle the hard cases. This is the no-bullshit end of AI.) [1] https://www.youtube.com/watch?v=RyBEUyEtxQo https://www.youtube.com/watch?v=RyBEUyEtxQo [2] https://max-ai.com/ https://max-ai.com/
KoolKat23 3y ago
I suspect it's less pre-trained than you think, the tweet is prefaced with a message that all actions are driven by neural networks indicating that it's probably adjusting for the objects, the environment etc.
- tintor 3y ago
  It is safe to assume it works only for that specific demo, and nothing else.
- modeless 3y ago
  I think it's true that it can adjust, somewhat. I expect that it could handle slight variations such as moving any of the items on the table around by a few inches, or adding or removing a couple of plates from the rack. However, I do not expect that it could handle larger variations like replacing the apple with a pineapple, picking trash out of the dish rack or the cup instead of the plate, replacing the cup with a coffee mug, or sliding the dish rack over to the other side of the robot. I'd love to be wrong but I expect that if they had that much flexibility in their controllers they would have demonstrated it.
  - KoolKat23 3y ago
    I'm sure there is definitely some limitation in this demo, but who knows, Google RT-2 exists after all. Enough shapes and object training will save the day, there's still objects me as a human would be new to handling.
    - modeless 3y ago
      Oh for sure these limitations will fall. It's only a matter of time (and a lot of work behind the scenes). Covariant's RFM-1 (announced Monday) is an interesting approach to generality. https://www.youtube.com/watch?v=1Go6HEC-bYU https://www.youtube.com/watch?v=1Go6HEC-bYU
fennecbutt 3y ago
At 1:37 in the video you can see it react by jerking its hand after the plate wobbles a bit. At 0:59 it also shifts the basket out of the way as a separate action, after its already put it down.

fragmede 3y ago

The Ok-robot demo shows that the technology for it to be fairly general is there, though no idea if figure one is using their technology or not. Simply being able to command a robot instead of moving a turtle with gcode is nothing short of astounding to those who aren’t deeply involved and tracking the sota progress in this area. https://ok-robot.github.io/ https://ok-robot.github.io/

Animats 3y ago

> Selecting one of a set of pre-trained actions by voice is cool but not exactly ground-breaking. Yes. Compare "Put That There" (1979).[1] > The way that guy is standing perfectly still makes me think that it would fail if everything wasn't set up just so. I'd like to see demos with more variation. Yes. Unstructured manipulation is hard. Structured robotic manipulation is pretty standard. Picking isolated objects is a solved problem. Here's a robot recycling sorting system, "Max-AI".[2] That's been in use for years. San Francisco recycling uses those robots. So do many other cities. (That's from "Bulk Handling Systems", a company which does exactly what their name says. Recycling and trash come in bulk, and their machines handle it. Shakers, magnets, screens, and vision-based air sorters do 95% of the sorting. The robots only handle the hard cases. This is the no-bullshit end of AI.) [1] https://www.youtube.com/watch?v=RyBEUyEtxQo https://www.youtube.com/watch?v=RyBEUyEtxQo [2] https://max-ai.com/ https://max-ai.com/

KoolKat23 3y ago

I suspect it's less pre-trained than you think, the tweet is prefaced with a message that all actions are driven by neural networks indicating that it's probably adjusting for the objects, the environment etc.

tintor 3y ago
It is safe to assume it works only for that specific demo, and nothing else.
modeless 3y ago
I think it's true that it can adjust, somewhat. I expect that it could handle slight variations such as moving any of the items on the table around by a few inches, or adding or removing a couple of plates from the rack. However, I do not expect that it could handle larger variations like replacing the apple with a pineapple, picking trash out of the dish rack or the cup instead of the plate, replacing the cup with a coffee mug, or sliding the dish rack over to the other side of the robot. I'd love to be wrong but I expect that if they had that much flexibility in their controllers they would have demonstrated it.
- KoolKat23 3y ago
  I'm sure there is definitely some limitation in this demo, but who knows, Google RT-2 exists after all. Enough shapes and object training will save the day, there's still objects me as a human would be new to handling.
  - modeless 3y ago
    Oh for sure these limitations will fall. It's only a matter of time (and a lot of work behind the scenes). Covariant's RFM-1 (announced Monday) is an interesting approach to generality. https://www.youtube.com/watch?v=1Go6HEC-bYU https://www.youtube.com/watch?v=1Go6HEC-bYU

fennecbutt 3y ago

At 1:37 in the video you can see it react by jerking its hand after the plate wobbles a bit. At 0:59 it also shifts the basket out of the way as a separate action, after its already put it down.