4 ms·

Do you want household androids? Because this kind of stuff is on the level of research a bery large step towards that. Think as it as ab example where we can ma

by golol 2y ago

Do you want household androids? Because this kind of stuff is on the level of research a bery large step towards that. Think as it as ab example where we can make a model understand a lot of physical common sense stuff, which is the goal for robotics right now.

suddenlybananas 2y ago
This is really not the avenue for house-hold robots. Interacting with the actual physical world is very different from creating a video game.
- deleted 2y ago
  [deleted]
- sangnoir 2y ago
  > Interacting with the actual physical world is very different from creating a video game The major difference being the former scales very poorly for generating training data compared to the latter. Genie 2 is not even a video game and has worse fidelity that video games, the upside is it probably scales even better than video games for generating training scenarios. If you want androids in teal life, Genie 2 (or similar systems) is how you bootstrap the agent AI. The training pipeline will be: raw video -> Genie 2 -> game engine with rules -> physical robot
  - youoy 2y ago
    > The training pipeline will be: raw video -> Genie 2 -> game engine with rules -> physical robot One of those arrows is not like the others
    - sangnoir 2y ago
      The final step is an oversimplification: purpose-built simulator -> deconstructed robot on a lab workbench -> controlled space -> "real world" with constraints -> real world Any model would have to succeed in one stage before it can proceed to the next one.
      - adverbly 2y ago
        At the risk of sounding repetitive, one of those arrows is not like the others.
        sangnoir 2y ago
        ...and?
  - mosdl 2y ago
    How does turning an image into a game help with robots? Robots don't need to guess what they can't see, they would have sensors to tell them exactly what is there (like a self driving car).
    - Chilko 2y ago
      I have no expertise in this area, but my assumption is that this could help for a broader sort of object/world permanence for robots - e.g. if something is no longer visible to the robot's sensors (e.g. behind an obstacle, smoke, etc) then it could use a model based on this type of tech to maintain a short-term estimate of its surroundings even when operating blind.
    - sangnoir 2y ago
      > Robots don't need to guess what they can't see, they would have sensors to tell them exactly what is there (like a self driving car). Self driving vars have cameras as part of their sensor suite, and have models to make sense of sensor data. Video will help with perception and classification (understanding the world) with no agency needed. Game-playing will help with planning, execution, and evaluation. Both functions are necessary, and those that come after rely on earlier capabilities
    - falcor84 2y ago
      To be able to plan ahead, robots do absolutely need to plan ahead (read: "guess" or even "imagine") what they might encounter before they sense it. In your self driving car example, for instance, it needs to come up with various scenarios for what might be around the corner ahead of a turn, and assign reasonable probabilities to these scenarios. I absolutely see how a system like this could help with it. For example, let's say that the car is approaching an intersection, and suddenly sees a puddle on the road to the left getting brighter - a visual world model like this might extrapolate a scenario that the brightness is the result of a car moving towards the intersection assigning this some probability, and signing another probably to a scenario that it's just a flickering headlight, and the car would then decide whether and how much to slow down. In this example there is a sensor, but it definitely doesn't tell the robot "exactly what is there", and while we could try to write rules about what it should do, the Bitter Lesson tells us it's better to just let it create its own model.
- j_timberlake 2y ago
  No, they actually did use a genie-like model to train robots on household chores. Page 8 of the Genie 1 paper: https://arxiv.org/abs/2402.15391 https://arxiv.org/abs/2402.15391
JTyQZSnP3cQGa8B 2y ago
I don't understand how that is relevant. I certainly would not want household androids unless I'm completely disabled.
- theshackleford 2y ago
  > I certainly would not want household androids unless I'm completely disabled. That's nice. I'm not completely disabled, but I am disabled, and I very much would appreciate them, as my capability to do things over the longer term is very much not going to go in the direction of improving. As it is, there are a lot of things I now rely on people for, that at one time, I did not. Whilst I recognise its probably not going to happen in a time span that is useful to me, I do wish it could, so that I could be less of a burden on those around me, and maintain a relative level of independence.
- furyofantares 2y ago
  A lot of people are disabled right now! Unless you have a young/quick death, there's a really good chance you will be, too.
- throwaway2037 2y ago
  No trolling: You wouldn't want robots to mow your lawn, maybe do the boring bits of cooking (prep! endless stirring!), clean your house, wash your clothes? Man, sign me up!

by golol 2y ago

suddenlybananas 2y ago

This is really not the avenue for house-hold robots. Interacting with the actual physical world is very different from creating a video game.

deleted 2y ago
[deleted]
sangnoir 2y ago
> Interacting with the actual physical world is very different from creating a video game The major difference being the former scales very poorly for generating training data compared to the latter. Genie 2 is not even a video game and has worse fidelity that video games, the upside is it probably scales even better than video games for generating training scenarios. If you want androids in teal life, Genie 2 (or similar systems) is how you bootstrap the agent AI. The training pipeline will be: raw video -> Genie 2 -> game engine with rules -> physical robot
- youoy 2y ago
  > The training pipeline will be: raw video -> Genie 2 -> game engine with rules -> physical robot One of those arrows is not like the others
  - sangnoir 2y ago
    The final step is an oversimplification: purpose-built simulator -> deconstructed robot on a lab workbench -> controlled space -> "real world" with constraints -> real world Any model would have to succeed in one stage before it can proceed to the next one.
    - adverbly 2y ago
      At the risk of sounding repetitive, one of those arrows is not like the others.
      - sangnoir 2y ago
        ...and?
- mosdl 2y ago
  How does turning an image into a game help with robots? Robots don't need to guess what they can't see, they would have sensors to tell them exactly what is there (like a self driving car).
  - Chilko 2y ago
    I have no expertise in this area, but my assumption is that this could help for a broader sort of object/world permanence for robots - e.g. if something is no longer visible to the robot's sensors (e.g. behind an obstacle, smoke, etc) then it could use a model based on this type of tech to maintain a short-term estimate of its surroundings even when operating blind.
  - sangnoir 2y ago
    > Robots don't need to guess what they can't see, they would have sensors to tell them exactly what is there (like a self driving car). Self driving vars have cameras as part of their sensor suite, and have models to make sense of sensor data. Video will help with perception and classification (understanding the world) with no agency needed. Game-playing will help with planning, execution, and evaluation. Both functions are necessary, and those that come after rely on earlier capabilities
  - falcor84 2y ago
    To be able to plan ahead, robots do absolutely need to plan ahead (read: "guess" or even "imagine") what they might encounter before they sense it. In your self driving car example, for instance, it needs to come up with various scenarios for what might be around the corner ahead of a turn, and assign reasonable probabilities to these scenarios. I absolutely see how a system like this could help with it. For example, let's say that the car is approaching an intersection, and suddenly sees a puddle on the road to the left getting brighter - a visual world model like this might extrapolate a scenario that the brightness is the result of a car moving towards the intersection assigning this some probability, and signing another probably to a scenario that it's just a flickering headlight, and the car would then decide whether and how much to slow down. In this example there is a sensor, but it definitely doesn't tell the robot "exactly what is there", and while we could try to write rules about what it should do, the Bitter Lesson tells us it's better to just let it create its own model.
j_timberlake 2y ago
No, they actually did use a genie-like model to train robots on household chores. Page 8 of the Genie 1 paper: https://arxiv.org/abs/2402.15391 https://arxiv.org/abs/2402.15391

JTyQZSnP3cQGa8B 2y ago

I don't understand how that is relevant. I certainly would not want household androids unless I'm completely disabled.

theshackleford 2y ago
> I certainly would not want household androids unless I'm completely disabled. That's nice. I'm not completely disabled, but I am disabled, and I very much would appreciate them, as my capability to do things over the longer term is very much not going to go in the direction of improving. As it is, there are a lot of things I now rely on people for, that at one time, I did not. Whilst I recognise its probably not going to happen in a time span that is useful to me, I do wish it could, so that I could be less of a burden on those around me, and maintain a relative level of independence.
furyofantares 2y ago
A lot of people are disabled right now! Unless you have a young/quick death, there's a really good chance you will be, too.
throwaway2037 2y ago
No trolling: You wouldn't want robots to mow your lawn, maybe do the boring bits of cooking (prep! endless stirring!), clean your house, wash your clothes? Man, sign me up!