3 ms·

I'm looking through the code, does it mean that the authors wrote all basic skills themselves and let LLM choose from them? So this approach can't be generalise

by eamag 2y ago

I'm looking through the code, does it mean that the authors wrote all basic skills themselves and let LLM choose from them? So this approach can't be generalised, can it? https://github.com/BAAI-Agents/Cradle/pull/44/files#diff-3f335abad14395112af49b9d0b684dd7ac30cf911fa1476a0764db862bb25e59 https://github.com/BAAI-Agents/Cradle/pull/44/files#diff-3f3...

huevosabio 2y ago
I think they seed it with some skills and then can generate more? The website allures to "skill generation" so arguably they are doing some level of code generation.
finnh 2y ago
Right ... the output isn't really "keyboard & mouse operations" as stated in the abstract. The output is a sequence of atomic commands ("shoot", "select_weapon") or composite commands ("follow") that are then translated into mouse actions by a human-written layer. Similarly, when I'm typing this post my output isn't "keyboard interrupts" nor "TCP frames". Those things do happen, but only by the grace of others' work. Still I like the basic framework idea & I suppose future work looks at pushing the provided layer lower & lower - so the system can learn those atomic commands, rather than having humans provide them - and pushing the logic higher & higher via AI-generated composite commands.
- fordacious 2y ago
  Many games do happen to be set up this way wrt their input systems if they happen to leverage Steam input. Player actions in game are defined as abstract verbs with concrete input device binding being left to the player / underlying system. Valve is trying to nudge games towards being platonic objects isolated from specific systems (so they'll be readily compatible steam deck, vr, pc etc...). Translating these verbs in an automated way with an agent is an interesting side effect.
  - miohtama 2y ago
    The problem is not triggering the action, the problem is being at the right place at the right time and looking into the right direction.
Fripplebubby 2y ago
They start with some basic skills but the model builds the more complex skills. The specific skills that they start with depends on which task, for example RDR2 they start with only how to turn, move forward, shoot, and select an item from the item equip wheel and then a couple composite skills that were not _necessary_ but saved a lot of time - for example, "follow npc", since the model could follow an npc without it but it required a huge amount of back and forth. However, a lot of skills were actually learned by the model from the in-game tutorial in RDR2 explaining how to do things! > So this approach can't be generalised, can it? The paper definitely demonstrates that the _approach_ can be generalized because they use the same approach across a variety of different environments and tasks, but you can also see that they did have to specialize the prompts, tools (like how the incoming screenshot was decorated with object detection / segmentation, stuff like that), and set of skills for each environment.

by eamag 2y ago

huevosabio 2y ago

I think they seed it with some skills and then can generate more? The website allures to "skill generation" so arguably they are doing some level of code generation.

finnh 2y ago

Right ... the output isn't really "keyboard & mouse operations" as stated in the abstract. The output is a sequence of atomic commands ("shoot", "select_weapon") or composite commands ("follow") that are then translated into mouse actions by a human-written layer. Similarly, when I'm typing this post my output isn't "keyboard interrupts" nor "TCP frames". Those things do happen, but only by the grace of others' work. Still I like the basic framework idea & I suppose future work looks at pushing the provided layer lower & lower - so the system can learn those atomic commands, rather than having humans provide them - and pushing the logic higher & higher via AI-generated composite commands.

fordacious 2y ago
Many games do happen to be set up this way wrt their input systems if they happen to leverage Steam input. Player actions in game are defined as abstract verbs with concrete input device binding being left to the player / underlying system. Valve is trying to nudge games towards being platonic objects isolated from specific systems (so they'll be readily compatible steam deck, vr, pc etc...). Translating these verbs in an automated way with an agent is an interesting side effect.
- miohtama 2y ago
  The problem is not triggering the action, the problem is being at the right place at the right time and looking into the right direction.

Fripplebubby 2y ago

They start with some basic skills but the model builds the more complex skills. The specific skills that they start with depends on which task, for example RDR2 they start with only how to turn, move forward, shoot, and select an item from the item equip wheel and then a couple composite skills that were not _necessary_ but saved a lot of time - for example, "follow npc", since the model could follow an npc without it but it required a huge amount of back and forth. However, a lot of skills were actually learned by the model from the in-game tutorial in RDR2 explaining how to do things! > So this approach can't be generalised, can it? The paper definitely demonstrates that the _approach_ can be generalized because they use the same approach across a variety of different environments and tasks, but you can also see that they did have to specialize the prompts, tools (like how the incoming screenshot was decorated with object detection / segmentation, stuff like that), and set of skills for each environment.