3 ms·
AGI is not needed to transform human instructions into tasks - GPT shows LLMs can do a reasonable job of taking task definitions and expressing possible ‘code’
by jameshart 4y ago
AGI is not needed to transform human instructions into tasks - GPT shows LLMs can do a reasonable job of taking task definitions and expressing possible ‘code’ solutions to them; prompt chaining is a useful approach for guiding LLMs to solve problems, so turning Dave shouting down ‘hey atlas, pass me that toolbag’ into ‘generate a series of robot execution steps to pass Dave the toolbag.’, together with some previously gathered context like ‘ Dave is on a scaffold 15 feet up; the toolbag is on the ground; there’s a platform about 5 feet off the ground next to the scaffold; there’s a plank here’, into an actual plan like ‘move the plank to make a ramp to the platform; get the toolbag; carry it up the ramp into the platform; throw the toolbag to Dave’ is actually not really beyond the realm of LLM capabilities
(Seriously, I tried this with ChatGPT and it doesn’t do badly)
Once you’ve got the model to produce a plan, you can do things like have Atlas explain it to Dave before attempting it (giving Dave plenty of opportunities to tell Atlas the plank won’t hold its weight, or that throwing a heavy bag at a person working at height is a clear OSHA violation; or, presumably to suggest Atlas could perform a sick flip off the platform at the end). Then once the plan’s approved again, the LLM can try to trim those steps into python code or whatever to call the actual atlas goal management APIs.
This seems feasible given LLM tech, but it’s likely there are better approaches - not everything here is a nail that the GPT hammer should be used to hit home. The point is that there seem to be pathways to combine LLMs into the mix of human - robot interaction that could solve the challenge that right now when Dave needs his toolbag handing to him, a team of MIT PhDs need to spend three months expressing that problem in subproblems of vision and manipulation that Atlas can understand.
But it’s also not AGI. It’s a human-robot interface mediated through LLM.
- kaba0 4y agoI mean, this misses this huge problem of having a (mental) model of the actual physical world. Sure, that LLM model have seen the word ladder used in conjunction with climbing many times, but it sure as hell don’t have any reasonable “idea” on the weight, weight distribution, usability of a physical ladder, ground, toolkit, the whole 3D coordinate system, orientations, gravity, etc. This is the same problem why FSD likely needs AGI, is that plastic bag in the air a reason to emergency brake on a motorway? Sure, it can be special cased but we have so many learned knowledge of the world around us that we don’t even think of as “knowledge”, that current models are just toys, comparatively. (And I don’t mean it in a dismissive way at all, don’t get me wrong! Research is on good track, but there is likely a lot ahead of us)
- cypress66 4y agoThis is mostly semantics I think, of what the actual definition of AGI is. You would probably need a multi modal AI that could do natural language, vision, and control of the robot. At that point it's probably as smart as a dog or any animal other than humans. And maybe you could argue there isn't any fundamental difference between dog level ai and human level ai, it's just a "scaled up" version. I think what you mention though is more like gluing an LLM with the current stack. But I doubt that will ever be enough, you probably need a multi modal model.
- fnbr 4y agoIf you can stitch something together though, you can start to get the data you need to train an end-to-end model. And once you get that at any reasonable level of proficiency, you should see the ML flywheel spinning. The hard part is stitching everything together.
- jameshart 4y agoI don’t think it’s just ‘semantics’ unless you believe that AGI can be achieved by stitching together LLM prompts under some sort of ‘executive planning’ expert system. I don’t think that produces anything like AGI, but it does let you build out of components that can tackle much fuzzier problems than typical computer programs can, like ‘figuring out what a human means’ and ‘identifying which things identified by an image classifier in an image of a complex environment might be relevant to a task’, or ‘summarizing the plan so far back to a human to get it approved’.
- ikekkdcjkfke 4y agoYou can convert the robots movement, including balancing with arms, to a language and make a LLM for that too :)