4 ms·

It does all start to feel like we'd get fairly close to being able to convincingly emulate a lot of human or at least animal behavior on top of the existing gen

by sho_hn 6mo ago

It does all start to feel like we'd get fairly close to being able to convincingly emulate a lot of human or at least animal behavior on top of the existing generative stack, by using brain-like orchestration patterns ... if only inference was fast enough to do much more of it. The gauge-reading example here is great, but in reality of course having the system synthesize that Python script, run the CV tasks, come back with the answer etc. is currently quite slow. Once things go much faster, you can also start to use image generation to have models extrapolate possible futures from photos they take, and then describe them back to themselves and make decisions based on that, loops like this. I think the assumption is that our brains do similar things unconsciously, before we integrate into our conscious conception of mind. I'm really curious what things we could build if we had 100x or 1000x inference throughput.

moonu 6mo ago
Idk if you've seen this already but Taalas does this interesting thing where they embed the model directly onto the chip, this leads to super-fast speeds (https://chatjimmy.ai https://chatjimmy.ai) but the model they're using is an old small Llama model so the quality is pretty bad. But they say that it can scale, so if that's really true that'd be pretty insane and unlock the inference you're talking about.
- lachlan_gray 6mo ago
  Robotics/control systems is exactly what came to mind when I saw this release! What struck me is the possibility of look ahead search in real time, a bit like alphazero's mcts.
- pstuart 6mo ago
  It's a fascinating proposition and no doubt they'll get bigger models in there, and likely be able to cluster multiple models for mega MOE. One thing that would really be great is if they could take the power requirements down -- the chip requires 2.5KW, which is modest in terms of what the big boys use but would be an issue on a battery powered robot.
  - fennecfoxy 6mo ago
    "The chip" no, a whole rack/deployment they offer takes 2.5kW. Not just one chip. Squeezing 2.5kW thru 1 chip would be mental.
    - pstuart 6mo ago
      My bad, thanks for the correction. Skimming FTL.
Kostic 6mo ago
Taalas showed that you could make LLMs faster by turning them into ASICs and get 10k+ token generation. It's a matter of time now.
- timmg 6mo ago
  Actually pretty interesting to think: in a few years you might buy a raspberry pi style computer board with an extra chip on it with one of these types of embodiment models and you can slap it in a rover or something.
LetsGetTechnicl 6mo ago
What if we put slop images into slop machines and got slop^2 back out
tootie 6mo ago
Is emulating human behavior really a valuable end goal though? Humans exist as the evolutionary endpoint of exhaustion hunting large pray and organic tool-making. We've built loads of industrial and residential automation tools in the last 100 years and none of them are humanoid. I'd imagine a household robot butler would be more like R2D2 with lots and lots of arms.
- hootz 6mo ago
  It is when the world was made to interface with us. We can't use robots for everything if they aren't emulating us, because we would have to adapt everything for the non-humanlike robots.
  - tootie 6mo ago
    We build our living spaces against the constraints of the human form, but that still doesn't imply the human form is optimal for anything. There's no reason a robot traveling over smooth surface should have legs instead of wheels or treads. There's no reason to have a head. Some kind of arm is a common design feature, but certainly no reason to have two. No reason to be symmetrical. A domestic robot may be constrained in terms of scale (ie see things at counter height) but not shape.
    - hgoel 6mo ago
      Really, the requirements are for the robot to move in predictable ways (if something looks like an arm, it ought to move like an arm, etc), and to have enough strength to be useful for difficult/tiring tasks while somehow also not being dangerous if something does go wrong.
    - famouswaffles 6mo ago
      >We build our living spaces against the constraints of the human form, but that still doesn't imply the human form is optimal for anything. We build just about everything we expect to interact with against the constraints of the human form, not just living spaces. And yes we because we built those spaces for the human body, the human body is by definition the optimal choice. >There's no reason a robot traveling over smooth surface should have legs instead of wheels or treads. There's a reason. The robot becomes useless for any surface that isn't smooth. What's it going to do about stairs ? You're not going to make a bespoke solution that generalizes for us better than 'feet that work'. Do you think it's better to built a million different complex robot bodies for every situation ? That defeats the purpose of being general purpose.

by sho_hn 6mo ago

moonu 6mo ago

Idk if you've seen this already but Taalas does this interesting thing where they embed the model directly onto the chip, this leads to super-fast speeds (https://chatjimmy.ai https://chatjimmy.ai) but the model they're using is an old small Llama model so the quality is pretty bad. But they say that it can scale, so if that's really true that'd be pretty insane and unlock the inference you're talking about.

lachlan_gray 6mo ago
Robotics/control systems is exactly what came to mind when I saw this release! What struck me is the possibility of look ahead search in real time, a bit like alphazero's mcts.
pstuart 6mo ago
It's a fascinating proposition and no doubt they'll get bigger models in there, and likely be able to cluster multiple models for mega MOE. One thing that would really be great is if they could take the power requirements down -- the chip requires 2.5KW, which is modest in terms of what the big boys use but would be an issue on a battery powered robot.
- fennecfoxy 6mo ago
  "The chip" no, a whole rack/deployment they offer takes 2.5kW. Not just one chip. Squeezing 2.5kW thru 1 chip would be mental.
  - pstuart 6mo ago
    My bad, thanks for the correction. Skimming FTL.

Kostic 6mo ago

Taalas showed that you could make LLMs faster by turning them into ASICs and get 10k+ token generation. It's a matter of time now.

timmg 6mo ago
Actually pretty interesting to think: in a few years you might buy a raspberry pi style computer board with an extra chip on it with one of these types of embodiment models and you can slap it in a rover or something.

LetsGetTechnicl 6mo ago

What if we put slop images into slop machines and got slop^2 back out

tootie 6mo ago

Is emulating human behavior really a valuable end goal though? Humans exist as the evolutionary endpoint of exhaustion hunting large pray and organic tool-making. We've built loads of industrial and residential automation tools in the last 100 years and none of them are humanoid. I'd imagine a household robot butler would be more like R2D2 with lots and lots of arms.

hootz 6mo ago
It is when the world was made to interface with us. We can't use robots for everything if they aren't emulating us, because we would have to adapt everything for the non-humanlike robots.
- tootie 6mo ago
  We build our living spaces against the constraints of the human form, but that still doesn't imply the human form is optimal for anything. There's no reason a robot traveling over smooth surface should have legs instead of wheels or treads. There's no reason to have a head. Some kind of arm is a common design feature, but certainly no reason to have two. No reason to be symmetrical. A domestic robot may be constrained in terms of scale (ie see things at counter height) but not shape.
  - hgoel 6mo ago
    Really, the requirements are for the robot to move in predictable ways (if something looks like an arm, it ought to move like an arm, etc), and to have enough strength to be useful for difficult/tiring tasks while somehow also not being dangerous if something does go wrong.
  - famouswaffles 6mo ago
    >We build our living spaces against the constraints of the human form, but that still doesn't imply the human form is optimal for anything. We build just about everything we expect to interact with against the constraints of the human form, not just living spaces. And yes we because we built those spaces for the human body, the human body is by definition the optimal choice. >There's no reason a robot traveling over smooth surface should have legs instead of wheels or treads. There's a reason. The robot becomes useless for any surface that isn't smooth. What's it going to do about stairs ? You're not going to make a bespoke solution that generalizes for us better than 'feet that work'. Do you think it's better to built a million different complex robot bodies for every situation ? That defeats the purpose of being general purpose.