16 ms·
Helix: A vision-language-action model for generalist humanoid control
- kubb 2y agoWow! This is something new.
- sandis 2y agoYouTube link for the video (for whatever reason the video hosted on their site kept buffering for me): https://www.youtube.com/watch?v=Z3yQHYNXPws https://www.youtube.com/watch?v=Z3yQHYNXPws
- bhouston 2y agoWhen doing robot control, how do you model in the control of the robot? Do you have tool_use / function calling at the top level model which then gets turned into motion control parameters via inverse kinematic controllers? What is the interface from the top level to the motors? I feel it can not just be a neural network all the way down, right?
- Philpax 2y agoHave a look at the post - it explains how it works. There are two models: a 7-9Hz 7B vision-language model, and a 200Hz 80M visuomotor model. The former produces a latent vector, which is then interpreted by the latter to drive the motors.
- NitpickLawyer 2y ago> a 7-9Hz 7B vision-language model, and a 200Hz 80M visuomotor model. huh. An interesting approach. I wonder if something like this can be used for other things as well, like "computer use" with the same concept of a "large" model handling the goals, and a "small" model handling clicking and stuff, at much higher rates, useful for games and things like that.
- whatever1 2y agoThis is typical in real time applications. A supervisor tries to guess in which region the system is currently and then invokes the correct set of lower level algorithms.
- imtringued 2y agoYou don't use function calling. You specifically train the neural network to directly encode the robot action as a token. There are many ways. You can output absolute positions, delta positions, relative trajectory. You can do this in joint space or end effector space. 200Hz is barely enough to control a motor, but it is good enough to send a reference signal to a motor controller. Usually what is done is that you have a neural network to learn complex high level behaviour and use that to produce a high level trajectory, then you have a whole body robot controller based on quadratic programming that does things like balancing, maintaining contacts when holding objects or pressing against things. This requires a model of the robot dynamics so that you know the relationship between torques and acceleration. Then after that you will need a motor controller that accepts reference acceleration/torque, velocity and position commands which then is turned into 10kHz to 100kHz pulse width modulated signals by the motor controller. The motor controller itself is driving MOSFETs so it can only turn them on or off, unless you are using expensive sinusoidal drivers.
- traverseda 2y ago"The first time you've seen these objects" is a weird thing to say. One presumes that this is already in their training set, and that these models aren't storing a huge amount of data in their context, so what does that even mean?
- jayd16 2y agoIt probably gives them confidence that they can accurately see a thing even though they don't know what that thing is. I could also imagine a lot of safety around leaving things outside of the current task alone so you might have to bend over backwards to get new objects worked on.
- thomastjeffery 2y agoThere is no such thing as "thing" here. These models are trained such that the given conditions (the visual input and the text prompt) will be continued with a desirable continuation (motor function over time). The only dimension accuracy can apply to is desirability.
- jayd16 2y agoYou don't think there's any segmentation going on?
- thomastjeffery 2y agoImplicitly, maybe. Does that matter if you don't know where?
- Symmetry 2y agoIt's normal to have a training set and a validation set and I interpreted that to mean that these items weren't in the training set.
- ygouzerh 2y agoSo from what I understand it actually means that they were for example never trained on a video of an apple. Maybe only on a video of bread, pineapple, chocolate. However, as it was trained using generic text data similarly to a normal LLM, it knows how an apple is supposed to look like. Similar than a kid that never saw a banana, but his parent described it to him.
- ziofill 2y agoThere’s nothing I want more than a robot that does house chores. That’s the real 10x multiplier for humans to do what they do best.
- dartos 2y agoHopefully in the next decade we’ll get there. Vision+language multimodal models seem to solve some of the hard problems.
- siavosh 2y agoWhat do humans do best?
- jayd16 2y agoEverything everything else is worse at.
- ziofill 2y agoI mean to use their time to pursue their passions and interests, not cleaning up the kitchen or making the bed or doing laundry...
- hooverd 2y agoGiven time to "pursue their passions and interests", most people chose to turn their brain to soup on social media.
- KolmogorovComp 2y agobut most people think they are better than most people.
- ein0p 2y agoBrowse Instagram, apparently.
- abraxas 2y ago
- deleted 2y ago[deleted]
- anentropic 2y agoVery impressive Why make such sinister-looking robots though...?
- jayd16 2y agoWith the way they move, they look like stoned teenagers interning at a Bond villain factory. Not to knock the tech but they're scary and silly at the same time.
- esafak 2y agoBlack was not the best color choice.
- anentropic 2y agoWell, when you put it like that I feel a bit uncomfortable... But it did seem like title of their mood board must have been "Black Mirror". Very uncanny valley, the glossy facelessness. It somehow looks neither purely utilitarian/industrial nor 'friendly'. I could see it being based on the aesthetic of laptops and phones, i.e. consumer tech, but the effect is so different when transposed onto a very humanoid form.
- aerodog 2y agoInteresting timing - same day MSFT releases https://microsoft.github.io/Magma/ https://microsoft.github.io/Magma/
- _1 2y agoCurrent discussion: https://news.ycombinator.com/item?id=43110265 https://news.ycombinator.com/item?id=43110265
- wwwtyro 2y agoUntil we get robots with really good hands, something I'd love in the interim is a system that uses _me_ as the hands. When it's time to put groceries away, I don't want to have to think about how to organize everything. Just figure out which grocery items I have, what storage I have available, come up with an optimized organization solution, then tell me where to put things, one at a time. I'm cautiously optimistic this will be doable in the near term with a combination of AR and AI.
- camjw 2y agoMaybe I don't understand exactly what you're describing but why would anyone pay for this? When I bring home the shopping I just... chuck stuff in the cupboards. I already know where it all goes. Maybe you can explain more?
- bear141 2y agoMaybe some people just assume there is a “best” or “optimal” way to do everything and AI will tell us what that is. Some things are just preference and I don’t mind the tiny amount of energy that goes into doing small things the way I like.
- jayd16 2y agoMaybe they're imagining more complex tasks like working on an engine.
- loudmax 2y agoOne use case I imagine is skilled workmanship. For example, putting on a pair of AR glasses and having the equivalent of an experienced plumber telling me exactly where to look for that leak and how to fix it. Or how to replace my brake pads or install a new kitchen sink. When I hire a plumber or a mechanic or an electrician, I'm not just paying for muscle. Most of the value these professionals bring is experience and understanding. If a video-capable AI model is able to assume that experience, then either I can do the job myself or hire some 20 year old kid at roughly minimum wage. If capabilities like this come about, it will be very disruptive, for better and for worse.
- sottol 2y agoImo, the Terminator movies would have been scarier if they moved like these guys - slow, careful, deliberate and measured but unstoppable. There's something uncanny about this.
- megous 2y agoUnfortunately, there'll be no time travel to save us. That was the lying part of the movie. Other stuff was true.
- ramenlover 2y agoWhy do they make “eye contact” after every hand off? Feels oddly forced.
- bear141 2y agoThis along with the writing style in the description is totally forced anthropomorphizing. It’s creepy.
- jimbohn 2y agoGotta hype up the investors somehow
- GoatInGrey 2y agoPerhaps they're exchanging knowing looks on how stupid they think the demo is. Solidarity between artificial brothers.
- pixl97 2y agoIf you're training robots to interact with humans this is the kind of behavior you'd want. Humans use a ton of nonverbal hints like this to register context.
- Symmetry 2y agoSo, there's no way you can have fully actuated control of every finger joint with just 35 degrees of freedom. Which is very reasonable! Humans can't individually control each of our finger joints either. But I'm curious how their hand setups work, which parts are actuated and which are compliant. In the videos I'm not seeing any in-hand manipulation other than just grasping, releasing, and maintaining the orientation of the object relative to the hand and I'm curious how much it can do / they plan to have it be able to do. Do they have any plans to try to mimic OpenAI's one handed rubics cube demo?
- pr337h4m 2y agoGoal 2 has been achieved, at least as a proof of concept (and not by OpenAI): https://openai.com/index/openai-technical-goals/ https://openai.com/index/openai-technical-goals/
- Symmetry 2y agoThey can put away clutter but if they could chop a carrot or dust a vase they'd have shown videos demonstrating that sort of capability. EDIT: Let alone chop an onion. Let me tell you having a robot manipulate onions is the worst. Dealing with loose onion skins is very hard.
- selfmodruntime 2y agoSure. But if you showed this video to someone 5 or 10 years ago, they'd say it's fiction.
- Symmetry 2y agoTelling a robot verbally "Put the cup on the counter" and having it figure out what the cup is, what the counter is in its field of view would have seemed like science fiction. The object manipulation itself is still well behind what we saw in the 2015 DARPA Grand Challenge though.
- squigz 2y agoThere's something hilarious to me about the idea of chopping onions being a sort of benchmark for robots.
- kla-s 2y agoDoes anyone know how long they have been at this? Is this mainly a reimplementation of the physical intelligence paper + the dual size/freq + the cooperative part?
- pr337h4m 2y ago"Over a year" according to the founder: https://x.com/adcock_brett/status/1892578309344502191 https://x.com/adcock_brett/status/1892578309344502191
- bilsbie 2y agoThey should have made them talk. It’s a little dehumanizing otherwise.
- bilsbie 2y agoThis is amazing but it also made me realize I just don’t trust these videos. Is it sped up? How much is preprogrammed? I now they claim there’s no special coding but did they practice this task? Special training? Even if this video is totally legit I’m but burned out by all the hype videos in general.
- bilsbie 2y agoI get the impression there’s a language model sending high level commands to a control model? I wonder when we can have one multimodal model that controls everything. The latest models seemed to be fluidly tied in with generating voice; even singing and laughing. It seems like it would be possible to train a multimodal that can do that with low level actuator commands.
- turnsout 2y agoIf you read the article, they describe a two-system approach; one "think fast" 80M parameter model running at 200hz to control motion, and one "think slow" 7B parameter model running at ~7-9hz for everything else (scene understanding, language processing, etc). If that sounds like a cheat, neuroscientists tell us this is how the human brain works.
- ge96 2y agoWonder what their vision stack is like. Depth via sensors or purely visual and the distance estimating of objects and inverse kinematics/proprioception, anyway it looks impressive.
- yurimo 2y agoI don't know, there has been so many overhyped and faked demos in humanoid robotics space over the last couple years, it is difficult to believe what is clearly a demo release for shareholders. Would love to see some demonstration in a less controlled environment.
- ge96 2y agoImagine they bring one out to a construction site and they treat the robot as a new rookie guy, go pick up those pipes. That would be an ultimate on the fly test to me.
- ortsa 2y agoPicking up a bundle of loose pipes actually seems like a great benchmark for humanoid robots. Especially if they're not in a perfect pile. A full test could be something like grabbing all the pipes, from the floor, and putting them into a truck bed, in some (hopefully) sane fashion
- sayamqazi 2y agoI have my personal multimodal benchmark for physical robots. You put a keyring with bunch of different keys in front of a robot and then instruct it pick it up and open a lock while you are describing which key is the correct one. Something like "Use the key with black plastic head and you need to put it in teeths facing down" I have low hopes of this being possibe in the next 20 years. I hope I am still alive to witness if it ever happens.
- m0llusk 2y agopick up that can, heh heh heh
- falcor84 2y agoI suppose the next big milestone is Wozniak's Coffee Test: A robot is to enter a random home and figure out how to make coffee with whatever they have.
- abraxas 2y agoIs this even reality or CGI? They really should show these things off in less sterile environemtns because this video has a very CGI feel to it.
- psb217 2y agoNatural, cluttered environments are a lot tougher to deal with. This near future-y minimalist environment has the dual benefits of looking stylish and being much closer to whatever they were able to simulate at scale for training the models.
- exe34 2y agoIs there a paper? I think I get how they did their training, but I'd like to understand it more. Does anyone know if this trained model would work on a different robot at all, or would it need retraining?
- swalsh 2y agoAt this point, this is enough autonomy to have a set of these guys man a howitzer (read as old stockpiles of weapons we already have). Kind of a scary thought. On one hand, I think the idea of moving real people out of danger in war is a good idea, and as an American i'd want Americans to have an edge... and we can't guarantee our enemies won't take it if we skip it, on the other hand I have a visceral reaction to machines killing people. I think we're at an inflection point now where AI and robotics can be used in warfare, and we need to start having that conversation.
- lyu07282 2y agoI don't understand we already saw exactly what happens with the emergence of drones and Israel is already using AI to select bombing targets and semi-autonomous turrets. What conversation? What kind of society do you think we are living in?
- 01100011 2y agoWe had sufficient AI to make death machines for decades. You don't need fancy LLMs to get a pretty good success rate for targeting. I have said for years that the only thing keeping us from "stabby the robot" is solving the power problem. If you can keep a drone going for a week, you have a killing machine. Use blades to avoid running out of ammo. Use IR detection to find the jugular. Stab, stab and move on. I'm guessing "traditional" vision algorithms are also sufficient to, say, identify an ethnicity and conduct ethnic cleansing. We are "solving the power problem" away from a new class of WMDs that are accessible to smaller states/groups/individuals.
- selfmodruntime 2y ago> We had sufficient AI to make death machines for decades And we already reached the peek here. Small drones that are cheaply mass produced, fly on SIM cards alone and explode when they reached a target. That's all there is to it. You don't need a gun mounted on a spot or a humanoid robot carrying a gun. Exploding swarms are enough.
- 2y ago
- causal 2y agoI'm always wondering at the safety measures on these things. How much force is in those motors? This is basically safety-critical stuff but with LLMs. Hallucinating wrong answers in text is bad, hallucinating that your chest is a drawer to pull open is very bad.
- cess11 2y agoNot a big deal on the battlefield.
- rizky05 2y ago[dead]
- causal 2y agoI'd say a very big deal when munitions and targeting are involved
- cess11 2y agoWhy do you think that? Battle fields are violent and exhausting, people get the shakes, make mistakes, hurt themselves and each other all the time. Munitions are generally designed to not explode from a bump in the road or getting dropped or squeezed, and targeting systems commonly support automatic tracking or similar, for this specific reason.
- silentwanderer 2y agoIn terms of low-level safety, they can probably back out forces on the robot from current or torque measurement and detect collisions. The challenge comes with faster motions carrying lots of inertia and behavioral safety (e.g. don't pour oil on the stove)
- mmh0000 2y agoThe thing in the video moves slower than the sloth in Zootopia. If you die by that robot, you probably deserve it.
- ianamo 2y agoAre we at a point now where Asimov’s laws are programmed into these fellas somewhere?
- thomastjeffery 2y agoNope. The article clearly spells out that it's end to end LLM. Text and video in, motor function out. Technically, the text model probably has a few copies, but they are nothing more than Asimov's narrative. Laws don't (and can't) exist in a model
- deleted 2y ago[deleted]
- porphyra 2y agoIt seems that end to end neural networks for robotics are really taking off. Can someone point me towards where to learn about these, what the state of the art architectures look like, etc? Do they just convert the video into a stream of tokens, run it through a transformer, and output a stream of tokens?
- vessenes 2y agoI was reading their site, and I too have some questions about this architecture. I'd be very interested to see what the output of their 'big model' is that feeds into the small model. I presume the small model gets a bunch of environmental input, and some input from the big model, and we know that the big model input only updates every 30 or 40 frames in terms of small model. Like, do they just output random control tokens from big model and embed those in small model and do gradient descent to find a good control 'language'? Do they train the small model on english tokens and have the big model output those? Custom coordinates tokens? (probably). Lots of interesting possibilities here. By the way, the dataset they describe was generated by a large (much larger presumably) vision model tasked with creating tasks from successful videos. So the pipeline is: * Video of robot doing something * (o1 or some other high end model) "describe very precisely the task the robot was given" * o1 output -> 7B model -> small model -> loss
- andiareso 2y agoSeriously, what's with all of these perceived "high-end" tech companies not doing static content worth a damn. Stop hosting your videos as MP4s on your web-server. Either publish to a CDN or use a platform like YouTube. Your bandwidth cannot handle serving high resolution MP4s. /rant
- unraveller 2y agohttps://www.youtube.com/watch?v=Z3yQHYNXPws https://www.youtube.com/watch?v=Z3yQHYNXPws the official figure yt release vid
- matteocontrini 2y agoWhat do you mean? Videos on that page are served by CloudFront. If you're seeing issues it may be that videos are not encoded for web playback (faststart, etc.) but I haven't checked.
- IAmNotACellist 2y agoI don't suppose this is open research and I can read about their model architecture?
- verytrivial 2y agoAre they claiming these robots are also silent? They seem to have "crinkle" sounds handling packaging, which if added in post seems needlessly smoke-and-mirror for what was a very impressive demonstration (of robots impersonating an extreme stoned human.)
- ein0p 2y agoThere's no way this is 100% real though. No startup demo ever is.
- the_other 2y agoIt’s funny… there a lot of comments here asking “why would anyone pay for this, when you could learn to do the thing, or organise your time/plans yourself.” That’s how I feel about LLMs and code.
- plipt 2y agoThe demo is quite interesting but I am mostly intrigued by the claim that it is running totally local to each robot. It seems to use some agentic decision making but the article doesn't touch on that. What possible combo of model types are they stringing together? Or is this something novel? The article mentions that the system in each robot uses two ai models. S2 is built on a 7B-parameter open-source, open-weight VLM pretrained on internet-scale data and the other S1, an 80M parameter cross-attention encoder-decoder transformer, handles low-level [motor?] control. It feels like although the article is quite openly technical they are leaving out the secret sauce? So they use an open source VLM to identify the objects on the counter. And another model to generate the mechanical motions of the robot. What part of this system understands 3 dimensional space of that kitchen? How does the robot closest to the refrigerator know to pass the cookies to the robot on the left? How is this kind of speech to text, visual identification, decision making, motor control, multi-robot coordination and navigation of 3d space possible locally? Figure robots, each equipped with dual low-power-consumption embedded GPUs Is anyone skeptical? How much of this is possible vs a staged tech demo to raise funding?
- bbor 2y agoI'm very far from an expert, but: What part of this system understands 3 dimensional space of that kitchen? The visual model "understands" it most readily, I'd say -- like a traditional Waymo CNN "understands" the 3D space of the road. I don't think they've explicitly given the models a pre-generated pointcloud of the space, if that's what you're asking. But maybe I'm misunderstanding? How does the robot closest to the refrigerator know to pass the cookies to the robot on the left? It appears that the robot is being fed plain english instructions, just like any VLM would -- instead of the very common `text+av => text` paradigm (classifiers, perception models, etc), or the less common `text+av => av` paradigm (segmenters, art generators, etc.), this is `text+av => movements`. Feeding the robots the appropriate instructions at the appropriate time is a higher-level task than is covered by this demo, but I think is pretty clearly doable with existing AI techniques (/a loop). How is this kind of speech to text, visual identification, decision making, motor control, multi-robot coordination and navigation of 3d space possible locally? If your question is "where's the GPUs", their "AI" marketing page[1] pretty clearly implies that compute is offloaded, and that only images and instructions are meaningfully "on board" each robot. I could see this violating the understanding of "totally local" that you mentioned up top, but IMHO those claims are just clarifying that the individual figures aren't controlled as one robot -- even if they ultimately employ the same hardware. Each period (7Hz?) two sets of instructions are generated. [1] https://www.figure.ai/ai https://www.figure.ai/ai What possible combo of model types are they stringing together? Or is this something novel? Again, I don't work in robotics at all, but have spent quite a while cataloguing all the available foundational models, and I wouldn't describe anything here as "totally novel" on the model level. Certainly impressive, but not, like, a theoretical breakthrough. Would love for an expert to correct me if I'm wrong, tho! EDIT: Oh and finally: Is anyone skeptical? How much of this is possible vs a staged tech demo to raise funding? Surely they are downplaying the difficulties of getting this setup perfectly, and don't show us how many bad runs it took to get these flawless clips. They are seeking to raise their valuation from ~$3B to ~$40B this month, sooooooo take that as you will ;) https://www.reuters.com/technology/artificial-intelligence/robotics-startup-figure-ai-talks-new-funding-395-billion-valuation-bloomberg-2025-02-14/ https://www.reuters.com/technology/artificial-intelligence/r...
- bbor 2y agoTo focus on something other than the obviously-terrifying nature of this and the skepticism that rightfully entails on our part: A fast reactive visuomotor policy that translates the latent semantic representations produced by S2 into precise continuous robot actions at 200 Hz Why 200Hz...? Any experts in here on robotics? Because to this layman that seems really often to update motor controls.
- dr_dshiv 2y agoWake me when robots can make a peanut butter sandwich
- kingkulk 2y agoAnyone have a link to their paper?
- Animats 2y ago"Pick up anything: Figure robots equipped with Helix can now pick up virtually any small household object, including thousands of items they have never encountered before, simply by following natural language prompts." If they can do that, why aren't they selling picking systems to Amazon by the tens of thousands?
- Frankion 2y agoPerhaps they are in the process of doing so? Perhaps it's possible to grip them but not to pack them?
- moshun 2y agoMost AI startups, like most startups in general, are in the business of selling futures. Much easier to get new seed $$$ once there’s a real hype around your demo. Not saying they aren’t being honest, just pointing out the logic of starting here and working your way up to a huge valuation.
- Animats 2y agoRight. As I point out occasionally, Tesla, as a car company, is overvalued by an order of magnitude. If they can find suckers who accept that valuation, it's much easier to exit as a billionaire than actually make it work. Visualize making it work. You build or buy a robot that has enough operating envelope for an Amazon picking station, provide it with an end-effector, and use this claimed general purpose software to control it. Probably just arms; it doesn't need to move around. Movement is handled by Amazon's Kiva-type AGV units. You set up a test station with a supply of Amazon products and put it to work. It's measured on the basis of picks per minute, failed picks, and mean time before failure. You spend months to years debugging standard robotics problems such as tendon wear, gripper wear, and products being damaged during picking and placing. Once it's working, Amazon buys some units and puts them to work in real distribution centers. More problems are found and solved. Now you have a unit that replaces one human, and costs maybe $20,000 to make in quantity. Amazon beats you down in price so you get to sell it for maybe $25,000 in quantity. You have to build manufacturing facilities and service depots. Success is Amazon buying 50,000 of them, for total income of $0.25 billion. This probably becomes profitable about five years from now, if it all works. By which time someone in China, Japan, or Taiwan is doing it cheaper and better.
- butifnot0701 2y agoIt's kinda eerie how they look at each other after handover
- ripped_britches 2y agoThis whole thread is just people who didn’t read the technical details or immediately doubt the video’s honesty. I’m actually fairly impressed with this because it’s one neural net which is the goal, and the two system paradigm is really cool. I don’t know much about robotics but this seems like the right direction.