10 ms·
Figure 01 robot demos its OpenAI integration
- peterhgb 3y ago[dead]
- mvkel 3y agoThe ability to translate between text to servo movement is unreal, and it looks like gpt4 vision + whisper are heavily used. They're also using the term "reasoning" which is... new. Can you call this an AI wrapper company? Kinda! The medium is a little different than an app, of course. Lots of amazing applications of AI even if frontier AI development froze today.
- jaffee 3y ago> text to servo movement yeah this was super impressive. If this is at the point where you can put an arbitrary object in front of it and ask it to move it somewhere, that's going to be huge for industrial automation type stuff I'd imagine. I do wonder how much of that demo was pre-baked/trained though. Could they repeat the same thing with a banana? What if the table was more cluttered? What if there were two people in the frame?
- mvkel 3y agoGreat question. Boston Dynamics has been demoing the pre-baked dance routine for 2+ decades at this point. Really hoping we can evolve past it.
- tintor 3y agoNot true. Boston Dynamics has been demoing long and complicated single take videos involving walking, running, jumping and manipulation, in less than ideal conditions.
- archermarks 3y agoKnowing as many people in the robotics space as I do, I suspect the demo may not be completely "pre-baked" but it is almost certaintly highly selected. Often they'll try the demo many many times until they get a clean run-through without mistakes. The circumstances are also likely pretty idealized, like they pick objects and settings that they know it performs well in.
- mvkel 3y agoInteresting! This would sort of explain the low energy of the human demonstrator in the video "Take 488... action!"
- superzamp 3y agoInteresting stutter at 0:53
- DalasNoin 3y agoSimilar at 1:47 I... I think It sounds so human, a person would also stutter at an introspective question like this. I wonder if their text to speech was trained on human data and produces these artifacts of human speech, or if it is intentional.
- mordymoop 3y agoI use ChatGPT voice a lot, and it is prone to this exact type of stutter. I don’t think it’s intentional. I think there are certain phonetic/tonal linkages that are naturally “weird” (uncommon in the training corpus) and that AI struggle with them. Why this struggle manifests as a very human-like stutter is a fascinating question.
- austinkhale 3y agoI'm not sure when OpenAI added them, but you can hear similar things when using the ChatGPT voice mode on iOS. Sometimes it feels almost like a latency stutter and other times it feels intentionally human.
- andoando 3y agoI would have added umms and hmms artificially just to make the latency less apparent, so Id say good chance thats what they did lol
- bigyikes 3y agoNow that you mention it, I, uh, also add umms when my speech pathways experience high latency.
- ilaksh 3y agoI believe it's Eleven Labs API with the Stability setting turned down a little bit. It is definitely trained on human speech and when you use a somewhat lower setting than default, it will insert those types of natural imperfections or pauses and is very realistic.
- deleted 3y ago[deleted]
- DalasNoin 3y agoThis humanoid form plus the voice really gives of a different feeling than the pure chat version. I think it will feel even more profound if they can add eyes and eye contact. Imagining demoing this to a random person.
- fnordpiglet 3y agoOn mobile chatgpt if you press the headphones you can voice chat with that voice already
- eigenvalue 3y agoThis shows the real utility of the Groq low latency inference. That delay in responding makes it much less impressive (it’s still really impressive obviously!)
- modeless 3y agoThat delay will be eliminated very soon. IMO low latency natural voice conversations are going to be bigger than ChatGPT. It's going to blow people's minds when they can converse with these AIs casually just like with their real life friends. It won't be anything like Siri or Alexa anymore. Here's a demo from a startup in this space. Still very early. https://deck.sindarin.tech/ https://deck.sindarin.tech/
- tomcam 3y agoI’ll be thrilled when Siri can spell my wife’s name correctly after 13 years of continuous usage and explicitly training it in her name. Admittedly her name is wildly complicated and totally unknown to the software folks at Apple: Ada Also my radio-trained voice is so generic a caller every week-ish assumes I am a bot, so I’m pretty sure the problem isn’t me enunciation or accent.
- margorczynski 3y agoLLMs can't really spell as the smallest "block" of information they operate on are tokenized whole words. Same issue with e.g. arithmetic.
- ta8645 3y agoBut they should be able to select the correct token for homophones, which amounts to the same thing.
- jxy 3y agoCan we stop this kind of misinformation? Training a model to map a token to individual letters are no harder than training a model to be fluent at English. Arithmetics with small number of digits are achievable as well. You can just try a small 7B model yourself. If you don't know where to start, try the mistral instruct v0.2, and this is how it goes, > [INST] Spell out the following word letter by letter: margorczynski [/INST] m - a - r - g - o - r - c - z - y - n - s - k - i > So, the word "margorczynski" spelled out letter by letter is: m-a-r-g-o-r-c-z-y-n-s-k-i. The text between `[INST]` and `[/INST]` is the input. The text after `[/INST]` is the output.
- daveguy 3y agoThe speech to servo movement is impressive as others pointed out. What strikes me as amazing is the speed with which it is performing tasks that require dexterity. This is the first object manipulation robot demo I have seen that didn't require speeding up the video for it to look "natural".
- huytersd 3y agoHow though. It’s probably just predefined actions that are triggered by the LLM output. At the same time it would be impressive if the LLM determined the right function to call in real time, was able to deal with the ambiguity in placement of the garbage and bonus points if it could do that in a scenario that wasn’t hardcoded to exactly standing behind that table in that spot.
- famouswaffles 3y agoThe robot movements are separate end-to-end neural networks. They're triggered by the LLM but aren't hardcoded.
- margorczynski 3y agoWhy does it then after performing an action or whole plan return to the "default" position with the hands in that strange stance? Looks kinda like there is some "hard coded" flow that simply uses the LLM(s) to perform actions.
- famouswaffles 3y agoThat's a closed loop operation. It's pretty common to train these robots on closed loop demonstrations.
- huytersd 3y agoDo you know this for a fact or are you speculating?
- gotrythis 3y agoSomeone, please ask OpenAI to stop artificially dumbing down ChatGPT by adding "um" to the audio output. I get that it is supposed to make it more human-like or something, but every time I hear it do that, I cringe and feel sad for humanity.
- niek_pas 3y agoFunny, I have the exact opposite response. It does, indeed, make it seem more human-like to me.
- gotrythis 3y agoI agree. That's the sad part. I want super-intelligence to sound intelligent and not artificially brought down to our level. :-)
- ilaksh 3y agoIf they are using Eleven Labs, this is just the Stability setting. Turning it down will make it more realistic and closer to the training data. That is what causes the pauses and imperfections. You can sign up and use their Voice Lab for free or maybe a few bucks and experiment with the slider for Stability and the other setting. In my opinion, turning Stability down just a little bit to demo extremely realistic speech is a no-brainer. They could have turned it up and made it ultra-smooth, but that makes no sense. Why make your robot demo less realistic deliberately?
- empath-nirvana 3y agoWhat if it just inserts filler words when the text generation is too slow, to make it sound more natural. It's exactly what people do when they're thinking about what to say next.
- deleted 3y ago[deleted]
- ChrisArchitect 3y agoRelated release about the deal: Figure Raises $675M at $2.6B Valuation;Signs Collaboration Agreement with OpenAI https://news.ycombinator.com/item?id=39553560 https://news.ycombinator.com/item?id=39553560
- thebiglebrewski 3y agoOmg. This is really impressive stuff! I'm sure it's a bit cherry-picked and they chose things it is good at. But it is already showing some useful stuff.
- d--b 3y agoNow that's a robot that will be fun jailbreaking. The safety hazards are robocop-level.
- iknownthing 3y agoInteresting how it put the trash into the basket when that wasn't explicitly asked for.
- fnordpiglet 3y agoOne of the remarkable things about LLM IMO is its ability to perform something akin to abductive semantic reasoning, which this is an example of. It inferred probabilistically from the semantic context that’s the appropriate action. This is an absolutely remarkable ability of LLMs that IMO is the most important capability they bring to the table over all prior AI techniques. (All usual caveats of stochastic parrots and non reasoning apply)
- bilsbie 3y agoI found this discussion interesting. Talking about this as a new form of labor. https://twitter.com/CernBasher/status/1767939757105991791 https://twitter.com/CernBasher/status/1767939757105991791
- ordinaryradical 3y agoReads like an ad written by GPT, honestly. But this guy is a professional advice giver, so to be expected? Wouldn’t surprise me if he outsourced his tweeting.
- lelag 3y agoIt's a really cool demo and I'm impressed by the dexterity of the robot, however I'm a bit underwhelm by what's shown here in the sense that the speech and reasoning capabilities is just obvious to anyone who's been paying attention and has experience with GPT4. The function calling was great, but it had a very simple "world" to interact with. It's really interesting to see it integrated with a robot that can interact with the world though. I think that what's really holding back the current crop of Gen AI is inference cost and speed. When we figure out to get thousands of token per seconds for cheap, I think we will be able to bruteforce many hard problems and actually start seeing amazing applications like this one (but in production rather than a cool demo).
- golol 3y agoYea of course this is just an LLM interacting through a very crude interface with some control algorithms, but I think it is amazing that we sort of have an approach for both ends of the complexity spectrum now: LLMs for high level, vague, common sense reasoning, and traditional robot control, planning, machine learning methods for the physicak execution of simple movements. We just have to gradually connect these two systems.
- mjfisher 3y agoI take your point about familiarity with GPT4 making this less immediately impressive - but just as an end to end demo it's absolutely mind-blowing how far we've come. Can you imagine seeing this ten years ago? Moving so far in such a short time frame would have been unbelievable. This gave me the "I'm actually living in the future" vibes that I always imagined I'd get from flying cars.
- deleted 3y ago[deleted]
- razodactyl 3y agoI think the big deal here is that it's an end to end system. It's very easy to forget the single line hallucinations of GPT3 not too long ago. Look into providers like Groq that are running real-time inference with Transformer models. It's only a matter of models improving then the hardware coming to market that this will be a major leap in robotic capabilities.
- marstall 3y agoafter that faked google gemini AI video from awhile back, I've got a healthy dose of skepticism about these next-gen demos. obviously they've done a lot though, kudos.
- asylteltine 3y ago[dead]
- unraveller 3y agoalternate vid source https://www.youtube.com/watch?v=Sq1QZB5baNw https://www.youtube.com/watch?v=Sq1QZB5baNw
- margorczynski 3y agoQuestion is how much this is cherry-picked as we all remember the "demo" of Bard. It would be nice to see it thrown into some random environment and then asked to do stuff, otherwise this has little value aside from marketing.
- modeless 3y agoSelecting one of a set of pre-trained actions by voice is cool but not exactly ground-breaking. Using GPT-4V to describe a scene is also pretty simple. The most impressive things here to me are the speed of picking up the trash and the fluid passing of objects between hands. It's unclear how general these movement policies are though. The way that guy is standing perfectly still makes me think that it would fail if everything wasn't set up just so. I'd like to see demos with more variation. I don't want to be too negative here though. I think it's a great demo and I can't wait to see more.
- fragmede 3y agoThe Ok-robot demo shows that the technology for it to be fairly general is there, though no idea if figure one is using their technology or not. Simply being able to command a robot instead of moving a turtle with gcode is nothing short of astounding to those who aren’t deeply involved and tracking the sota progress in this area. https://ok-robot.github.io/ https://ok-robot.github.io/
- Animats 3y ago> Selecting one of a set of pre-trained actions by voice is cool but not exactly ground-breaking. Yes. Compare "Put That There" (1979).[1] > The way that guy is standing perfectly still makes me think that it would fail if everything wasn't set up just so. I'd like to see demos with more variation. Yes. Unstructured manipulation is hard. Structured robotic manipulation is pretty standard. Picking isolated objects is a solved problem. Here's a robot recycling sorting system, "Max-AI".[2] That's been in use for years. San Francisco recycling uses those robots. So do many other cities. (That's from "Bulk Handling Systems", a company which does exactly what their name says. Recycling and trash come in bulk, and their machines handle it. Shakers, magnets, screens, and vision-based air sorters do 95% of the sorting. The robots only handle the hard cases. This is the no-bullshit end of AI.) [1] https://www.youtube.com/watch?v=RyBEUyEtxQo https://www.youtube.com/watch?v=RyBEUyEtxQo [2] https://max-ai.com/ https://max-ai.com/
- KoolKat23 3y agoI suspect it's less pre-trained than you think, the tweet is prefaced with a message that all actions are driven by neural networks indicating that it's probably adjusting for the objects, the environment etc.
- zvmaz 3y agoYes, perhaps the demonstration can be "demystified," but I can't resist but be astounded by the robot. A few years ago this was unimaginable and only seen in science fiction movies. Truly flabbergasted.
- oliwary 3y agoThis is cool. But why would the plate go in the drying rack? It is obviously dirty since there was trash and an apple on it. It should have been washed first.
- educaysean 3y agoI hadn't even noticed this leap in logic. Good catch.
- basil-rash 3y agoI’m certainly not eating at Figure 1’s house!
- tintor 3y agoBecause the robot likely doesn't have any memory. It sees empty plate after the trash got removed.
- userabchn 3y agoOne of my colleagues predicted, when ChatGPT was first released, that AI would reduce the value of knowledge work relative to manual labour, but I disagreed as I think the main thing holding back robots from replacing many manual labour jobs at the moment is the difficulty communicating with them. I argued that ChatGPT indicated that we were not too far away from being able to tell a robot to pick the apples from a particular row and put them in the green barn as the usual red barn was being painted today. This video suggests that I was right, and I in fact suspect that such manual labour is a more realistic type of work for AI (with robots) to substantially replace first than most knowledge work, where I think it will remain as an assistant for some time.
- iamflimflam1 3y agoI think we are going to find that the Moravec paradox is wrong. Interesting times…
- imtringued 3y agoAssuming you already have a robot, it is going to be easier to teach it to clean your toilet than to make it write a scientific paper that is worth publishing.
- trashtester 3y agoWhen this is over, the relative value of manual labor to knowledge work will be 0:0.
- digitalsalvatn 3y agoThe singularity is nigh! We must work towards it so humans can be free from the drudgery of work and can work towards whatever their heart desires! Join us on our quest for Digital Salvation.
- sciencesama 3y agoThought he is going to give it knives and ask it to cut the apple !!
- ProfessorZoom 3y agome when my chatgpt wrapper robot can't respond cause the wifi is down
- nemothekid 3y agoThe most impressive part of this demo, to me, is the robot "seeing" and picking up objects with human-like appendages. I must have missed something, but I was under the impression that this was very hard. As I understand it inverse kinematics is pretty hard - did they solve it with NNs?
- tintor 3y agoTry poking it with a hockey stick (Boston Dynamic style)
- beefnugs 3y agoI wish I was a 5 year old that didn't actually know how infeasible and useless this all is, and could just be positive about the future for once. But humans cant even figure out you can't operate an "imaginary number go up" underneath the basic human requirement of rent for shelter, there is no way they can make this technology useful or affordable or reliable or good
- Solvency 3y agoFully agreed. This is going to turn into novelty nonsense, like some fashion designer will have a robot runway event and it'll drum up a bunch of stupid press. Then the military will take the rest over into advancing drone technology.
- golol 3y agoI don't understand this negative sentiment. We are finally making progress on real physical robot labor. This has nothing to do with "imaginary number go up", this is about more labor being available to assemble, clean, transport, cook, cut, harvest, build, etc. i.e. things that really matter and are the bottlenecks to our livves being more comfortable. This stuff is not infeasible, these robots can be built for the price of a car. There is still a ways to go from this to androids, but it is legitimately technologically feasible now.
- ShamelessC 3y agoIs the music and color correction deliberately designed to give off creepy sci fi vibes?
- meindnoch 3y agoThe robot is rendered. Focus depth is off in multiple places, the movements lack real-world inertia, and the shading is too sterile. HN you're not supposed to be this gullible...