7 ms·
Gemini Robotics On-Device brings AI to local robotic devices
- suninsight 1y agoThis will not end well.
- sajithdilshan 1y agoI wonder what kind of guardrails (like Three Laws of Robotics) there are to prevent the robots going crazy while executing the prompts
- hn_throwaway_99 1y agoA power cord?
- sajithdilshan 1y agowhat if they are battery powered?
- msgodel 1y agoUsually I put master disconnect switches on my robots just to make working on them safe. I use cheap toggle switches though I'm too cheap for the big red spiny ones.
- pixl97 1y ago[Robot learns to superglue the switch open]
- msgodel 1y agoIt's only going to do that if you RL it with episodes that include people shutting it down for safety. The RL I've done with my models are all simulations that don't even simulate the switch.
- pixl97 1y agoWhich will likely work for only on machine AI, but it seems to me any very complicated actions/interactions with the world may require external interactions with LLMs which know these kind of actions. Or in the future the models will be far larger and more expansive on device containing this kind of knowledge. For example, what if you need to train the model to keep unauthorized people from shutting it off?
- msgodel 1y agoHaving a robot near people with no master off switch sounds like a dumb idea.
- bigyabai 1y agoThat's what we use twelve gauge buckshot for, here in America.
- ctoth 1y agoThe laws of robotics were literally designed to cause conflict and facilitate strife in a fictional setting--I certainly hope no real goddamn system is built like that,. > To ensure robots behave safely, Gemini Robotics uses a multi-layered approach. "With the full Gemini Robotics, you are connecting to a model that is reasoning about what is safe to do, period," says Parada. "And then you have it talk to a VLA that actually produces options, and then that VLA calls a low-level controller, which typically has safety critical components, like how much force you can move or how fast you can move this arm."
- conception 1y agoOf course someone will. The terror nexus doesn’t build itself, yet, you know.
- hlfshell 1y agoThe generally accepted term for the research around this in robotics is Constitutional AI (https://arxiv.org/abs/2212.08073 https://arxiv.org/abs/2212.08073) and has been cited/experimented with in several robotics VLAs.
- JumpCrisscross 1y agoIs there any evidence we have the technical ability to put such ambiguous guardrails on LLMs?
- hlfshell 1y agoNo. We can add guardrails, but nothing that has been proven to work reliably.
- asadm 1y agoin practice, those laws are bs.
- Symmetry 1y agoCurrent guardrails are more IEC 61508 than anything like the three laws.
- suyash 1y agoWhat sort of hardware does the SDK runs on, can it run on a modern Raspberry Pi ?
- ethan_smith 1y agoAccording to the blog post, it requires an NVIDIA Jetson Orin with at least 8GB RAM, and they've optimized for Jetson AGX Orin (64GB) and Orin NX (16GB) modules.
- v9v 1y agoCould you quote where in the blog post they claim that? CTRL+F "Jetson" gave no results in TFA.
- moffkalast 1y agoYeah they didn't really mention anything, I was almost getting my hopes up that Google might be announcing a modernized Coral TPU for the transformer age, but I guess not. It's probably all just API calls to their TPUv6 data centers lmao.
- martythemaniak 1y agoYou can think of these as essentially multi-modal LLMs, which is to say you can have very small/fast ones (SmolVLA - 0.5B params) that are good at specific tasks, and larger/slower more general ones (OpenVLA - a finetuned llama2 7B). So a rpi could be used for some very specific tasks, but even the more general ones could run on beefy consumer hardware.
- estormy 1y agoThere's a post on x from one of the project contributors that says it fits on a 4090: https://x.com/sippeyxp/status/1937520297789497668 https://x.com/sippeyxp/status/1937520297789497668
- Toritori12 1y agoDoes Anyone know how easy is to join the "trusted tester program" and if they offer modules that you can easily plug-in to run the sdk?
- technotony 1y agoThere's a sign up button at the bottom of the article...
- martythemaniak 1y agoI've spent the last few months looking into VLAs and I'm convinced that they're gonna be a big deal, ie they very well might be the "chatgpt moment for robotics" that everyone's been anticipating. Multimodal LLMs already have a ton of built-in understanding of images and text, so VLAs are just regular MMLLMs that are fine-tuned to output a specific sequence of instructions that can be fed to a robot. OpenVLA, which came out last year, is a Llama2 fine tune with extra image encoding that outputs a 7-tuple of integers. The integers are rotation and translation inputs for a robot arm. If you give a vision llama2 a picture of a an apple and a bowl and say "put the apple in the bowl", it already understands apples, bowls, knows the end state should apple in bowl etc. What missing is a series of tuples that will correctly manipulate the arm to do that, and the way they did it is through a large number of short instruction videos. The neat part is that although everyone is focusing on robot arms manipulating objects at the moment, there's no reason this method can't be applied to any task. Want a smart lawnmower? It already understands "lawn" "mow", "don't destroy toy in path" etc, just needs a finetune on how to corectly operate a lawnmower. Sam Altman made some comments about having self-driving technology recently and I'm certain it's a chat-gpt based VLA. After all, if you give chatgpt a picture of a street, it knows what's a car, pedestrian, etc. It doesn't know how to output the correct turn/go/stop commands, and it does need a great deal of diverse data, but there's no reason why it can't do it. https://www.reddit.com/r/SelfDrivingCars/comments/1le7iq4/sam_altman_i_think_we_have_some_new_technology/ https://www.reddit.com/r/SelfDrivingCars/comments/1le7iq4/sa... Anyway, super exciting stuff. If I had time, I'd rig a snowblower with a remote control setup, record a bunch of runs and get a VLA to clean my driveway while I sleep.
- ckcheng 1y agoVLA = Vision-language-action model: https://en.wikipedia.org/wiki/Vision-language-action_model https://en.wikipedia.org/wiki/Vision-language-action_model Not https://public.nrao.edu/telescopes/VLA/ https://public.nrao.edu/telescopes/VLA/ :( For completeness, MMLLM = Multimodal Large language model.
- generalizations 1y agoI will be surprised if VLAs stick around, based on your description. That sounds far too low-level. Better hand that off to the 'nervous system' / kernel of the robot - it's not like humans explicitly think about the rotation of their hip & ankle when they walk. Sounds like a bad abstraction.
- baron816 1y agoI’m optimistic about humanoid robotics, but I’m curious about the reliability issue. Biological limbs and hands are quite miraculous when you consider that they are able to constantly interact with the world, which entails some natural wear and tear, but then constantly heal themselves.
- marinmania 1y agoIt does either get very exciting or very spooky thinking of the possibilities in the near future. I had always assumed that such a robot would be very specific (like a cleaning robot) but it does seem like by the time they are ready they will be very generalizable. I know they would require quite a few sensors and motors, but compared to self-driving cars their liability would be less and they would use far less material.
- fragmede 1y agoThe exciting part comes when two robots are able to do repairs on each other.
- pryelluw 1y ago2 bots 1 bolt ?
- marinmania 1y agoI think this is the spooky part. I feel dumb saying it, but is there a point where they are able to coordinate and build a factory to build chips/more of themselves? Or other things entirely?
- bamboozled 1y agoOf course there is
- ta988 1y agoBut this still has a massive cost. Replacing or repairing an actuator isn't cheap, in material and in time of unavailability.
- zzzeek 1y agoTHANK YOU. Please make robots. LLMs should be put to work for *manual* tasks, not art/creative/intellectual tasks. The goal is to improve humanity. not put us to work putting screws inside of iphones (five years later) what do you mean you are using a robot for your drummer
- Workaccount2 1y agoI continued to be impressed how Google stealth releases fairly groundbreaking products, and then (usually) just kind of forgets about them. Rather than advertising blitz and flashy press events, they just do blog posts that tech heads circulate, forget about, and then wonder 3-4 years later "whatever happened to that?" This looks awesome. I look forward to someone else building a start-up on this and turning it into a great product.
- fusionadvocate 1y agoBecause the whole purpose of these kinds of projects at Google is to keep regulators at bay. They don't need these products in the sense of making money from them. They will just burn some money and move on, exactly the way they did hundreds of times. But what kind of company has such a free pass to burning money? The kind of company that is a monopoly. Monopolies are THAT profitable.
- jagger27 1y agoThese are going to be war machines, make absolutely no mistake about it. On-device autonomy is the perfect foil to escape centralized authority and accountability. There’s no human behind the drone to charge for war crimes. It’s what they’ve always dreamed of. Who’s going to stop them? Who’s going to say no? The military contracts are too big to say no to, and they might not have a choice. The elimination of toil will mean the elimination of humans all together. That’s where we’re headed. There will be no profitable life left for you, and you will be liquidated by “AI-Powered Automation for Every Decision”[0]. Every. Decision. It’s so transparent. The optimists in this thread are baffling. 0: https://www.palantir.com/ https://www.palantir.com/
- mateus1 1y agoMIT spinoff Google-owned Boston Dynamics pledged not to militarize their robots. Which is very hard to believe given they're backed by DARPA, the DoD/Military investment arm.
- JumpCrisscross 1y ago> These are going to be war machines, make absolutely no mistake about it Of course they will. Practically everything useful has a military application. I'm not sure why this is considered a hot take.
- polskibus 1y agoWhat is the model architecture? I'm assuming it's far away from LLMs, but I'm curious about knowing more. Can anyone provide links that describe architectures for VLA?
- KoolKat23 1y agoActually very close to one I'd say. It's a "visual language action" VLA model "built on the foundations of Gemini 2.0". As Gemini 2.0 has native language, audio and video support, I suspect it has been adapted to include native "action" data too, perhaps only on output fine-tuning rather than input/output at training stage (given its Gemini 2.0 foundation). Natively multimodal LLM's are basically brains.
- martythemaniak 1y agoOpenVLA is basically a slightly modified, fine-tuned llama2. I found the launch/intro talk by lead author to be quite accessible: https://www.youtube.com/watch?v=-0s0v3q7mBk https://www.youtube.com/watch?v=-0s0v3q7mBk
- m00x 1y agoA more modern one, smolVLA is similar and uses a VLM but skips a few layers and uses an action adapter for outputs. Both are from HF and run on LeRobot. https://arxiv.org/abs/2506.01844 https://arxiv.org/abs/2506.01844 Explanation by PhosphoAI: https://www.youtube.com/watch?v=00A6j02v450 https://www.youtube.com/watch?v=00A6j02v450
- KoolKat23 1y agoIn the paper at the bottom of googles page, this VLA says it is built on the foundations of Gemini 2.0 (hence my quotations). They'd be using Gemini 2.0 rather than llama. https://arxiv.org/pdf/2503.20020 https://arxiv.org/pdf/2503.20020
- quantumHazer 1y ago> Natively multimodal LLM's are basically brains. Absolutely not.
- moelf 1y agoThe MuJoCo link actually points to https://github.com/google-deepmind/aloha_sim https://github.com/google-deepmind/aloha_sim
- westurner 1y agomujoco_menagerie has Mujoco MJCF XML models of various robots. google-deepmind/mujoco_menagerie: https://github.com/google-deepmind/mujoco_menagerie https://github.com/google-deepmind/mujoco_menagerie mujoco_menagerie/aloha: https://github.com/google-deepmind/mujoco_menagerie/tree/main/aloha https://github.com/google-deepmind/mujoco_menagerie/tree/mai...
- antonkar 1y agoThe only way to prevent robots from being jailbroken and set to rob banks is to move GPUs to private SOTA secure GPU clouds
- TZubiri 1y agoNice. I work with some students younger than 13, so most cloud and llms are quite tricky to work with, local only models like vertex are nice for this use case. I will try this as a replacement for chatgpt as Computer Vision in robotics like Lego Mindstorm
- MidoriGlow 1y agoElon Musk said in last week’s Starship Update: the very first Mars missions are planned to be flown by Optimus humanoid robots to scout and build basic infrastructure before humans arrive (full transcript + audio: https://transpocket.com/share/oUKhep6cUl3s/ https://transpocket.com/share/oUKhep6cUl3s/). If Gemini Robotics On-Device can truly adapt to new tasks with ~50–100 demos, pairing that with mass-produced Optimus bodies and Starship’s lift capacity could be powerful—offline autonomy, zero-latency control, and the ability to ship dozens of robots per launch.
- lm28469 1y agoElon Musk said in 2016 that we'd have fully autonomous cars by the end of the year and we'd be on Mars by 2018, with manned missions by 2024. Fast forward to 2025, weeks have no self driving cars, and nothing is even close to getting to Mars, let alone manned
- san1927 1y agomeanwhile i will drink a coffee while it loads a reply from the API