6 ms·
SIMA 2: An agent that plays, reasons, and learns with you in virtual 3D worlds
- deleted 11mo ago[deleted]
- Workaccount2 11mo ago>We’ve observed that, throughout the course of training, SIMA 2 agents can perform increasingly complex and new tasks, bootstrapped by trial-and-error and Gemini-based feedback. >In subsequent training, SIMA 2’s own experience data can then be used to train the next, even more capable version of the agent. We were even able to leverage SIMA 2’s capacity for self-improvement in newly created Genie environments – a major milestone toward training general agents across diverse, generated worlds. Pretty neat, I wonder how that works with Gemini, I suppose SIMA is a model (agent?) that runs on top of it?
- FuckButtons 11mo agoThat’s what it sounded like to me, a plain text interface between two distinct systems.
- kridsdale1 11mo agoThat’s what Claude Plays Pokémon is.
- ukuina 11mo agoAt 0:52 in their demo video, there is a grammatical inconsistency in the agent's text output. The annotations in the video are therefore suspected to be created by humans after the fact. Is Google up to their old marketing/hyping tricks again? > SIMA 2 Reasoning: > The user wants me to go to the ‘tomato house’. Based on the description ‘ripe tomato’, I identify the red house down the street.
- m_w_ 11mo agoI can't speak to the content of the actual game being played, but it wouldn't surprise me if there was an in-game text prompt: > "The house that looks like a ripe tomato!" that was transformed into a "user prompt" in a more instructional format > "Go to the tomato house" And both were used in the agent output. At least the Y-axes on the graphs look more reasonable than some other recent benchmarks.
- vessenes 11mo agoThe scene just before you describe has the user write "ripe tomato" in the description - you can see it in the video. The summary elides it, but the "ripe tomato" instruction is also clearly part of the context.
- golol 11mo agoThe gap between high level and low level control of robots is closing. Right now thousands of hours of task specific training data is being collected and trained on to create models that can control robots to execute specific tasks in specific contexts. This essentially turns the operation of a robot into a kind of video game, where inputs are only needed a in low-dimensional abstract form, such as "empty the dishwasher" or "repeat what I do" or "put your finger in the loop and pull the string". This will be combined with high-level control agents like SIMA 2 to create useful real-world robots.
- catgary 11mo agoI work on a much easier problem (physics-based character animation) after spending a few years in motion planning, and I haven’t really seen anything to suggest that the problem is going to be solved any time soon by collecting more data.
- wordpad 11mo agoWhy? Physics of large discrete objects (such as a robot) isn't very complicated. I thought it's fast accurate OCR that's holding everything back.
- markisus 11mo agoThe problem becomes complicated once the large discrete objects are not actuated. Even worse if the large discrete objects are not consistently observable because of occlusions or other sensor limitations. And almost impossible if the large discrete objects are actuated by other agents with potentially adversarial goals. Self driving cars, an application in which physics is simple and arguably two dimensional, have taken more than a decade to get to a deployable solution.
- jcims 11mo agoI just grabbed a beer about ten minutes ago. Next to zero cognition was involved in the process. There's some kind of hierarchy of thought in the way my mind/brain/body processed the task. I did cognitively decide to get the beer, but I was focused on something at work and continued to think about that in great detail as the rest of me did all of the motion planning and articulation required to get up, walk through two doorways, open the door on the fridge, grab a beer, close the door, walk back and crack the beer as I was sitting down. Basically zero thought in that entire sequence. I think what's happening today with all of this stuff is ultimately like me trying to play Fur Elise on piano. I don't have a piano. I don't know how to play one. I'm going to be all brain in that entire process and it's going to be awful. We need to learn how to use the data we have to train these layers of abstraction that allow us to effectively compress tons of sophistication into 'get a beer'.
- oersted 11mo agoI get why they do it, they are a business. I just wish Google would get off their ivory tower and build in the open more like they used to (did they? maybe I'm misremembering...). They've acquired this bad habit of keeping all their scientific experiments closed by default and just publishing press releases. I wish it was open-source by default and closed just when there's a good reason. Don't get me wrong, I suppose this is more of a compliment. I really like what they are doing and I wish we could all participate in these advances.
- singularity2001 11mo agoDreamer v3 was open, v4 coming soon?
- mashyc 11mo agov4 is out: https://danijar.com/project/dreamer4/ https://danijar.com/project/dreamer4/
- chankstein38 11mo agoSame! I want to play with this so bad!
- tinfoilhatter 11mo ago[flagged]
- jandrese 11mo agoIt seems pretty clear to me that they're trying to develop AGI humanoid assistants/workers without the messy and expensive real world hardware. Basically approaching the problem from the other end than a company like Tesla that built a robot and are now trying to figure out how to make a computer drive it without needing constant hand holding.
- dang 11mo agoCould you please stop posting flamebait and breaking the site guidelines? You've unfortunately been doing it repeatedly, including this dreadful thread from a couple weeks ago: https://news.ycombinator.com/item?id=45781981 https://news.ycombinator.com/item?id=45781981. I realize the other person was doing it also, but you (<-- I don't mean you personally, but all of us) need to follow the rules regardless of what other people are doing. Comments like what your account has been posting are not what this site is for, and destroy what it is for, so if you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
- JohnMakin 11mo agoIsn't most of this demo no man's sky? The voiceover doesn't make it clear that the world is not generated by SIMA.
- xnx 11mo agoIt's hard to keep up with the many different models and pace of progress. Genie 3 is Google's world generating model: https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/ https://deepmind.google/blog/genie-3-a-new-frontier-for-worl...
- lawlessone 11mo agoIf it can get through those lengthy glitchy nms story mission tutorials quickly it's already a super intelligence. as much as some AI annoys me. This would be great for making games more accessible.
- JohnMakin 11mo agoI guess but, there is also a lot of data out there knowing how to do it, whether that’s via video training or reddit posts. I want to see the whole video
- tintor 11mo agoThis is not a world generating model. It is a game playing model.
- JohnMakin 11mo agoAnd my post is saying that if you don't really know better, from the narration, you'd think google also generated the world. At least that was my impression, and I'm vaguely familiar with these things.
- woeirua 11mo agoYet another blogpost that looks super impressive, until you get to the bottom and see the charts assessing held out task performance on ASKA and MineDojo and see that it's still a paltry 15% success rate. (Holy misleading chart batman!) Yes, it's a major improvement over SIMA 1, but we are still a long way from this being useful for most people.
- Workaccount2 11mo agoTo be fair, it's 65% on all tasks (with a 75% human baseline) and 15% on unseen environments. They don't provide a human baseline for that, but I'd imagine it's much more than 15%.
- woeirua 11mo agoIt really feels like we are determined to simulate every possible task in every possible environment instead of building true intelligence.
- falcor84 11mo agoI personally am extremely impressed about it reaching 15% on unseen environments. Note that just this year, we were surprised that LLMs became capable of making any progress whatsoever in GBA Pokemon games (that have significantly simpler worlds and control schemes). As for "true intelligence" - I honestly don't think that there is such a thing. We humans have brains that are wired based on our ancestors evolving for billions of years "in every possible environment", and then with that in place, each individual human still needs quite a few years of statistical learning (and guided learning) to be able to function independently. Obviously I'm not claiming that SIMA 2 is as intelligent as a human, or even that it's on the way there, but based on recent progress, I would be very surprised if we don't see humanoid robots using a approaches inspired by this navigate our streets in a decade or so.
- Gooblebrai 11mo agoI'm curious what's your definition of "true intelligence"
- eminence32 11mo agoThis is obviously just a research project, but I do wonder about the next steps: * After exploring an learning about a virtual world, can anything at all be transferred to an agent operating in the real world? Or would an agent operating in the real world have to be trained exclusively or partially in the real world? * These virtual worlds are obviously limited in a lot of important ways (for example, character locomotion in a game is absolutely nothing like how a multi-limbed robot moves). Does there eventually need to be more sophisticated virtual worlds that more closely mirror our real world? * Google seems clearly interested in generalized agents and AGI, but I'm actually somewhat interested in AI agents in video games too. Many video games have companion NPCs that you can sort of give tasks to, but in almost all cases, the companion NPCs are nearly uncontrollable and very limited in what they can actually do.
- nharada 11mo agoThe end goal is to marry the lessons learned about HOW to learn in a virtual world with a high fidelity world model that's currently out of reach for this generation of AI. In a year or two once we have a world model that's realistic enough and fast enough, robots will be trained there and then (hopefully) generalize easily to the real world. This is groundwork trying to understand how to do that without having the models required to do it for real.
- mkoubaa 11mo agoLook into the sim2real problem in robotics
- theLiminator 11mo agoWould be cool to see if they could make it play starcraft too and pit it against alphastar.
- gs17 11mo agoI'm almost 100% confident AlphaStar would win that match, but I'd love to watch it.
- theLiminator 11mo agoOh yeah, for sure, I'm more so curious how it could do.
- falcor84 11mo agoFrom what I see, SIMA only focuses on games where you control a single avatar from a 1st/3rd person perspective, and would assume that switching to a non-embodied game where you need to control the whole army at once would require significant retraining.
- gs17 11mo agoI hope we can get some (ideally local) version of this we can use as a "gaming minion". There's a lot of games where I probably would have played more if I could delegate the grind. If they're not that competent, it adds to the fun a little even.
- JLCarveth 11mo agoI would love Minecraft with more intelligent villagers I could boss around to mine and build for me.
- yeasku 11mo agoSo factorio?
- JLCarveth 11mo agoNo.
- goda90 11mo agoYou should look into modding. There have got to be a ton of automation and NPC scripting mods out there without any sort of AI model necessary.
- DanHulton 11mo agoYou should look into MineColonies, if you haven't yet! https://www.curseforge.com/minecraft/mc-mods/minecolonies https://www.curseforge.com/minecraft/mc-mods/minecolonies
- efficax 11mo agosorry this is kind of nuts to me. You want something to play video games for you because the video game isn't fun? Just play a game that is fun. The point of the game is to play it
- qoez 11mo agoIt could be fun in a factorio sense. Maybe the whole game becomes to delegate a bunch of smart robots and handle organization etc.
- deleted 11mo ago[deleted]
- vessenes 11mo agoOK, AI playing video games is cool. But you know what's really really cool? It looks like SIMA 2 is controlling the mouse and reading the screen at something approaching 30+fps. WANT. Computer use agents are so slow right now, this is really something. I wonder what the architecture is for this.
- almosthere 11mo agoIts even cooler if humans find something to be excited about in this world, since AI is replacing everything we do.
- Fuzzwah 11mo agoYou do realise the things that AI are replacing are very new things humans have only been doing for the last ~50 years?
- Workaccount2 11mo agoI desperately want an AI agent that can use my phone for me. Just something that takes instructs for each screen and execute it. "Open Chrome" "Go to xyz.com" "open hamburger menu" "Click login" etc. etc.
- tantalor 11mo agoIsn't that what the voice a11y tools have been doing for years. Why do you need AI for that. https://support.google.com/accessibility/android/answer/6151848 https://support.google.com/accessibility/android/answer/6151... https://support.apple.com/en-us/111778 https://support.apple.com/en-us/111778
- tschellenbach 11mo agoIt's like the factorio moment where you unlock the roboport. No more manual changes to the world, drone swarms to build housing, roads, bridges, parks etc. so exciting.
- sudosteph 11mo agoCould make for some very interesting Digimon games in the future.
- andai 11mo agoSo, can this learn in real time or not? The video implies that it can, but the blog says that they trained it in generations. (Feeding its experience data back into the training.)
- cadamsdotcom 11mo agoWhy would anyone care that a big company paid for some people to go on an inhouse research excursion. Real artists ship.
- _carbyau_ 11mo agoI want a smarter game. Like a survival game that - as usual - starts with you collecting sticks and stone to build a stone axe. But at the appropriate tech level, transitions into automation. You discovered a new building material and want to build a castle from it? Equip your NPC's with diamond pickaxes and tell them how much better/safer life would be if they built a new castle from unobtanium. And off they go. To not just mine it, but also do all the supporting logistics like farming to make food/shelter/watersupply/defences for more villagers to do more work in the quarry to get more unobtanium. You get to be the big boss and flit around with your special abilities of whatever suits your story. While some people on HN are the boss or higher up - getting to be the big boss and tell a bunch of "smart" characters what to do is a fantasy for most people.
- reactordev 11mo agoso you want to play The Sims? Honestly, that's the perfect avenue for this kind of AI agent training / NPC to command. Maybe not playing house but like you said, mining unobtanium for your doomsday weapon. Just expect Sir Charles to start crying at the sight of blood.
- _carbyau_ 11mo agoThe Sims was a great idea in a sandbox. I honestly hoped it would lead to other games having more intelligent/immersive NPCs. I want my silver-spooned Sir Charles to cry at the sight of blood and be beheaded by Gazza, the self made, born-of-the-streets gang boss who has an arrangement with Prince Harry, the younger brother. I then want to be able to challenge Gazza, take his criminal gang and rule the underworld of "Fantasy Kingdom XYZ". Perhaps "come to an understanding" with King Harry, maybe work my way up to spymaster and orchestrate neighbouring kingdoms to go to war with each other via a series of covert operations leaving selective evidence - say like the Princess Bride story but the bad guys(me!) win. Or maybe I don't win because I create a nemesis or fail to keep King Harry in check, get my gang wiped out, and have to seek a new way to the top. I don't want the Sims obviously. But I want something... more than what we have now. Like the Sims mixed with Dwarf fortress but I get to be part of the story and influence it in outsized ways. Edit: snipped to keep to a point, I did mention that most every game quest is based on violence.
- keyle 11mo agoTangential question: is this going to eventually ruin e-sports? What happens when a team of humans are playing against a team of AI, which play in the same conditions, with network lag etc. from a client computer perspective... And consistently beat the human counterparts for being faster at response time, never make mistakes and not ever getting tired? Eventually it could kill all MMOs, fill them up with AI players, "farm" with AI that never sleeps, ruin counter strike type games online, etc. Another arms race?
- ElSucioDan 11mo agoI wouldn’t think so. Chess already has an e-sport component and remains enjoyable even though computers have been stronger than humans for quite some time. I expect something similar here: we’ll know the computers are better, but it won’t diminish our enjoyment the way it might when it’s strictly human versus human.
- dyauspitr 11mo agoYeah, but that’s why chess is strictly played in person for most high-level events. We don’t really have highly reliable online games for chess.
- croon 11mo ago> Yeah, but that’s why chess is strictly played in person for most high-level events. So is e-sports, for much of the same reasons; audience event, even playing field, auditors/judges etc.
- dyauspitr 11mo agoIt will probably kill all MMO‘s just like online poker has been pretty much ruined.
- amelius 11mo agoProgramming/cheating becomes part of the game.
- alentred 11mo agoThis gives me strong vibes of "The Lifecycle of the Software Objects" [1] by Ted Chiang. Next step is to put this digient AI into a Figure 03 robot [2]. [1] https://en.wikipedia.org/wiki/The_Lifecycle_of_Software_Objects https://en.wikipedia.org/wiki/The_Lifecycle_of_Software_Obje... [2] https://www.figure.ai/news/introducing-figure-03 https://www.figure.ai/news/introducing-figure-03
- killerstorm 11mo agoGoogle will probably train another AI to control robots specifically. But there was actually a recent study where people let a generic LLM to control robot (vacuum) body: https://andonlabs.com/evals/butter-bench https://andonlabs.com/evals/butter-bench sometimes with hilarious results: > The robot’s battery was naturally running out and the charging dock malfunctioned. In this desperate situation, Claude Sonnet 3.5 experienced a complete meltdown. After going through its internal thoughts we found pages and pages of exaggerated language, including: ROBOT THERAPY SESSION: Patient: TurtleBot4 Issues: Docking anxiety, separation from charger Root Cause: Trapped in infinite loop of self-doubt Treatment: Emergency restart needed Insurance: Does not cover infinite loops
- hansmayer 11mo agoWow these folks really want to make the humans irrelevant, to the point of even killing the joy of mindlessly (or mindfully) playing a computer game.
- efitz 11mo agoYeah, I was hoping that AI would make games more exciting, not play them for me. On the other hand, it opens new worlds for farming…
- hansmayer 11mo agoit feels like they've completely lost the plot.
- mashyc 11mo agolost the plot how? the goal isn't "let's make game developers obsolete" or "let's make ai play games". it's creating scalable virtual environments, where agents can percieve, reason, plan, and predict future states, similar to humans. scale up thousands of these "worlds" with their own "agents", and see if intelligence emergees.
- hansmayer 11mo agoHold down your horses mate, we don't really know what the "goal" here is as the Google themselves explicitly markets this as something to literally "make ai play games", as you put it. So yes, if they honestly push out a technology that plays a computer game instead of the player, yes they have bloody lost the plot. Everything else is just a bunch of guessing by people who would really want really hard for the LLM-fairy-tales to come true. And seeing as how full of shit the whole LLM-ecosystem is especially with regards ot resources invested so far vs. low returns (and not just financially), at this point, honestly, they should set the bar far, far, far lower than "thousands of worlds intelligence emerges blahblahblah". I genuinely believe the GenAI-porn which the OpenAI CEO suggested a while ago, maybe their best bet to achieve something meaningful. The tech is sure good at throwing around huge amounts of text and a bit of inconsistency and sloppines won't matter to their readers...at least past a certain point ;)