13 ms·

Project Genie: Experimenting with infinite, interactive worlds

675 points by meetpateltech 9mo ago

meetpateltech 9mo ago
Google Deepmind Page: https://deepmind.google/models/genie/ https://deepmind.google/models/genie/ Try it in Google Labs: https://labs.google/projectgenie https://labs.google/projectgenie (Project Genie is available to Google AI Ultra subscribers in the US 18+.)
nickandbro 9mo ago
This could be the future of film. Instead of prompting where you don't know what the model will produce, you could use fine-grained motion controls to get the shot you are looking for. If you want to adjust the shot after, you could just checkpoint the model there, by taking a screenshot, and rerun. Crazy.
- JKCalhoun 9mo ago
  I feel like people are already currently doing this. Essentially storyboarding first. This guy a month ago for example: https://youtu.be/SGJC4Hnz3m0 https://youtu.be/SGJC4Hnz3m0
- deleted 9mo ago
  [deleted]
mosquitobiten 9mo ago
Every character goes forward only, permanence is still out of reach apparently.
- mikelevins 9mo ago
  I've been experimenting with that from a slightly different angle: teaching Claude how to play and referee a pencil-and-paper RPG that I developed over about 20 years starting in the mid 1970s. Claude can't quite do it yet for reasons related to permanence and learning over time, but it can do surprisingly well up until it runs into those problems, and it's possible to help it past some obstacles. The game is called "Explorers' Guild", or "xg" for short. It's easier for Claude to act as a player than a director (xg's version of a dungeon master or game master), again mainly because of permance and learning issues, but to the extent that I can help it past those issues it's also fairly good at acting as a director. It does require some pretty specific stuff in the system prompt to, for example, avoid confabulating stuff that doesn't fit the world or the scenario. But to really build a version of xg on Claude it needs better ways to remember and improve what it has learned about playing the game, and what it has learned about a specific group of players in a specific scenario as it develops over time.
- matt_LLVW 9mo ago
  Looks good to me: https://youtu.be/15KtGNgpVnE?t=648&si=urWJGEFWuN5veh43 https://youtu.be/15KtGNgpVnE?t=648&si=urWJGEFWuN5veh43
  - mosquitobiten 9mo ago
    the castles multiply and change positions in the background
montebicyclelo 9mo ago
Reminds me of this [1] HN post from 9 months ago, where the author trained a neural network to do world emulation from video recordings of their local park — you can walk around in their interactive demo [2]. I don't have access to the DeepMind demo, but from the video it looks like it takes the idea up a notch. (I don't know the exact lineage of these ideas, but a general observation is that it's a shame that it's the norm for blog posts / indie demos to not get cited.) [1] https://news.ycombinator.com/item?id=43798757 https://news.ycombinator.com/item?id=43798757 [2] https://madebyoll.in/posts/world_emulation_via_dnn/demo/ https://madebyoll.in/posts/world_emulation_via_dnn/demo/
- deleted 9mo ago
  [deleted]
- ollin 9mo ago
  Yup, similar concepts! Just at two opposite extremes of the compute/scaling spectrum. - That forest trail world is ~5 million parameters, trained on 15 minutes of video, scoped to run on a five-year-old iPhone through a twenty-year old API (WebGL GPGPU, i.e OpenGL fragment shaders). It's the smallest '3D' world model I'm aware of. - Genie 3 is (most likely) ~100 billion parameters trained on millions of hours of video and running across multiple TPUs. I would be shocked if it's not the largest-scale world model available to the public. There are lots of neat intermediate-scale world models being developed as well (e.g. LingBot-World https://github.com/robbyant/lingbot-world https://github.com/robbyant/lingbot-world, Waypoint 1 https://huggingface.co/blog/waypoint-1 https://huggingface.co/blog/waypoint-1) so I expect we'll be able to play something of Genie quality locally on gaming GPUs within a year or two.
- danielwmayer 9mo ago
  I was immediately struck when I looked down at just the boardwalk how similar it felt to being on LSD. I am continually astounded with how similar these systems end up seeming to how our brain works. May just be happy coincidences but I am pretty sold on there being true parallels that will only become more and more apparent.
  - 9mo ago
0xcb0 9mo ago
I keep on repeating myself, but it feels like I'm living in the future. Can't wait to hook this up to my old Oculus glasses and let Genie create a fully realistic sailing simulator for me, where I can train sailing with realistic conditions. On boats I'd love to sail. If making games out of these simulations work, it't be the end for a lot of big studios, and might be the renaissance for small to one person game studios.
- neom 9mo ago
  ...and then, the pneumatics in your living room.
- jsheard 9mo ago
  Isn't this still essentially "vibe simulation" inferred from videos? Surface-level visual realism is one thing, but expecting it to figure out the exact physical mechanics of sailing just by watching boats, and usefully abstract that into a gamified form, is another thing entirely.
  - falcor84 9mo ago
    Why wouldn't it just hook it into something like physx?
    - jsheard 9mo ago
      Google has made it clear that Genie doesn't maintain an explicit 3D scene representation, so I don't think hooking in "assists" like that is on the table. Even if it were, the AI layer would still have to infer things like object weight, density, friction and linkages correctly. Garbage in, garbage out.
      - Morromist 9mo ago
        Google could build try to build an actual 3d scene with ai using meshes or metaballs or something. That would allow for more persistance, but I expect makes the ai more brittle and limited, and, because it doesn't really understand the rules for the 3d meshes it created, it doesn't know how to interact with them. It can only be fluffy-mushy dream images.
srameshc 9mo ago
What’s the endgame here? For a small gaming studio, what are the actual implications?
- educasean 9mo ago
  I understand the ultimate end goal to be simulation of life. A near perfect replica of the real world we can use to simulate and test medicine, economy, and social impact.
- aurumque 9mo ago
  I would think that building a environment which can be managed by a game engine is the first pass. In a few years when we are able to render more than 60 seconds it could very well replace the game engine entirely by just rendering everything in realtime based on user interactions. The final phase is just prompts which turn directly into interactive games, maybe even multiplayer. When I see the progress we've made on things like DOOM, where it can infer the proper rendering of actions like firing weapons and even updating scores on hits and such it doesn't feel like we're very far off, a few years at most. For a game studio that could mean cutting out almost everything between keyboard and display, but for now just replacing the asset pipeline is huge.
  - mikewittie 9mo ago
    We seem to think that Genie is good at the creative part, but bad at the consistency and performance part. How hard would it be to take 60 seconds of Genie output and pipe it into a model that generates a consistent and performant 3D environment?
- hiccuphippo 9mo ago
  It seems to be generating images in real time, not 3d scenes. It might still be useful for prototyping.
  - saberience 9mo ago
    There are collisions though and physics seemingly, so it doesn't seen to be a huge stretch that this could be used for games.
sy26 9mo ago
I have been confused for a long time why FB is not motivated enough to invest in world models, it IS the key to unblock their "metaverse" vision. And instead they let go Yann LeCun.
- phailhaus 9mo ago
  Most people don't like putting on VR headsets, no matter what the content is. It just never broke out of the tech enthusiast niche.
- observationist 9mo ago
  LeCun wasn't producing results. He was obstinate and insistent on his own theories and ideas which weren't, and possibly aren't, going anywhere. He refused to engage with LLMs and compete in the market that exists, and spent all his effort and energy on unproven ideas and research, which split the company's mission and competitiveness. They lost their place as one of the top 4 AI companies, and are now a full generation behind, in part due to the split efforts and lack of enthusiastic participation by all the Meta AI team. If you look at the chaos and churn at the highest levels across the industry, there's not a lot of room for mission creep by leadership, and LeCun thoroughly demonstrated he wasn't suited for the mission desired by Meta. I think he's lucky he got out with his reputation relatively intact.
  - halfmatthalfcat 9mo ago
    Were you there or just an attentive outsider?
    - observationist 9mo ago
      Attentive outsider and acquaintance of a couple people who are or were employed there. Nothing I'm saying is particularly inside baseball, though, it's pretty well covered by all the blogs and podcasts.
      - richard___ 9mo ago
anxtyinmgmt 9mo ago
Demis stays cooking
RivieraKid 9mo ago
This would be really cool if polished and integrated with VR.
- cpeth 9mo ago
  Exactly this, it would essentially be a STTNG Holodeck
  - qingcharles 9mo ago
    The engineers said in an interview today that full reality simulation is their goal, aka holodeck.
ofrzeta 9mo ago
I don't know ... it's impressive and all but the result always looks kind of dead.
- api 9mo ago
  It's super cool but I see it as a much more flexible open ended take on the idea of procedurally generated worlds where hard-coded deterministic math and rendering parameters are replaced by prompt-able models. The deadness you're talking about is there in procedural worlds too, and it stems from the fact that there's not actually much "there." Think of it as a kind of illusion or a magic trick with math. It replicates some of the macro structure of the world but the true information content is low. Search YouTube for procedural landscape examples. Some of them are actually a lot more visually impressive than this, but without the interactivity. It's a popular topic in the demo scene too where people have made tiny demos (e.g. under 1k in size) that generate impressive scenes. I expect to see generative AI techniques like this show up in games, though it might take a bit due to their high computational cost compared to traditional procedural generation.
- saberience 9mo ago
  This reminds me of the comments by programmers roughly two years ago: "Sure it can write a single function but the code is terrible when it tries to write a whole class..."
  - sfn42 9mo ago
    You say that as if those programmers aren't still right
phailhaus 9mo ago
I have no idea why Google is wasting their time with this. Trying to hallucinate an entire world is a dead-end. There will never be enough predictability in the output for it to be cohesive in any meaningful way, by design. Why are they not training models to help write games instead? You wouldn't have to worry about permanence and consistency at all, since they would be enforced by the code, like all games today. Look at how much prompting it takes to vibe code a prototype. And they want us to think we'll be able to prompt a whole world?
- deleted 9mo ago
  [deleted]
- asim 9mo ago
  Take the positive spin. What if you could put in all the inputs and it can simulate real world scenarios you can walk through to benefit mankind e.g disaster scenarios, events, plane crashes, traffic patterns. I mean there's a lot of useful applications for it. I don't like the framing at this time, but I also get where it's going. The engineer in me is drawn to it, but the Muslim in me is very scared to hear anyone talk about creating worlds.... But again I have to separate my view from the reality that this could have very positive real world benefits when you can simulate scenarios. So I could put in a 2 pager or 10 page scenario that gets played out or simulated and allow me to walk through it. Not just predictive stuff but let's say things that have happened so I can map crime scenes or anything. In the end this performance art is because they are a product company being Benchmarked by wall street and they'll need customers for the technology but at the same time they probably already have uses for it internally.
  - jsheard 9mo ago
    > What if you could put in all the inputs and it can simulate real world scenarios you can walk through to benefit mankind e.g disaster scenarios, events, plane crashes, traffic patterns. This is only a useful premise if it can do any of those things accurately, as opposed to dreaming up something kinda plausible based on an amalgamation of every vaguely related YouTube video.
  - 9mo ago
ollin 9mo ago
Really great to see this released! Some interesting videos from early-access users: - https://youtu.be/15KtGNgpVnE?si=rgQ0PSRniRGcvN31&t=197 https://youtu.be/15KtGNgpVnE?si=rgQ0PSRniRGcvN31&t=197 walking through various cities - https://x.com/fofrAI/status/2016936855607136506 https://x.com/fofrAI/status/2016936855607136506 helicopter / flight sim - https://x.com/venturetwins/status/2016919922727850333 https://x.com/venturetwins/status/2016919922727850333 space station, https://x.com/venturetwins/status/2016920340602278368 https://x.com/venturetwins/status/2016920340602278368 Dunkin' Donuts - https://youtu.be/lALGud1Ynhc?si=10ERYyMFHiwL8rQ7&t=207 https://youtu.be/lALGud1Ynhc?si=10ERYyMFHiwL8rQ7&t=207 simulating a laptop computer, moving the mouse - https://x.com/emollick/status/2016919989865840906 https://x.com/emollick/status/2016919989865840906 otter airline pilot with a duck on its head walking through a Rothko inspired airport
- RaftPeople 9mo ago
  I liked that first one and I hope someone creates one of going back to dinosaur age, i want to see that.
  - echelon 9mo ago
    Tim is awesome. Ironically, he covered PixVerse's world model last week and it came close to your ask: https://youtu.be/SAjKSRRJstQ?si=dqybCnaPvMmhpOnV&t=371 https://youtu.be/SAjKSRRJstQ?si=dqybCnaPvMmhpOnV&t=371 (Earlier in the video it shows him live prompting.) World models are popping up everywhere, from almost every frontier lab.
  - post-it 9mo ago
    One step closer to the science-based dinosaur MMO we were promised.
- Valk3_ 9mo ago
  Any thoughts about Project Genie?
  - ollin 9mo ago
ge96 9mo ago
Damn that was crazy the picture of the tabletop setup/cardboard robot and it becomes 3D interactive.
WarmWash 9mo ago
The actual breakthrough with Genie is being able to turn around and look back, and seeing the same scene that was there before. A few other labs have similar world simulators, but they all struggle badly with keeping coherence of things not in view. Hence why they always walk forwards and never look around.
- sfn42 9mo ago
  And what if I go somewhere then go back there a week later?
  - jsheard 9mo ago
    Best they can do is 60 seconds, for now at least.
    - autonomousErwin 9mo ago
      Makes you wonder what the TTL caching for our universe is.
      - dabbz 9mo ago
        Whatever the speed of light is I would imagine
        jacquesm 9mo ago
        No, that's just an optimization that saved on computing resources. It effectively allows the party that runs this simulation to have a limited world to simulate. Dark matter is the other half of that trick. Both were invented by one Bebele Zropaxhodb after a particularly interesting party in the universe just above this one...
        autonomousErwin 9mo ago
        That's the rendering speed.
- deleted 9mo ago
moohaad 9mo ago
everyone will make his own game now
JaiRathore 9mo ago
I now believe we live in a simulation
- moomoo11 9mo ago
  clearly
cloudflare728 9mo ago
We will probably see Ready Player One in a few decades. Hoping to stay alive till then.
- HardCodedBias 9mo ago
  Decades? I mean, yes, the probability of having that level of tech in decades is quite high. But the technology is moving very fast right now. It sounds crazy, but I think that there is a 50% chance of having ready player one level technology before 2030. It's absolutely possible it will take more time to become economical.
  - qingcharles 9mo ago
    Right. This thing has come light years in two years and money is pouring into the space now. Never mind the sheer amount of competition that is driving innovation in world simulation. Just within a couple of years this thing will be wild.
- lexandstuff 9mo ago
  The mass-poverty and climate changed ravaged world parts, I could definitely see.
krunck 9mo ago
The more of this I see the more I want to spend time away from screens and doing those things I love to do in the real world.
- MillionOClock 9mo ago
  I love AI but I also hope it will paradoxically make people realize the value of real life experiences and human relationships.
  - avaer 9mo ago
    Or maybe it will just make people realize the value of fake life experiences and human relationships.
    - boogrpants 9mo ago
      The American public is no different than an American corporation; trying to extract as much allegiance and loyalty as possible for as little compensation as possible. Your neighbors in the street protesting for comprehensive single payer healthcare? Yeah they're perfectly fine leaving your existence up to "market forces". Copy-paste office workers everywhere reciting memorized platitudes and compliance demands. You're telling me I could interact even less with such selfish (and often useless given their limited real skillset) people? Deal. America needs to rethink the compensation package if it wants to survive as a socio-political meme. Happy to call myself Canadian or Chinese if their offer is better. No bullets needed.
      - jplusequalt 9mo ago
        >I think we'll eventually get to the point where these are real time and have consistent representations You have a dangerously low opinion of your fellow man, and while I sympathize with your frustration, I would humbly suggest you direct that anger at owners of companies/politicians, rather than aim it at your everyday citizen.
in-silico 9mo ago
Everyone here seems too caught up in the idea that Genie is the product, and that its purpose is to be a video game, movie, or VR environment. That is not the goal. The purpose of world models like Genie is to be the "imagination" of next-generation AI and robotics systems: a way for them to simulate the outcomes of potential actions in order to inform decisions.
- avaer 9mo ago
  Soft disagree; if you wanted imagination you don't need to make a video model. You probably don't need to decode the latents at all. That seems pretty far from information-theoretic optimality, the kind that you want in a good+fast AI model making decisions. The whole reason for LLMs inferencing human-processable text, and "world models" inferencing human-interactive video, is precisely so that humans can connect in and debug the thing. I think the purpose of Genie is to be a video game, but it's a video game for AI researchers developing AIs. I do agree that the entertainment implications are kind of the research exhaust of the end goal.
  - in-silico 9mo ago
    Sufficiently informative latents can be decoded into video. When you simulate a stream of those latents, you can decode them into video. If you were trying to make an impressive demo for the public, you probably would decode them into video, even if the real applications don't require it. Converting the latents to pixel space also makes them compatible with existing image/video models and multimodal LLMs, which (without specialized training) can't interpret the latents directly.
    - soulofmischief 9mo ago
      At which point you're training another model on top of the first, and it becomes clear you might as well have made one model from the start!
  - SequoiaHope 9mo ago
    Didn’t the original world models paper do some training in latent space? (Edit: yes[1]) I think robots imagining the next step (in latent space) will be useful. It’s useful for people. A great way to validate that a robot is properly imagining the future is to make that latent space renderable in pixels. [1] “By using features extracted from the world model as inputs to an agent, we can train a very compact and simple policy that can solve the required task. We can even train our agent entirely inside of its own hallucinated dream generated by its world model, and transfer this policy back into the actual environment.” https://arxiv.org/abs/1803.10122 https://arxiv.org/abs/1803.10122
rationalfaith 9mo ago
[dead]
analog8374 9mo ago
If creating an infinite world is so trivially easy (relatively speaking) then occam suggests that this world is generated.
- avaer 9mo ago
  I don't know if that's the simplest explanation, considering how insanely complex the generation is (these world models might literally be the most complex things to ever be created). But I do think it's a partial existence proof.
  - analog8374 9mo ago
    One big simplifier is to only render what you're looking at. I wonder how one might demonstrate that.
adventured 9mo ago
This is as good of a place to mark it as any. Humanity goes into the box and it never comes back out. It's better in there than it is out there for 99% of the population.
TacoCommander 9mo ago
[dead]
speak_on 9mo ago
Compared to DeepMind's Genie 3 demo, this appears to have more morphing issues and less user interactivity with environmental consistency. Is this a stripped down version?
- qingcharles 9mo ago
  Yes. Some features were removed from the version demoed last year.
binsquare 9mo ago
It's ability to simulate physics intact is actually a huge breakthrough. I can't even fathom what it would be like for the future of simulation and physical world when it gets far more accurate and realistic.
- Bjorkbat 9mo ago
  Ironically the physics are kind of my biggest criticism. They call these "world models", but I think it's more accurate to call them "video game models" because they employ "video game physics" rather than real world physics, among other things This is most evident in the way things collide.
  - binsquare 9mo ago
    It's getting better staggeringly fast, just a year ago I wouldn't expect it to be at even video game physics level so quickly. If there is a possibility where it continue to improve at a similar rate with llms. A way to simulate fluid dynamics or structural dynamics with reasonable accuracy and speed can unlock much faster pace of innovation in the physical world. (And validated with rigorous scientific methods)
    - MITSardine 9mo ago
      Numerical simulation is a well explored field, we know how to do all sorts of things, the issues lie rather in the tooling and robustness of it all put together (from geometry to numerical results) than in conceptual barriers. Finite Differences have existed since the 1700's! What hadn't for the longest time, is the computational power to crunch billions of operations per simulation. A nice thing about numerical simulation from first principles, is it innately supports arbitrary speed/precision, that's in fact the backbone of the mathematical analysis for why it works. In some cases, as is the case for CFD, we're actually mathematically screwed because you just have to resolve the small scales to get the macro dynamics. So the standard remains a kind of hack, which is to introduce additional equations (turbulence models) that steer the dynamics in place of the small (unresolved) scales. We know how to do better though (DNS), but it costs an arm and a leg (like years to milenia on a super computer).
gambiting 9mo ago
>>How we’re building responsibly How are you justifying the enormous energy cost this toy is using, exactly? I don't find anything "responsible" about this. And it doesn't even seem like something that has any actual use - it's literally just a toy.
- Morromist 9mo ago
  I think everyone has forgotten who google is here. Google makes cool things and then destroys them. Like Stadia, as someone else mentioned. This thing is already just hovering over the trash bin. Of course, maybe its a bridge to something else, but all I see is an image generator that's working really fast, so nothing novel.
bpiche 9mo ago
This is the plot of The Peripheral, right? Love the way the second half of that book turned out. Never finished Agency..
- qingcharles 9mo ago
  Agency was great. I read it when it came out and never realized until a couple of weeks ago that it was a sequel to The Peripheral.
mupuff1234 9mo ago
If only Google had the technology for game streaming... Oh wait RIP Stadia.
- WhereIsTheTruth 9mo ago
  Stadia was lightyears ahead, but pro-Microsoft media assassinated it with FUD While "journalists" were busy bootlicking a laggy 720p Android only xCloud beta, Stadia was already delivering flawless 4K@60FPS in a web browser They killed the only platform that actually worked just to protect Microsoft This will be a textbook case study in how a legacy monopoly kills innovation to protect its own mediocrity Microsoft won't survive the century, they are a dinosaur on borrowed time that has already lost the war in mobile, AI, and robotics They don't create,, they just buy marrket share to suffocate the competition and ruin every product they touch Even their cloud dominance is about to end, as they are already losing their grip on the European market to antitrust and sovereign alternatives
  - reneberlin 9mo ago
    I remember a video where they went through scyscrapers zooming into a room, where life was moving on and there was a screen inside a room and there was something running on it. I never understood how this was tanked. It was revolutionary.
dominick-cc 9mo ago
Finally all my anime figurines will come to life
bigblind 9mo ago
Anyone else going to try it and just keep getting a 404 page?
- CamperBob2 9mo ago
  It came up for me and accepted a photo, but it has just been stuck in the "Don't go anywhere, it's almost ready" state for 10+ minutes. No idea how long it is supposed to take. They can pull a 3D world out of thin air but they apparently can't vibe-code a progress bar... Edit: Now it's saying "We'll notify you when it's ready, and you'll have 30 seconds to enter your world. You are 37th in the queue." Go to restroom, come back 1 minute later: "The time to enter your world ran out." Lame-o.
  - CamperBob2 9mo ago
    Edit 2: It works, but not in Firefox. You have to use Chrome, and no, it doesn't tell you this. I don't know what I expected...
artisin 9mo ago
Best case, Google DeepMind cracks AGI by letting agents learn for themselves inside simulated worlds. Worst case, they've invented the greatest, most expensive screensaver generator in human history.
- rook_line_sinkr 9mo ago
  Could you explain how AGI is a best case? Last time I checked, "if anyone builds it everyone dies". Oh, is this the joke?
pedalpete 9mo ago
This is what we were building in 2018 with Ayvri, starting from 3d tiles with the aim of building a real-world view by using AI to essentailly re-paint and add detail to what was essentially a high-resolution and faster loading Google Earth (for outside cities, we didn't have building data). We saw a very diverse group of users, the common uses was paragliders, gliders, and pilots who wanted to view their or other peoples flights. Ultramarathons, mountain bike and some road-races where it provided an interactive way to visualize the course from any angle and distance. Transportation infrastructure to display train routes to be built. The list goes on.
jlhawn 9mo ago
Now I can't stop thinking about _The Experience Machine_ by Andy Clark. It theorizes that this is how humans navigate and experience the real world: Our brains generate what we think the world around is like and our senses don't so much directly process visual information but instead act like a kind of loss function for our internal simulations. Then we use that error to update our internal model of the world. In this view, we are essentially living inside a high-fidelity generative model. Our brains are constantly 'hallucinating' a predicted reality based on past experience and current goals. The data from our senses isn't the source of the image; it's the error signal used to calibrate that internal model. Much like Genie 3 uses latent actions and frames to predict the next state of a world, our brains use 'Active Inference' to minimize the gap between what we expect and what we experience. It suggests that our sense of 'reality' isn't a direct recording of the world, but a highly optimized, interactive simulation that is continuously 'regularized' by the photons hitting our retinas.
- shagie 9mo ago
  A kurzgesagt on this: Why Your Brain Blinds You For 2 Hours Every Day https://youtu.be/wo_e0EvEZn8 https://youtu.be/wo_e0EvEZn8 and the sources for that video - https://sites.google.com/view/sources-reality-is-not-real/ https://sites.google.com/view/sources-reality-is-not-real/
- cfiggers 9mo ago
  Another analogy that kinda fits in with what you're saying is the post-processing on smartphone "photos." At what point does the processing become so strong that it's less a photograph and more a work of computational impressionism?
  - direwolf20 9mo ago
    At the point where Samsung detects a photo of a white circle while the phone is pointing upwards and substitutes a high resolution picture of the moon. This actually happened.
- tracerbulletx 9mo ago
  I think this is pretty well established as far as neurologists are concerned and explains a lot of things. Like dreaming for instance.. just something like the model running without sensory input constraining it.
sebasv_ 9mo ago
I am stumped. Am I misreading, or are the folks at Google deliberately confounding two interpretations of "world model"? Dont get me wrong, this is really cool, and it will undoubtedly have its use. But what I am seeing is an LLM that can generate textures to be fed into a human-coded 3d engine (the "world model" that is demonstrated), and I fail to see how that brings us closer to AGI. For AGI we need "world models" as in "belief systems". The AI model must be able to reason about (learned) dynamics, which I dont see reflected in the text or video.
- littlekey 9mo ago
  >an LLM that can generate textures to be fed into a human-coded 3d engine I'm not certain but I think the LLM is also generating the physics itself. It's generating rules based on its training data, e.g. watch a cat walk enough and you can simulate how the cat moves in the generated "world".
artur_makly 9mo ago
let's reboot Leisure Suit Larry ;-)
user_hn_827 9mo ago
[dead]
Havoc 9mo ago
Are world models from the perspective of an observer in the world or zoomed out? Or in gaming terms do these models think FPS or RTS? Text models and pixel grid vision models is easy but struggling to wrap my head around what world model "sees" so to speak.
almosthere 9mo ago
So what is it doing in the real world, microwaving an elephant on high with 80kw every second and pouring out all the water in an sub-saharan African well every 4 minutes?
seanmozeik 9mo ago
[dead]
spullara 9mo ago
has the person who designed the movement control ever played a video game?
throwaway314155 9mo ago
The "How we're building responsibly" section has nothing to do with acting responsibly. It should be called "Limitations" instead. Section reads LLM generated honestly.
jacquesm 9mo ago
Isn't that more or less the theme of the movie 'the Thirteenth floor?' https://www.youtube.com/watch?v=Cbjhr-H2nxQ https://www.youtube.com/watch?v=Cbjhr-H2nxQ
hn_user_9876 9mo ago
[dead]
cyrusradfar 9mo ago
Now let's cross this with the game of life with a lot more processing and see what happens.
deleted 9mo ago
[deleted]
dangoodmanUT 9mo ago
this will go crazy for kids - being able to run around as a doll or action figure in their room
0x1ceb00da 9mo ago
Someone please create a world with this: https://giphy.com/gifs/6pUjuQQX9kEfSe604w https://giphy.com/gifs/6pUjuQQX9kEfSe604w
consumer451 9mo ago
Related: > Diego Rivas, Shlomi Fruchter, and Jack Parker-Holder from the Project Genie team join host Logan Kilpatrick for an in-depth discussion on Google DeepMind’s latest breakthrough in world models. Project Genie is an experimental research prototype that allows users to generate, explore, and interact with infinitely diverse, photorealistic worlds in real-time. Learn more about the shift from passive video generation to interactive media, the technical challenges of maintaining world consistency and memory, and how these models serve as an essential training ground for AI agents. https://www.youtube.com/watch?v=Ow0W3WlJxRY https://www.youtube.com/watch?v=Ow0W3WlJxRY
uyribackgy 9mo ago
[dead]
kittikitti 9mo ago
This is really great and to me, it feels like another ChatGPT moment. Thank you Google! This product can easily leap them over the competition. I had originally dismissed Yan Lecun's take on world models and I now feel foolish.
lysace 9mo ago
Prediction: In true Google fashion they will never spend enough time on this really cool tech demo to make it really useful in any way. In 6-12 months they will announce another really cool tech demo. And so on. They have been doing this for decades. To us this seems like the starting point of something really cool. To them it's a delivery; finally time to move on to something else.
kimchidude 8mo ago
Anyone else consistently get Error 404?
reneberlin 9mo ago
The subscribers to simulations from the pr0n-industry and the billions of lonely humanoids will suffocate in their VR-headsets, if we don't think about sensors to watch their oxygen-levels.
forrestthewoods 9mo ago
SPIN THE CAMERA 1080 DEGREES YOU COWARDS The only test I ever want to see with these frame-gen models is a full 1080 degree camera spin. Miss me with that 30 degree back and forth crap. I want multiple full turns. Some jitter and a little back-and-forth wobble is fine. But I want multiple full spins. I’m pretty sure I know why they don’t do this :)
- pcthrowaway 9mo ago
  > SPIN THE CAMERA 1080 DEGREES YOU COWARDS > a full 1080 degree camera spin Do you mean 3 full turns, or do you mean 180 (one half-turn)?
  - forrestthewoods 9mo ago
    I mean three full turns. I promise it will fail catastrophically.
lurker616 9mo ago
Are there any open source projects like this?
z3t4 9mo ago
When I try to run it I get: "We don't know what you're looking for, but we hope you find it"
BatteryMountain 9mo ago
Can't wait to have a game that can be an all-in-one game: rpg, roleplaying, rts, space, orcs, magic, cyberworlds with infinite story lines/worlds, items, dialogs etc. Ready Player One vibes. Like I want to take my skyrim character, drop it into Diablo 2, then drop Diablo (the demon) into Need for Speed, then have my need for speed car show up on another planet and upgrade it into a space ship, while the space ship takes me to fight some mega aliens. All while offering a coherent & unique experience. As you play, the game saves major forks in your story & game genre, so you can invite/share your game recipe with other humans to enjoy. Also, when are we getting a new Spore game? This game is a sleeping giant waiting to be awaken.
emsign 9mo ago
The holodeck in Star Trek seemed so cool, but when I see this I get some very bad feelings somehow. Probably because society in Star Trek seemed so much more mature and beneficial than BigTech America.
numerogeek 9mo ago
Am I the only one who thinks it's as beautiful as it is useless?
brador 9mo ago
It’s a controllable camera traversing a 3D world. Notice we didn’t see the cat go behind the couch. Maladaptive cut. They also don’t mention that the longer it runs over around 15 seconds the more hallucination until it’s a garbled mess.
cipherself 9mo ago
It's quite exciting how far we've come from the modern exposition of world models by David Ha and Jürgen Schmidhuber in 2018 https://worldmodels.github.io/ https://worldmodels.github.io/
unicorn_cowboy 9mo ago
Can someone explain the actual use cases for this that folks are excited about? Nobody wants to be responsible for writing the film they're watching, as they watch it. Nobody wants to engage with a virtual world that has no story, narrative, meaning, purpose. What is the actual use case for this, once we move beyond the tech demo stage? The idea it will wholesale replace existing 3D graphics/rendering pipelines and processes any time soon seems so far-fetched to me (even just logistically) that I can't wrap my head around what people are thinking here. This strikes me as a fancy parlor trick. Unless I'm mistaken, we need proper world models to do the things people are erroneously assuming this to be capable of one day.
- caspar 9mo ago
  Presumably others will write the prompts (or equivalent directing mechanism) that will steer the generation, such that you can act out whatever fantasies interest you.
dysoco 9mo ago
I'm both scared and excited for this. I really don't want games to turn into a soulless, AI generated mush. This could also bring a huge amount of slop-generated content flooding the game market. On the other hand I think this could be very cool from some specific use cases like outdoor scenarios in simulators. I've always wanted a game like Euro Truck Simulator where I can drive a car around a whole 1:1 representation of a country and this might just allow that, obviously I don't care about an accurate representation of every building or tree or hallucinations, just for it to be believable enough. I wonder if it can be integrated into already existing engines though, because it seems like a big stretch to write actual game logic as an LLM prompt.
maieuticagent 9mo ago
PSA to Driving Sim vibe coders and their enablers - Your racing game data ≠ real-world driving training data. Training FSD on gaming data = teaching AI that driving is a competitive sport where the worst outcome is hitting "restart". Real roads have pedestrians, weather, unpredictable humans, and actual consequences. Gaming AI would learn all the wrong lessons. Imaginary gameplay telemetry might be valuable, but not for a world of street festivals, farmers markets, and school zones.

675 points by meetpateltech 9mo ago

Google Deepmind Page: https://deepmind.google/models/genie/ https://deepmind.google/models/genie/ Try it in Google Labs: https://labs.google/projectgenie https://labs.google/projectgenie (Project Genie is available to Google AI Ultra subscribers in the US 18+.)

nickandbro 9mo ago

This could be the future of film. Instead of prompting where you don't know what the model will produce, you could use fine-grained motion controls to get the shot you are looking for. If you want to adjust the shot after, you could just checkpoint the model there, by taking a screenshot, and rerun. Crazy.

JKCalhoun 9mo ago
I feel like people are already currently doing this. Essentially storyboarding first. This guy a month ago for example: https://youtu.be/SGJC4Hnz3m0 https://youtu.be/SGJC4Hnz3m0
deleted 9mo ago
[deleted]

mosquitobiten 9mo ago

Every character goes forward only, permanence is still out of reach apparently.

mikelevins 9mo ago
I've been experimenting with that from a slightly different angle: teaching Claude how to play and referee a pencil-and-paper RPG that I developed over about 20 years starting in the mid 1970s. Claude can't quite do it yet for reasons related to permanence and learning over time, but it can do surprisingly well up until it runs into those problems, and it's possible to help it past some obstacles. The game is called "Explorers' Guild", or "xg" for short. It's easier for Claude to act as a player than a director (xg's version of a dungeon master or game master), again mainly because of permance and learning issues, but to the extent that I can help it past those issues it's also fairly good at acting as a director. It does require some pretty specific stuff in the system prompt to, for example, avoid confabulating stuff that doesn't fit the world or the scenario. But to really build a version of xg on Claude it needs better ways to remember and improve what it has learned about playing the game, and what it has learned about a specific group of players in a specific scenario as it develops over time.
matt_LLVW 9mo ago
Looks good to me: https://youtu.be/15KtGNgpVnE?t=648&si=urWJGEFWuN5veh43 https://youtu.be/15KtGNgpVnE?t=648&si=urWJGEFWuN5veh43
- mosquitobiten 9mo ago
  the castles multiply and change positions in the background

montebicyclelo 9mo ago

Reminds me of this [1] HN post from 9 months ago, where the author trained a neural network to do world emulation from video recordings of their local park — you can walk around in their interactive demo [2]. I don't have access to the DeepMind demo, but from the video it looks like it takes the idea up a notch. (I don't know the exact lineage of these ideas, but a general observation is that it's a shame that it's the norm for blog posts / indie demos to not get cited.) [1] https://news.ycombinator.com/item?id=43798757 https://news.ycombinator.com/item?id=43798757 [2] https://madebyoll.in/posts/world_emulation_via_dnn/demo/ https://madebyoll.in/posts/world_emulation_via_dnn/demo/

deleted 9mo ago
[deleted]
ollin 9mo ago
Yup, similar concepts! Just at two opposite extremes of the compute/scaling spectrum. - That forest trail world is ~5 million parameters, trained on 15 minutes of video, scoped to run on a five-year-old iPhone through a twenty-year old API (WebGL GPGPU, i.e OpenGL fragment shaders). It's the smallest '3D' world model I'm aware of. - Genie 3 is (most likely) ~100 billion parameters trained on millions of hours of video and running across multiple TPUs. I would be shocked if it's not the largest-scale world model available to the public. There are lots of neat intermediate-scale world models being developed as well (e.g. LingBot-World https://github.com/robbyant/lingbot-world https://github.com/robbyant/lingbot-world, Waypoint 1 https://huggingface.co/blog/waypoint-1 https://huggingface.co/blog/waypoint-1) so I expect we'll be able to play something of Genie quality locally on gaming GPUs within a year or two.
danielwmayer 9mo ago
I was immediately struck when I looked down at just the boardwalk how similar it felt to being on LSD. I am continually astounded with how similar these systems end up seeming to how our brain works. May just be happy coincidences but I am pretty sold on there being true parallels that will only become more and more apparent.
- 9mo ago

0xcb0 9mo ago

I keep on repeating myself, but it feels like I'm living in the future. Can't wait to hook this up to my old Oculus glasses and let Genie create a fully realistic sailing simulator for me, where I can train sailing with realistic conditions. On boats I'd love to sail. If making games out of these simulations work, it't be the end for a lot of big studios, and might be the renaissance for small to one person game studios.

neom 9mo ago
...and then, the pneumatics in your living room.
jsheard 9mo ago
Isn't this still essentially "vibe simulation" inferred from videos? Surface-level visual realism is one thing, but expecting it to figure out the exact physical mechanics of sailing just by watching boats, and usefully abstract that into a gamified form, is another thing entirely.
- falcor84 9mo ago
  Why wouldn't it just hook it into something like physx?
  - jsheard 9mo ago
    Google has made it clear that Genie doesn't maintain an explicit 3D scene representation, so I don't think hooking in "assists" like that is on the table. Even if it were, the AI layer would still have to infer things like object weight, density, friction and linkages correctly. Garbage in, garbage out.
    - Morromist 9mo ago
      Google could build try to build an actual 3d scene with ai using meshes or metaballs or something. That would allow for more persistance, but I expect makes the ai more brittle and limited, and, because it doesn't really understand the rules for the 3d meshes it created, it doesn't know how to interact with them. It can only be fluffy-mushy dream images.

srameshc 9mo ago

What’s the endgame here? For a small gaming studio, what are the actual implications?

educasean 9mo ago
I understand the ultimate end goal to be simulation of life. A near perfect replica of the real world we can use to simulate and test medicine, economy, and social impact.
aurumque 9mo ago
I would think that building a environment which can be managed by a game engine is the first pass. In a few years when we are able to render more than 60 seconds it could very well replace the game engine entirely by just rendering everything in realtime based on user interactions. The final phase is just prompts which turn directly into interactive games, maybe even multiplayer. When I see the progress we've made on things like DOOM, where it can infer the proper rendering of actions like firing weapons and even updating scores on hits and such it doesn't feel like we're very far off, a few years at most. For a game studio that could mean cutting out almost everything between keyboard and display, but for now just replacing the asset pipeline is huge.
- mikewittie 9mo ago
  We seem to think that Genie is good at the creative part, but bad at the consistency and performance part. How hard would it be to take 60 seconds of Genie output and pipe it into a model that generates a consistent and performant 3D environment?
hiccuphippo 9mo ago
It seems to be generating images in real time, not 3d scenes. It might still be useful for prototyping.
- saberience 9mo ago
  There are collisions though and physics seemingly, so it doesn't seen to be a huge stretch that this could be used for games.

sy26 9mo ago

I have been confused for a long time why FB is not motivated enough to invest in world models, it IS the key to unblock their "metaverse" vision. And instead they let go Yann LeCun.

phailhaus 9mo ago
Most people don't like putting on VR headsets, no matter what the content is. It just never broke out of the tech enthusiast niche.
observationist 9mo ago
LeCun wasn't producing results. He was obstinate and insistent on his own theories and ideas which weren't, and possibly aren't, going anywhere. He refused to engage with LLMs and compete in the market that exists, and spent all his effort and energy on unproven ideas and research, which split the company's mission and competitiveness. They lost their place as one of the top 4 AI companies, and are now a full generation behind, in part due to the split efforts and lack of enthusiastic participation by all the Meta AI team. If you look at the chaos and churn at the highest levels across the industry, there's not a lot of room for mission creep by leadership, and LeCun thoroughly demonstrated he wasn't suited for the mission desired by Meta. I think he's lucky he got out with his reputation relatively intact.
- halfmatthalfcat 9mo ago
  Were you there or just an attentive outsider?
  - observationist 9mo ago
    Attentive outsider and acquaintance of a couple people who are or were employed there. Nothing I'm saying is particularly inside baseball, though, it's pretty well covered by all the blogs and podcasts.
    - richard___ 9mo ago

anxtyinmgmt 9mo ago

Demis stays cooking

RivieraKid 9mo ago

This would be really cool if polished and integrated with VR.

cpeth 9mo ago
Exactly this, it would essentially be a STTNG Holodeck
- qingcharles 9mo ago
  The engineers said in an interview today that full reality simulation is their goal, aka holodeck.

ofrzeta 9mo ago

I don't know ... it's impressive and all but the result always looks kind of dead.

api 9mo ago
It's super cool but I see it as a much more flexible open ended take on the idea of procedurally generated worlds where hard-coded deterministic math and rendering parameters are replaced by prompt-able models. The deadness you're talking about is there in procedural worlds too, and it stems from the fact that there's not actually much "there." Think of it as a kind of illusion or a magic trick with math. It replicates some of the macro structure of the world but the true information content is low. Search YouTube for procedural landscape examples. Some of them are actually a lot more visually impressive than this, but without the interactivity. It's a popular topic in the demo scene too where people have made tiny demos (e.g. under 1k in size) that generate impressive scenes. I expect to see generative AI techniques like this show up in games, though it might take a bit due to their high computational cost compared to traditional procedural generation.
saberience 9mo ago
This reminds me of the comments by programmers roughly two years ago: "Sure it can write a single function but the code is terrible when it tries to write a whole class..."
- sfn42 9mo ago
  You say that as if those programmers aren't still right

phailhaus 9mo ago

I have no idea why Google is wasting their time with this. Trying to hallucinate an entire world is a dead-end. There will never be enough predictability in the output for it to be cohesive in any meaningful way, by design. Why are they not training models to help write games instead? You wouldn't have to worry about permanence and consistency at all, since they would be enforced by the code, like all games today. Look at how much prompting it takes to vibe code a prototype. And they want us to think we'll be able to prompt a whole world?

deleted 9mo ago
[deleted]
asim 9mo ago
Take the positive spin. What if you could put in all the inputs and it can simulate real world scenarios you can walk through to benefit mankind e.g disaster scenarios, events, plane crashes, traffic patterns. I mean there's a lot of useful applications for it. I don't like the framing at this time, but I also get where it's going. The engineer in me is drawn to it, but the Muslim in me is very scared to hear anyone talk about creating worlds.... But again I have to separate my view from the reality that this could have very positive real world benefits when you can simulate scenarios. So I could put in a 2 pager or 10 page scenario that gets played out or simulated and allow me to walk through it. Not just predictive stuff but let's say things that have happened so I can map crime scenes or anything. In the end this performance art is because they are a product company being Benchmarked by wall street and they'll need customers for the technology but at the same time they probably already have uses for it internally.
- jsheard 9mo ago
  > What if you could put in all the inputs and it can simulate real world scenarios you can walk through to benefit mankind e.g disaster scenarios, events, plane crashes, traffic patterns. This is only a useful premise if it can do any of those things accurately, as opposed to dreaming up something kinda plausible based on an amalgamation of every vaguely related YouTube video.
- 9mo ago

ollin 9mo ago

Really great to see this released! Some interesting videos from early-access users: - https://youtu.be/15KtGNgpVnE?si=rgQ0PSRniRGcvN31&t=197 https://youtu.be/15KtGNgpVnE?si=rgQ0PSRniRGcvN31&t=197 walking through various cities - https://x.com/fofrAI/status/2016936855607136506 https://x.com/fofrAI/status/2016936855607136506 helicopter / flight sim - https://x.com/venturetwins/status/2016919922727850333 https://x.com/venturetwins/status/2016919922727850333 space station, https://x.com/venturetwins/status/2016920340602278368 https://x.com/venturetwins/status/2016920340602278368 Dunkin' Donuts - https://youtu.be/lALGud1Ynhc?si=10ERYyMFHiwL8rQ7&t=207 https://youtu.be/lALGud1Ynhc?si=10ERYyMFHiwL8rQ7&t=207 simulating a laptop computer, moving the mouse - https://x.com/emollick/status/2016919989865840906 https://x.com/emollick/status/2016919989865840906 otter airline pilot with a duck on its head walking through a Rothko inspired airport

RaftPeople 9mo ago
I liked that first one and I hope someone creates one of going back to dinosaur age, i want to see that.
- echelon 9mo ago
  Tim is awesome. Ironically, he covered PixVerse's world model last week and it came close to your ask: https://youtu.be/SAjKSRRJstQ?si=dqybCnaPvMmhpOnV&t=371 https://youtu.be/SAjKSRRJstQ?si=dqybCnaPvMmhpOnV&t=371 (Earlier in the video it shows him live prompting.) World models are popping up everywhere, from almost every frontier lab.
- post-it 9mo ago
  One step closer to the science-based dinosaur MMO we were promised.
Valk3_ 9mo ago
Any thoughts about Project Genie?
- ollin 9mo ago

ge96 9mo ago

Damn that was crazy the picture of the tabletop setup/cardboard robot and it becomes 3D interactive.

WarmWash 9mo ago

The actual breakthrough with Genie is being able to turn around and look back, and seeing the same scene that was there before. A few other labs have similar world simulators, but they all struggle badly with keeping coherence of things not in view. Hence why they always walk forwards and never look around.

sfn42 9mo ago
And what if I go somewhere then go back there a week later?
- jsheard 9mo ago
  Best they can do is 60 seconds, for now at least.
  - autonomousErwin 9mo ago
    Makes you wonder what the TTL caching for our universe is.
    - dabbz 9mo ago
      Whatever the speed of light is I would imagine
      - jacquesm 9mo ago
        No, that's just an optimization that saved on computing resources. It effectively allows the party that runs this simulation to have a limited world to simulate. Dark matter is the other half of that trick. Both were invented by one Bebele Zropaxhodb after a particularly interesting party in the universe just above this one...
      - autonomousErwin 9mo ago
        That's the rendering speed.
deleted 9mo ago

moohaad 9mo ago

everyone will make his own game now

JaiRathore 9mo ago

I now believe we live in a simulation

moomoo11 9mo ago
clearly

cloudflare728 9mo ago

We will probably see Ready Player One in a few decades. Hoping to stay alive till then.

HardCodedBias 9mo ago
Decades? I mean, yes, the probability of having that level of tech in decades is quite high. But the technology is moving very fast right now. It sounds crazy, but I think that there is a 50% chance of having ready player one level technology before 2030. It's absolutely possible it will take more time to become economical.
- qingcharles 9mo ago
  Right. This thing has come light years in two years and money is pouring into the space now. Never mind the sheer amount of competition that is driving innovation in world simulation. Just within a couple of years this thing will be wild.
lexandstuff 9mo ago
The mass-poverty and climate changed ravaged world parts, I could definitely see.

krunck 9mo ago

The more of this I see the more I want to spend time away from screens and doing those things I love to do in the real world.

MillionOClock 9mo ago
I love AI but I also hope it will paradoxically make people realize the value of real life experiences and human relationships.
- avaer 9mo ago
  Or maybe it will just make people realize the value of fake life experiences and human relationships.
  - boogrpants 9mo ago
    The American public is no different than an American corporation; trying to extract as much allegiance and loyalty as possible for as little compensation as possible. Your neighbors in the street protesting for comprehensive single payer healthcare? Yeah they're perfectly fine leaving your existence up to "market forces". Copy-paste office workers everywhere reciting memorized platitudes and compliance demands. You're telling me I could interact even less with such selfish (and often useless given their limited real skillset) people? Deal. America needs to rethink the compensation package if it wants to survive as a socio-political meme. Happy to call myself Canadian or Chinese if their offer is better. No bullets needed.
    - jplusequalt 9mo ago
      >I think we'll eventually get to the point where these are real time and have consistent representations You have a dangerously low opinion of your fellow man, and while I sympathize with your frustration, I would humbly suggest you direct that anger at owners of companies/politicians, rather than aim it at your everyday citizen.

in-silico 9mo ago

Everyone here seems too caught up in the idea that Genie is the product, and that its purpose is to be a video game, movie, or VR environment. That is not the goal. The purpose of world models like Genie is to be the "imagination" of next-generation AI and robotics systems: a way for them to simulate the outcomes of potential actions in order to inform decisions.

avaer 9mo ago
Soft disagree; if you wanted imagination you don't need to make a video model. You probably don't need to decode the latents at all. That seems pretty far from information-theoretic optimality, the kind that you want in a good+fast AI model making decisions. The whole reason for LLMs inferencing human-processable text, and "world models" inferencing human-interactive video, is precisely so that humans can connect in and debug the thing. I think the purpose of Genie is to be a video game, but it's a video game for AI researchers developing AIs. I do agree that the entertainment implications are kind of the research exhaust of the end goal.
- in-silico 9mo ago
  Sufficiently informative latents can be decoded into video. When you simulate a stream of those latents, you can decode them into video. If you were trying to make an impressive demo for the public, you probably would decode them into video, even if the real applications don't require it. Converting the latents to pixel space also makes them compatible with existing image/video models and multimodal LLMs, which (without specialized training) can't interpret the latents directly.
  - soulofmischief 9mo ago
    At which point you're training another model on top of the first, and it becomes clear you might as well have made one model from the start!
- SequoiaHope 9mo ago
  Didn’t the original world models paper do some training in latent space? (Edit: yes[1]) I think robots imagining the next step (in latent space) will be useful. It’s useful for people. A great way to validate that a robot is properly imagining the future is to make that latent space renderable in pixels. [1] “By using features extracted from the world model as inputs to an agent, we can train a very compact and simple policy that can solve the required task. We can even train our agent entirely inside of its own hallucinated dream generated by its world model, and transfer this policy back into the actual environment.” https://arxiv.org/abs/1803.10122 https://arxiv.org/abs/1803.10122

rationalfaith 9mo ago

[dead]

analog8374 9mo ago

If creating an infinite world is so trivially easy (relatively speaking) then occam suggests that this world is generated.

avaer 9mo ago
I don't know if that's the simplest explanation, considering how insanely complex the generation is (these world models might literally be the most complex things to ever be created). But I do think it's a partial existence proof.
- analog8374 9mo ago
  One big simplifier is to only render what you're looking at. I wonder how one might demonstrate that.

adventured 9mo ago

This is as good of a place to mark it as any. Humanity goes into the box and it never comes back out. It's better in there than it is out there for 99% of the population.

TacoCommander 9mo ago

[dead]

speak_on 9mo ago

Compared to DeepMind's Genie 3 demo, this appears to have more morphing issues and less user interactivity with environmental consistency. Is this a stripped down version?

qingcharles 9mo ago
Yes. Some features were removed from the version demoed last year.

binsquare 9mo ago

It's ability to simulate physics intact is actually a huge breakthrough. I can't even fathom what it would be like for the future of simulation and physical world when it gets far more accurate and realistic.

Bjorkbat 9mo ago
Ironically the physics are kind of my biggest criticism. They call these "world models", but I think it's more accurate to call them "video game models" because they employ "video game physics" rather than real world physics, among other things This is most evident in the way things collide.
- binsquare 9mo ago
  It's getting better staggeringly fast, just a year ago I wouldn't expect it to be at even video game physics level so quickly. If there is a possibility where it continue to improve at a similar rate with llms. A way to simulate fluid dynamics or structural dynamics with reasonable accuracy and speed can unlock much faster pace of innovation in the physical world. (And validated with rigorous scientific methods)
  - MITSardine 9mo ago
    Numerical simulation is a well explored field, we know how to do all sorts of things, the issues lie rather in the tooling and robustness of it all put together (from geometry to numerical results) than in conceptual barriers. Finite Differences have existed since the 1700's! What hadn't for the longest time, is the computational power to crunch billions of operations per simulation. A nice thing about numerical simulation from first principles, is it innately supports arbitrary speed/precision, that's in fact the backbone of the mathematical analysis for why it works. In some cases, as is the case for CFD, we're actually mathematically screwed because you just have to resolve the small scales to get the macro dynamics. So the standard remains a kind of hack, which is to introduce additional equations (turbulence models) that steer the dynamics in place of the small (unresolved) scales. We know how to do better though (DNS), but it costs an arm and a leg (like years to milenia on a super computer).

gambiting 9mo ago

>>How we’re building responsibly How are you justifying the enormous energy cost this toy is using, exactly? I don't find anything "responsible" about this. And it doesn't even seem like something that has any actual use - it's literally just a toy.

Morromist 9mo ago
I think everyone has forgotten who google is here. Google makes cool things and then destroys them. Like Stadia, as someone else mentioned. This thing is already just hovering over the trash bin. Of course, maybe its a bridge to something else, but all I see is an image generator that's working really fast, so nothing novel.

bpiche 9mo ago

This is the plot of The Peripheral, right? Love the way the second half of that book turned out. Never finished Agency..

qingcharles 9mo ago
Agency was great. I read it when it came out and never realized until a couple of weeks ago that it was a sequel to The Peripheral.

mupuff1234 9mo ago

If only Google had the technology for game streaming... Oh wait RIP Stadia.

WhereIsTheTruth 9mo ago
Stadia was lightyears ahead, but pro-Microsoft media assassinated it with FUD While "journalists" were busy bootlicking a laggy 720p Android only xCloud beta, Stadia was already delivering flawless 4K@60FPS in a web browser They killed the only platform that actually worked just to protect Microsoft This will be a textbook case study in how a legacy monopoly kills innovation to protect its own mediocrity Microsoft won't survive the century, they are a dinosaur on borrowed time that has already lost the war in mobile, AI, and robotics They don't create,, they just buy marrket share to suffocate the competition and ruin every product they touch Even their cloud dominance is about to end, as they are already losing their grip on the European market to antitrust and sovereign alternatives
- reneberlin 9mo ago
  I remember a video where they went through scyscrapers zooming into a room, where life was moving on and there was a screen inside a room and there was something running on it. I never understood how this was tanked. It was revolutionary.

dominick-cc 9mo ago

Finally all my anime figurines will come to life

bigblind 9mo ago

Anyone else going to try it and just keep getting a 404 page?

CamperBob2 9mo ago
It came up for me and accepted a photo, but it has just been stuck in the "Don't go anywhere, it's almost ready" state for 10+ minutes. No idea how long it is supposed to take. They can pull a 3D world out of thin air but they apparently can't vibe-code a progress bar... Edit: Now it's saying "We'll notify you when it's ready, and you'll have 30 seconds to enter your world. You are 37th in the queue." Go to restroom, come back 1 minute later: "The time to enter your world ran out." Lame-o.
- CamperBob2 9mo ago
  Edit 2: It works, but not in Firefox. You have to use Chrome, and no, it doesn't tell you this. I don't know what I expected...

artisin 9mo ago

Best case, Google DeepMind cracks AGI by letting agents learn for themselves inside simulated worlds. Worst case, they've invented the greatest, most expensive screensaver generator in human history.

rook_line_sinkr 9mo ago
Could you explain how AGI is a best case? Last time I checked, "if anyone builds it everyone dies". Oh, is this the joke?

pedalpete 9mo ago

This is what we were building in 2018 with Ayvri, starting from 3d tiles with the aim of building a real-world view by using AI to essentailly re-paint and add detail to what was essentially a high-resolution and faster loading Google Earth (for outside cities, we didn't have building data). We saw a very diverse group of users, the common uses was paragliders, gliders, and pilots who wanted to view their or other peoples flights. Ultramarathons, mountain bike and some road-races where it provided an interactive way to visualize the course from any angle and distance. Transportation infrastructure to display train routes to be built. The list goes on.

jlhawn 9mo ago

Now I can't stop thinking about _The Experience Machine_ by Andy Clark. It theorizes that this is how humans navigate and experience the real world: Our brains generate what we think the world around is like and our senses don't so much directly process visual information but instead act like a kind of loss function for our internal simulations. Then we use that error to update our internal model of the world. In this view, we are essentially living inside a high-fidelity generative model. Our brains are constantly 'hallucinating' a predicted reality based on past experience and current goals. The data from our senses isn't the source of the image; it's the error signal used to calibrate that internal model. Much like Genie 3 uses latent actions and frames to predict the next state of a world, our brains use 'Active Inference' to minimize the gap between what we expect and what we experience. It suggests that our sense of 'reality' isn't a direct recording of the world, but a highly optimized, interactive simulation that is continuously 'regularized' by the photons hitting our retinas.

shagie 9mo ago
A kurzgesagt on this: Why Your Brain Blinds You For 2 Hours Every Day https://youtu.be/wo_e0EvEZn8 https://youtu.be/wo_e0EvEZn8 and the sources for that video - https://sites.google.com/view/sources-reality-is-not-real/ https://sites.google.com/view/sources-reality-is-not-real/
cfiggers 9mo ago
Another analogy that kinda fits in with what you're saying is the post-processing on smartphone "photos." At what point does the processing become so strong that it's less a photograph and more a work of computational impressionism?
- direwolf20 9mo ago
  At the point where Samsung detects a photo of a white circle while the phone is pointing upwards and substitutes a high resolution picture of the moon. This actually happened.
tracerbulletx 9mo ago
I think this is pretty well established as far as neurologists are concerned and explains a lot of things. Like dreaming for instance.. just something like the model running without sensory input constraining it.

sebasv_ 9mo ago

I am stumped. Am I misreading, or are the folks at Google deliberately confounding two interpretations of "world model"? Dont get me wrong, this is really cool, and it will undoubtedly have its use. But what I am seeing is an LLM that can generate textures to be fed into a human-coded 3d engine (the "world model" that is demonstrated), and I fail to see how that brings us closer to AGI. For AGI we need "world models" as in "belief systems". The AI model must be able to reason about (learned) dynamics, which I dont see reflected in the text or video.

littlekey 9mo ago
>an LLM that can generate textures to be fed into a human-coded 3d engine I'm not certain but I think the LLM is also generating the physics itself. It's generating rules based on its training data, e.g. watch a cat walk enough and you can simulate how the cat moves in the generated "world".

artur_makly 9mo ago

let's reboot Leisure Suit Larry ;-)

user_hn_827 9mo ago

[dead]

Havoc 9mo ago

Are world models from the perspective of an observer in the world or zoomed out? Or in gaming terms do these models think FPS or RTS? Text models and pixel grid vision models is easy but struggling to wrap my head around what world model "sees" so to speak.

almosthere 9mo ago

So what is it doing in the real world, microwaving an elephant on high with 80kw every second and pouring out all the water in an sub-saharan African well every 4 minutes?

seanmozeik 9mo ago

[dead]

spullara 9mo ago

has the person who designed the movement control ever played a video game?

throwaway314155 9mo ago

The "How we're building responsibly" section has nothing to do with acting responsibly. It should be called "Limitations" instead. Section reads LLM generated honestly.

jacquesm 9mo ago

Isn't that more or less the theme of the movie 'the Thirteenth floor?' https://www.youtube.com/watch?v=Cbjhr-H2nxQ https://www.youtube.com/watch?v=Cbjhr-H2nxQ

hn_user_9876 9mo ago

[dead]

cyrusradfar 9mo ago

Now let's cross this with the game of life with a lot more processing and see what happens.

deleted 9mo ago

[deleted]

dangoodmanUT 9mo ago

this will go crazy for kids - being able to run around as a doll or action figure in their room

0x1ceb00da 9mo ago

Someone please create a world with this: https://giphy.com/gifs/6pUjuQQX9kEfSe604w https://giphy.com/gifs/6pUjuQQX9kEfSe604w

consumer451 9mo ago

Related: > Diego Rivas, Shlomi Fruchter, and Jack Parker-Holder from the Project Genie team join host Logan Kilpatrick for an in-depth discussion on Google DeepMind’s latest breakthrough in world models. Project Genie is an experimental research prototype that allows users to generate, explore, and interact with infinitely diverse, photorealistic worlds in real-time. Learn more about the shift from passive video generation to interactive media, the technical challenges of maintaining world consistency and memory, and how these models serve as an essential training ground for AI agents. https://www.youtube.com/watch?v=Ow0W3WlJxRY https://www.youtube.com/watch?v=Ow0W3WlJxRY

uyribackgy 9mo ago

[dead]

kittikitti 9mo ago

This is really great and to me, it feels like another ChatGPT moment. Thank you Google! This product can easily leap them over the competition. I had originally dismissed Yan Lecun's take on world models and I now feel foolish.

lysace 9mo ago

Prediction: In true Google fashion they will never spend enough time on this really cool tech demo to make it really useful in any way. In 6-12 months they will announce another really cool tech demo. And so on. They have been doing this for decades. To us this seems like the starting point of something really cool. To them it's a delivery; finally time to move on to something else.

kimchidude 8mo ago

Anyone else consistently get Error 404?

reneberlin 9mo ago

The subscribers to simulations from the pr0n-industry and the billions of lonely humanoids will suffocate in their VR-headsets, if we don't think about sensors to watch their oxygen-levels.

forrestthewoods 9mo ago

SPIN THE CAMERA 1080 DEGREES YOU COWARDS The only test I ever want to see with these frame-gen models is a full 1080 degree camera spin. Miss me with that 30 degree back and forth crap. I want multiple full turns. Some jitter and a little back-and-forth wobble is fine. But I want multiple full spins. I’m pretty sure I know why they don’t do this :)

pcthrowaway 9mo ago
> SPIN THE CAMERA 1080 DEGREES YOU COWARDS > a full 1080 degree camera spin Do you mean 3 full turns, or do you mean 180 (one half-turn)?
- forrestthewoods 9mo ago
  I mean three full turns. I promise it will fail catastrophically.

lurker616 9mo ago

Are there any open source projects like this?

z3t4 9mo ago

When I try to run it I get: "We don't know what you're looking for, but we hope you find it"

BatteryMountain 9mo ago

Can't wait to have a game that can be an all-in-one game: rpg, roleplaying, rts, space, orcs, magic, cyberworlds with infinite story lines/worlds, items, dialogs etc. Ready Player One vibes. Like I want to take my skyrim character, drop it into Diablo 2, then drop Diablo (the demon) into Need for Speed, then have my need for speed car show up on another planet and upgrade it into a space ship, while the space ship takes me to fight some mega aliens. All while offering a coherent & unique experience. As you play, the game saves major forks in your story & game genre, so you can invite/share your game recipe with other humans to enjoy. Also, when are we getting a new Spore game? This game is a sleeping giant waiting to be awaken.

emsign 9mo ago

The holodeck in Star Trek seemed so cool, but when I see this I get some very bad feelings somehow. Probably because society in Star Trek seemed so much more mature and beneficial than BigTech America.

numerogeek 9mo ago

Am I the only one who thinks it's as beautiful as it is useless?

brador 9mo ago

It’s a controllable camera traversing a 3D world. Notice we didn’t see the cat go behind the couch. Maladaptive cut. They also don’t mention that the longer it runs over around 15 seconds the more hallucination until it’s a garbled mess.

cipherself 9mo ago

It's quite exciting how far we've come from the modern exposition of world models by David Ha and Jürgen Schmidhuber in 2018 https://worldmodels.github.io/ https://worldmodels.github.io/

unicorn_cowboy 9mo ago

Can someone explain the actual use cases for this that folks are excited about? Nobody wants to be responsible for writing the film they're watching, as they watch it. Nobody wants to engage with a virtual world that has no story, narrative, meaning, purpose. What is the actual use case for this, once we move beyond the tech demo stage? The idea it will wholesale replace existing 3D graphics/rendering pipelines and processes any time soon seems so far-fetched to me (even just logistically) that I can't wrap my head around what people are thinking here. This strikes me as a fancy parlor trick. Unless I'm mistaken, we need proper world models to do the things people are erroneously assuming this to be capable of one day.

caspar 9mo ago
Presumably others will write the prompts (or equivalent directing mechanism) that will steer the generation, such that you can act out whatever fantasies interest you.

dysoco 9mo ago

I'm both scared and excited for this. I really don't want games to turn into a soulless, AI generated mush. This could also bring a huge amount of slop-generated content flooding the game market. On the other hand I think this could be very cool from some specific use cases like outdoor scenarios in simulators. I've always wanted a game like Euro Truck Simulator where I can drive a car around a whole 1:1 representation of a country and this might just allow that, obviously I don't care about an accurate representation of every building or tree or hallucinations, just for it to be believable enough. I wonder if it can be integrated into already existing engines though, because it seems like a big stretch to write actual game logic as an LLM prompt.

maieuticagent 9mo ago

PSA to Driving Sim vibe coders and their enablers - Your racing game data ≠ real-world driving training data. Training FSD on gaming data = teaching AI that driving is a competitive sport where the worst outcome is hitting "restart". Real roads have pedestrians, weather, unpredictable humans, and actual consequences. Gaming AI would learn all the wrong lessons. Imaginary gameplay telemetry might be valuable, but not for a world of street festivals, farmers markets, and school zones.