23 ms·
Genie 2: A large-scale foundation world model
- josvdwest 2y agoI understand the value of infinite NPC dialogues and story arcs, but why do we need live scene generation? Don't we already get that with procedural generation?
- jjice 2y agoI don't understand this space very well, but this seems incredible. Something I find interesting about generative AI is how it adds a huge layer of flexibility, but at the cost of lots of computation, while a very narrow set of constraints (a traditional program) is comparatively incredibly efficient. If someone spent a ton of time building out something simple in Unity, they could get the same thing running with a small fraction of the computation, but this has seemingly infinite flexibility based on so little and that's just incredible. The reason I mention it is because I'm interested in where we end up using these. Will traditional programming be used for most "production" workloads with gen AI being used to aid in the prototyping and development of those traditional programs, or will we get to the point where our gen AI is the primary driver of software? I assume that concrete code will always be faster and the best way to have deterministic results, but I really have to idea how to conceptualize what the future looks like now.
- Retric 2y agoLonger term computation isn’t really the limiting factor for generative AI, it’s training data. Generative AI is like Google search before the web responded to their search engine existing. There’s a huge quantity of high quality training data which nobody had any reason to pollute ready for the scrapping. But modern search is hampered by people responding to algorithmic indexes. Algorithms responding to metadata without directly evaluating content enabled a world of SEO and low quality websites suddenly being discoverable as long as they narrow their focus enough. So longer term it’s going to be an arms race between the output of Generative AI and people trying to keep updating their models. In 20 years people will get much better at using these tools, but the tools themselves may be less useful. I wouldn’t be surprised if eventually someone sneaks advertising into the output of someone else’s model etc.
- Miraste 2y agoThis has already happened. Search google for a few random terms, and go through the first page of web and image results. A decent chunk will be AI-generated.
- deleted 2y ago[deleted]
- golol 2y agoI disagree. With more computation you can train a bigger model on the same size training data and it will be better. There is a lot if knowledge on the internet that GPT-4 etc. have not yet learned.
- Retric 2y agoThe issue is the training data isn’t some constant. Let’s suppose OpenAI had 10x the computing power but a vastly worse dataset, do you expect a better or worse result? The question is ambiguous without defining how much worse the dataset is.
- golol 2y agoBut why would the dataset be worse, they can just use the same one as before.
- Retric 2y agoA dataset that’s 2 years old is worse than one that’s 20 years old even if it contains the same data. Even facts age. In 2004, Pluto was still classified as a planet. Not such a big deal on its own, but stale data gets a little bit worse every day.
- danans 2y ago> I assume that concrete code will always be faster and the best way to have deterministic results, but I really have to idea how to conceptualize what the future looks like now. It will likely be a mix of both concrete code and live AI generated experiences, but even the concrete code will likely be partially AI generated and modified. The ratio will depend on how reliable vs creative the software needs to be. For example, no AI generated code running pacemakers or power plants. But game world experiences could easily be made more dynamic by generative AI.
- singularity2001 2y agoMakes me wonder if there's any company which is trying to train a model to produce three D worlds within Unity (not as a video like oasis).
- teamonkey 2y agoThis at least is a bit more realistic than what’s being presented by Google. There are already a number of techniques for procedurally-generating a world (including Markov-based systems). The problem with replacing procedural world generation with LLM generation are a) you need to obtain a data set to train it, which doesn’t commercially exist, or train it yourself, b) there’s a fundamental need to iterate on the design, which LLMs do not cope with well at all, c) you need to somehow debug issues and fix them. That’s quite apart from the quality issues, cost and power usage.
- throwaway2037 2y agoI was hoping that Midjourney would make the leap from 2D to 3D, then start to provide the 3D model. A bit further, you could tell the Midjourney LLM to create a small scene, like: The character runs fast. Then, Midjourney LLM could output whatever script is necessary to make the 3D model "run". If Midjourney doesn't do it first, I am sure someone else will.
- sbarre 2y ago> Will traditional programming be used for most "production" workloads with gen AI being used to aid in the prototyping and development of those traditional programs I mean we're already there with Copilot, Cursor and other tools that use LLMs to assist in coding tasks.
- ganzuul 2y ago> ...this has seemingly infinite flexibility based on so little and that's just incredible. What makes it little? This is the difference between von Neumann architecture and Harvard architecture.
- me551ah 2y agoSo when I can try this?
- ilaksh 2y agoIt's Google so I assume never. No model release, no product, no API, no detailed paper. There was another quite similar model from a different group within the last month or so. I can't remember if they released any weights or anything or the name of it. But it was the same concept.
- vessenes 2y agoYou'll need to wait until Baidu or AliBaba or Nvidia publish a competing model, unfortunately, if history is any guide.
- mhld 2y agoProbably when Genie 10 will get integrated on a Pixel phone.
- vessenes 2y agoThis is.. super impressive. I'd like to know how large this model is. I note that the first thing they have it do is talk to agents who can control the world gen; geez - even robots get to play video games while we work. That said; I cannot find any: - architecture explanation - code - technical details - API access information Feels very DeepMind / 2015, and that's a bummer. I think the point of the "we have no moat" email has been taken to heart at Google, and they continue to be on the path of great demos, bleh product launches two years later, and no open access in the interim. That said, just knowing this is possible - world navigation based on a photo and a text description with up to a minute of held context -- is amazing, and I believe will inspire some groups out there to put out open versions.
- justlikereddit 2y ago[flagged]
- xnx 2y ago> the squealing carcass called Gemini Have you used Gemini? It seems every bit as good as ChatGPT.
- auggierose 2y agoIt is not nearly as good. I tried the free trial and cancelled it before it was over.
- Mistletoe 2y agohttps://www.cnet.com/tech/services-and-software/chatgpt-vs-google-gemini/ https://www.cnet.com/tech/services-and-software/chatgpt-vs-g... https://www.tomsguide.com/ai/google-gemini-vs-openai-chatgpt https://www.tomsguide.com/ai/google-gemini-vs-openai-chatgpt It won these shootouts and that's been my experience also, when I need to use AI (extremely rare) I just use the Google Gemini free one. I feel like this is how most people will use AI and why it is doomed to be the ultra low margin grocery store business instead of the huge cash cow business people think it will be.
- artninja1988 2y agoLooking at the list of authors, is this from their open endedness team? I found their position paper on it super convincing https://arxiv.org/abs/2406.02061 https://arxiv.org/abs/2406.02061
- warkdarrior 2y agoDid you link the wrong Arxiv paper? https://arxiv.org/abs/2406.02061 https://arxiv.org/abs/2406.02061 does not look like a position paper nor does it share any authors with this Genie 2 work.
- artninja1988 2y agoYes, I meant this paper https://arxiv.org/abs/2406.04268 https://arxiv.org/abs/2406.04268 Should have double checked, sorry and thank you for pointing it out
- mdrzn 2y agoWow.. I can't even imagine where we'll be in 5 or 10 years from now. Seems that it's only "consistent" up to a minute, but if the progress keeps the same rate.. just wow.
- netdevphoenix 2y agoProgress is not linear. For all we know, in 2027 things will slow down to a virtual halt for the next 30 years. Look at how much big science progressed in the first 20 years of the 19th century/20th century and look how little it has progressed in the first 20 years of this century. We are on the downlow compared to the last centuries and even if you look at crisp or deep learning, they are not as impactful NOW as let's say the germ theory of disease, evolution, the discovery of the double helix structure or general relativity was. Almost a quarter of a century gone and we don't have much to show for it. For reference: 19th century evolution by natural selection as science electromagnetism germ theory of disease first law of thermodynamics -------------------------------------------- 20th century general relativity quantum mechanics dna structure penicillin big bang theory -------------------------------------------- 21st century crisp deep learning
- dooglius 2y agoThe things you list for previous centuries aren't limited to the first 20 years
- netdevphoenix 2y ago19th century: electromagnetism, the voltaic pile, the double slit experiment for the light wave theory 20th century: general/special relativity, radioactive decay, discovery of the electron 21st century: crisp and deep learning Hard to argue that the big science of the first 20 years of the previous century looks way more impact than crisp and deep learning put TOGETHER.
- dekhn 2y agoits called crispr, not crisp.
- lionkor 2y ago> deepmind.google uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more. Yippee finally google posts a non confirming cookie popup with no way to reject the ad cookies!
- wildermuthn 2y agoThe technology is incredible, but the path to AGI isn't single-player. Qualia is the missing dataset required for AGI. See attention-schema theory for how social pressures lead to qualia-driven minds capable of true intelligence.
- deleted 2y ago[deleted]
- simonw 2y agoRelated recent project you can try out yourself (Chrome only) which hallucinates new frames of a Minecraft style game: https://oasis.decart.ai/ https://oasis.decart.ai/ That one would reimagine the world any time you look at the sky or ground. Sounds like Genie2 solves that: "Genie 2 is capable of remembering parts of the world that are no longer in view and then rendering them accurately when they become observable again."
- echelon 2y agoThis blows Decart's Oasis (which raised $25 million at $500 million valuation) and World Labs (which raised $230 million in complete stealth) out of the water. Google is firing warning shots to kill off interest in funding competing startups in this space. I suspect that in 6 months it won't matter as we'll have completely open source Chinese world models. They're already starting to kill video foundation model companies' entire value prop by releasing open models and weights. Hunyuan blows Runway and OpenAI's Sora completely out of the water, and it's 100% open source. How do companies like Pika compete with free? Meta and Chinese companies are not the leaders in the space, so they're salting the earth with insanely powerful SOTA open models to prevent anyone from becoming a runaway success. Meta is still playing its cards close to its chest so they can keep the best pieces private, but these Chinese companies are dropping innovation left and right like there's no tomorrow. The game theory here is that if you're a foundation model "company", you're dead - big tech will kill you. You don't have a product, and you're paying a lot to do research that isn't necessarily tied to customer demand. If you're a leading AI research+product company, everyone else will release their code/research to create a thousand competitors to you.
- Workaccount2 2y agoI strongly suspect that like open ai and O1, for profit companies are going to start locking down whatever advances they find. There is still an enormous amount of long hanging fruit that anyone can harvest right now, but eventually big advances are going to require big budgets and I can only imagine how technically tight lipped they will be with those.
- senko 2y ago
- bix6 2y agoGenuine question: What is the point of telling us about this if we can’t use it? Is it just to flex on everyone?
- mhld 2y agoSome kind of marketing strategy that actually nobody understands
- jazzyjackson 2y agoIt's not that opaque, it's recruitment. Basically same marketing as a univeristy. "We do state of the art research here. If you are a talented researcher who wants to advance the field, you'll want to work here" Now, how Google plans to make money with all this bleeding edge research, that's the mystery.
- deleted 2y ago[deleted]
- tootie 2y agoIt's PR but it's also meant to entice. Let the world know Google is #1 for Gen AI, convince researchers to join Google, convince investors to boost the stock price, make Elon Musk grit his teeth. That kind of thing. In the short term, it may provide a bump in interest for existing AI products from Google.
- ChrisArchitect 2y agoThe best minds of a generation went from thinking about how to make people click ads to how to generate 3d video game worlds.
- adventured 2y agoThe best minds were never working on getting people to click on ads. That was an internal industry narrative so people could feel better about themselves.
- rvz 2y agoHmmm.... But we were told on HN that "Google is dying" remember? in reality, is it isn't. We'll see which so-called AI-companies are really "dying" when either a correction, market crash or a new AI winter arrives.
- bearjaws 2y ago> Genie 2 is capable of remembering parts of the world that are no longer in view and then rendering them accurately when they become observable again. This is huge, the Minecraft demos we saw recently we're just toys because you couldn't actually do anything in them.
- psb217 2y agoIt's worth keeping in mind that "there exists X such that Y is true" is not the same as "Y is true for all X". People love using these sorts of statements since they're technically true as written, but most people will read them in a way that's false. Eg, the statement is true for the Minecraft demos, and for any model which doesn't exhibit literally zero persistence for (temporarily) non-visible state.
- stoicjumbotron 2y agoDo people within Google get to try it? If yes, how long is the approval process?
- xcodevn 2y agoOn a very similar theme, here is the work from World Lab (founded by Fei-Fei Li, ImageNet dataset, et al.) about creating 3D worlds: https://www.worldlabs.ai/blog https://www.worldlabs.ai/blog
- momojo 2y agoI find this work much more exciting. They're not just teaching a model to hallucinate given WASD input. They're generating durable, persistent point clouds. It looks so similar to Genie2 yet they're worlds apart.
- moralestapia 2y agoNot even a month ago HN was discussing Ben Affleck's take on actors and AI, somehow taking a side with him and arguing how the tech "it's just not there, etc...". I'll keep my stance, give it two years and very realistic movies, with plot and everything, will be generated on demand.
- tartoran 2y agoAi can't generate images without awkward hallucinations yet. From that to movies that make sense to movies that people would want to watch (comparable to feature films) beyond the initial curiosity factor is a long way, if there is one.
- moralestapia 2y agoChatGPT (no Sora, no World Generation, etc...) was released two years ago almost to the date. What you're talking about is a minor jump from the SOTA, much smaller than what we've already see in these two years.
- Sateeshm 2y agoI'll take that bet
- moralestapia 2y agoEmail on profile! I'll match any 5-figure amount you propose. I also know an escrow service we can trust.
- moralestapia 2y agoDay two of the two year bet: https://x.com/mrjonfinger/status/1865161230706520472 https://x.com/mrjonfinger/status/1865161230706520472 Let's do this, Shasseem.
- moralestapia 2y ago
- binalpatel 2y agoThis is super impressive. Interesting they're framing this more from the world model/agent environment angle, when this seems like the best example so far of generative games. 720p realtime mostly consistent games for a minute is amazing, considering stable diffusion was originally released 2ish years ago.
- uoaei 2y agoPixelspace is an awful place to be generating 3D assets and maintaining physical self-consistency.
- jeroenvlek 2y agoUltimately even conventional 3d assets are rendered into pixelspace. It all comes down to the constraints in the model itself.
- psb217 2y agoA key strength of conventional 3d assets is that their form is independent of the scenes in which they will be rendered. Models that work purely in pixel space avoid the constraints imposed by representing assets in a fixed format, but they have to do substantial extra work to even approximate the consistency and recomposability of conventional 3d assets. It's unclear whether current approaches to building and training purely pixel-based models will be able to achieve a practically useful balance between their greater flexibility and higher costs. World Labs, for example, seems to be betting that an intermediate point of generating worlds in a flexible but structured format (NERFs, gauss splats, etc) may produce practical value more quickly than going straight for full freedom and working in pixel space.
- deleted 2y ago[deleted]
- BoredPositron 2y agoI don’t know I get the excitement but as soon as you turn around and there is something completely different behind you it breaks the immersion.
- jdlyga 2y agoIt's very cool, but we've gotten too many of these big bold announcements with no payoff. All it takes is a very limited demo and we'd be much happier.
- rishabhparikh 2y agoI'm guessing it would be far too expensive to make a free demo
- sergiotapia 2y agoWill the GPU go the way of the soundcard, and we will all purchase an "LPU"? Language Processing Unit for AIs to run fast? I remember there was a brief window where some gamers bought a Physx card for high fidelity physics in games. Ultimately they rolled that tech in to the CPUs themselves right?
- 0x1ceb00da 2y agoThe graphics stuff in modern gpus is just a software layer on top of a generic processing unit. The name is a misnomer.
- jsheard 2y agoPartially true, a significant chunk of modern GPUs are really just very wide general purpose processors, but they do still have fixed-function silicon specifically for graphics and probably will for the foreseeable future. Intel tried to lean into doing as much as possible in general purpose compute with their Larrabee GPU project but even that still had fixed-function texture units... and the concept was ultimately a failure which hasn't been revisited.
- k2xl 2y agoThis is impressive, but why are they all looking still like a video game? Could they have this render movie scenes with realistic looking humans? I wonder if it is due to the training set they use being mostly video games?
- xnx 2y ago> This is impressive, but why are they all looking still like a video game? Many of the current AI models have their roots in games: Chess, Go, etc.
- nonameiguess 2y agoI highly doubt it. While there is no ceiling in principle on how good rendering can get, even with perfect knowledge of the physics of optics, the cost to compute that physics is too high not to cut some corners. Nature gives you this for free. Every photon is deflected at exactly the right angle and frequency without anything needing to be computed. All you need is a camera to record it. At least for now, this is why every deep fake, digital de-aging, AI upscaling, grafting Carrie Fisher's face onto a different actor, and CGI in general inevitably occupies the uncanny valley.
- devonsolomon 2y agoYesterday I laughed with my brother about how harsh people on the internet were about World Labs launch (“you can only walk three steps, this demo sucks!”). I was thinking, “this was unthinkable a few years ago, this is incredible”. People of the internet, you were right. Now, this is incredible.
- bilbo0s 2y agoWorld Labs was kind of laughable. But at least you laughed. Now? I mean, I don't know man? With this Genie 2 sneak peak, it all just makes World Labs' efforts look sad. Did they really think better funded independents and majors would all not be interested in generating 3D worlds? This is a GUBA moment. If you're old enough to know, then you know.
- corysama 2y agoFor quite a while now David Holz of Midjourney has mused that videogames will be AI generated. Like a theoretical PlayStation 7 with an AI processor replacing the GPU. But, I didn’t expect this much progress towards that quite this fast…
- kypro 2y agoAgreed. All I'd say is that these demos look quite limited in their creativity and depth. Good video games are far more than some graphics with a movable character and action states. A good video game is far more the world building, the story, the creativity or "uniqueness" of the experience, etc. Currently this seems to generate fairly generic looking and shallow experiences. Not hating though. It's early days obviously.
- doctorpangloss 2y agoIf only it were that simple. Google spent $10b developing Stadia, where was the big hit game from that? These DeepMind guys play Factorio, they don't play Atari games or shooters, so why aren't they thinking about that? Or maybe they are, and because they know a lot about Factorio, they see how hard it is to make? There's a lot of "musing" as you say.
- gcr 2y agoI've had the idea for a Backrooms-style hallucinatory generative videogame for a while. Imagine being able to wander through infinitely generated surreal indoor buildingscapes that were rendered in close-to-realtime. It would play to the medium's strengths -- any "glitches" the player experiences could be seen as diagetic corruptions of reality. The moment we get parameterized NeRF models running in close-to-realtime, I want to go for it.
- maxglute 2y ago2000s graphics vibes.
- YeGoblynQueenne 2y agoHey, DeepMind folks, are you listening? Listen. We believe you: you can conquer any virtual world you put your mind to. Minecraft, Starcraft, Warcraft (?), Atari, anything. You can do it! With the power of RL and Neural Nets. Well done. What you haven't been able to do so far, after many years of trying, is to go from the virtual, to the real. Go from Arcanoid to a robot that can play, I dunno, squash, without dying. A robot that can navigate an arbitrary physical location without drowning, or falling off a cliff, or getting run over by a bus. Or build any Lego kit from instructions. Where's all that? You've conquered games. Bravo! Now where's the real world autonomy?
- sdenton4 2y agohttps://sites.research.google/palm-saycan https://sites.research.google/palm-saycan
- YeGoblynQueenne 2y agoTech demo, doesn't generalise.
- sdenton4 2y agoWell, Waymo.
- YeGoblynQueenne 2y ago"Well Waymo" is not DeepMind. Look. The other poster also said "Waymo" but I'm talking about DeepMind. It's DeepMind that promises to conquer the world with Deep Reinforcement Learning, and it's DeepMind that keeps showing us how great their DRL agents work in virtual worlds, like minecraft or starcraft, or how well they work on Chess and Go, but still haven't been able to demonstrate the application of those powerful learning approaches to real-world environments, except for very strictly controlled ones. Waymo's stuff works in the real world (although they do have remote safety drivers much as they try to downplay the fact) but they're also not pretending that they'll do it all with one big DRL "generalist" agent. That's DeepMind's schtick. For example, it was, I believe, DeepMind that recently publicised some results about legged robot football, where the robots were controlled by agents trained with DRL in a simulation. That's robot football: two robots (yeah, no teams) kicking a ball in the safest of safe environments: a (reduced-size) football field with artificial grass, probably padded underneath (because robots) and no other objects in the play area (except anxious researchers who have to pull the robots on their feet once in a while). Running in the physical world in principle, but in practice nothing but a tech demo. Or take the other Big Idea, where they had a few dozen robot arms reaching for various little plastic bits in a (specially-made) box to try and learn object manipulation by real-world DRL. I can find a link to those things if you want, but that robot arm project was a few years ago and you haven't heard anything from them since because it was a whole load of overpromising and it failed. That kind of thing just doesn't generalise. More than that: it's a total waste of time and money. And yet DeepMind keeps banging the drum. They keep trying to convince everyone and themselves that training DRL agents in virtual environments has anything to do with the real world, and that it's somehow the road to AGI. "Reward is all you need". Yeah, OK. Btw, Waymo is not using DRL, at least not exclusively. They use all sorts of techniques but from what I understand they do a hell of a lot of good, old-fashioned, manual programming to deal with all the stuff that magickal deep learning in the sky can't deal with.
- aithrowawaycomm 2y agoIt is jaw-dropping and dismaying how for-profit AI companies use long-standing terms like "world model" and "physics" when they mean "video game model" and "video game physics." Or, as you can plainly see, "models gravity" when they mean "models Red Dead Redemption 2's gravity function, along with its cinematic lighting effects and Rockstar's distinctively weighty animations." Which is to say Google is not modeling gravity at all. I will add the totally inconsistent backgrounds in the "prototyping" example suggests the AI is simply cribbing from four different games with a flying avatar, which makes it kind of useless unless you're prototyping cynical AI slop. And what are we even doing here by calling this a "world model" if the details of the world can change on a whim? In my world model I can imagine a small dragon flying through my friend's living room without needing to turn her electric lights into sconces and fireplaces. To state the obvious: if you train your model on thousands of hours of video games, you're also gonna get a bunch of stuff like "leaves are flat and don't bend" or "sometimes humans look like plastic" or "sometimes dragons clip through the scenery," which wouldn't fly in an actual world model. Just call it "video game world model!" Google is intentionally misusing a term which (although mysterious) has real meaning in cognitive science. I am sure Genie 2 took an awful lot of work and technical expertise. But this advertisement isn't just unscientific, it's an assault on language itself.
- empath75 2y ago> It is jaw-dropping and dismaying how for-profit AI companies use long-standing terms like "world model" and "physics" when they mean "video game model" and "video game physics." Or, as you can plainly see, "models gravity" when they mean "models Red Dead Redemption 2's gravity function, along with its cinematic lighting effects and Rockstar's distinctively weighty animations." Which is to say Google is not modeling gravity at all. That's because it's using video game data for training footage because it's cheap and easy to generate. It would not be simulating video game gravity if it was training on real world video inputs.
- ricardobeat 2y agoRemembering off-screen objects, generating spatially consistent features, modeling physical interactions and lights, understanding what "up the stairs" means, all seem to warrant talking about a world model, because that's exactly what's required to do these things compared to simply hallucinating video sequences.
- brink 2y agoWhat is actually of value here? There's no actual game, it's incredibly expensive to compute, the behavior is erratic.. It's cool because it's new - but that will quickly wear off, and once that's gone, what's left? There's insane amounts of money being spent on this, and for what?
- deleted 2y ago[deleted]
- adverbly 2y ago> What is actually of value here? Noone knows yet. AI technology like this is closer to scientific research than it is to product development. AI is basically new magic, and people are in a "discovery" phase where we are still trying to figure out what is possible. Nothing of value was immediately created when they discovered DNA. Productization came much later when it was combined with other technologies to fit a particular use case.
- Menu_Overview 2y agoWell, what's next? Beyond prototyping, I imagine this is an early step towards more practical agents building their own world model. Better problem solving. Prompt: Here's a blueprint of my new house and a photo of my existing furniture. Show me some interior design options.
- ilaksh 2y agoIt's an obviously amazing research development. You just don't like AI. It can be used for training agents, prototyping, video generation, and is quite possibly a glimpse of a whole new type of entertainment or a new way to create video games. What's the point of the massive amount of money spent on video games in general? Or all of the energy spent moving people back and forth to an office? Or expensive meals at restaurants? Or trillions in weaponry? Or television shows or movies?
- nightski 2y agoVideo games bring billions of real people joy. This is sitting in some lab at Google inaccessible to anyone.
- KaoruAoiShiho 2y agoThis is where the GPU limits on China really hurts, Chinese companies have been dropping great proof of concepts but because they have been so compute bottlenecked they can't ever really make something actually competitive or transformative.
- tigerlily 2y agoI can.. see this being used to solve crime, even solving unsolved mysteries and cold cases, among other alternative applications.
- phtrivier 2y agoI don't understand your line of reasoning here. Are you picturing a situation where you would take a photo of a crime scene, and "jump" into a virtual model created from the photo, to help generate intuitions about where to go look for clues ? Kinda like the CSI "enhance quality" meme, but on steroids ? That would be fun to use, but ultimately pointless. An AI model will generate things that are _statistically plausible_ ; solving crimes usually requires finding out the _truth_.
- tigerlily 2y agoYou nailed it, and yes I was being lamely ironic. I am however terrified of a future where this type of thing happens, and people just go along with it instead of stating the obvious facts the way you just did.
- mosdl 2y agoRemake Blade Runner but with the twist that the snake scale was never actually there.
- phtrivier 2y agoIt's easy to be on wrong side of the malfunction, and it would be obvious enough that people would do something about it. Again, the robocop glitch scene - in real life, Kinney's family would have suied, I guess ?
- rndmize 2y agoThese clips feels like watching someone dream in real time. Particularly the door ones, where the environment changes in wild fashion, or the middle NPC one, where you see a character walk into shadow and mostly disappear and a different character walks out.
- m3kw9 2y ago“ Generating unlimited diverse training environments for future general agents” it may seem unlimited but up to a limited point there will be a pattern. I don’t buy that an AI can use a static model and train itself with data generated from it
- diimdeep 2y agowhat for world models be equivalent of ChatGPT for LLM to really blow up in utility?
- singularity2001 2y agotext to roblox maybe?
- rationalfaith 2y agoAs impressive as this might seem let's think about fundamentals. Statistical models will output a compressed mishmash of what they were trained on. No matter how hard they try to cover that inherent basic reality, it is still there. Not to mention the upkeep of training on new "creative" material on a regular basis and the never ending bugs due to non-determinism. Aside from contrived cases for looking up and synthesizing information (Search Engine 2.0). The Tech Industry is over investing in this area exposing an inherent bias towards output rather than solving actual problems for humanity.
- notsylver 2y agoI doubt it, but it would be interesting if they recorded Stadia sessions and trained on that data (... somehow removing the hud?), seems like it would be the easiest way for them to get the data for this.
- blixt 2y agoSeems somewhat likely to me. They probably even trained a model to do both frame generation and upscaling to allow the hardware to work more efficiently while being able to predict the future based on user input (to reduce perceived latency). Seems like Genie is just that but extrapolated much further.
- anyer1 2y ago[dead]
- anyer1 2y ago[dead]
- worldmerge 2y agoThis looks really cool. How can I use it? Like can I mix it with Unity/Unreal?
- cptroot 2y agoFor all that this is lauded as a "prototyping tool", it's frustrating to see Genie2 discarding entire portions of the concept art demo. The original images drawn by Max Cant have these beautiful alien creatures. Large ones floating, and small ones being herded(?). Genie2 just ignores these beautiful details entirely: > That large alien? That's a tree. > That other large alien? It's a bush. > That herd of small creatures? Fugghedaboutit > The lightning storm? I can do one lightning pole. > Those towering baobob/acacia hybrids? Actually only two stories tall. It feels so insulting to the concept artist to show those two videos off.
- Kiro 2y agoThat's an odd thing to complain about. Focusing on such a minor issue feels overly critical at this stage, like anything less than a pixel perfect 3D world representation of the source image is unacceptable. Insulting? Come on... Max Cant works at DeepMind so I'm sure he's fine.
- thirdacc 2y ago> That's an odd thing to complain about. Focusing on such a minor issue feels overly critical at this stage Welcome to HackerNews.
- wongarsu 2y agoYeah, those two demos fell flat for me. The model performing badly on inputs far outside the training data is fine, but those two videos belong in the outtakes section or maybe a limitations section, not next to text lauding the "out-of-distribution generalization capabilities". The videos show the opposite of what's claimed.
- mlsu 2y agoYes, and it should be treated as a front-and-center limitation. Generative text models can kinda ape creativity, because the amount of creativity in the training data is so huge. They still are interpolating across text and cannot generalize well, but the interpolation works to most of us because the data is so varied. It's quite easy to write text so if you have a thought you think is original, odds are someone on the internet wrote about it at some point, which makes the model seem quite capable of originality! But these video game models I think are a lot less capable, because there just aren't that many video games out there, they aren't all that different from one another, and they're all just finite state machines. WASD, desert, jungle, ruins, city. Hell half of them share the very same game engine! How many massive, cohesive, open world games are there? Red Dead and GTA5... Gee, I wonder why so many of their examples look like that?
- zja 2y agoI love the outtakes section in the bottom. It made me laugh but it also feels more transparent than a lot of GenAI stuff that’s being announced.
- tsunamifury 2y agoI'm guessing from the demo sophisticated indoor architectures do not work yet.
- CaptainFever 2y agoAs a game developer, I'm impressed and thinking of ideas of what to do with this kind of tech. The sailboat example was my favourite. Depending on how controllable the tech ends up being, I suppose. Could be anywhere from a gimmick (which is still nice) to a game engine replacement.
- echelon 2y agoYou could compress down a game to run on cheap hardware acceleration. No more Unreal Engine with crazy requirements. Once the hallucinations are fixed, you even get better lighting. This is the Unreal Engine killer. Give it five years.
- noch 2y ago> This is the Unreal Engine killer. Give it five years. We need to calm down with the clickbait-addled thinking that "this new thing kills this established powerful tested useful thing." :-) Game developers have been discussing these tools at length, after all, they are the group of software developers who are most motivated to improve their workflow. No other group of software developers comes close to gamedevs' efficiency requirements. The 1 thing required for serious developers is control. As such, game engines like Unreal and in-house engines won't die. Generative tools will instead open up a whole new, but quite different, way of creating interactive media and games. Those who need maximum control over every frame and every millisecond and CPU cyle will still use engines. The rest who don't will be productive with generative tools.
- echelon 2y ago> gamedevs' efficiency requirements These models won't need you to retopo meshes, write custom shaders, or optimize Nanite or Lumen gameplay. They'll generate the final frames, sans traditional graphics processing pipeline. > The 1 thing required for serious developers is control Same with video and image models, and there's tremendous work being done there as we speak. These models will eventually be trained to learn all of human posture and animation. And all other kinds of physics as well. Just give it time. > Those who need maximum control over every frame and every millisecond and CPU cyle will still use engines. Why do you think that's true? These techniques can already mimic the physics of optics better than 80 years of doing it with math. And they're doing anatomy, fluid dynamics, and much more. With far better accuracy than game engines. These will get faster and they will get controllable.
- esotericimpl 2y ago[dead]
- Const-me 2y agoThe scrolling doesn’t work in my MS Edge so I opened the page in Firefox. Firefox has “Open Video in New Tab” context menu command. When viewed that way, the videos are not that impressive. Horrible visual quality, Egyptian pyramids of random shapes which cast round shadows, etc. I have a feeling many AI researchers are trying to fix things which are not broken. Game engines are not broken, no reasonable amount of AI TFlops going to approach a professional with UE5. DAWs are not broken, no reasonable amount of AI TFlops going to approach a professional with Steinberg Cubase and Apple Logic. I wonder why so many AI researchers are trying to generate the complete output with their models, as opposed to training model to generate some intermediate representation and/or realtime commands for industry-standard software?
- rougka 2y agoWaiting for OpenAI to take this concept and make it into a product
- deleted 2y ago[deleted]
- qwertox 2y agoThis is... something different. It will be interesting to see how we will integrate our current 3D tooling into that prompt-based world. Sometimes a "place a button next to the the door" isn't the same as selecting a button and then clicking on the place next to the door, as it is today, or to sculpt a terrain with a brush, all heavily 3D oriented operations, involving transformation matrix calculations, while that promt-based world is build through words. The current tooling we have is just way too good to just discard it, think of Maya, Blender and the like. How will these interfaces, with the tools they already provide, enable sculpting these word-based worlds? I wonder if some kind of translator will be required, one which precisely instructs "User holds a brush pointing 33° upwards and 56° to the left of the world's x-axis with a brush consisting of ... applied with a strength of ...", or how this will be translated into embeddings or whatever that will be required to communicate with that engine. This is probably the most exciting time for the CG industry in decades, and this means a lot, since we've been seeing incredible progress in every area of traditional CG generation. Also a scary time for those who learned the skills and will now occasionally see some random persons doing incredible visuals with zero knowledge of the entire CG pipeline.
- freedryk 2y agoForget video games. This is a huge step forward for AGI and Robotics. There's a lot of evidence from Neurobiology that we must be running something like this in our brains--things like optical illusions, the editing out of our visual blind spot, the relatively low bandwidth measured in neural signals from our senses to our brain, hallucinations, our ability to visualize 3d shapes, to dream. This is the start of adding all those abilities to our machines. Low bandwidth telepresence rigs. Subatomic VR environments synthesized from particle accelerator data. Glasses that make the world 20% more pleasant to look at. Schizophrenic automobiles. One day a power surge is going to fry your doorbell camera and it'll start tripping balls.
- pmayrgundter 2y agoI can't wait for Schizophrenic automobiles
- sa-code 2y agoThere is a fleshed out realisation of this in Cyberpunk 2077. The cab AI is called Delamain > Delamain was a non-sentient AI created by the company Alte Weltordnung. His core was purchased by Delamain Corporation of Night City to drive its fleet of taxicabs in response to a dramatic increase in accidents caused by human drivers and the financial losses from the resulting lawsuits. The AI quickly returned Delamain Corp to profitability and assumed other responsibilities, such as replacing the company's human mechanics with automated repair drones and transforming the business into the city's most prestigious and trusted transporting service. However, Delamain Corp executives underestimated their newest employee's potential for growth and independence despite Alte Weltordnung's warnings, and Delamain eventually bought out his owners and began operating all aspects of the company by himself. Although Delamain occupied a legal gray area in Night City due to being an AI, his services were so reliable and sought after that Night City's authorities were willing to turn a blind eye to his status. https://cyberpunk.fandom.com/wiki/Delamain_(AI) https://cyberpunk.fandom.com/wiki/Delamain_(AI)
- dekhn 2y agoProbably my favorite side quest in the whole game.
- erulabs 2y agoIt’s interesting to me that we continue to see such pressure on video and world generation, despite the fact that for years now we’ve gotten games and movies that have beautiful worlds filled with lousy, limited, poorly written stories. Star Wars movies have looked phenomenal for a decade, full of bland stories we’ve all heard a thousand times. Are there any game developers working on infinite story games? I don’t care if it looks like Minecraft, I want a Minecraft that tells intriguing stories with infinite quest generation. Procedural infinite world gen recharged gaming, where is the procedural infinite story generation? Still, awesome demo. I imagine by the time my kids are in their prime video game age (another 5 years or so) we will be in a new golden age of interactive story telling. Hey siri, tell me the epic of Gilgamesh over 40 hours of gameplay set 50,000 years in the future where genetic engineering has become trivial and Enkidu is a child’s creation.
- levkk 2y agoNo Man's Sky is kind of what you're looking for, except you may notice its quests (and worlds) become redundant quickly...I say quickly, but that became the case for me after like 30 hours of game play.
- jsheard 2y agoThat's the kicker, LLM driven stories are likely to fall into the same trap that "infinite" procedurally generated games usually do - technically having infinite content to explore doesn't necessarily mean that content is infinitely engaging. You will get bored when you start to notice the same patterns coming up over and over again. Procgen games mainly work when the procedural parts are just a foundation for hand-crafted content to sit on, whether that's crafted by the players (as in Minecraft) or the developers (as in No Mans Sky after they updated it a hundred times, or Rougelikes in general).
- est31 2y agoYeah, generative AI can create cool looking pictures and video but so far it hasn't managed to create infinitely engaging stories. The models aren't there yet.
- Stevvo 2y agoYou can see artifacts common in screen-space reflections in the videos. I suspect they are not due to the model rendering reflections based on screen-space information, but the model being trained on games that render reflections in such a manner.
- xavirodriguez 2y agouoou
- enbugger 2y agoJust like with the images, this will never be at good shape to actually use it for real product as it discards details completely leaving generic 3rd person controller animation. What this should say to you instead is that stuff is really bad on training data side if you start scraping billions of game streams on internet - hard to imagine if there is a bigger chunk of training data than this. Stagnation incoming.
- ata_aman 2y agoWe're about to have on-demand video content and games simply based on prompts. My prediction is we'll have "prompt marketplaces" where you can gen content based on 3rd party prompts (or your own). 3-5 years.
- smusamashah 2y agoIts so much like my lucid dreams where world sometimes stays consistent for a while when I take its control. It's a strange feeling seeing computer hallucinating a world just like I hallucinate a world in dreams. This also means that my dreams will keep looking like this iteration of Genie 2, but computer will scale up and the worlds won't look anything like my dreams anymore in next versions (its already more colorful anyway). I remember image generation use to look like dreams too in the beginning. Now it doesn't look anything like that.
- MrTrvp 2y agoSoon enough I imagine we'll have dream state to cohesive reality models. Our desires and world events can be dissected and analyzed by fine grain and hint authorities to your intent before you know what they mean to you /s.
- jckahn 2y agoAt first I was excited to see a new model, but then I saw no indication that the model is open source so I closed the page.
- anthonymax 2y agoWow, is this artificial intelligence creating this already?
- bbstats 2y agowho is asking for this?
- ddtaylor 2y agoThis is very impressive technology and I am active in this space. Very active. I make an (unreleased) Steam game that helps users create their own games from not knowing how to program. I also (unknowingly) co-authored tools that K12 and university are using to teach game programming. For the time being I will gloss over the fact this might just be a consumer facing product for Google that ends up having nothing to do with younger developers. I'm torn between two ideas: a. Show kids awesome stuff that motivates them to code b. Show kids how to code something that might not be as awesome, but they actually made it On the one hand you want to show kids something cool and get them motivated. What Google is doing here is certainly capable of doing that. On the other hand I want to show kids what they can actually do and empower them. The days of making a game on your own in your basement are mostly dead, but I don't think that means the idea of being someone who can control a large amount of your vision - both technical and non-technical - is important. Not everyone is the same either. I have met kids that would never spend a few hours to learn some Python with pygame to get a few rectangles and sprites on screen that might get more interested if they saw something this flashy. But experience also tells me those kids are extremely less likely to get much value from a tool like this beyond entertainment. I have a 14 year old son myself and I struggle to understand how he sees the world in this capacity sometimes. I don't understand what he thinks is easy or hard and it warps his expectations drastically. I come from a time period where you would grind for hours at a terminal pecking in garbage from a magazine to see a few seconds of crappy graphics. I don't think there should be meaningless labor attached to programming for no reason, but I also think that creating a "cost" to some degree may have helped us. Given two programs to peck into the terminal, which one do you peck? Very few of us had the patience (and lack of sanity) to peck them all.
- empiricus 2y agoFeed it the inputs from the real world and then it will recreate in its mind a mirror of the world. Some say this is what we do also, we live in a virtual reality created by our minds.
- wg0 2y agoGoogle is not coming slow... This is magic. As a casual gamer and someone wanting to make my own game, this is black magic. Lighting, gravity, character animation and what not internalized by the model... from a single image...!
- nopinsight 2y agoThe real goal of this research is developing models that match or exceed human understanding of the 3D world -- a key step toward AGI. A key reason why current Large Multimodal Models (LMMs) still have inferior visual understanding compared to humans is their lack of deep comprehension of the 3D world. Such understanding requires movement, interaction, and feedback from the physical environment. Models that incorporate these elements will likely yield much more capable LMMs. As a result, we can expect significant improvements in robotics and self-driving cars in the near future. Simulations + Limited robot data from labs + Algorithms advancement --> Better spatial intelligence which will lead to a positive feedback loop: Better spatial intelligence --> Better robots --> More robot deployment --> Better spatial intelligence --> ...
- andelink 2y agoIs this type of on-the-fly graphics generation more expensive than purely text based LLMs? What is the inference energy impact of these types of models?
- dartos 2y ago> Genie 2 can generate consistent worlds for up to a minute, with the majority of examples shown lasting 10-20s.
- Coraline456 2y ago[flagged]
- lacoolj 2y agoOpenAI launches Sora (quite a while ago now), Google needs to fire back with something else groundbreaking. I love the advancement of the tech but this still looks very young and I'd be curious what the underlying output code looks like (how well it's formatted, documented, organized, optimized, etc.) Also, this seems oddly related to the recent post from WorldLabs https://www.worldlabs.ai/blog https://www.worldlabs.ai/blog. Wonder if this was timed to compete directly and overtake the related news cycle.
- whiplash451 2y agoI also find the timing vs World Labs demo disturbing.
- alphabetting 2y agoWhat's disturbing? In all likelihood the close timing was world labs rushing to get their demo out the door knowing this was coming because they wouldn't get nearly the hype they did if this came before.
- swyx 2y agoi was wondering when genie 1 was and... it didtn seem to get much love? https://news.ycombinator.com/item?id=39509937 https://news.ycombinator.com/item?id=39509937 @dang was there a main thread here?
- brap 2y agoWhile this is very (very) cool, what is the upside to having a model render everything at runtime, vs. having it render the 3D assets during development (or even JIT), and then rendering it as just another game? I can think of many reasons why the latter is preferable.
- gavmor 2y agoTo me, keeping a world state in sync with rapidly changing external state is the most compelling application. Something like dockercraft: https://github.com/docker/dockercraft https://github.com/docker/dockercraft
- aussieguy1234 2y agoIf it can play video games that simulate the laws of physics, could it control a robot in the physical world?
- dangoodmanUT 2y agothis page loads like shit
- lifeformed 2y agoNeat tech but people might mistake this as being useful for game development, where it'll be less helpful than useless. Games are about interactions, and this actively works against it. You don't want the model to infer mechanics, the designer needs deep control over every aspect of it. People mentioned using this for prototyping a game, but that's completely meaningless. What would it even mean to use this to prototype something? It doesn't help you figure out anything mechanically or visually. It's just, "what if you were an avatar in a world?" What do you do after you run around with your random character controller in your random environments? I think the most useful part of this is the world generation part, not the mechanics inference part.
- serf 2y ago>It doesn't help you figure out anything mechanically or visually. people sell entire franchises off of a few pre-rendered generic-fantasy still images -- I would have to disagree with the premise that this is useless as a visual concept tool. I agree with your notions about integration into an existing game.
- aya700 2y ago[flagged]
- beeflet 2y agoThese game-video models remind me of the dream-like "Mind Game" game described in Ender's Game, because of how it has to spontaneously come up with a new environment to address player input. The game in that book is also described as an AI.
- mdtrooper 2y agoYeah, I think the same it.
- ingen0s 2y agoSo when is Google Glass coming back to spawn this for my pleasure?
- david_shi 2y ago"On the back part of the step, toward the right, I saw a small iridescent sphere of almost unbearable brilliance. At first I thought it was revolving; then I realised that this movement was an illusion created by the dizzying world it bounded. The Aleph's diameter was probably little more than an inch, but all space was there, actual and undiminished. Each thing (a mirror's face, let us say) was infinite things, since I distinctly saw it from every angle of the universe. I saw the teeming sea; I saw daybreak and nightfall; I saw the multitudes of America; I saw a silvery cobweb in the center of a black pyramid; I saw a splintered labyrinth (it was London); I saw, close up, unending eyes watching themselves in me as in a mirror; I saw all the mirrors on earth and none of them reflected me; I saw in a backyard of Soler Street the same tiles that thirty years before I'd seen in the entrance of a house in Fray Bentos; I saw bunches of grapes, snow, tobacco, lodes of metal, steam; I saw convex equatorial deserts and each one of their grains of sand..."
- jerpint 2y agoI have a sneaking suspicion OpenAI will announce something very similar in a few days
- r721 2y agoRelated? https://news.ycombinator.com/item?id=42328798 https://news.ycombinator.com/item?id=42328798
- mrbungie 2y agoGoogle doing the "look how we can do this but you can't and you won't with our help" with more force than ever.
- ganzuul 2y agoWe are repeating the COVID virus scare but this time with software. Most people don't know the difference so this is respectful.
- taneq 2y agoI don't see any mention of DIAMOND (https://diamond-wm.github.io/ https://diamond-wm.github.io/) which does something pretty similar, training a model to predict a game or otherwise 3D world based on videos of gameplay plus corresponding user inputs. It's fascinating how much understanding of the world is being extracted and learned by these models in order to do this. (For the 'that's not really understanding' crowd, what definition of 'understanding' are you using?)
- amaurose 2y agoI am wondering if this sort of thing could be used in the real world, in particular, as navigation helper for a blind pedestrian. Products like Orcam have shown a cam + headphones can more or less easily be packed onto some glasses (for OCR). Navigation helper tools exist since the 80s, but all they basically did until now is scan the environment in a primitive way and use some sort of vibration to alert the user. This is very unspecific, and mostly useless in real life. However, having a vision AI that looks down the path of a blind person could potentially revolutionize this sort of application. For obstacle detection and navigation help. From "Careful, construction site on the sidewalk, 20 meters ahead" to "tactile paving 1 meter to your left". Lets take the game to the streets! If the tech is there, that sounds like a good startup idea...
- nine_k 2y agoWhile cool, this also seems utterly wasteful. Video games offer known "analytical" solutions for the interactions that the model provides as a "statistical approximation", so to say. I would consider a different approach, when the training phase watches games (or video recordings) and refines the formulas that describe its physics, the geometry of the area, the optics, etc. The result would be a "map" that is "playable" without much if any inference involved, and with no time limitation dictated by the size of the context to keep. Very certainly, video game map generation by AI is a thing, and creating models of motion by watching and then fitting reasonably simple functions (fewer than millions of parameters) is also known. I cannot be the first person to think about such possibilities, so I wonder what does the current SOTA look like there.
- furyofantares 2y ago> Video games offer known "analytical" solutions for the interactions that the model provides as a "statistical approximation", so to say. I think this is precisely why they're doing it. Video games are where the data is, because the analytical solutions can generate it. They aren't trying to make a video game. They're trying to make an android.
- halflings 2y ago> I cannot be the first person to think about such possibilities Differentiable Rendering [1] is the closest thing to what you are describing. And yes, people have been working on this for the same reason you outline, it is more data/compute efficient and hence should generalize better. [1] https://blog.qarnot.com/article/an-overview-of-differentiable-rendering https://blog.qarnot.com/article/an-overview-of-differentiabl... But also: > While cool, this also seems utterly wasteful. Video games offer known "analytical" solutions for the interactions that the model provides as a "statistical approximation", so to say. A bit of the same debate as people calling LLMs a "blurry JPEG of the web" and hence useless. Yes this is a statistical approximation to an analytical problem... but that's a very reductive framing to what is going on. To find the symbolic/analytical solution here would require to constrain the problem greatly: not all things on the screen have a differentiable representation, for example complex simulations might involve some kind of custom internal loop/simulation. You waste compute to get a solution that can just be trained on billions of unlabeled (synthetic) examples, and then generalize to previously unseen prompts/environments.
- asdaqopqkq 2y agoFirst think that comes to mind is what about multiplayer? Can we let another models generate in this models's world and vice versa? What if both output in a single instance of a world? What if both output in their own private world and only share data about location and some other metrics?
- andsoitis 2y agoWill the agents in these worlds realize the worlds were sparked by humans?
- ganzuul 2y agoThey have nowhere to go if they do so no. Realization is transcendental.
- fowlie 2y agoOne cool use case for this could be "generative hybrid video meetings"; when I participate in a teams meeting and the majority is in the same physical room, the video conference software could read the wall camera video feed and generate individual video streams of each person as if they sat just in front of me. Of all things this must be the most boring use case for this crazy looking new technology. But hybrid video meetings have always annoyed me and I think to myself that surely there must be a better way (and why hasn't it arrived yet?).
- mrhbh 2y ago[dead]
- baalimago 2y agoTo me, this is a bit like web3: Can't we already do this? What's the benefit?
- woctordho 2y agoWe can already program Minecraft. Also we can already program GTA6. But imagine interpolating Minecraft and GTA6 such that all buildings are destroyable. It may be easier to achieve using AI rather than traditional programming
- robblbobbl 2y agoRelease please
- Ava4333 2y ago[flagged]
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- De_333 2y agolooks amazing!
- infinite-hugs 2y agoDo you want the matrix because this is how you get the matrix
- kouru225 2y agoConsidering the new American Vice President publicly stated he was primarily politically influenced by a guy who wants “a humane alternative genocide” using virtual reality… yea that’s what they want