9 ms·

Oasis: A Universe in a Transformer

• petersonh 2y ago
  Very cool - has a very dreamlike quality to it
• gunalx 2y ago
  Really cool tech demo. What for the most part impressed me is the inference speed. But I don't really see any use for this unless a way to store worldstate to avoid the issue of it forgetting what it just said.
• jmartin2683 2y ago
  Why? Seems like a very expensive way to vaguely clone a game.
  • deleted 2y ago
    [deleted]
  • haccount 2y ago
    It's an early demo of interactive realtime inference but it appears to have a promise of promptable game worlds and mechanisms. Or "scriptable dynamic imagination" if you will. The answer to "why?" when DeepDream demoed hallucinated dog faces in 2015 was contemporary diffusion models.
    • dartos 2y ago
      > it appears to have a promise of promptable game worlds Is it a world if there’s no permanence? We’ve seen demos like this for a while now (granted not as fast) but the core problem is continuity. (Or some kind of object permanence) It’s a problem for image generators as well. I’d be more interesting if that was any closer to being solved than to have a real time Minecraft screenshot generator. I may have missed it, but I didn’t see anything about prompting. I’d be surprised if this model could generalize beyond Minecraft at all.
      • thirdacc 2y ago
        >It’s a problem for image generators as well. It was, about a year ago. It's a solved problem.
        • dartos 2y ago
          I haven’t seen any image generator maintain spatial consistency for more than a few seconds of video (maybe a min for runway actually) Definitely nothing that maintains it for hours on end. If you have, please link them. I’m very interested.
    • hnlmorg 2y ago
      The dog demo was introducing something new. This isn't. I don't want to be negative about someone else's project but I can completely understand why people are underwhelmed by this. What I think will be the real application for AI in gaming isn't creating poorer versions of native code, it will be creating virtual novels that evolve with the player. Where characters are actual AI rather than following a predefined script. Where you, as the player, can change and shape the story as you wish. Think Star Trek Holodeck "holo-novels" or MMORPGs but can be played fully offline. Rendering the pixels is possibly the worst application for AI at this stage because AI lacks reliable frame by frame continuity, rendering speed, nor an understanding of basic physics, which are all the bare minimum for any modern games engine.
      • stale2002 2y ago
        > What I think will be the real application for AI in gaming isn't creating poorer versions of native code "Why would you use this basic image diffusion model? It just creates a poorer version of existing images!" Is what your statement sounds like. Obviously, you'd do the same thing with this game engine as is done with images. You combine together multiple world models, and thats almost the same as creating a new game. Imagine playing minecraft and you could add a prompt "Now make complex rules for building space ships that can fly to different planets!" and it just mostly works, on the fly.
        • hnlmorg 2y ago
          > “Why would you use this basic image diffusion model? It just creates a poorer version of existing images!" Is what your statement sounds like. I’m not the OP and that’s not what my reply “sounds like”. You’re not reading my comment charitably. > Obviously, you'd do the same thing with this game engine as is done with images. You combine together multiple world models, and thats almost the same as creating a new game. That was actually my point. Adding some VFX to a game engine doesn’t alter the game engines state. What some of the proponents of this project are describing is akin to saying “ray tracing alters the game mechanics”. It doesn’t matter how smart nor performant the AI gets at drawing pixels, it’s still just stateless pixels. This is why I was discussed practical applications of AI that can define the game engines state rather than just what you see on screen. Image diffusion models specifically are not the right applications of AI to create and alter a persistent game state. And that’s why people are underwhelmed by this project.
  • int_19h 2y ago
    It also seems like a pretty decent way to investigate emergent world models in NNs.
• blixt 2y ago
  Super cool, and really nice to see the continuous rapid progress of these models! I have to wonder how long-term state (building a base and coming back later) as well as potentially guided state (e.g. game rules that are enforced in traditional code, or multiplayer, or loading saved games, etc) will work. It's probably not by just extending the context window or making the model larger, though that will of course help, because fundamentally external state and memory/simulation are two different things (right?). Either way it seems natural that these models will soon be used for goal-oriented imagination of a task – e.g. imagine a computer agent that needs to find a particular image on a computer, it would continuously imagine the path between what it currently sees and its desired state, and unlike this model which takes user input, it would imagine that too. In some ways, to the best of my understanding, this already happens with some robot control networks, except without pixels.
  • aithrowawaycomm 2y ago
    There's not even the slightest hint of state in this demo: if you hold "turn left" for a full rotation you don't end up where you started. After a few rotations the details disappear and you're left in the middle of a blank ocean. There's no way this tech will ever make a playable version of Mario, let alone Minecraft.
    • blixt 2y ago
      There's plenty of evidence of state, just a very short-term memory. Examples: - The inventory bar is mostly consistent throughout the play - State transitions in response to key presses - Block breakage over time is mostly consistent - Toggling doors / hatches works as expected - Jumping progresses with correct physics Turning around and seeing approximately the same thing you saw a minute ago is probably just a matter of extending a context window, but it will inherently have limits when you get to the scale of an entire world even if we somehow can make context windows have excellent compression of redundant data (which would greatly help LLM transformers too). And I guess what I'm mostly wondering about is how would you synchronize this state with a ground truth so that it can be shared between different instances of the agent, or other non-agent entities. And again, I think it's important to remember games are just great for training this type of technology, but it's probably more useful in non-game fields such as computer automation, robot control, etc.
• whism 2y ago
  Allow the user to draw into the frame buffer during play and feed that back, and you could have something very interesting.
  • dartos 2y ago
    It’d probably break wildly since it’s really hard to draw Minecraft by hand.
• thrance 2y ago
  > It's a video game, but entirely generated by AI I ctrl-F'ed the webpage and saw 0 occurrence of "Minecraft". Why? This isn't a video game, this is a poor copy of a real video game you didn't even bother to say the name of, let alone credit it.
  • dartos 2y ago
    Yeah it is strange how they make the model sound like it can generate any environment, but only shows demos of the most data-available game ever.
  • deleted 2y ago
    [deleted]
  • armchairhacker 2y ago
    It seems like an interesting attempt to get around legal issues. They can't say "Minecraft" because that's a Microsoft trademark, but they can use Minecraft images as training data, because everyone (including Microsoft) is using proprietary data to train diffusion models. There's the issue that the outputs obviously resemble Minecraft, but Microsoft has its own problems with Bing and DALL-E generating images that obviously resemble trademarked things (despite guardrails).
    • chefandy 2y ago
      Avoiding legal and ethical issues. This stuff was made by a bunch of real people, and people still get their name in movie and game credits even if they got a paycheck from it. Microsoft shamelessly vacuuming up proprietary content didn't change the norms of the way people get credited in these mediums. It's sad to see how thoughtlessly so many people using generative AI disregard the models' source material as "data" while the models (and therefore their creators) almost always get prominently credited for putting in a tiny fraction of the effort. The dubious ethical defense against crediting source works— that models learn about media the same way humans do and adapt it to suit their purposes— is obliterated when it is trained on one work to reproduce that work. That this is equated to generating an image on Midjourney it's a blatant example of a common practice— people want to get credit for other people's work, but when it's time to take responsibility the way a human artist would have to, "The machine did it! It's not my responsibility!"
• brap 2y ago
  Waiting line is too long so I gave up. Can anyone tell me, are the pixels themselves generated by the model, or does it just generate the environment which is rendered by “classical” means?
  • yorwba 2y ago
    If it were to generate an environment rendered by classical means, it would most likely have object permanence instead of regenerating something new after briefly looking away: https://oasis-model.github.io/3_second_memory.webp https://oasis-model.github.io/3_second_memory.webp
  • yokto 2y ago
    It generates the pixels, including the blurry UI at the bottom.
  • naed90 2y ago
    Every pixel is generated! User actions go in, pixels come out -- and there is only a transformer in the middle :) Why is this interesting? Today, not too interesting (Oasis v1 is just a POC). In the future (and by future I literally mean a few months away -- wait for future versions of Oasis coming out soon), imagine that every single pixel you see will be generated, including the pixels you see as you're reading this message. Why is that interesting? It's a new interface for communication between humans and machines. It's like why LLMs are interesting for chat, because they provide humans and machines an ability to interact in a way humans are used to (chat) -- here, computers will be able to see the world as we do and show back stuff to us in a way we are used to. TLDR: imagine telling your computer "create a pink elephant" and just seeing it popup in a game you're playing.
• GaggiX 2y ago
  Kinda hyped to see how this model (or a much bigger one) will run on Etched's transformer ASIC, Sohu, if it ever comes out.
• sigh_again 2y ago
  [flagged]
  • kookamamie 2y ago
    Agreed. Looks like utter garbage, while the tech could be "groundbreaking".
  • dartos 2y ago
    It’s a cool concept, but I’d be very surprised if it gets much better than this. “Playable” statistical models seem to miss the point of games entirely
• 0xTJ 2y ago
  Seems like a neat idea, but too bad that the demo it doesn't work on Firefox.
  • naed90 2y ago
    we really wanted it too! but webtrc was giving us lots of trouble on FF :( trust me, most of the team here is on FF too, and we're bummed we can't play it there haha
  • tayiorrobinson 2y ago
    // ==UserScript== // @name oasis.decart.ai // @match https://oasis.decart.ai/* // @run-at document-start // ==/UserScript== chrome = true;
    • piperly 2y ago
      Haha, it really works. Thanks!
• redblacktree 2y ago
  "If you were dreaming in Minecraft" is the impression that I get. It feels very much like a dream with the lack of object permanence. Also interesting is light level. If you stare at something dark for a while or go "underwater" and get to the point where the screen is black, it's difficult to get back to anything but a black screen. (I didn't manage it in my one playthrough) Very odd sensation indeed.
• vannevar 2y ago
  If anyone has ever read Tad Williams' Otherland series, this is basically the core idea. "The dream that is dreaming us."
• jiwidi 2y ago
  So basically trained a model on minecraft. This is not generalistic at all or whatsoever. Is not like the game comes from a prompt, it probably comes from a bunch of finetuning and gigadatasets from playing minecraft. Would love to see some work like this but with world/games coming from a prompt.
  • naed90 2y ago
    wait for Oasis v2, coming out soon :) (Disclaimer: I'm from the Oasis team)
• xyzal 2y ago
  Maybe we should train models on Mario games to make Nintendo fight for the "Good Cause".
• duendefm 2y ago
  It's not a videogame, it's a fast minecraft screenshot simulator where the prompt between each frame is the state of the input and the previous frames, with something of a resemblance of coherence.
• joshdavham 2y ago
  Incredible work! I think once we’re able to solidly emulate these tiny universes, we can then train agents within them to make even more intelligent AI.
• deleted 2y ago
  [deleted]
• th0ma5 2y ago
  This feels like a nice preview at the bottom of the kinds of unsolvable issues these things will always have to some degree.
• robotresearcher 2y ago
  I don't see how you design and ship a game like this. You can't design a game by setting model weights directly. I do see how you might clone a game, eventually without all the missing things like object permanence and other long-term state. But the inference engine is probably more expensive to run than the game engine it (somewhat) emulates. What is this tech useful for? Genuine question from a long-time AI person.
  • naed90 2y ago
    Yep! Which is why a key point for our next models is to get to a state that you can "code" a new world using "prompting". I agree that these tools become insanely useful only once there is a very good way for creators to "develop" new worlds/games on top of these systems and then users could interact with those worlds. At the end of the day, it should provide the same "API" as a game engine does: creators develop worlds, users interact with those worlds. The nice thing is that if AI can actually fill this role, then it would be: 1. Potentially much easier to create worlds/games (you could just "talk" to the AI -- "add a flying pink elephant here") 2. Users could interact with a world that could change to fit each game session -- this is truly infinite worlds Last point: are we there yet? Ofc not! Oasis v1 is a first POC. Wait just a bit more for v2 ;)
  • andoando 2y ago
    I suppose you could potentially take a movie like avatar and create a somewhat interactive experience with it?
    • deleted 2y ago
      [deleted]
  • notfed 2y ago
    Obviously this tool is not going to generate a "ship"pable game for you. AI is a long way off from that. As for "design", I don't find it very hard to see how incredibly useful being able to rapidly prototype a game would be, even if it requires massive GPU usage. And papers like these are only stepping stones to getting there.
• therein 2y ago
  Queue makes it untestable. It isn't running client-side? What's with the queueing?
• gessha 2y ago
  I find this extremely disappointing. A diffusion transformer trained on Minecraft frames and accelerated on an ASIC... Okay? From the demo(that doesn't work on Firefox) you can see that it's overfit to the training set and it doesn't have a consistent state transition. If you define it as a Markov decision process with states being images, actions being keyboard/mouse inputs, the probability transition being the transformer model, the model is a very poor one. Turning the mouse around shouldn't result in a completely different world, it should result in the exact same point of space from different camera orientation. You can fake it by fudging with the training data and augmenting with walking a bit, doing a 360 camera rotation and continuing the exploration but that will just overfit to that specific seed. The page says their ASICs model inference supports 60+ players. Where are they shown playing together? What's the point of touting multiplayer performance when realistically, the poor state transition will mean those 60+ players are playing single player DeepDream Minecraft?
• deleted 2y ago
  [deleted]
• mrtnl 2y ago
  Very cool tech demo! Curious to see if we continue to generate environments in this level or move more to generating the physics
• robblbobbl 2y ago
  Me gusta!
• amiramer 2y ago
  So cool! Curious to see how it evolves.. seems like a portal into fully generated content, 0 applications. So exciting. Will it also be promptable at some point?
• aaladdin 2y ago
  How would you verify that real world physics actually hold here? Otherwise, such breaches could be maliciously and unfairly exploited.
• TalAvner 2y ago
  This is next level! I can't believe it's all AI generated in real time. Can't wait to see what's next.
• goranim 2y ago
  Love it! this virtual world looks so goo and it is also changing really fast so seems like a very powerful model!
• keidartom 2y ago
  So cool!
• shanim_ 2y ago
  Could you explain how the interaction between the spatial autoencoder (ViT-based) and the latent diffusion backbone (DiT-based) enables both rapid response to real-time input and maintains temporal stability across long gameplay sequences? Specifically, how does dynamic noising integrate with these components to mitigate error compounding over time in an autoregressive setup?
• jhonj 2y ago
  tried their not-a-game and it was SICK to play knowing it's not a game engine. really sick. When did these Decart ppl started working on that. must be f genius ppl
• hesyechter 2y ago
  Very very cool, i love it Good luck
• djhworld 2y ago
  I think this is really cool as a sort of art piece? It's very dreamlike and unsettling, especially with the music
• drdeca 2y ago
  This apparently currently only supports chrome. I hope it will support non-chrome browsers in the future.
• pka 2y ago
  Negative comments are so weird, it's like people forgot what GPT 2 was like. I know this isn't completely new, but it's a world simulation inside a goddamn LLM. Not perfect, not coherent over longer time periods, but still insane. I swear if tomorrow magic turned out to be real and wizards start controlling the literal fabric of the universe people will be like "meh" before the week ends :D
• Daroar 2y ago
  I can see where they are going with it and wow! Truly the proof that we are all indeed in a simulation.
• duan2112 2y ago
  Love it!!!
• piperly 2y ago
  From a research perspective, this approach isn’t new; David Ha and Danijar Hafner explored similar ideas years ago. However, the technique itself and the achievement of deploying it for testing by hundreds of users is commendable. It feels more like an experimental prototype than a viable replacement for mainstream gaming.