8 ms·
Minecraft with object impermanence
- ryanisnan 2y agoI enjoyed this read far more than I expected. It’s kind of a dark glimpse into a weird future.
- sedatk 2y agoThat's pretty much how my dreams work. For example, I can read in my dreams (like signs, books), but it's almost always one word at a time, and the rest of the text becomes impossible to focus while I'm reading that word, no matter how hard I try. It's like the "permanence module" is disabled and my brain falls back to a primitive generative AI for the next frame. And, I can't reason about the logic of my experience, the lack of causality, or persistence of objects at the time. Everything feels "as they should be" despite how awkward things get. I guess that's why nightmares are so convincing or hard to get out of, because I can't reason about them. The whole article feels like describing a typical dream to me.
- kqr 2y ago> It's like the "permanance module" is disabled and my brain falls back to a primitive generative AI for the next frame. What if there is no such module and whenever you believe you have experienced permanence it is because your brain confabulated it from what it actually observed?
- sedatk 2y agoI'd still call that permanence module :) AFAIK, that the way our vision works is that our eyes only see a narrowly focused region in detail and the rest of our periphery is generated by our brain (based on what we saw there before). There are optical illusions based on that.
- xeonmc 2y ago"Attention", perhaps?
- lordnacho 2y agoAll you need
- TeMPOraL 2y agoExcept for those times when you're facing off against creatures that can keep track of your saccades, as to move only during the brief moments your brain is ignoring input from your eyes.
- zhengyi13 2y agoI see you too like to read terrifying science fiction. https://en.wikipedia.org/wiki/Blindsight_(Watts_novel) https://en.wikipedia.org/wiki/Blindsight_(Watts_novel) for those who need new stuff in their TBR list.
- diggan 2y ago> AFAIK, that the way our vision works is that our eyes only see a narrowly focused region in detail and the rest of our periphery is generated by our brain (based on what we saw there before). I know it's a over-simplification, but maybe too much of it? 100% of your periphery isn't "generated" by your brain based only on what you've seen before, otherwise it wouldn't be possible to see stuff "at the corner of your eyes" or "looking while not looking" where you can see stuff without actually focusing straight on on the thing itself.
- sedatk 2y agoYou're right, there is a peripheral sensory mechanism involved that generates a rough rendering, but the details are filled in by our brain.
- EZ-E 2y ago> I can read in my dreams (like signs, books), but it's almost always one word at a time, and the rest of the text becomes impossible to focus while I'm reading that word, no matter how hard I try. It's like the "permanence module" is disabled Techniques for lucid dreaming rely on this fact for you to be able to "detect" that you are in a dream. There is one where you need to look at time, look away and look again. The time will change or be in an impossible format if you're dreaming. Then if you're aware you're dreaming you can start to "control" the dream.
- lucidlucia 2y agoFunny enough, I never really needed these tricks. Or I guess, I do have one? It's when I realize the events are scripted. I don't know why, but many of my dreams are perfectly sensical and linear, and sometimes I notice and hijack them. What I can do with them tho, depends on how close I am to waking up. Saddest one was when I realized I was walking my dead dog. She was actually exhibiting behaviors I know she had been used to have, but not ones I had consciously remembered much when thinking about her after her death. I lingered on that one a bit. Still, dreams are dreams, and there was some weirdness, and I knew them for what they were. But that time, I did not fight the script.
- thedanbob 2y ago> It's when I realize the events are scripted. > But that time, I did not fight the script. That's a good way of putting it. In the past when I realized I was dreaming, I would take advantage of the situation and change things, give myself superpowers, etc. But in recent years my few lucid dreams have had really interesting plots, so I consciously chose to let things be and just see where they go.
- sedatk 2y ago> I was walking my dead dog I can relate. I saw my dead dog several times in my dream, and in the dream, it felt like she was never dead in the first place, not like I reunited with her, but it was just another of those regular days with her. So, even though I enjoyed the dream, I didn't have that euphoric joy of reuniting with her. I perceive everything as 100% absolute truth and only realize what's wrong after I wake up.
- HappMacDonald 2y agoMy experience with nightmares has largely been that a part of my subconscious is trying to conduct the dream in an "exciting" direction, to engender thrill on a par with a carnival ride .. but then it slips or gets out of hand and turns into a spiral of self-fulfilling terror instead. I feel like ruminating on that process some during waking time has helped my dream producing instincts to instead steer away from unnecessary sources of fear often before I even perceive (during the dream) that things might have been about to get scary (though I'm often able to assess that pattern after the fact).
- deleted 2y ago[deleted]
- torginus 2y agoThe AI dungeon thing is a salient point - GPT2 (which it started out with) could barely make a coherent paragraph, and GPT3 could write about half a page of text that more or less made sense. With modern LLMs, they still get occasionally tripped up, but you could go for pages without a minor detail not making sense. Something similar might happen with these game models, given enough time.
- TeMPOraL 2y agoAI Dungeon was the text equivalent of the Minecraft thing in TFA. I still remember the distinct feeling I had after getting immersed in the interactive story experience for an hour - for the rest of the day, I felt like I woke up from an intense fever dream.
- itsTyrion 2y agoOooor we could not waste our time with game models that serve no purpose after a minute of "that’s cool"
- skykooler 2y agoThis was a fun read. But some of the things attributed to a glitchy AI are just normal Minecraft stuff. For example: "One time I was swimming across a lake and noticed that the reflections at the water's edge were looking weirdly spiky." That's kelp, it looks like that in the game because the chunks beyond haven't loaded so it stands out from the sky. Also, the "natural blocks of rare glowstone" are honeycomb blocks, which she thinks might be new "cheese" blocks but in fact have been in the game for over five years.
- kqr 2y ago> over five years I think there is a generation of Minecraft players that participated in the alpha, beta, and early 1.0 releases and then grew out of it. For these people, "in the game for over five years" is basically the same as "new". (I belong to that group. I recently started introducing my four-year old to Minecraft and there is so much stuff that was not there when I played!)
- lynndotpy 2y agoInfinidev here :) It's wild to think that some people played Minecraft, had kids, and some of their kids are older than I was when I started playing Minecraft.
- bakugo 2y agoCan confirm. Whenever I'm reminded of the existence of things like Villagers, or the End, my brain still goes "oh yeah, that new thing they added" before reality catches up and I realize just how long those things have been in the game. Part of it is because 80%+ of my playtime was before they were added, but I think another part of it is that those things have never really quite fit into the game in my mind.
- imtringued 2y agoThe End is akin to an inside joke or maybe a pun. You cannot finish a sandbox game, so Notch deliberately added a dimension that you have to go out of your way for, that is difficult to find, unless someone tells you or you read about it. The End dimension isn't meant to fit in the game. It's meant to subvert the idea of a sandbox game. You can say "I've reached the end of Minecraft" and it is true by the letter of the words, but untrue by the spirit of the words. Beating the Ender Dragon does not mean you finished all the content in Minecraft, since there is content even beyond the Ender Dragon.
- fdb 2y agoJanelle Shane’s work has been really inspiring. Her talk at Strange Loop from a couple of years ago — right before DALL-E, before ChatGPT — was a turning point, and very funny: https://youtu.be/yneJIxOdMX4?feature=shared https://youtu.be/yneJIxOdMX4?feature=shared
- baq 2y agoI wonder if this is what Zelazny was thinking about when he was writing the Amber series.
- folli 2y agoSo the AI is trained from the previous frame and the input and tries to predict the next frame, correct? How could you achieve object permanence this way? Will it 'automatically' appear given more training data or more hidden layers? How is this handled in other approaches?
- 383toast 2y agothe same way LLMs are trained to predict next tokens from prev tokens, just longer context = better memory = object permanence
- echoangle 2y agoBut then you can’t just give the previous frame, with the LLM analogy you would have to give the last few thousand frames (that’s the context window, right?). If you only give the previous frame, that’s like having an LLM that only gets the single previous token and has to predict the next one.
- int_19h 2y agoIndeed. Although more recently they figured out a way to feed the hidden state as the new input, which basically allows the model to "continue thinking" in vectors without round-tripping it via words (or pixels). Presumably if you were to take that and build a large enough NN to accommodate all the necessary state it needs to carry and all the rules it needs to be able to execute, then after training it on enough game input you'd have a proper world simulation. Of course, as the article rightly notes, then you have just successfully reimplemented Minecraft in a way that is orders of magnitude more computationally expensive...
- _flux 2y agoPerhaps the trick used by text-based LLMs could be used: when the context window starts filling up, the LLM is asked to summarize the existing data in the context, thus compressing it (lossily..) into smaller space.
- Sharlin 2y ago
- findthewords 2y agoJust like a dream. Electrons running through our brains have no permanence, atoms in the real world do.
- Matthyze 2y agoI'm puzzled by your comment. Do our brains not exist in the real world?
- thiago_fm 2y agoI think he means that the brain does exist, but that our brains don't have a perfect model of the real world or physics, hence our dreams look like this. Also, in a way, comparing GenAI/LLMs to how our brain neural nets work (which isn't exact, but have similar outputs).
- findthewords 2y agoWhen we are awake, we see with our eyes the ground truth. When we are asleep, our eyes are closed.
- omcnoe 2y agoInteresting blog post from the creators with more details about the model: https://oasis-model.github.io https://oasis-model.github.io Interactive demo: https://oasis.decart.ai/ https://oasis.decart.ai/
- lostmsu 2y agoAs far as I can see, the demos in the blog post are cherry-picked. In most cases that I tried the model could not understand that I just mined a block and it should go to my inventory. Still very impressive.
- HelloUsername 2y agoRelated? https://news.ycombinator.com/item?id=42041611 https://news.ycombinator.com/item?id=42041611 https://news.ycombinator.com/item?id=42030787 https://news.ycombinator.com/item?id=42030787
- 01100011 2y agoFirst there is a mountain, then there is no mountain, then there is...
- lou1306 2y ago> Things became duplicated in Tlön; they also tend to become effaced and lose their details when they are forgotten. A classic example is the doorway which survived so long as it was visited by a beggar and disappeared at his death. At times some birds, a horse, have saved the ruins of an amphitheater. JL Borges, Tlön, Uqbar, Orbis Tertius
- aktau 2y agoI highly recommend this to anyone reading HN. I wouldn't say it changed my life, but I think about Borges and Fictions/The Aleph at least once per year.
- lou1306 2y agoI would say that TUOT is absolute essential reading (even more than the -- justifiably -- better-known Library of Babel), since the fact that things disappear unless kept in memory is exactly how the digital world works.
- Matthyze 2y agoI really adore this. AI art is at its best when it's strange and uncanny. Another example (watch till the end): https://www.youtube.com/watch?v=XqLm0qbLJqY https://www.youtube.com/watch?v=XqLm0qbLJqY edit: I'll add this great take too: https://www.youtube.com/watch?v=fQs0GwIlEKM https://www.youtube.com/watch?v=fQs0GwIlEKM
- p0w3n3d 2y agoI love those happy cheerful people burning with a blazing fire in 00:02
- rollcat 2y agoThat's the difference between a deterministic and a stochastic system. Using one where you need the other leads to exactly these kinds of results. Recent trend in LLMs is to gravitate towards more determinism (like fact checking), I'm afraid in the end we will come full circle and end up with a fancy frontend to a database.
- rogual 2y agoThere's a bit in Ender's Game where he plays this computer game that has a dreamlike quality to it. It seemed it could simulate a response to any player action, and continue the story along that path. I think in the book it was a tool for introspection, like dreams can be? I can't really remember. Anyway, as a kid I thought, silly author, that's not how computer games work. But it turns out Orson Scott Card is smarter than me. Give this kind of thing a few years and we'll have it.
- UltraSane 2y agoI think that "game" was a tool used to test the kids.
- kridsdale1 2y agoOr guide them towards their true goal.
- _def 2y agoI kind of want to see this with Pokemon gen 1 games now.
- cubefox 2y agoApparently Oasis Minecraft is what's called a Markov process: The next state only depends on the current state, but not on the past states. To "fix" this, Oasis Minecraft would need a memory of the past frames.
- api 2y agoWhat fascinates me about this is this is what dreams are like. The human brain tries to loosely predict reality, leading to object impermanence and all kinds of trippy weirdness we call dream logic. It’s also why writing good fiction is hard. You have an idea and try to put it down and it’s a mess because your idea is just a loose prediction of what you want to achieve. Then it has to be thought out deliberately or the edges don’t meet up. I’m sure the same goes for painting, etc.
- mkoubaa 2y agoThe idea always starts as a mess. Putting it to paper is a sort of gradient descent to make it less messy. Like many optimization algorithms it's an iterative process Source: am a fiction writer
- lionkor 2y ago> the first step in our research towards more complex interactive worlds As we know, object permanence is not really needed in any videogame, so this makes sense. It's also easy to add in, they just didn't do it because they didn't feel like it. Snark aside, not sure what that is supposed to mean. This isn't interactive in the same sense that games are, its not a world because it doesn't have any properties of a world, its just a video generator you can steer with keys. Cool, but not what we would call an interactive world
- Jeep_ 2y agoI would argue that a video generator you can steer with keys is an interactive world. In this case the way you "steer" has a direct effect on your experience which makes it, in my sense, interactive.
- Dilettante_ 2y agoIt's an interactive experience, but you are not interacting with, having any impact on, a world.
- RedNifre 2y agoYou can interact with it, by placing or breaking blocks, harvesting crops etc.
- p0w3n3d 2y agoTo me AI generated videos (this kind) makes my brain untwist its wrinkles. I've seen this video https://www.youtube.com/watch?v=7Jd-Rr9cJYo https://www.youtube.com/watch?v=7Jd-Rr9cJYo - a Minecraft gamer plays the Oasis AI version. There is literally zero coherence and dreamy trippy style
- npteljes 2y agoThis seems to be in line with a previous submission, where one can "play" Counter Strike in a similar way with AI. Diffusion for World Modeling https://news.ycombinator.com/item?id=41826402 https://news.ycombinator.com/item?id=41826402 Direct link to a video, apologies for the xitter source: https://x.com/Sentdex/status/1845146540555243615?mx=2 https://x.com/Sentdex/status/1845146540555243615?mx=2
- mkoubaa 2y agoThis reminds me of dreaming. Either there's a similarity between dreaming and what this AI is doing or my human brain is stereotypically finding patterns where there are none.
- djmips 2y agoYeah, I tend to agree. I think there is a familiarity and if you read up on how AI generates images it kind of works like that where it builds and amplifies noise or existing structure as a seed of a plausible image. Our perception of reality probably works like this except we are getting some good consistent information from our eyes - using the same apparatus with blank inputs or noise or shifting memories then you get some interesting output.
- MarioMan 2y agoI felt the same way using GPT-2 and AI Dungeon back in the day. The surrealism and incredibly short-term memory was a big part of the fun.
- flessner 2y agoAs a longstanding Minecraft player, that's played less and over the years, I simply don't know how I feel about this. A side of me hopes that this technology gets used to drive progress in self-driving cars, robotics etc. My intuition says that having a sandbox that looks similar yet functions unexpectedly could greatly help with sniffing out edge cases. On the other side I fear that this ends up in the next generation of social media - never ending generative short-form content. Obviously, in its current state the effects on the brain regarding object permanence could be... devastating?
- whywhywhywhy 2y ago>I fear that this ends up in the next generation of social media - never ending generative short-form content. Tiktok already is never ending generative short form content, it's just humans doing the grunt work of figuring out which video performed well and making more like it or pushing the weighting of what they think made it successful higher.
- treyd 2y ago> One time I was swimming across a lake and noticed that the reflections at the water's edge were looking weirdly spiky. This is probably the model trying to generate kelp is the ocean. When water is in chunks on the edge of the loaded world, you can see through the water to the sky at that angle, so you also see the outline of the kelp kinda like that. This is pretty common to see in the game nowadays, but the AI model might be having a hard time since kelp was only added a few years ago and isn't as well represented in the training data.
- whywhywhywhy 2y agoFelt very dreamlike playing the simulated Minecraft, as in they are a place where you can turn around and be somewhere else.
- ThomW 2y agoI'm guessing the author didn't play Cyberpunk at launch -- the cars and pedestrians would blink out of existence if you turned around. I couldn't help but think that Rockstar had figured this out 19 years prior in GTA3. :/
- chronogram 2y agoI recall that Vice City would also remove police from the game when they're out of sight, but that was to increase performance.
- ryanisnan 2y agoI've experienced the same thing even looking straight at vehicles.
- throwawaymobule 2y agoFlicking the camera down to despawn cars is a common trick in speedrunning those 3d GTA games. They did it better than cyberpunk, but it was not figured out.
- nottorp 2y agoIf you smoke the right mushroom and start vanilla MC, you won't need this AI right?
- atahanacar 2y agoSlightly (or very) off topic, but I think it fits so I will write what I feel about people talking about their AI experiences. Feel free to ignore my opinion. I just wanted to know if there is anyone who feels the same way. Listening to someone talk about "their chat with ChatGPT" etc. feels very annoying. It's like they are explaining a dream they had in detail but even less informative or useful because a dream might suggest something about their state of mind. However, content like this (to be fair, this specific one isn't that annoying to me especially compared to "AI said this and that" kind of stuff) has absolutely no constructive value (to me) and feels almost schizophrenic. Maybe I feel this way because people around me, unlike the average HN user's social circle, have only superficial idea about how AI works but are very interested in using it.
- bongodongobob 2y agoEven more annoying is listening to people complain about people being interested in a new paradigm.
- haltcatchfire 2y agoReminds me of Weeping Angels from Dr. Who. Great concept! https://en.m.wikipedia.org/wiki/Weeping_Angel https://en.m.wikipedia.org/wiki/Weeping_Angel
- gwern 2y ago> The problem, as far as I can tell, is that if all this works perfectly, this will be simply the human-programmed Minecraft we already have, except far more expensive to run. The original Minecraft is already infinitely playable, thanks to the way it randomly generates a 7x-larger-than-planet-Earth landscape with each new game. You can't use generative AI like this to get a new kind of game, only a better simulation of the game you trained it on. Oh come now, that's not true and is showing a lack of imagination, especially from someone who's been covering generative models this long. There's lots of things you can do with an accurate differentiable model of Minecraft - just like there's lots of things you can do with a LLM or a diffusion model beyond just 'generate a random sample'. (Just think about all the things you can do with CLIP steering a diffusion model...) Imagine describing GPT-3 as saying 'as far as I can tell, if this all works perfectly, this will simply be Common Crawl, except far more expensive to read'!
- graypegg 2y agoMaybe this is extrapolating a bit too much from this, but this would be sort of neat as a game streaming compression system right? If the service only needed to send a frame every 1-2 seconds, and the interim could be interpolated locally, still but still processing key strokes, it might be juuuuust smooth enough to not notice. Sort of like DLSS, but game specific, and "upscaling" a frame over time instead of resolution. That is of course dependant on running inference locally being somehow less intensive than just running the game. But there we go, it's an interesting thought at least!
- itsTyrion 2y agoMan, I’m glad I’m using paper straws and turning off the PC the second I leave the room for the environment while some "researchers" elsewhere waste god knows how many kWh for a barely functional "AI" version of a game. ♥
- johann8384 2y agoThis is like navigating Steve's dream world while he is sleeping. Just some conjured up wonderland in his subconscious.