12 ms·
> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected To temper this a bit, you may want to pay close attention to t
by wavemode 2y ago
> Apparently there’s more cause, effect, and sequencing in diffusion models than what I expected
To temper this a bit, you may want to pay close attention to the demo videos. The player rarely backtracks, and for good reason - the few times the character does turn around and look back at something a second time, it has changed significantly (the most noticeable I think is the room with the grey wall and triangle sign).
This falls in line with how we'd expect a diffusion model to behave - it's trained on many billions of frames of gameplay, so it's very good at generating a plausible -next- frame of gameplay based on some previous frames. But it doesn't deeply understand logical gameplay constraints, like remembering level geometry.
- hoosieree 2y agoSmall objects like powerups appear and disappear as the player moves (even without backtracking), the ammo count is constantly varying, getting shot doesn't deplete health or armor, etc.
- mensetmanusman 2y agoThat is kind of cool though, I would play like being lost in a dream. If on the backend you could record the level layouts in memory you could have exploration teams that try to find new areas to explore.
- debo_ 2y agoIt would be cool for dream sequences in games to feel more like dreams. This is probably an expensive way to do it, but it would be neat!
- Groxx 2y agoThere's an example right at the beginning too - the ammo drop on the right changes to something green (I think that's a body?)
- codeflo 2y agoEven purely going forward, specks on wall textures morph into opponents and so on. All the diffusion-generated videos I’ve seen so far have this kind of unsettling feature.
- bee_rider 2y agoIt it like some kind of weird dream doom.
- whiteboardr 2y agoBut does it need to be frame-based? What if you combine this with an engine in parallel that provides all geometry including characters and objects with their respective behavior, recording changes made through interactions the other model generates, talking back to it? A dialogue between two parties with different functionality so to speak. (Non technical person here - just fantasizing)
- bee_rider 2y agoIn that case, the title of the article wouldn’t be true anymore. It seems like a better plan, though.
- beepbooptheory 2y agoWhat would the model provide if not what we see on the screen?
- whiteboardr 2y agoThe environment and everything in it. “Everything” would mean all objects and the elements they’re made of, their rules on how they interact and decay. A modularized ecosystem i guess, comprised of “sub-systems” of sorts. The other model, that provides all interaction (cause for effect) could either be run artificially or be used interactively by a human - opening up the possibility for being a tree : ) This all would need an interfacing agent that in principle would be an engine simulating the second law of thermodynamics and at the same time recording every state that has changed and diverged off the driving actor’s vector in time. Basically the “effects” model keeping track of everyones history. In the end a system with an “everything” model (that can grow overtime), a “cause” model messing with it, brought together and documented by the “effect” model. (Again … non technical person, just fantasizing) : )
- mplewis 2y agoWhat you’re asking for doesn’t make sense.
- 2y ago
- dewarrn1 2y agoGreat observation. And not entirely unlike normal human visual perception which is notoriously vulnerable to missing highly salient information; I'm reminded of the "gorillas in our midst" work by Dan Simons and Christopher Chabris [0]. [0]: https://en.wikipedia.org/wiki/Inattentional_blindness#Invisible_Gorilla_Test https://en.wikipedia.org/wiki/Inattentional_blindness#Invisi...
- bamboozled 2y agoAre you saying if I turn around, I’ll be surprised at what I find ? I don’t feel like this is accurate at all.
- matheusd 2y agoIf a generic human glances at an unfamiliar screen/wall/room, can they accurately, pixel-perfectly reconstruct every single element of it? Can they do it for every single screen they have seen in their entire lives?
- bamboozled 2y agoI never said pixel perfect, but I would be surprised if whole objects , like flaming lanterns suddenly appeared. What this demo demonstrates to me is how incredible willing we are to accept what seems familiar to us as accurate. I bet if you look closely and objectively you will see even more anomalies. But at first watch, I didn’t see most errors because I think accepting something is more efficient for the brain.
- ben_w 2y agoYou'd likely be surprised by a flaming lantern unless you were in Flaming Lanterns 'R Us, but if you were watching a video of a card trick and the two participants changed clothes while the camera wasn't focused on them, you may well miss that and the other five changes that came with that.
- dewarrn1 2y ago
- alickz 2y agois that something that can be solved with more memory/attention/context? or do we believe it's an inherent limitation in the approach?
- nmstoker 2y agoI saw a longer video of this that Ethan Mollick posted and in that one, the sequences are longer and they do appear to demonstrate a fair amount of consistency. The clips don't backtrack in the summary video on the paper's home page because they're showing a number of district environments but you only get a few seconds of each. If I studied the longer one more closely, I'm sure inconsistencies would be seen but it seemed able to recall presence/absence of destroyed items, dead monsters etc on subsequent loops around a central obstruction that completely obscured them for quite a while. This did seem pretty odd to me, as I expected it to match how you'd described it.
- wavemode 2y agoYes it definitely is very good for simulating gameplay footage, don't get me wrong. Its input for predicting the next frame is not just the previous frame, it has access to a whole sequence of prior frames. But to say the model is simulating actual gameplay (i.e. that a person could actually play Doom in this) is far fetched. It's definitely great that the model was able to remember that the gray wall was still there after we turned around, but it's untenable for actual gameplay that the wall completely changed location and orientation.
- dr_dshiv 2y agoIt's an empirical question, right? But they didn't do it...
- TeMPOraL 2y ago> it's untenable for actual gameplay that the wall completely changed location and orientation. It would in an SCP-themed game. Or dreamscape/Inception themed one. Hell, "you're trapped in Doom-like dreamscape, escape before you lose your mind" is a very interesting pitch for a game. Basically take this Doom thing and make walking though a specific, unique-looking doorway from the original game to be the victory condition - the player's job would be to coerce the model to generate it, while also not dying in the Doom fever dream game itself. I'd play the hell out of this. (Implementation-wise, just loop in a simple recognition model to continously evaluate victory condiiton from last few frames, and some OCR to detect when player's hit points indicator on the HUD drops to zero.) (I'll happily pay $100 this year to the first project that gets this to work. I bet I'm not the only one. Doesn't have to be Doom specifically, just has to be interesting.)
- TeMPOraL 2y agoSo for the next iteration, they should add a minimap overlay (perhaps on a side channel) - it should help the model give more consistent output in any given location. Right now, the game is very much like a lucid dream - the universe makes sense from moment to moment, but without outside reference, everything that falls out of short-term memory (few frames here) gets reimagined.
- Workaccount2 2y agoI don't see this as something that would be hard to overcome. Sora for instance has already shown the ability for a diffusion model to maintain object permanence. Flux recently too has shown the ability to render the same person in many different poses or images.
- idunnoman1222 2y agoWhere does a sora video turn around backwards? I don’t even maintain such consistency in my dreams.
- idunnoman1222 2y agoWhere does a sora video turn around backwards? I can’t maintain such consistency in my own dreams.
- Workaccount2 2y agoI don't know of an example (not to say it doesn't exist) but the problem is fundamentally the same as things moving out of sight/out of frame and coming back again.
- Jensson 2y ago> the problem is fundamentally the same as things moving out of sight/out of frame and coming back again Maybe it is, but doing that with the entire scene instead of just a small part of it makes the problem massively harder, as the model needs to grow exponentially to remember more things. It isn't something that we will manage anytime soon, maybe 10-20 years with current architecture and same compute progress. Then you make that even harder by remembering a whole game level? No, ain't gonna happen in our lifetimes without massive changes to the architecture. They would need to make a different model keep track of level state etc, not just an image to image model.
- Workaccount2 2y ago10 to 20 years sounds wildly pessimistic In this sora video the dragon covers half the scene, and its basically identical when it is revealed again ~5 seconds later, or about 150 frames later. The is lots of evidence (and some studies) that these models are in fact building internal world models. https://www.youtube.com/watch?v=LXJ-yLiktDU https://www.youtube.com/watch?v=LXJ-yLiktDU Buckle in, the train is moving way faster. I don't think there would be much surprise if this is solved in the next few generations of video generators. The first generation is already doing very well.
- nielsbot 2y agoYou can also notice in the first part of the video the ammo numbers fluctuate a bit randomly.