6 ms·

The Matrix: Infinite-Horizon World Generation with Real-Time Interaction

210 points by lnyan 2y ago

jerpint 2y ago
I’m really excited for where this is going. From the demo videos, it seems to be a step up from Oasis, which itself came out only 2 weeks ago. I expect to see a lot of innovative use cases in this field
shmerl 2y ago
> Click to play Clicking - nothing works.
- tsaoyu 2y ago
  Click to play [...the video]
ftyhbhyjnjk 2y ago
[flagged]
- echelon 2y ago
  Yeah, no. It's posted at midnight on Thursday (eastern time). It's mobile unfriendly, hard to read, and has no videos. The other models had playable demos and videos, and they were posted in the middle of the day so we could think about it during work. The hype wave for this stuff is going to require bigger splashes for each new model. New image-to-3D models garner a yawn, and it's going to be the same here soon. These folks put a lot of thought into their branding (and CSS), but they kind of let the excitement fizzle as there's nothing to look at and evaluate. We just have to trust that they did things? It's a bunch of pictures of a car and green text. It's far too late to open the paper. Basically they just don't excel at marketing. 3/10. Edit: I had no idea this was Chinese until you said it. The page doesn't mention names at top, and it didn't suck me into the paper.
  - sunnybeetroot 2y ago
    Tap the images to watch the videos.
- alignmint 2y ago
  [flagged]
- wellthisisgreat 2y ago
  lol had no idea it's chinese nor cared which country produced it. nationalism in any form is repulsive was intrigued by the post but couldn't get anything to play
  - Sohcahtoa82 2y ago
    > lol had no idea it's chinese Same. There are some people that REALLY want to find racism even when it's not there. The "everything is racist" crowd is just as insufferable as the "nothing is racist" crowd.
- simojo 2y ago
  See also: timezones
- anonzzzies 2y ago
  Really don't think that's the case. I don't care who makes things at first; I first want to see if they are interesting and then maybe dig deeper. Looks cool, upvoted and bookmarked to wait for the playable demo.
- ben_w 2y ago
  Here's my list of submissions: https://news.ycombinator.com/submitted?id=ben_w https://news.ycombinator.com/submitted?id=ben_w Of those ten: • 6 had zero comments, 1 had only my own comment • 1 had 7 • 1 had 17 • 1 had 148 I have no reason to think there's a nationality thing here, stuff just falls off the top fast and most people don't comment or upvote… same as with most comments themselves.
- Cthulhu_ 2y ago
  ? You're making a generalization / judgment based on... the amount of comments left only two hours after it was posted, late at night in the US and early in the morning in Europe. It's also very much a meta-comment instead of something substantial about the subject at hand, which isn't encouraging discussion about the thing itself but actually draws the focus on it being Chinese, which most people weren't even aware of until you drew attention to it.
efitz 2y ago
Prediction: in 20 years, I’m going to be reading about some dude who wrote a program to drive the car continuously until it ran into some surreal edge condition, and finally hit it. There will be a subculture of “matrix glitchers” who spend much of their time doing these kinds of experiments.
- soulofmischief 2y ago
  People have been doing that with Minecraft for over a decade. In the old days, once you got far away enough, the terrain generation would go haywire. Lots of videos from that time period of people exploring the "edge of the world". Personally, these were the kind of glitches which made games feel magical and "real" to me as a kid. Being able to analyze a system by breaking it made it seem so much more tangible, like an actual place I had an NTSC-sized porthole into.
  - billisonline 2y ago
    Cf. the "Minus World" in Super Mario Bros. for the NES. https://en.wikipedia.org/wiki/Minus_World https://en.wikipedia.org/wiki/Minus_World
    - soulofmischief 2y ago
      Ha! I remember being either 5 or 6 when my uncle showed me Minus World and it blowing my mind. That might have actually been my first exposure to "backrooms" glitches like that. What an amazing glitch. It even worked on my combo Super Mario Bros / Duck Hunt cartridge MissingNo. is another good example. I have fond memories spending untold hours in my favorite game engines trying to break free. The Jak and Daxter series were some of my favorite to break, due to the uniqueness and flexibility of the engine and the weird ways that the chunk loading system could be broken.
      - axus 2y ago
        Ahh, "Mountain King" on the Atari 2600 was the game for me finding a cool bug. If you bounced just right, you'd soar over the mountain into the glitches far above. Games didn't crash, they just worked with what they had.
grodes 2y ago
unreadable website
ribcage 2y ago
Someone should ban "AI" articles on Hacker News.
- ferguu_ 2y ago
  But then there would be no articles!
- Cthulhu_ 2y ago
  Why? It's a mostly democratic news aggregator site without much editorial overview, if you don't like it, downvote / don't upvote it, or write a client that filters topics you don't like out based on some keywords. You didn't need to open this page and comment on it if you don't like it, nobody's making you read things you don't like.
- steego 2y ago
  I don't know if you're actually new here, or you've been reading HN for years, but your account is only 11 months old. We talk about a lot of things here, but when we do talk about AI, we tend to prefer talk about things with code or papers. This particular project has a paper. They're expecting to publish their code soon. Here's the paper: https://thematrix1999.github.io/article/the_matrix.pdf https://thematrix1999.github.io/article/the_matrix.pdf You don't have to read it, but you may want to consider it if you want to learn something and/or contribute something meaningful to the conversation.
- olddustytrail 2y ago
  Yes, and that someone should be you. Be the change you want to be.
- brcmthrowaway 2y ago
  Check lobsters
ilaksh 2y ago
I'm into VR and mixed reality, and I think this is headed to making the Holodeck real in an immersive way. That's the concept of the Matrix and what they are demoing, just in 2d. I am guessing the main thing holding this stuff back in terms of fidelity and consistency or generalization is just compute. But the new techniques they have here have just dramatically lowered the compute costs and increased the generalization. Maybe just something like the giant Cerebras SRAM chips will get to the next 10 X in scale that smooths this out and pushes it closer to Star Trek. Or maybe some new paradigm like memristors. But I'm looking forward to within just a few years being able to put on some fairly comfortable mixed reality glasses and just asking for whatever or whoever I want to appear in my home (for example) according to my whim. Or, train it on a lot of how-to videos such as cooking. It just materializes an example of someone showing you exactly what you need to do right in your kitchen. Here's another crazy idea: train on videos and interactions with productivity applications rather than games. In the future, for small businesses, we skip having the AI generate source code and just describe how the application works. The data and program state are just stored in a giant context window, and the application functionality changes the instant you make a request.
- scotty79 2y ago
  I think we need devices that can produce gaussian splat videos to become as common as 2d cameras.
- chamomeal 2y ago
  I think fever dream AI stuff like this would be nightmarishly terrifying in VR lol. Still, would be super cool to try
alkonaut 2y ago
What does "world" mean here? How does the spatiality fit into some latent space? Or what constitutes the "world"? If the answer is, there is none, the world is just frames of video and any consistency quickly blurs out after a few seconds. That's not a world generation, that's just generation of video frames following frames. Not that it isn't cool, but it has almost zero usability for generating a "world" simulation. The key to a realistic world is that you can reliably navigate it. Visit and revisit places. If you modify anything, those modifications are persisted. If you leave a room and re-enter it hours later, the base expectation is that the same objects are in that room. Wouldn't a working approach be to just create a really low resolution 3D world in the traditional "3D game world" sense to get the spatial consistency. Then this crude map with attributes is fed into frame generation to create the resulting world? It wouldn't be infinite, but on the other hand no one has a need for an infinite world either. A spherical world solves the border issue pretty handily. As I understood it, there was some element of that in the new FS2024 (discussed yesterday on HN).
- QuantumG 2y ago
  What you really want is a story telling engine informing the world model and the humans need to generate power.
- scotty79 2y ago
  It's basically a dream. Maybe that's what dreams are? World prediction engine running with no sensor input and memory in read only mode?
  - withinboredom 2y ago
    When you go to sleep, your brain stem disconnects from your body and your brain enters a feedback loop. The sensors are very much connected, just to whatever. Hence why you can grow dragon wings in your dreams and feel them. Memories can be made as well. I used to be really into lucid dreaming and time compression. My longest dream was nine years compressed into a 12-hour sleep period.
    - 2y ago
abricq 2y ago
This is surely really cool. Just a bit sad that, as phrased by the authors, the "First Real-Time" virtual world created for the demo is a fat & fast SUV driving on virgin lands.
- spuz 2y ago
  Why is that sad?
  - ferguu_ 2y ago
    When you think of "The Matrix" - An infinitely generated and extremely complex world, I don't think most people would picture this, and especially not in a car like that. The car choice makes sense when you consider there aren't any roads at all! Of course, it is a very new technology, but then that does bring into question the paper author's frankly ambitious title choice. Interestingly, this website does remind me of something I played a while back - check it out! [https://slowroads.io/ https://slowroads.io/]. It has roads!
    - spuz 2y ago
      What would you expect it to look like based on the title? (I don't understand the title and have no expectations)
      - ferguu_ 2y ago
        Well, the Matrix movie showed a perfect depiction of the world stuck in 1999, with people given freedom to do as they wish (or what they thought was freedom). So... I guess more than just driving on barren plains haha.
- pmayrgundter 2y ago
  Sigh, agreed!
- deleted 2y ago
  [deleted]
akutlay 2y ago
Why would you want to generate all the pixels using this model instead of generating all the art, physics, and objects in the world using a game engine? The engine does so much of the physics and keeps everything stable for very cheap.
- 01HNNWZ0MV43FF 2y ago
  Because you wouldn't have to wait for a human to write the engine and write the game
  - drusepth 2y ago
    We generate 2d art, 3d models, some other in-world data, and soon animations at our game studio. The huge problems of generating pure pixels without a persistent game state behind it is illustrated well by the recent MinecraftAI craze (where it's a cool demo, but absolutely unplayable for anything other than novelty), and exacerbated on the dev side by missing out on tons of "free" stuff you get from existing engines that already do almost every kind of physics ops for you.
- FeepingCreature 2y ago
  Generating coherent objects in an infinite world is probably harder.
- deleted 2y ago
  [deleted]
wavemode 2y ago
I wish researchers would spend more time on using generative models to create level geometry, rather than trying to generate video from scratch. It would be both cheaper and more effective for stable gameplay.
- jebarker 2y ago
  There's so many researchers working in AI now that we can afford to explore all avenues. This probably isn't going to lead to a fully generative AI game engine, but I bet there'll be useful learnings along the way.
- kridsdale1 2y ago
  I think Ubisoft are leaders in that space. I saw algorithmic generation of New York style geometry for some behind the scenes video for The Division 2. I wouldn’t be surprised if epic has similar.
dangsux 2y ago
[dead]
pedalpete 2y ago
This is the future I was trying to pitch in 2018 when we had built Ayvri and had every paraglider in the world, the world's largest ultramarathons, drone operators, and lots of other users of our real-world 3D environment. Though we were using map tiles at the time, we were developing a model that took photos and a GPS track to add information that better matched environmental conditions (cloud, better lighting, etc). People still ask me to open-source or give them our source code, but the code was acquired, so that isn't possible. But I do regularly say that if I were to rebuild Ayvri today, I'd do it as an interactive video rather than loading tiles.
darepublic 2y ago
I didn't fully grok what this was about from the website. Though just last night I was talking to a friend about that quote from the Matrix that Morpheus tells Neo, so some nice synchronicity there. The sense I got from this is that they are developing a triple AAA type virtual world that can get generated on the fly based on text prompts? When the authors say frame level control do they mean that at any point, the next frame can be manipulated, either to be completely new or to influence the current story or context that is being played out?
m3kw9 2y ago
Definitely used Cyberpunk2077 footage to train
petermcneeley 2y ago
"Welcome to the Matrix" with matrix-like rain seems like an invitation for Warner Bros to sue you into oblivion.
airbreather 2y ago
No source, no playable demo, just promises of. Could be total vaporware for all we know. It is an ad, a statement of achievement in case someone else states it first, or what? Seems like it would be better on Youtube, it really doesn't offer much of use right now.
deleted 2y ago
[deleted]

210 points by lnyan 2y ago

I’m really excited for where this is going. From the demo videos, it seems to be a step up from Oasis, which itself came out only 2 weeks ago. I expect to see a lot of innovative use cases in this field

shmerl 2y ago

> Click to play Clicking - nothing works.

tsaoyu 2y ago
Click to play [...the video]

ftyhbhyjnjk 2y ago

[flagged]

echelon 2y ago
Yeah, no. It's posted at midnight on Thursday (eastern time). It's mobile unfriendly, hard to read, and has no videos. The other models had playable demos and videos, and they were posted in the middle of the day so we could think about it during work. The hype wave for this stuff is going to require bigger splashes for each new model. New image-to-3D models garner a yawn, and it's going to be the same here soon. These folks put a lot of thought into their branding (and CSS), but they kind of let the excitement fizzle as there's nothing to look at and evaluate. We just have to trust that they did things? It's a bunch of pictures of a car and green text. It's far too late to open the paper. Basically they just don't excel at marketing. 3/10. Edit: I had no idea this was Chinese until you said it. The page doesn't mention names at top, and it didn't suck me into the paper.
- sunnybeetroot 2y ago
  Tap the images to watch the videos.
alignmint 2y ago
[flagged]
wellthisisgreat 2y ago
lol had no idea it's chinese nor cared which country produced it. nationalism in any form is repulsive was intrigued by the post but couldn't get anything to play
- Sohcahtoa82 2y ago
  > lol had no idea it's chinese Same. There are some people that REALLY want to find racism even when it's not there. The "everything is racist" crowd is just as insufferable as the "nothing is racist" crowd.
simojo 2y ago
See also: timezones
anonzzzies 2y ago
Really don't think that's the case. I don't care who makes things at first; I first want to see if they are interesting and then maybe dig deeper. Looks cool, upvoted and bookmarked to wait for the playable demo.
ben_w 2y ago
Here's my list of submissions: https://news.ycombinator.com/submitted?id=ben_w https://news.ycombinator.com/submitted?id=ben_w Of those ten: • 6 had zero comments, 1 had only my own comment • 1 had 7 • 1 had 17 • 1 had 148 I have no reason to think there's a nationality thing here, stuff just falls off the top fast and most people don't comment or upvote… same as with most comments themselves.
Cthulhu_ 2y ago
? You're making a generalization / judgment based on... the amount of comments left only two hours after it was posted, late at night in the US and early in the morning in Europe. It's also very much a meta-comment instead of something substantial about the subject at hand, which isn't encouraging discussion about the thing itself but actually draws the focus on it being Chinese, which most people weren't even aware of until you drew attention to it.

efitz 2y ago

Prediction: in 20 years, I’m going to be reading about some dude who wrote a program to drive the car continuously until it ran into some surreal edge condition, and finally hit it. There will be a subculture of “matrix glitchers” who spend much of their time doing these kinds of experiments.

soulofmischief 2y ago
People have been doing that with Minecraft for over a decade. In the old days, once you got far away enough, the terrain generation would go haywire. Lots of videos from that time period of people exploring the "edge of the world". Personally, these were the kind of glitches which made games feel magical and "real" to me as a kid. Being able to analyze a system by breaking it made it seem so much more tangible, like an actual place I had an NTSC-sized porthole into.
- billisonline 2y ago
  Cf. the "Minus World" in Super Mario Bros. for the NES. https://en.wikipedia.org/wiki/Minus_World https://en.wikipedia.org/wiki/Minus_World
  - soulofmischief 2y ago
    Ha! I remember being either 5 or 6 when my uncle showed me Minus World and it blowing my mind. That might have actually been my first exposure to "backrooms" glitches like that. What an amazing glitch. It even worked on my combo Super Mario Bros / Duck Hunt cartridge MissingNo. is another good example. I have fond memories spending untold hours in my favorite game engines trying to break free. The Jak and Daxter series were some of my favorite to break, due to the uniqueness and flexibility of the engine and the weird ways that the chunk loading system could be broken.
    - axus 2y ago
      Ahh, "Mountain King" on the Atari 2600 was the game for me finding a cool bug. If you bounced just right, you'd soar over the mountain into the glitches far above. Games didn't crash, they just worked with what they had.

grodes 2y ago

unreadable website

ribcage 2y ago

Someone should ban "AI" articles on Hacker News.

ferguu_ 2y ago
But then there would be no articles!
Cthulhu_ 2y ago
Why? It's a mostly democratic news aggregator site without much editorial overview, if you don't like it, downvote / don't upvote it, or write a client that filters topics you don't like out based on some keywords. You didn't need to open this page and comment on it if you don't like it, nobody's making you read things you don't like.
steego 2y ago
I don't know if you're actually new here, or you've been reading HN for years, but your account is only 11 months old. We talk about a lot of things here, but when we do talk about AI, we tend to prefer talk about things with code or papers. This particular project has a paper. They're expecting to publish their code soon. Here's the paper: https://thematrix1999.github.io/article/the_matrix.pdf https://thematrix1999.github.io/article/the_matrix.pdf You don't have to read it, but you may want to consider it if you want to learn something and/or contribute something meaningful to the conversation.
olddustytrail 2y ago
Yes, and that someone should be you. Be the change you want to be.
brcmthrowaway 2y ago
Check lobsters

ilaksh 2y ago

I'm into VR and mixed reality, and I think this is headed to making the Holodeck real in an immersive way. That's the concept of the Matrix and what they are demoing, just in 2d. I am guessing the main thing holding this stuff back in terms of fidelity and consistency or generalization is just compute. But the new techniques they have here have just dramatically lowered the compute costs and increased the generalization. Maybe just something like the giant Cerebras SRAM chips will get to the next 10 X in scale that smooths this out and pushes it closer to Star Trek. Or maybe some new paradigm like memristors. But I'm looking forward to within just a few years being able to put on some fairly comfortable mixed reality glasses and just asking for whatever or whoever I want to appear in my home (for example) according to my whim. Or, train it on a lot of how-to videos such as cooking. It just materializes an example of someone showing you exactly what you need to do right in your kitchen. Here's another crazy idea: train on videos and interactions with productivity applications rather than games. In the future, for small businesses, we skip having the AI generate source code and just describe how the application works. The data and program state are just stored in a giant context window, and the application functionality changes the instant you make a request.

scotty79 2y ago
I think we need devices that can produce gaussian splat videos to become as common as 2d cameras.
chamomeal 2y ago
I think fever dream AI stuff like this would be nightmarishly terrifying in VR lol. Still, would be super cool to try

alkonaut 2y ago

What does "world" mean here? How does the spatiality fit into some latent space? Or what constitutes the "world"? If the answer is, there is none, the world is just frames of video and any consistency quickly blurs out after a few seconds. That's not a world generation, that's just generation of video frames following frames. Not that it isn't cool, but it has almost zero usability for generating a "world" simulation. The key to a realistic world is that you can reliably navigate it. Visit and revisit places. If you modify anything, those modifications are persisted. If you leave a room and re-enter it hours later, the base expectation is that the same objects are in that room. Wouldn't a working approach be to just create a really low resolution 3D world in the traditional "3D game world" sense to get the spatial consistency. Then this crude map with attributes is fed into frame generation to create the resulting world? It wouldn't be infinite, but on the other hand no one has a need for an infinite world either. A spherical world solves the border issue pretty handily. As I understood it, there was some element of that in the new FS2024 (discussed yesterday on HN).

QuantumG 2y ago
What you really want is a story telling engine informing the world model and the humans need to generate power.
scotty79 2y ago
It's basically a dream. Maybe that's what dreams are? World prediction engine running with no sensor input and memory in read only mode?
- withinboredom 2y ago
  When you go to sleep, your brain stem disconnects from your body and your brain enters a feedback loop. The sensors are very much connected, just to whatever. Hence why you can grow dragon wings in your dreams and feel them. Memories can be made as well. I used to be really into lucid dreaming and time compression. My longest dream was nine years compressed into a 12-hour sleep period.
  - 2y ago

abricq 2y ago

This is surely really cool. Just a bit sad that, as phrased by the authors, the "First Real-Time" virtual world created for the demo is a fat & fast SUV driving on virgin lands.

spuz 2y ago
Why is that sad?
- ferguu_ 2y ago
  When you think of "The Matrix" - An infinitely generated and extremely complex world, I don't think most people would picture this, and especially not in a car like that. The car choice makes sense when you consider there aren't any roads at all! Of course, it is a very new technology, but then that does bring into question the paper author's frankly ambitious title choice. Interestingly, this website does remind me of something I played a while back - check it out! [https://slowroads.io/ https://slowroads.io/]. It has roads!
  - spuz 2y ago
    What would you expect it to look like based on the title? (I don't understand the title and have no expectations)
    - ferguu_ 2y ago
      Well, the Matrix movie showed a perfect depiction of the world stuck in 1999, with people given freedom to do as they wish (or what they thought was freedom). So... I guess more than just driving on barren plains haha.
pmayrgundter 2y ago
Sigh, agreed!
deleted 2y ago
[deleted]

akutlay 2y ago

Why would you want to generate all the pixels using this model instead of generating all the art, physics, and objects in the world using a game engine? The engine does so much of the physics and keeps everything stable for very cheap.

01HNNWZ0MV43FF 2y ago
Because you wouldn't have to wait for a human to write the engine and write the game
- drusepth 2y ago
  We generate 2d art, 3d models, some other in-world data, and soon animations at our game studio. The huge problems of generating pure pixels without a persistent game state behind it is illustrated well by the recent MinecraftAI craze (where it's a cool demo, but absolutely unplayable for anything other than novelty), and exacerbated on the dev side by missing out on tons of "free" stuff you get from existing engines that already do almost every kind of physics ops for you.
FeepingCreature 2y ago
Generating coherent objects in an infinite world is probably harder.
deleted 2y ago
[deleted]

wavemode 2y ago

I wish researchers would spend more time on using generative models to create level geometry, rather than trying to generate video from scratch. It would be both cheaper and more effective for stable gameplay.

jebarker 2y ago
There's so many researchers working in AI now that we can afford to explore all avenues. This probably isn't going to lead to a fully generative AI game engine, but I bet there'll be useful learnings along the way.
kridsdale1 2y ago
I think Ubisoft are leaders in that space. I saw algorithmic generation of New York style geometry for some behind the scenes video for The Division 2. I wouldn’t be surprised if epic has similar.

dangsux 2y ago

[dead]

pedalpete 2y ago

This is the future I was trying to pitch in 2018 when we had built Ayvri and had every paraglider in the world, the world's largest ultramarathons, drone operators, and lots of other users of our real-world 3D environment. Though we were using map tiles at the time, we were developing a model that took photos and a GPS track to add information that better matched environmental conditions (cloud, better lighting, etc). People still ask me to open-source or give them our source code, but the code was acquired, so that isn't possible. But I do regularly say that if I were to rebuild Ayvri today, I'd do it as an interactive video rather than loading tiles.

darepublic 2y ago

I didn't fully grok what this was about from the website. Though just last night I was talking to a friend about that quote from the Matrix that Morpheus tells Neo, so some nice synchronicity there. The sense I got from this is that they are developing a triple AAA type virtual world that can get generated on the fly based on text prompts? When the authors say frame level control do they mean that at any point, the next frame can be manipulated, either to be completely new or to influence the current story or context that is being played out?

m3kw9 2y ago

Definitely used Cyberpunk2077 footage to train

petermcneeley 2y ago

"Welcome to the Matrix" with matrix-like rain seems like an invitation for Warner Bros to sue you into oblivion.

airbreather 2y ago

No source, no playable demo, just promises of. Could be total vaporware for all we know. It is an ad, a statement of achievement in case someone else states it first, or what? Seems like it would be better on Youtube, it really doesn't offer much of use right now.

deleted 2y ago

[deleted]