4 ms·

I earnestly think this is where all gaming will go in the next five years - it’s going to be so compelling that stuff already under development will likely see

I earnestly think this is where all gaming will go in the next five years - it’s going to be so compelling that stuff already under development will likely see a shift to using diffusion models. As this is demonstrating, a sufficiently honed model can produce realtime graphics - and some of the demos floating around where people are running GTA San Andreas through non-realtime models hint as to where this will go. I give it the same five years before there are games entirely indistinguishable from reality, and I don’t just mean graphical fidelity - there’s no reason that the same or another model couldn’t provide limitless physics - bust a hole through that wall, set fire to this refrigerator, whatever.

advael 2y ago
I'm not sure that's a warranted assumption based on this result, exciting as it is, we are still seeing replication of an extant testable world model, rather than extrapolation that can produce novel mechanics without them being in the training data. I'm not saying this isn't a stepping stone to that, I just think your prediction's a little optimistic based on the scope of that problem
casenmgreen 2y ago
Not a chance. There are fundamental limitations with what are in the end all essentially neural nets; there is no understanding, only prediction. Prediction alone is not enough to emulate reality, which is why for example genuinely self-driving cars have not, and will not, emerge. A fundamental advance in AI technology will be required for that, something which leads to genuine intelligence, and we are no closer to that than ever we were.
- jiggawatts 2y ago
  For simulations like games, it's a trivial matter to feed the neural game engine pixel-perfect metadata. Instead of rendering the final shaded and textured pixels, the engine would output just the material IDs, motion vectors, and similar "meta" data that would normally be the inputs into a real-time shader. The AI can use this as inputs to render a photorealistic output. It can be trained using offline-rendered "ground-truth" raytraced scenes. Potentially, video labelled in a similar way could be used to give it a flair of realism. This is already what NVIDIA DLSS and similar AI upscaling tech uses. The obvious next step is not just to upscale rendered scenes, but to do the rendering itself.
- fancyfredbot 2y ago
  Looking at the examples of 2600 games in the paper I'm not sure you can tell that they are just predictions. Have you considered how you'd tell the difference between a prediction and understanding in practice?
- francoisfleuret 2y ago
  "there is no understanding, only prediction" I have no idea what this means.
  - tourmalinetaco 2y ago
    The MLM has no idea what it’s making, where you are in the map, what you left behind, and what you picked up. It can accurately predict what comes next, but if you pick up an item and do a 360° turn the item will be back and you can repeat the process.
  - GaggiX 2y ago
    When a human does it, it's understanding, when an AI does it, it's prediction, I thinks it's very clear /s
    - therouwboat 2y ago
      Does what? In normal game world things tend to stay where they are without player having to do anything.
      - GaggiX 2y ago
        We are talking about neural networks in general, not this one or that one, if you train a bad model or the model is untrained it would not indeed understand much or anything.
  - nonrandomstring 2y ago
    > > "there is no understanding, only prediction" > I have no idea what this means. You can throw a ball up in the air and predict that it will fall again and bounce. You have no understanding of mass, gravity, acceleration, momentum, impulse, elasticity... You can press a button that makes an Uber car appear in reality and take you home. You have no understanding of apps, operating systems, radio, internet, roads, wheels, internal combustion engines, driving, GPS, maps... This confusion of understanding and prediction affects a lot of people who use technology in a "machine-like" way, purely instrumental and utilitarian... "how does this get me what I want immediately?" You can take any complex reality and deflate it, abstract it, reduce it down to a mere set of predictions that preserve all the utility for a narrow task (in this case visual facsimile) but strip away all depth of meaning. The models, of both the system and the internal working model of the user are flattened. In this sense "AI" is probably the greatest assault on actual knowledge since the book burning under totalitarian regimes of the mid 20th century.
- killerstorm 2y ago
  That's bs. You have no understanding of understanding. Hooke's law was pure curve-fitting. Hooke definitely did not understand the "why". And yet we don't consider that bad physics. Newton's laws can be derived from curve fitting. How is that different from "understanding"?
  - madaxe_again 2y ago
    Einstein couldn’t even explain why general relativity occurred. Sure, spacetime is curved by mass, but why? What a loser.
    - killerstorm 2y ago
      It's very illustrative to look into the history of discovery of laws of motion, as it's quite well documented. People have an intuitive understanding of motion - we see it literally every day, we throw objects, etc. And yet it took literally thousands of years since discovery of mathematics (geometry, etc.) to formulate a concept of force, momentum, etc. Ancient Greek mathematicians could do integration, so they were not lacking mathematical sophistication. And yet their understanding of motion was so primitive: Aristotle, an extremely smart man, was muttering something about "violent" and "natural" motion: https://en.wikipedia.org/wiki/Newton%27s_laws_of_motion#Antiquity_and_medieval_background https://en.wikipedia.org/wiki/Newton%27s_laws_of_motion#Anti... People started to understand the conservation of quantity of motion only in 17th century. So we have two possibilities: * everyone until 17th century was dumb af (despite being able to do quite impressive calculations) * scientific discovery is really a heuristic-driven search process where people try various things until they find a good fit I.e. millions of people were somehow failing to understand motion for literally thousands of years until they collected enough assertions about motion that they were able to formulate the rule of conservation, test it, and confirm it fits. And only then it became understanding. You can literally see conservation of momentum on a billiard table: you "violently" hit one ball, it hits other balls and they start to move, but slower, etc. So you really transfer something from one ball to the rest. And yet people could not see it for thousands of years. What this shows is that there's nothing fundamental about understanding: it's just a sense of familiarity, it is a sense that your model fits well. Under the hood it's all prediction and curve fitting. We literally have prediction hardware in our brains: cerebellum has specialized cells which can predict, e.g. motion. So people with damaged cerebellum have impaired movement: they still can move, but their movement are not precise. When do you think we find specialized understanding cells in the human brain?
- madaxe_again 2y ago
  Yet we have no understanding, only prediction. We can describe a great many things in detail, how they interact - and we can claim to understand things, yet if you recursively ask “why?” everybody, and I mean everybody, will reach a point where they say “I don’t know” or “god”. An incomplete understanding is no understanding at all, and I would argue that we can only predict, and we can certainly emulate reality, otherwise we would not be able to function within it. A toddler can emulate reality, anticipate causality - and they certainly can’t be said to be in possession of a robust grand unified theory.
- esafak 2y ago
  Ilya Sutskever, among others, believe that better prediction is achieved through understanding, and that models can be learn to understand. As we have. We are not born understanding everything.
TinkersW 2y ago
It requires a monster GPU to run 10 fps at what looks like sub 720p... I think it may be abit more than 5 years..
viraptor 2y ago
It's not that great yet. Given a model which can generate the game view in ~real time and a model which can generate the models and textures, why would you ever use the first option, apart from a cool tech demo? I'm sure there's space for new dreamy games where invisible space behind you transforms when you turn around, but for other genres... why? Destructible environment has been possible for quite a while, but once you allow that everywhere, you can get games into unplayable state. They need to be designed around that mechanic to work well: Noita, Worms, Teardown, etc. I don't believe the "limitless physics" would matter after a few minutes.
qayxc 2y ago
I think you're missing the most important point: these models need to be trained on something and that something is a fully developed, working game. You're basically saying that game development would need to do the work twice: step 1: develop a fully functional game, step 2: spend ridiculous effort (in terms of time and compute) on training a model to emulate the game in a half-baked fashion. It's a solution looking for a problem.
- manmal 2y ago
  The world model can still be rendered in very low res, and then the diffusion skin/remaster is applied. And this would also be an exciting route to go at remastering old games. I‘d pay a lot to play NFS Porsche again, with photorealism. Or imagine Command & Conquer Red Alert, „rendered“ with such a model.
  - qayxc 2y ago
    NVIDIA's RTX Remix [1] suite of tools already does that. It doesn't require any model training or dozens of hours of pre-recorded gameplay either. You can drop in low-res textures and have AI tools upscale them. Models can be replaced, as well as lighting and the best part: it's all under your control. You're not at the merci of obscure training material that might or might not result in a consistent look-and-feel. More knobs, more control, less compute required. [1] https://www.nvidia.com/en-us/geforce/rtx-remix/ https://www.nvidia.com/en-us/geforce/rtx-remix/
    - manmal 2y ago
      TIL, thanks for posting. The workflow I was sketching out is simpler though: Render a legacy game or low fidelity modern game as-is, and run it through a diffusion model in real time.
- FeepingCreature 2y ago
  You can crosstrain on reality.
Arch485 2y ago
It seems extremely unlikely to me that ML models will ever run entire games. Nobody wants a game that's "entirely indistinguishable from reality" anyways. If they did, they would go outside. I think it's possible specific engine components could be ML-driven in the future, like graphics or NPC interactions. This is already happening to a certain degree. Now, I don't think it's impossible for an ML model to run an entire game. I just don't think making + running your game in a predictive ML model will ever be more effective than making a game the normal way.
- jsheard 2y ago
  Yep, the fuzziness and opaqueness of ML models makes developing an entire game state inside one a non-starter in my opinion. You need precise rules, and you need to be able to iterate on those rules quickly, neither of which are feasible with our current understanding of ML models. Nobody wants a version of CS:GO where fundamental constants like weapon damage run on dream logic. If ML has any place in games it's for specific subsystems which don't need absolute precision, NPC behaviour, character animation, refining the output of a renderer, that kind of thing.

advael 2y ago

I'm not sure that's a warranted assumption based on this result, exciting as it is, we are still seeing replication of an extant testable world model, rather than extrapolation that can produce novel mechanics without them being in the training data. I'm not saying this isn't a stepping stone to that, I just think your prediction's a little optimistic based on the scope of that problem

casenmgreen 2y ago

Not a chance. There are fundamental limitations with what are in the end all essentially neural nets; there is no understanding, only prediction. Prediction alone is not enough to emulate reality, which is why for example genuinely self-driving cars have not, and will not, emerge. A fundamental advance in AI technology will be required for that, something which leads to genuine intelligence, and we are no closer to that than ever we were.

jiggawatts 2y ago
For simulations like games, it's a trivial matter to feed the neural game engine pixel-perfect metadata. Instead of rendering the final shaded and textured pixels, the engine would output just the material IDs, motion vectors, and similar "meta" data that would normally be the inputs into a real-time shader. The AI can use this as inputs to render a photorealistic output. It can be trained using offline-rendered "ground-truth" raytraced scenes. Potentially, video labelled in a similar way could be used to give it a flair of realism. This is already what NVIDIA DLSS and similar AI upscaling tech uses. The obvious next step is not just to upscale rendered scenes, but to do the rendering itself.
fancyfredbot 2y ago
Looking at the examples of 2600 games in the paper I'm not sure you can tell that they are just predictions. Have you considered how you'd tell the difference between a prediction and understanding in practice?
francoisfleuret 2y ago
"there is no understanding, only prediction" I have no idea what this means.
- tourmalinetaco 2y ago
  The MLM has no idea what it’s making, where you are in the map, what you left behind, and what you picked up. It can accurately predict what comes next, but if you pick up an item and do a 360° turn the item will be back and you can repeat the process.
- GaggiX 2y ago
  When a human does it, it's understanding, when an AI does it, it's prediction, I thinks it's very clear /s
  - therouwboat 2y ago
    Does what? In normal game world things tend to stay where they are without player having to do anything.
    - GaggiX 2y ago
      We are talking about neural networks in general, not this one or that one, if you train a bad model or the model is untrained it would not indeed understand much or anything.
- nonrandomstring 2y ago
  > > "there is no understanding, only prediction" > I have no idea what this means. You can throw a ball up in the air and predict that it will fall again and bounce. You have no understanding of mass, gravity, acceleration, momentum, impulse, elasticity... You can press a button that makes an Uber car appear in reality and take you home. You have no understanding of apps, operating systems, radio, internet, roads, wheels, internal combustion engines, driving, GPS, maps... This confusion of understanding and prediction affects a lot of people who use technology in a "machine-like" way, purely instrumental and utilitarian... "how does this get me what I want immediately?" You can take any complex reality and deflate it, abstract it, reduce it down to a mere set of predictions that preserve all the utility for a narrow task (in this case visual facsimile) but strip away all depth of meaning. The models, of both the system and the internal working model of the user are flattened. In this sense "AI" is probably the greatest assault on actual knowledge since the book burning under totalitarian regimes of the mid 20th century.
killerstorm 2y ago
That's bs. You have no understanding of understanding. Hooke's law was pure curve-fitting. Hooke definitely did not understand the "why". And yet we don't consider that bad physics. Newton's laws can be derived from curve fitting. How is that different from "understanding"?
- madaxe_again 2y ago
  Einstein couldn’t even explain why general relativity occurred. Sure, spacetime is curved by mass, but why? What a loser.
  - killerstorm 2y ago
    It's very illustrative to look into the history of discovery of laws of motion, as it's quite well documented. People have an intuitive understanding of motion - we see it literally every day, we throw objects, etc. And yet it took literally thousands of years since discovery of mathematics (geometry, etc.) to formulate a concept of force, momentum, etc. Ancient Greek mathematicians could do integration, so they were not lacking mathematical sophistication. And yet their understanding of motion was so primitive: Aristotle, an extremely smart man, was muttering something about "violent" and "natural" motion: https://en.wikipedia.org/wiki/Newton%27s_laws_of_motion#Antiquity_and_medieval_background https://en.wikipedia.org/wiki/Newton%27s_laws_of_motion#Anti... People started to understand the conservation of quantity of motion only in 17th century. So we have two possibilities: * everyone until 17th century was dumb af (despite being able to do quite impressive calculations) * scientific discovery is really a heuristic-driven search process where people try various things until they find a good fit I.e. millions of people were somehow failing to understand motion for literally thousands of years until they collected enough assertions about motion that they were able to formulate the rule of conservation, test it, and confirm it fits. And only then it became understanding. You can literally see conservation of momentum on a billiard table: you "violently" hit one ball, it hits other balls and they start to move, but slower, etc. So you really transfer something from one ball to the rest. And yet people could not see it for thousands of years. What this shows is that there's nothing fundamental about understanding: it's just a sense of familiarity, it is a sense that your model fits well. Under the hood it's all prediction and curve fitting. We literally have prediction hardware in our brains: cerebellum has specialized cells which can predict, e.g. motion. So people with damaged cerebellum have impaired movement: they still can move, but their movement are not precise. When do you think we find specialized understanding cells in the human brain?
madaxe_again 2y ago
Yet we have no understanding, only prediction. We can describe a great many things in detail, how they interact - and we can claim to understand things, yet if you recursively ask “why?” everybody, and I mean everybody, will reach a point where they say “I don’t know” or “god”. An incomplete understanding is no understanding at all, and I would argue that we can only predict, and we can certainly emulate reality, otherwise we would not be able to function within it. A toddler can emulate reality, anticipate causality - and they certainly can’t be said to be in possession of a robust grand unified theory.
esafak 2y ago
Ilya Sutskever, among others, believe that better prediction is achieved through understanding, and that models can be learn to understand. As we have. We are not born understanding everything.

TinkersW 2y ago

It requires a monster GPU to run 10 fps at what looks like sub 720p... I think it may be abit more than 5 years..

viraptor 2y ago

It's not that great yet. Given a model which can generate the game view in ~real time and a model which can generate the models and textures, why would you ever use the first option, apart from a cool tech demo? I'm sure there's space for new dreamy games where invisible space behind you transforms when you turn around, but for other genres... why? Destructible environment has been possible for quite a while, but once you allow that everywhere, you can get games into unplayable state. They need to be designed around that mechanic to work well: Noita, Worms, Teardown, etc. I don't believe the "limitless physics" would matter after a few minutes.

qayxc 2y ago

I think you're missing the most important point: these models need to be trained on something and that something is a fully developed, working game. You're basically saying that game development would need to do the work twice: step 1: develop a fully functional game, step 2: spend ridiculous effort (in terms of time and compute) on training a model to emulate the game in a half-baked fashion. It's a solution looking for a problem.

manmal 2y ago
The world model can still be rendered in very low res, and then the diffusion skin/remaster is applied. And this would also be an exciting route to go at remastering old games. I‘d pay a lot to play NFS Porsche again, with photorealism. Or imagine Command & Conquer Red Alert, „rendered“ with such a model.
- qayxc 2y ago
  NVIDIA's RTX Remix [1] suite of tools already does that. It doesn't require any model training or dozens of hours of pre-recorded gameplay either. You can drop in low-res textures and have AI tools upscale them. Models can be replaced, as well as lighting and the best part: it's all under your control. You're not at the merci of obscure training material that might or might not result in a consistent look-and-feel. More knobs, more control, less compute required. [1] https://www.nvidia.com/en-us/geforce/rtx-remix/ https://www.nvidia.com/en-us/geforce/rtx-remix/
  - manmal 2y ago
    TIL, thanks for posting. The workflow I was sketching out is simpler though: Render a legacy game or low fidelity modern game as-is, and run it through a diffusion model in real time.
FeepingCreature 2y ago
You can crosstrain on reality.

Arch485 2y ago

It seems extremely unlikely to me that ML models will ever run entire games. Nobody wants a game that's "entirely indistinguishable from reality" anyways. If they did, they would go outside. I think it's possible specific engine components could be ML-driven in the future, like graphics or NPC interactions. This is already happening to a certain degree. Now, I don't think it's impossible for an ML model to run an entire game. I just don't think making + running your game in a predictive ML model will ever be more effective than making a game the normal way.

jsheard 2y ago
Yep, the fuzziness and opaqueness of ML models makes developing an entire game state inside one a non-starter in my opinion. You need precise rules, and you need to be able to iterate on those rules quickly, neither of which are feasible with our current understanding of ML models. Nobody wants a version of CS:GO where fundamental constants like weapon damage run on dream logic. If ML has any place in games it's for specific subsystems which don't need absolute precision, NPC behaviour, character animation, refining the output of a renderer, that kind of thing.