3 ms·

Where it gets really interesting is if we can train a model on the latest GTA, plus maybe related real life footage, and then use it to live upgrade the visuals

Where it gets really interesting is if we can train a model on the latest GTA, plus maybe related real life footage, and then use it to live upgrade the visuals of an old game like Vice City. The lack of temporal consistency will still make it feel pretty dreamlike, but it won't matter that much, because the base is consistent and it will look amazing.

davedx 2y ago
A game like GTA has way too much functionality and complex branching for this to work I think (beyond eg doing aimless drives around the city — which would be very cool though)
- DrSiemer 2y ago
  Gta 5 has everything Vice City has and more. In the Doom AI dream it's possible to shoot people. Maybe in this CS model as well? I think the model does not have to know anything about the functionality. It can just dream up what is most probable to happen based on the training data.
InsideOutSanta 2y ago
Just redrawing images drawn by an existing game engine works, and generates amazing results, although like you point out, temporal consistency is not great. It might interpret the low-res green pixels on a far-away mountain as fruit trees in one frame, and as pines in the next. Here's a demo from 2021 doing something like that: https://www.youtube.com/watch?v=3rYosbwXm1w https://www.youtube.com/watch?v=3rYosbwXm1w
sorenjan 2y ago
In addition to the sibling comment's older example there's new work done with GTA too. https://www.reddit.com/r/aivideo/comments/1fx6zdr/gta_iv_with_a_photorealistic_filter_with_runway/ https://www.reddit.com/r/aivideo/comments/1fx6zdr/gta_iv_wit...
- DrSiemer 2y ago
  Cool! Looks fairly consistent as well. I wonder if this type of AI upscaling could eventually also fix things like slightly janky animations, but I guess that would be pretty hard without predetermined input and some form of look ahead. Limiting character motion to only allow correct, natural movement would introduce a strange kind of input lag.
  - sorenjan 2y ago
    Nvidia has done work in AI driven character animation too. https://www.nvidia.com/en-us/on-demand/session/gtcspring22-d4107/ https://www.nvidia.com/en-us/on-demand/session/gtcspring22-d...
skydhash 2y ago
Why not just creating the assets with higher resolution?
- DrSiemer 2y ago
  Because that is a lot more work, will only work for a single game, potentially requires more resources to run and will not get you the same level of realism.
  - skydhash 2y ago
    Based on what I know about 3D games, I can not imagine that being more efficient than current methods. Especially if you take the 3D world data into account.
taneq 2y ago
Using it as a visual upgrade is pretty close to what DLSS does so that sounds plausible.
empath75 2y ago
People focusing on the use of this in video games baffles me. The point isn't that it can regenerate a videogame world, the point is that it can simulate the _real world_. They're using video game footage to train it because it's cheap and easy to synthesize the data they need. This system doesn't know it's simulating a game. You can give it thousands or millions of hours of real world footage and agent input and get a simulation of the real world.
- Tostino 2y ago
  Reading this thread has been extremely frustrating because so many commenters seem to be stuck on "why do this when we already have a CS:GO simulation at home?". You hit the nail on the head.

davedx 2y ago

A game like GTA has way too much functionality and complex branching for this to work I think (beyond eg doing aimless drives around the city — which would be very cool though)

DrSiemer 2y ago
Gta 5 has everything Vice City has and more. In the Doom AI dream it's possible to shoot people. Maybe in this CS model as well? I think the model does not have to know anything about the functionality. It can just dream up what is most probable to happen based on the training data.

InsideOutSanta 2y ago

Just redrawing images drawn by an existing game engine works, and generates amazing results, although like you point out, temporal consistency is not great. It might interpret the low-res green pixels on a far-away mountain as fruit trees in one frame, and as pines in the next. Here's a demo from 2021 doing something like that: https://www.youtube.com/watch?v=3rYosbwXm1w https://www.youtube.com/watch?v=3rYosbwXm1w

sorenjan 2y ago

In addition to the sibling comment's older example there's new work done with GTA too. https://www.reddit.com/r/aivideo/comments/1fx6zdr/gta_iv_with_a_photorealistic_filter_with_runway/ https://www.reddit.com/r/aivideo/comments/1fx6zdr/gta_iv_wit...

DrSiemer 2y ago
Cool! Looks fairly consistent as well. I wonder if this type of AI upscaling could eventually also fix things like slightly janky animations, but I guess that would be pretty hard without predetermined input and some form of look ahead. Limiting character motion to only allow correct, natural movement would introduce a strange kind of input lag.
- sorenjan 2y ago
  Nvidia has done work in AI driven character animation too. https://www.nvidia.com/en-us/on-demand/session/gtcspring22-d4107/ https://www.nvidia.com/en-us/on-demand/session/gtcspring22-d...

skydhash 2y ago

Why not just creating the assets with higher resolution?

DrSiemer 2y ago
Because that is a lot more work, will only work for a single game, potentially requires more resources to run and will not get you the same level of realism.
- skydhash 2y ago
  Based on what I know about 3D games, I can not imagine that being more efficient than current methods. Especially if you take the 3D world data into account.

taneq 2y ago

Using it as a visual upgrade is pretty close to what DLSS does so that sounds plausible.

empath75 2y ago

People focusing on the use of this in video games baffles me. The point isn't that it can regenerate a videogame world, the point is that it can simulate the _real world_. They're using video game footage to train it because it's cheap and easy to synthesize the data they need. This system doesn't know it's simulating a game. You can give it thousands or millions of hours of real world footage and agent input and get a simulation of the real world.

Tostino 2y ago
Reading this thread has been extremely frustrating because so many commenters seem to be stuck on "why do this when we already have a CS:GO simulation at home?". You hit the nail on the head.