12 ms·

GAN Theft Auto [video]

388 points by ALittleLight 5y ago

dividuum 5y ago
Impressive. Makes you wonder if at some point in the future there isn't a game engine any more but tons of training material and you play in a generated dream.
- jsiepkes 5y ago
  Certainly impressive. And sure, maybe in a distant future. Though I think this is like one of those things where creating a working prototype that is 75% complete is the "easy" part. The other 25% (which you need for an actual working product) will take forever. Like self driving cards, nuclear fusion, etc.
  - viraptor 5y ago
    Text dungeon was very much a 75% complete thing, yet it's greatly entertaining in its own right. I would happily play a dream game which just falls apart sometimes.
- slver 5y ago
  Maybe not entirely, because just like a dream, the rules of a neural network tend to drift and be somewhat fuzzy. Unless it's a high-concept game whose very goal is offering you a dream environment. But I do believe neural networks will get into everything. They're the last missing piece of our compute model.
Randomoneh 5y ago
I fail to see novelty here. What's the size difference between the model and and all of the 64x32 image training data? If the difference is not significant, you're basically almost just scrubbing a video, right?
- sentdex 5y ago
  The GAN model is the game environment. You're playing a neural network. The novelty is no game engine, no rules, just learned how to represent the game and you can play it.
  - rasz 5y ago
    What he meant is you overfitted the network with video footage. There is no game, just seemingly clever stitching and playback of learned footage similar concept applied to animations and implemented in a state machine https://www.youtube.com/watch?v=KSTn3ePDt50 https://www.youtube.com/watch?v=KSTn3ePDt50 and optimized with nn https://www.youtube.com/watch?v=16CHDQK4W5k https://www.youtube.com/watch?v=16CHDQK4W5k
    - ShamelessC 5y ago
      The first link provided seems to need a very detailed human-provided cost function for specific development needs. The second one is indeed interesting research and seems to be a combination of the prior learned motion mapping working in tandem with a generative model. I suppose you could say that the automation of the dataset is considered as "augmentation"; but the difference here is that the dataset is just pixels and inputs rather than all that animation info and simulation data. Yes, a simulation is running; but the GAN only gets the pixels and the input. There's a similarity there though; you're right. In either case; the explicit goal of the video you posted is to combat runtime constraints of generative models. I'm not certain it's a fair comparison. The latter video and sentdex's result both seem to generalize to unique scenarios not present in the training set. This may mean they are creating an efficient representation of the underlying data in order to predict future samples more easily than simply overfitting. The top level comment here is a shallow dismissal and Randomoneh could have answered these questions themselves before throwing out a smug comment like "I fail to see novelty here" when it's at the very least the first large-scale GAN successfully trained on GTA V.
      - rasz 5y ago
        The first link exposes the trick employed by your model. >animation info and simulation data but did your model learn any of that? >explicit goal of the video you posted is to combat runtime constraints The trick to motion mapping is feeding a lot of data with accompanying inputs to build an atlas you can reference during playback. >first large-scale GAN successfully trained on GTA V Its really cool. The problem I had is in the presentation. I immediately felt insincerity bordering on scamming the audience, because I assume someone working in this field would know how the sausage is made. From the YT clip: "the shadow and reflection works", "modeling of physics works". Do they? or did your model build an atlas of video frames it can play back according to the fed input? Im guessing weather/time of day was locked when recording training data - perfect shadow and constant sun position for a nice reflection. Searching for 1:1 matches of generated output in the training set would be interesting and pretty revealing.
senkora 5y ago
Someone did a similar project with the exact same name for an ML art project at CMU a few years ago. https://m.youtube.com/watch?v=eP5hHKne_gE&feature=youtu.be https://m.youtube.com/watch?v=eP5hHKne_gE&feature=youtu.be Full list of projects: https://sites.google.com/site/artml2018/showcase/final-project https://sites.google.com/site/artml2018/showcase/final-proje...
- sentdex 5y ago
  Jeez, scared me. Same name yep, totally different project. That project is pix2pix. That is not a GAN-based game engine that you play within.
  - senkora 5y ago
    Oh, yeah, definitely a different thing but kinda neat that the name has occurred twice. Sorry for the scare!
  - godelski 5y ago
    Honestly "GAN Theft Auto" is the obvious choice for the name of the project (your project).
ShamelessC 5y ago
Great work! Hacker News still seems to have a deeply skeptical culture with regard to machine learning - not sure why. There's always someone saying it's "not novel" and it's "just doing x". Overfitting is a known issues in machine learning, people. If you still think all neural networks are doing is memorizing the dataset completely in the year 2021 - you might want to revisit the topic. It is one of the first concerns anyone training a deep model will have and to assume this model is overfit _without_ providing specific examples is arguing in bad faith. Sentdex has shown his GAN is able to generalize various game logic like collision/friction with vehicles and also learns aspects of rendering such as a proper reflection of the sun on the back of the car. He also showed weak points where the model is incapable of handling some situations and even did the impossible task of "splitting a car in two" to try and solve a head-on collision. Even though this is a failure case; it should at least provide you with some intuition that the GAN isn't just spitting out frames memorized from the dataset because that never happens in the dataset. You will need to apply a little more rigor before outright dismissing these weights as merely overfit. @sentdex Have you considered a guided diffusion approach now that that's all the rage? It's all rather new still but I believe it could be applied to these concepts as well.
- rasz 5y ago
  One of the main problems with ML/NN is it often works like magic, aka the trick works as long as audience doesnt know the secret behind it. Its fascinating to gullible audience, mundane bordering on boring to practitioners. My Tiger repelling rock^^^^^^leopard detection model works great on all animal pictures ... until you feed it a sofa https://web.archive.org/web/20150703094328/http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.html https://web.archive.org/web/20150703094328/http://rocknrolln... >able to generalize various game logic like collision/friction with vehicles and also learns aspects of rendering such as a proper reflection of the sun on the back of the car id did none of that, what this model did is learn all the frames of video and their chronological order according to the input. > impossible task of "splitting a car in two" to try and solve a head-on collision. it played back both learned versions at once, like reporting confidence of round thing being 50% ball and 50% orange.
  - 5y ago
4dahalibut 5y ago
Hey sentdex this is absolutely awesome! Playing with exotic target types like generating games is IMO where the fun is in ml :) Do you see yourself taking this train of play further?
- sentdex 5y ago
  We'd like to try some further GTA stuff, as well as some IRL stuff. Have seen some recent IRL GAN stuff, and it looks super interesting. There's just something about AI-based environments that is particularly intriguing!
darepublic 5y ago
Cool stuff. Looking forward to more realistic NPCs and player decision driven stories in an open world sandbox
someperson 5y ago
It should be possible to take arbitrary video training data (whether from a game or real-life) and automatically reconstruct the 3D models of all vehicles in the scene (and the skybox) and "play back" the scene in a video game engine. This is the direction virtual and augmented reality is headed (Facebook Codec Avatar, and their room reconstruction technology).
emptyparadise 5y ago
One throwaway line about GAN operating systems now made me want to see a shell GAN. Keypresses as inputs, 80x24 terminal screens as outputs. Could a neural network dream of Unix?
- reasonabl_human 5y ago
  This exists via recent NLP models, I’ll see if I can dig up a link… Edit: https://www.reddit.com/r/linux/comments/mtnld7/programmer_created_an_aipowered_linux_shell_using/ https://www.reddit.com/r/linux/comments/mtnld7/programmer_cr...
  - emptyparadise 5y ago
    But this is converting natural language input to commands, right? It's not actually dreaming up the entire shell and the output.
  - riveducha 5y ago
    As the original creator of that video, it’s a little sad to see people download and then re-upload the entire video to Reddit. You can find the original video as well as written commentary on my web page: https://riveducha.onfabrica.com/openai-powered-linux-shell https://riveducha.onfabrica.com/openai-powered-linux-shell
    - jallbrit 5y ago
      Wow, what an incredible video and showcase. This really puts GPT-3's power into perspective. I can't wait till the public has access to something that powerful- or maybe I should enjoy not receiving GPT-3 phishing emails in my inbox.
- sentdex 5y ago
  I don't see why not. Might be something fun to try tbh.
  - toxik 5y ago
jwilber 5y ago
I like your YouTube videos in general and think this content is a great benefit to the community. I wouldn’t take the few negative comments personally - I’ve seen many GAN architectures that heavily overfit (including my own bobross pix2pix) get a lot of praise, while ‘less violating’ models (like yours) get more skepticism. Skepticism isn’t bad! But I’d wager in your case it may be because you’re a YouTuber, and other ml YouTubers are notorious for ripping off content (eg Siraj). Not really related to this, but I’d personally love to see the difference in training times it would take an RL agent to adequately learn to drive a car in gta versus adequately flying a helicopter.
- ALittleLight 5y ago
  Because you're using the word "your" I just feel the need to clarify that I didn't create this. I just saw it on YouTube and thought it was neat.
  - jwilber 5y ago
    Oh my bad! The dude who made it (sentdex) has replied a ton in this thread, I just assumed he was the op as well.
whalesalad 5y ago
GPUs: am I a joke to you? Instead of using them to render polygons, let’s use them to train neural networks that produce models that make them unnecessary. I’m oversimplifying - but pretty wild nonetheless.
- slver 5y ago
  Well, neural networks run (fastest) on GPUs.
- Sharlin 5y ago
  Wait, you mean that "GPU" doesn't mean "GAN Processing Unit"? ;)
nitrogen 5y ago
Something I'd like to see is a visualization of subsets of the network's internal state that correlate with simple quantities like compass direction, velocity, position, etc. It'd be really fascinating to see where in the model these things are being learned, whether they are concentrated in a small area or spread out, and whether this is somewhat consistent across different iterations of the model.
- ludwigschubert 5y ago
  Me too! In a much simpler setting a former colleague of mine, Jacob Hilton, tried such an exploration for the vision part of a OpenAI CoinRun model. It’s the first part of this paper: https://distill.pub/2020/understanding-rl-vision/ https://distill.pub/2020/understanding-rl-vision/
philipswood 5y ago
The GitHub repo is here: https://github.com/sentdex/GANTheftAuto/ https://github.com/sentdex/GANTheftAuto/
junon 5y ago
This is incredible. Took me a minute to realize this isn't an image transform of some kind. Really well done.
black_puppydog 5y ago
Woah dude! sudo python3 inference.py? Really? :D
deleted 5y ago
[deleted]
okamiueru 5y ago
Can someone explain a bit more on the long term applicability, or maybe other use cases that might be easier to appreciate? The reason why I ask is that it seems very challenging to generate the training data for such systems. Could someone explain how this can go further than to just replicating X? So, if assuming some creative freedom, could you give an idea of what the long term application of this would be? NB: please take my questions at face value without thinking I'm implying this isn't cool for what it is. I'm all for people having fun. I'm all for projects not needing to tackle some grander issue.
- tiborsaas 5y ago
  In the future we might have a fourth common media format besides pictures, videos and audio: GAN records.
  - okamiueru 5y ago
    That is an interesting thought. I don't fully understand how though. The main challenge is the training data. If you need to first create the interactive experience... What would the added value be?
    - mdale 5y ago
      https://openai.com/blog/dall-e/ https://openai.com/blog/dall-e/ For example is creating novel combination based on (large) training set. If the network had enough weights on what is realistic looking could create novel game experiences based on a prompt of say a film or book.
TinkersW 5y ago
Looks interesting, if very far from practical-- too bad it requires a "DGX" station to train It seems to flicker/fade things in alot, like the random poles that keep appearing and disappearing, it seems like there is not enough focus on temporal consistency or something?
- bruce343434 5y ago
  If you see the source output image before it was upscaled you noticed the resolution is too low and thin objects "fall between" the pixels. The upscaler then interprets it as air, it seems.
boyadjian 5y ago
WOW, this is awesome. The video gives the impression we are dreaming
HerrmannM 5y ago
Great work! I'm curious about what could be achieved in this space in the future. I'm curious about why you cannot share the GTA5 mod and collection script? I'm curious about that part too -- obtaining good data is always hard. Cheers and all the best!

388 points by ALittleLight 5y ago

Impressive. Makes you wonder if at some point in the future there isn't a game engine any more but tons of training material and you play in a generated dream.

jsiepkes 5y ago
Certainly impressive. And sure, maybe in a distant future. Though I think this is like one of those things where creating a working prototype that is 75% complete is the "easy" part. The other 25% (which you need for an actual working product) will take forever. Like self driving cards, nuclear fusion, etc.
- viraptor 5y ago
  Text dungeon was very much a 75% complete thing, yet it's greatly entertaining in its own right. I would happily play a dream game which just falls apart sometimes.
slver 5y ago
Maybe not entirely, because just like a dream, the rules of a neural network tend to drift and be somewhat fuzzy. Unless it's a high-concept game whose very goal is offering you a dream environment. But I do believe neural networks will get into everything. They're the last missing piece of our compute model.

Randomoneh 5y ago

I fail to see novelty here. What's the size difference between the model and and all of the 64x32 image training data? If the difference is not significant, you're basically almost just scrubbing a video, right?

sentdex 5y ago
The GAN model is the game environment. You're playing a neural network. The novelty is no game engine, no rules, just learned how to represent the game and you can play it.
- rasz 5y ago
  What he meant is you overfitted the network with video footage. There is no game, just seemingly clever stitching and playback of learned footage similar concept applied to animations and implemented in a state machine https://www.youtube.com/watch?v=KSTn3ePDt50 https://www.youtube.com/watch?v=KSTn3ePDt50 and optimized with nn https://www.youtube.com/watch?v=16CHDQK4W5k https://www.youtube.com/watch?v=16CHDQK4W5k
  - ShamelessC 5y ago
    The first link provided seems to need a very detailed human-provided cost function for specific development needs. The second one is indeed interesting research and seems to be a combination of the prior learned motion mapping working in tandem with a generative model. I suppose you could say that the automation of the dataset is considered as "augmentation"; but the difference here is that the dataset is just pixels and inputs rather than all that animation info and simulation data. Yes, a simulation is running; but the GAN only gets the pixels and the input. There's a similarity there though; you're right. In either case; the explicit goal of the video you posted is to combat runtime constraints of generative models. I'm not certain it's a fair comparison. The latter video and sentdex's result both seem to generalize to unique scenarios not present in the training set. This may mean they are creating an efficient representation of the underlying data in order to predict future samples more easily than simply overfitting. The top level comment here is a shallow dismissal and Randomoneh could have answered these questions themselves before throwing out a smug comment like "I fail to see novelty here" when it's at the very least the first large-scale GAN successfully trained on GTA V.
    - rasz 5y ago
      The first link exposes the trick employed by your model. >animation info and simulation data but did your model learn any of that? >explicit goal of the video you posted is to combat runtime constraints The trick to motion mapping is feeding a lot of data with accompanying inputs to build an atlas you can reference during playback. >first large-scale GAN successfully trained on GTA V Its really cool. The problem I had is in the presentation. I immediately felt insincerity bordering on scamming the audience, because I assume someone working in this field would know how the sausage is made. From the YT clip: "the shadow and reflection works", "modeling of physics works". Do they? or did your model build an atlas of video frames it can play back according to the fed input? Im guessing weather/time of day was locked when recording training data - perfect shadow and constant sun position for a nice reflection. Searching for 1:1 matches of generated output in the training set would be interesting and pretty revealing.

senkora 5y ago

Someone did a similar project with the exact same name for an ML art project at CMU a few years ago. https://m.youtube.com/watch?v=eP5hHKne_gE&feature=youtu.be https://m.youtube.com/watch?v=eP5hHKne_gE&feature=youtu.be Full list of projects: https://sites.google.com/site/artml2018/showcase/final-project https://sites.google.com/site/artml2018/showcase/final-proje...

sentdex 5y ago
Jeez, scared me. Same name yep, totally different project. That project is pix2pix. That is not a GAN-based game engine that you play within.
- senkora 5y ago
  Oh, yeah, definitely a different thing but kinda neat that the name has occurred twice. Sorry for the scare!
- godelski 5y ago
  Honestly "GAN Theft Auto" is the obvious choice for the name of the project (your project).

ShamelessC 5y ago

Great work! Hacker News still seems to have a deeply skeptical culture with regard to machine learning - not sure why. There's always someone saying it's "not novel" and it's "just doing x". Overfitting is a known issues in machine learning, people. If you still think all neural networks are doing is memorizing the dataset completely in the year 2021 - you might want to revisit the topic. It is one of the first concerns anyone training a deep model will have and to assume this model is overfit _without_ providing specific examples is arguing in bad faith. Sentdex has shown his GAN is able to generalize various game logic like collision/friction with vehicles and also learns aspects of rendering such as a proper reflection of the sun on the back of the car. He also showed weak points where the model is incapable of handling some situations and even did the impossible task of "splitting a car in two" to try and solve a head-on collision. Even though this is a failure case; it should at least provide you with some intuition that the GAN isn't just spitting out frames memorized from the dataset because that never happens in the dataset. You will need to apply a little more rigor before outright dismissing these weights as merely overfit. @sentdex Have you considered a guided diffusion approach now that that's all the rage? It's all rather new still but I believe it could be applied to these concepts as well.

rasz 5y ago
One of the main problems with ML/NN is it often works like magic, aka the trick works as long as audience doesnt know the secret behind it. Its fascinating to gullible audience, mundane bordering on boring to practitioners. My Tiger repelling rock^^^^^^leopard detection model works great on all animal pictures ... until you feed it a sofa https://web.archive.org/web/20150703094328/http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.html https://web.archive.org/web/20150703094328/http://rocknrolln... >able to generalize various game logic like collision/friction with vehicles and also learns aspects of rendering such as a proper reflection of the sun on the back of the car id did none of that, what this model did is learn all the frames of video and their chronological order according to the input. > impossible task of "splitting a car in two" to try and solve a head-on collision. it played back both learned versions at once, like reporting confidence of round thing being 50% ball and 50% orange.
- 5y ago

4dahalibut 5y ago

Hey sentdex this is absolutely awesome! Playing with exotic target types like generating games is IMO where the fun is in ml :) Do you see yourself taking this train of play further?

sentdex 5y ago
We'd like to try some further GTA stuff, as well as some IRL stuff. Have seen some recent IRL GAN stuff, and it looks super interesting. There's just something about AI-based environments that is particularly intriguing!

darepublic 5y ago

Cool stuff. Looking forward to more realistic NPCs and player decision driven stories in an open world sandbox

someperson 5y ago

It should be possible to take arbitrary video training data (whether from a game or real-life) and automatically reconstruct the 3D models of all vehicles in the scene (and the skybox) and "play back" the scene in a video game engine. This is the direction virtual and augmented reality is headed (Facebook Codec Avatar, and their room reconstruction technology).

emptyparadise 5y ago

One throwaway line about GAN operating systems now made me want to see a shell GAN. Keypresses as inputs, 80x24 terminal screens as outputs. Could a neural network dream of Unix?

reasonabl_human 5y ago
This exists via recent NLP models, I’ll see if I can dig up a link… Edit: https://www.reddit.com/r/linux/comments/mtnld7/programmer_created_an_aipowered_linux_shell_using/ https://www.reddit.com/r/linux/comments/mtnld7/programmer_cr...
- emptyparadise 5y ago
  But this is converting natural language input to commands, right? It's not actually dreaming up the entire shell and the output.
- riveducha 5y ago
  As the original creator of that video, it’s a little sad to see people download and then re-upload the entire video to Reddit. You can find the original video as well as written commentary on my web page: https://riveducha.onfabrica.com/openai-powered-linux-shell https://riveducha.onfabrica.com/openai-powered-linux-shell
  - jallbrit 5y ago
    Wow, what an incredible video and showcase. This really puts GPT-3's power into perspective. I can't wait till the public has access to something that powerful- or maybe I should enjoy not receiving GPT-3 phishing emails in my inbox.
sentdex 5y ago
I don't see why not. Might be something fun to try tbh.
- toxik 5y ago

jwilber 5y ago

I like your YouTube videos in general and think this content is a great benefit to the community. I wouldn’t take the few negative comments personally - I’ve seen many GAN architectures that heavily overfit (including my own bobross pix2pix) get a lot of praise, while ‘less violating’ models (like yours) get more skepticism. Skepticism isn’t bad! But I’d wager in your case it may be because you’re a YouTuber, and other ml YouTubers are notorious for ripping off content (eg Siraj). Not really related to this, but I’d personally love to see the difference in training times it would take an RL agent to adequately learn to drive a car in gta versus adequately flying a helicopter.

ALittleLight 5y ago
Because you're using the word "your" I just feel the need to clarify that I didn't create this. I just saw it on YouTube and thought it was neat.
- jwilber 5y ago
  Oh my bad! The dude who made it (sentdex) has replied a ton in this thread, I just assumed he was the op as well.

whalesalad 5y ago

GPUs: am I a joke to you? Instead of using them to render polygons, let’s use them to train neural networks that produce models that make them unnecessary. I’m oversimplifying - but pretty wild nonetheless.

slver 5y ago
Well, neural networks run (fastest) on GPUs.
Sharlin 5y ago
Wait, you mean that "GPU" doesn't mean "GAN Processing Unit"? ;)

nitrogen 5y ago

Something I'd like to see is a visualization of subsets of the network's internal state that correlate with simple quantities like compass direction, velocity, position, etc. It'd be really fascinating to see where in the model these things are being learned, whether they are concentrated in a small area or spread out, and whether this is somewhat consistent across different iterations of the model.

ludwigschubert 5y ago
Me too! In a much simpler setting a former colleague of mine, Jacob Hilton, tried such an exploration for the vision part of a OpenAI CoinRun model. It’s the first part of this paper: https://distill.pub/2020/understanding-rl-vision/ https://distill.pub/2020/understanding-rl-vision/

philipswood 5y ago

The GitHub repo is here: https://github.com/sentdex/GANTheftAuto/ https://github.com/sentdex/GANTheftAuto/

junon 5y ago

This is incredible. Took me a minute to realize this isn't an image transform of some kind. Really well done.

black_puppydog 5y ago

Woah dude! sudo python3 inference.py? Really? :D

deleted 5y ago

[deleted]

okamiueru 5y ago

Can someone explain a bit more on the long term applicability, or maybe other use cases that might be easier to appreciate? The reason why I ask is that it seems very challenging to generate the training data for such systems. Could someone explain how this can go further than to just replicating X? So, if assuming some creative freedom, could you give an idea of what the long term application of this would be? NB: please take my questions at face value without thinking I'm implying this isn't cool for what it is. I'm all for people having fun. I'm all for projects not needing to tackle some grander issue.

tiborsaas 5y ago
In the future we might have a fourth common media format besides pictures, videos and audio: GAN records.
- okamiueru 5y ago
  That is an interesting thought. I don't fully understand how though. The main challenge is the training data. If you need to first create the interactive experience... What would the added value be?
  - mdale 5y ago
    https://openai.com/blog/dall-e/ https://openai.com/blog/dall-e/ For example is creating novel combination based on (large) training set. If the network had enough weights on what is realistic looking could create novel game experiences based on a prompt of say a film or book.

TinkersW 5y ago

Looks interesting, if very far from practical-- too bad it requires a "DGX" station to train It seems to flicker/fade things in alot, like the random poles that keep appearing and disappearing, it seems like there is not enough focus on temporal consistency or something?

bruce343434 5y ago
If you see the source output image before it was upscaled you noticed the resolution is too low and thin objects "fall between" the pixels. The upscaler then interprets it as air, it seems.

boyadjian 5y ago

WOW, this is awesome. The video gives the impression we are dreaming

HerrmannM 5y ago

Great work! I'm curious about what could be achieved in this space in the future. I'm curious about why you cannot share the GTA5 mod and collection script? I'm curious about that part too -- obtaining good data is always hard. Cheers and all the best!