9 ms·
Python plays Grand Theft Auto V
- cactus_frenzy 10y agoyo dawg! I heard U like GTA V and Python tutorials, so I put a tutorial inside your tutorial so you can play GTA V while you're learning!
- krzrak 10y agoThis is really cool. I love how those "stuping, time wasting" games turn out to be - with a help from some smart individual - great learning tools.
- coupdejarnac 10y agoGreat tutorial, watched the first few when he did them a week or so ago. I was one of the people suggesting he use PID controllers. While this is a ways off, it is fun to think of a project like this evolving into an open source self driving car package. pip install jalopy, or something like that.
- ZeroCool2u 10y ago>pip install jalopy Poetry.
- sentdex 10y agoAny chance you know how to implement PID control? I'm still throwing errors. Last night I was looking through a pull request that threw a breaking error and realized it was due to two uses of D (one for the key, one for the D in PID). Still having issues though after fixing that. If you have any ideas: https://github.com/Sentdex/pygta5/pull/3 https://github.com/Sentdex/pygta5/pull/3
- coupdejarnac 10y agoYes, I do. Also, I was the guy who suggested using two PI controllers- you can probably get by without the D term. I will look at it later tonight.
- jackdh 10y agoNow this is is a great way to get people interested in coding. I'd of loved to have something like this when I was learning!
- roystonvassey 10y agoWow, just wow. I am going to try doing this with my GTA SA. I am amazed with how versatile and fun Python is!
- sorenjan 10y agoWould it be possible to read the game's framebuffer directly to get rid of the screen grab step? I'm thinking that might be faster.
- bitL 10y agoWith source code obviously yes, though it would need interfacing between Python and C++ (likely used in GTA V), then internal screen resizing, and as most of the time is spent in OpenCV anyway it might not give you the effect you wish to get.
- sorenjan 10y agoObviously with source code, I was thinking of something like RenderDoc but in the form of a Python library.
- chatmasta 10y agoYou should look into method swizzling/hooking, to hook into a c++ method called on every frame, and add a line of code to save the frame buffer to disk. You can either do this at runtime via some kind of code injection, or at compilation time if you can manage to re-sign the game. I don't know much about c++ reverse engineering but I've done this when reversing iOS apps (and this is what "tweaking" means in the jailbreak community). The same concepts should apply. The first step is obtaining a class dump (what it's called with iOS) that shows you all the header files and gives you the information necessary to understand what's going on in a debugger so you can determine which method to hook. Edit: this should point you in the right direction - https://rafalcieslak.wordpress.com/2013/04/02/dynamic-linker-tricks-using-ld_preload-to-cheat-inject-features-and-investigate-programs/ https://rafalcieslak.wordpress.com/2013/04/02/dynamic-linker...
- anqh4 10y agoThis being such a popular multiplayer game, I'm sure the memory offsets of the position in the map/acceleration/etc are documented and you can just hook the process and read/write these
- Keyframe 10y agoSelf-playing games ought to be newer generation's Logo's turtle!
- nojvek 10y agoThere's something magical about creating a game AI. It's you're you're a god and you built a divine being in your diving game world.
- pacman83 10y agoTo make an artificial player for a simulated world we'd built was part of an assignment in Dr. Robert Harper's 15-212 class when I took it at CMU. On each tick of time, each of them would randomly do one of the things it could do. To see these characters randomly pick up change, put it into vending machines, and get out guns, ammo, and more of themselves, and then collectively gun me down, was a thrilling illustration of an AI control problem. I felt like a god being killed by his creations. Mindchildren?
- pacman83 10y agoBy the way, in case anyone knows, I would LOVE to track down the origins of this idea. A dozen years later, when I first thought to ask Dr. Harper, he did not remember the source, probably one of his TA's circa Fall 1992.
- cr0sh 10y agoIt probably goes back further than that; one could argue that things like Conway's LIFE implement certain aspects of the idea, as well as military "tabletop" game simulations (which go back a long ways). Other kinds of fantasy RPGs are similar, and then you have older video games on systems in the 1970s that could be considered. But I think military and other more "physical" simulations likely pre-date all of this; these systems have their own rule systems for the various "pieces" on the "board", and use things like dice and other random number generators to determine how things progress. There's also probably more than a few fictional stories which set up this idea as well.
- frik 10y agoI wish GTA V has the GTA IV vehicle physics engine settings. Driving in GTA IV feels so realistic, with so many attention to details from pedestrians that really react to small little things like misfiring of half-broken car engine and so much more. (it's the same engine (RAGE with bullet physics), just with dumbed down settings for more noob-friendly/casual gameplay) And not only the vehicle physics but also the damage model has been dumped down - the GTA IV one was superb. (Only totally different and a lot of more resource intensive damage models in RigsofRods/DriveNG are even more realistic.) They had to decrease the simulation level for GTA V as the PS3/Xbox360 weren't capable to offer the richness of GTA IV on bigger scale. Sadly they never added the details back with the PS4/XboxOne/PC HD re-mastered release. So to get the most detailed simulation one has to use GTA IV and install mods to drive around on the GTA V map. See YouTube videos, it's possible and a lot of fun.
- floatboth 10y agoIt's realistic but it's so annoying to play, you have to slow down A LOT to make a turn, compared to III/VC/SA where driving was actually fun.
- LordKano 10y agoIV was such a disappointment to me that I haven't even tried V yet. SA felt so expansive and open to me that when I played IV, it felt like I was trapped on story-telling rails. Have they reversed that trend in V?
- nickpsecurity 10y agoYou should play V. I don't know if you'll enjoy it but it's the biggest, loaded world they've ever done. Controls way better. The characters and dialogue are super-fun, too, even when missions make it pretty linear. Hell, even my mom liked watching me play for the characters. You also get some replay value on missions from ability to switch characters in real-time. You can try different ways of doing the mission with different people. The world itself obviously has piles of stuff to do as in the trailer that I luckily still have bookmarked since the trailer itself was put together so well. https://www.youtube.com/watch?v=N-xHcvug3WI https://www.youtube.com/watch?v=N-xHcvug3WI My problem with it was that the single-player was great. Lots of people and things I ran into. Lots of fun random things. Online was broken for that first month so couldn't do that. A glitch also made my cars disappear out of garages. I eventually just kind of got done with it after a month or two. Great experience but just a month or two given I wasn't going to go hunting, endlessly customize my character, play golf, etc. Might as well be on The Witcher or Skyrim for that kind of thing. People that like setting up crazy stunts, chain explosions, etc still had more fun but I just watch the best ones on YouTube to skip the work part of it. :) So, you should get at least a month of enjoyment out of it with it being pretty cheap by now. Trevor was worth it by himself since I'm pretty sure hypothesis is true that he's the first character modeled after the way most GTA players play GTA.
- omeid2 10y agoAnyone know of a similar game that you can run under Linux? I want to play with this but rather not spend the time to setup a vm and go through the whole pass-through dance or setup a dev env on windows.
- bko 10y agoCheck out openai. They have a lot of Atari games (https://gym.openai.com/envs#atari https://gym.openai.com/envs#atari) and a car racing game (https://gym.openai.com/envs/CarRacing-v0 https://gym.openai.com/envs/CarRacing-v0). For the Atari/car racing games you get direct pixel output of the game so using a neural network for image processing is required. But other games have a much simpler environment where you can focus on reinforcement learning. There's even classic doom. https://gym.openai.com/envs#doom https://gym.openai.com/envs#doom
- omeid2 10y agoThank you.
- poppingtonic 10y agoYou could use SuperTuxCart. I'm setting this up, actually.
- OrangeTux 10y agoI was thinking of Euro Truck simulator. Or SuperTuxKart.
- FranOntanaya 10y agoLiftoff (quadcopter flying sim) would be interesting.
- chatmasta 10y agoLook into the dolphin emulator for running GameCube and wii games. It's very well maintained with a smart community behind it, and I'm pretty sure it runs on Linux. I've used it on my mac for playing Mario kart and it was great. I actually found it through a post on here where someone was doing AI training on Mario kart through dolphin.
- jprissi 10y agoThe tutorial is great and the other website resources seem to be really interesting. Glad it was shared here !
- StavrosK 10y agoHere's a faster way to get a frame using PyGTK: https://www.pastery.net/xafjmn/ https://www.pastery.net/xafjmn/ Takes around 5ms per frame for me, rather than 50. Slightly shorter code (still using the same method): https://askubuntu.com/a/400384/73044 https://askubuntu.com/a/400384/73044
- sentdex 10y agoThanks for sharing this, will definitely look into it. With the ever improving FPS, this AI's reflexes are going to be insane.
- StavrosK 10y agoNo problem, glad to have helped!
- gejose 10y agoIs there a cross platform way to do this? I'd like to be able to do this on a Mac, but would prefer not to have to set up GTK (if it even works).
- nbrochu 10y agoThank you so much for sharing this! I'm working on something similar right now (a framework to build game agents using computer vision tools) and I just went from capturing at 10-15fps with ImageGrabber, to 15-25fps with FFMpeg and finally 70-80 fps with this technique. Insane!
- StavrosK 10y agoYou're quite welcome! I used it to capture a single pixel for my Super Hexagon LED strip (https://www.stavros.io/posts/wifi-enabled-rgb-led-strip-controller/ https://www.stavros.io/posts/wifi-enabled-rgb-led-strip-cont...) and it works really well.
- nbrochu 10y agoReally nice and can see why you needed the performance for this project. Funny that you mention Super Hexagon because it's the game I'm using for my POCs!
- creo 10y agoI would love to have public API for Witcher3 or GTAV. Imagine all those crazy things you could do.
- falcolas 10y agoIn a way, you do. The inputs are well defined (mouse and keyboard), and the outputs (graphics) can be captured fairly easily. Interpreting those outputs is a lot harder than interpreting a stream of "character A at point (X,Y,Z), character B at point...", I agree.
- sentdex 10y agoPart of my idea for this course was that you could use these methods on any game. You don't need anyone to make an API, you read the frames in, send direct keys, and you're all set. That said, there are various things like scripthookpy that let you communicate with GTA V to do things other than AI with Python. I am personally more curious about the AI aspect, so I figure any game that I can play visually, an AI can too with these methods.
- cr0sh 10y agoWhat I want is a full public API to control all aspects of the current view and character. At a minimum, I want to be able to send positional data to control the orientation and position of the head, and at least one of the hands (ragdoll physics and IK can take care of the rest). That way, I could control where I look with my head, and where I point or grab with my arm - independently. Ideally - you'd have both arms/hands free - plus you'd need inputs to control other aspects. For the head, if you crouch down, it would know this, and adjust the body model as well (or perhaps those parts could be controlled as well). The goal? Better immersion for VR - but platform agnostic; the thing is, I don't want future games to be beholden to a particular or set of HMDs (ie, this game only works with the RIFT, but not the Vive, etc). I also own a nice older kit of magnetic trackers (an old Ascension Flock) that I could use, along with some older HMDs (as a contributor to the KS, I also have the original DK1, plus the later KS backer CV1 - but both of these aren't well supported under Linux, which ticks me off). Basically, let me (or anyone else) control the character in full, and just provide a way to output two stereo frames (don't even distort them - let third-party software deal with that - because you might have an older HMD that doesn't need the distortion). Then get out of the way. I doubt that we'll see such a development though.
- pavel_lishin 10y agoA question for the author of this, or other authors who create step-by-step guides like this: do you write it as you go along, or fiddle around and then go back and write the blogpost? Even with small commits, I usually have a hard time reconstructing the history of what I've done well enough to write it up like this.
- Ajedi32 10y agoNot the author, but usually when writing step-by-step stuff like this I alternate between short bursts of messing around and then going back to write down what I just learned. So I'll fiddle for 10-15 minutes, occasionally pausing to write small notes and important details down (like the CLI commands I used to achieve some particular result), but only after I achieve some significant amount of progress will I go back and write about what I just did in more verbose language.
- sentdex 10y agoHey there, author here. It's just me who does the entire site of pythonprogramming.net and the youtube channel. The way I have gone about things has changed over time, but I have found the worst way to do documentation is to write 100% of the code, the full series in this case, then go back and document/write a tutorial on it. I used to just do the videos, but had lots of requests to do write-ups too, so I started that and really hated it at first, because I was just timing it all wrong. So now, I usually will do maybe 1-5 videos, and then make sure to do the write-ups on them before continuing...otherwise the write-ups suffer significantly. I also try to do the write-ups in the same day, and before I personally progress on to the next topics. It's just hard, because the last thing I want to do after I've made something that I think is cool is document it. Other times I will actually work locally, and just either save my scripts in a step-by-step manner, or just work in an ipython notebook to save the steps I took in development, and then film the video, and then go back to the notebook to do the documentation. Not sure that really helps much, it's an "it depends," but the main thing is to not get too far ahead of any accompanying documentation.
- cdnsteve 10y agoVery neat! I can't wait to take a look at this and give it a shot. The author mentioned it would be better to offload the screen capture to the GPU, would this be achieved through something like CUDA?
- sentdex 10y agoWow, hello hacker news! I haven't seen pythonprogramming.net load this slow in a ... ever. Thanks for sharing my silly project! Any ideas, pull requests, or critiques are welcome. (https://github.com/sentdex/pygta5/ https://github.com/sentdex/pygta5/) I'm currently working on PID control, and also contemplating switching the model to a DQN, with a reward for speed (whether this is perceived, or read from the game directly) IF we're on a paved surface. If you know about either, don't be shy. If you hate long loading, you can watch just the videos on youtube: https://www.youtube.com/playlist?list=PLQVvvaa0QuDeETZEOy4VdocT7TOjfSA8a https://www.youtube.com/playlist?list=PLQVvvaa0QuDeETZEOy4Vd... edit: added link to github, as well as links to just the videos since pythonprogramming.net is going slow from traffic.
- happy-go-lucky 10y agoHi, I watched some of your videos at https://www.youtube.com/user/sentdex https://www.youtube.com/user/sentdex and I found them helpful. Keep up the good work :)
- simonorlovsky 10y agoYour videos are great/inspirational. Keep it up!!
- stillsut 10y agoI'm always curious when someone shows up with a super impressive project, but has never registered a user name on Hacker News. Can I ask "where have you been all this time" - Do you have some other aggregator or blogroll that you follow? Do you try to avoid time wasted surfing different articles and comments? Did you know about HN before and avoid it due to its reputation, etc?
- markovbling 10y agoHe's been posting great Python tutorials on YouTube for years under the same channel name as his username - super impressive
- nickpsecurity 10y ago
- oculusthrift 10y agoi think the main contribution from this is just finding a way to get the frames from the game into opencv. Would be interesting to see a more "real" self driving ai method with this. I.E. a CNN
- Houshalter 10y agoCould you run GTA5 and a CNN at the same time? I think that would fry your GPU.
- cr0sh 10y agoIt can work - I won't re-post everything again here, but I've already mentioned in this thread how I used my 750 ti for training a CNN based on NVidia's End-to-End model to self-drive a virtual vehicle around a simulated track (this was part of Udacity's Self-Driving Vehicle Engineer Nanodegree). Udacity supplied the simulator, I set up CUDA and such to work with TensorFlow and Keras under Python. I had to drop the res down on the simulator (windowed at 800x600 IIRC), and training was done without the simulator (I used the simulator to generate the datasets - this is built into the simulator). The resulting model was actually fairly small (less than 300 kbytes); I scaled down the input images (originally 640x480, scaled to 160x120) to limit the number of initial parameters to the CNN, then applied aggressive dropout and other tricks to further keep things in check thru the layers as I trained (I could have used batch processing, too). The resulting model worked well with the simulator afterward - and it had no problem with memory or anything else in order to keep up.
- mandeepj 10y agoThis is also relevant - https://github.com/ai-tor/DeepGTAV https://github.com/ai-tor/DeepGTAV (A plugin for GTAV that transforms it into a vision-based self-driving car research environment.)
- kyberias 10y agoUsing python for all those individual WIN32 calls looks awkward. This interfacing would be easier in pure C.