21 ms·
Full Autopilot in GTA Using TensorFlow
- tinus_hn 6y agoWasn’t there a Twitch channel running this?
- hans1729 6y agoMaybe sentdex [0] streamed a session on twitch? He has a series on automatic driving in GTA iirc https://www.youtube.com/user/sentdex https://www.youtube.com/user/sentdex
- littlemtman 6y agoYes but what he did was behavioural cloning, I do lane and path prediction that's a whole different level
- tikej 6y agoAnd to me this is the essence of hacking and hacker culture, not “how do I create MVP from my idea and become an entrepreneur” or “this and that in company XYZ” mindset. So glad to see people doing silly things that in fact require skill. This is why I come to hackernews. Keep up great work with this wonderful attitude!
- skrebbel 6y agoI see your point and I agree with it. But I also want to remark that in many ways, building a company is also a silly thing that in fact requires skill. I mean, I'm pretty sure I'd be way richer right now if I had just taken a job at some bigco. Or kept doing consulting. I don't feel that what I'm doing right now (building a startup because I can) is fundamentally different from my demoscene past (coding computer graphics because I could). Sure, some startups make some people obscenely rich, but the vast majority don't :-) It's not too different from art or AI hobbying or gaming or game modding in that sense.
- alexeichemenda 6y ago> But I also want to remark that in many ways, building a company is also a silly thing that in fact requires skill. It is - Tikej's point isn't that it's not a skill - but rather that it's not the right place to share these. Think of the difference between a "Startup News" and "Hacker News". Hacker news used to be very deep on tech topics, now those deep topic have become more rare.
- littlemtman 6y agoHopefully I could bring you a deep tech topic !
- owlninja 6y agoVery true, but it is hosted by ycombinator which exists to fund startups.
- bravura 6y agoI remember an early pg quote where he noted that being a hacker isn’t supposed to be limit to coding or tech even, but all topics that hackers are drawn to and can ply their hacking in. Social hacking to bootstrap a userbase is a on-topic as far as that quote is concerned [edit: errant autocorrect]
- edouard-harris 6y agoInterestingly, Hacker News used to be called Startup News in the early days.
- BrandonM 6y agoIndeed, we’ve come full (half?) circle. IIRC, it was renamed to Hacker News in part to discourage meta-discussion on intellectually stimulating posts, questioning why the front page had so many posts that weren’t about startups. Now we are instead discussing startup post fatigue.
- dang 6y ago
- littlemtman 6y agoThank you very much !
- Erlich_Bachman 6y agoComma.ai which is used in the demo is a for-profit company that makes a sellable product.
- littlemtman 6y agoYes the comma two https://comma.ai/shop/products/comma-two-devkit https://comma.ai/shop/products/comma-two-devkit. If you have a newer Honda, Toyota etc. you can try it in real life ! Won't be working on your Aviato car, I am sorry :)
- nunja 6y agoBut please don't try it on a real road. Find a spot where it can't endanger fellows. It's possible to find those places.
- dicknuckle 6y agoThere's hundreds of people using these on public roadways. Check out the official Discord and the various Community Discords centered around it.
- commaai 6y agoOver 1,000 people just today. A couple thousand every week. In addition to running on a laptop, openpilot also runs on a phone for easy mounting in your car. https://github.com/commaai/openpilot https://github.com/commaai/openpilot
- Miraste 6y agoYour FAQ says Comma "openpilot performs the functions of Adaptive Cruise Control (ACC) and Automated Lane Centering (ALC)" and my 2020 Corolla is supported-but it already has those features. I read through the site and I don't see the value proposition. Is it that Comma does a better job of it?
- 6y ago
- smt88 6y agoNot sure why you need to put down entrepreneurs while praising this kid. Hacking a company together is really hard. Like tech hacking, it requires intelligence, creativity, and persistence. There's no reason we can't read about and respect both on this site.
- kick 6y agoThe concern is in the appropriation of the term 'hacker'. He's not putting anything or anyone down, he's saying that this is hacking, and MBA posts of acronym soup aren't. I think his implied conclusion is probably wrong-ish (both are arguably right for HN, even if I vastly prefer this), but he's right in that this is hacking.
- rapind 6y agoOne could argue WeWork hacked unicorns.
- toofy 6y agoOne could argue anything they like, it doesn’t mean their argument (or sophistry in general) leads to a better acquiring of knowledge through discussion. In my opinion, the gp’s discussion is one worth having.
- ilrwbwrkhv 6y ago100% agree. Winamp was a hacker's application. iTunes is a company's application.
- ddevault 6y agoStartup culture is basically the opposite of hacker culture. As kick hints to, the term has been appropriated. This site has been dubiously hackish from the start.
- enos_feedler 6y agoHacking together a product takes intelligence, creativity and persistence. Hacking together a company takes empathy, honesty, respect, humility, etc. It's about the people and building a team. I don't associate this with "hacking" at all.
- renewiltord 6y agoUgh, so obnoxious.
- umeshunni 6y agoWSJ had an interesting article the other day about why there are fewer startups now - https://www.wsj.com/articles/six-theories-on-why-fast-growing-startups-seem-to-be-disappearing-11588986419 https://www.wsj.com/articles/six-theories-on-why-fast-growin... tl;dr is that people are now more motivated by the idea of being an entrepreneur than by having a great business concept.
- holografix 6y agoI think Scott Galloway has some very valuable input on this topic. I might do it a disservice by trying to to;dr it here but basically there is very little if any room for innovation left when the big monopolies are currently very efficient in “featuring” your whole startup and through economies of scale (not having to turn a profit) they’ll eliminate you from the market it acquire you if they like the talent.
- deleted 6y ago[deleted]
- nagyf 6y agoReading the title my first thought was someone is driving a driverless car in Toronto, how cool is that. I was disappointed to see it's about the game. (Great article though)
- braunshedd 6y agoI'd love to see this done in Forza, although I'm sure that's optimizing for the wrong kind of driving behavior (lol) Amazing work!
- tudelo 6y agoYeah... I bet that would make for some really exciting gameplay :D
- mhh__ 6y agoA'I' in racing games is already a cracked problem I think (unless you mean forza horizon?). The AIs in Rfactor2 are even pretty good wheel to wheel on some tracks - definitely a lot better than online rookies who are quick but blind. The only noticeable thing is that they don't learn over the course of a race. They seem to be programmed to make mistakes realistically, i.e. if the car loses grip in the wet they don't lock back onto the track.
- bowmessage 6y agoSure, but this is a different kind of agent - one that takes screen frames as input, rather than one that has direct access to game state.
- gfrangakis 6y agoSeems like Forza would be easier to interface with too, example: this guy built a virtual cockpit[0] using VW parts -- the shifter, tach/speedometer all work [0] - https://www.youtube.com/channel/UCSEqO_2kdxLtslBK9FLqtxA https://www.youtube.com/channel/UCSEqO_2kdxLtslBK9FLqtxA
- SupriseAnxiety 6y agoI wonder, does this automatically turn off if the system that’s running it messes up? Also, in that case, if someone were to screw up GPS navigation or sensors, it would crash. Someone could fake a set of objects making the car take a modified prediction path, like “Virtual” barriers. Then, another worry, if all that can be avoided 100%, there’s still driver input. The driver should have the last say in what’s going on. Much like how a tesla alerts you rapidly as it’s predicting, Incase it’s prediction is wrong, thus, you SHOULD BE WATCHING, even if it’s “Autonomous” I’m sure the Tesla’s that have crashed have crashed because someone was able to remotely hack one. Which is why tesla sues someone for a ALOT if they hack their algorithm
- savrajsingh 6y agoGreat work! Now let's drive faster than 25 mph -- it's called Grand Theft Auto for a reason :)
- littlemtman 6y agoYes I will add a speeding video !
- rvz 6y agoGrand Theft AutoPilot. Now that would be a great name for this project. :)
- littlemtman 6y agoYes that's true :)
- dicknuckle 6y agoYour work here is done.
- bzb3 6y agoMy heart broke when I saw it wasn't one of the renderware era GTA games.
- mendelmaleh 6y agosentdex did something similar a year or two ago: https://www.youtube.com/playlist?list=PLQVvvaa0QuDeETZEOy4VdocT7TOjfSA8a https://www.youtube.com/playlist?list=PLQVvvaa0QuDeETZEOy4Vd...
- littlemtman 6y agoBehavioural cloning, doesnt work in the real world ! I use path and lane detection
- numpad0 6y agoFor inputs, roundabout but easiest ways I know is to either just get a Pro Micro and use the gamepad library in Arduino IDE to make it a serial-to-gamepad adapter, or use it in conjunction with a cheapest no-brand knockoff Xbox 360 controller(which are easier than genuine controllers to mod as well) and solder digital/analog out from Arduino to digital/analog sensing pads for the same effect. The latter allows real Xbox 360 to be used(don’t go online with it, that is cheating). Xbox 360 is past EOL so I bet they can be had for cheap, steps to capture is well explained by millions of wannabe streamers, and probably less tedious to set up at the same time than a PC.
- bluesign 6y agoOr full software solution for PlayStation is Remote Play.
- wigl 6y agoWhat ever happened to the OpenAI/universe GTA sandbox? Is there any reason Rockstar shouldn’t support research using their game?
- littlemtman 6y agoIf there was a nice solution for it, I would have done it but there isn't. Also the action space for that would be enormous. Maybe I will be making a MuZero Bot drive in GTA, that would be fun !
- wigl 6y agoDid some searching around and found a hidden fork, have you tried this? https://github.com/Alpha-Drive/universe-windows-envs/tree/master/vnc-gtav https://github.com/Alpha-Drive/universe-windows-envs/tree/ma...
- littlemtman 6y agoYes I have, but I would prefer a really clean Gym Env for just vehicles in GTA, with an understandable action space.
- rzzzt 6y agoI was also wondering about this. There are two issues in OpenAI's GitHub repo related to the removal, with not much information: - https://github.com/openai/universe/issues/126 https://github.com/openai/universe/issues/126 - https://github.com/openai/universe/issues/160 https://github.com/openai/universe/issues/160
- numpad0 6y agoFor some reason they really didn't like GTA being seen as a research platform and C&D'd projects.
- vhold 6y agoPerhaps because it could lead eventually to the most undetectable cheats yet and undermine their Skinner box cash extractors.
- verelo 6y agoCan it drive a plane / the tank on the road? How do motorbikes go? So curious.
- littlemtman 6y agoYes a tank would be possible. Both only from the front camera view, the model is not used to see the car it is driving otherwise !
- pugworthy 6y agoInteresting. I have a World of Tanks twitch streamer's video playing on the side as I read this, and also thought tanks.
- z3t4 6y agoSince this is GTA, it would be cool to add pedestrians to the mix...
- littlemtman 6y agoI tried it in the city center it was going nowhere, the bigger and complex the models get the more they learn "intuitively" i.e. avoiding pedestrians, avoiding crashes etc. Like the video where Autopilot predicted a crash 3 seconds before it actually occured.
- jabbanobodder 6y agoWhat ya doing? Given birth?!
- littlemtman 6y agoContext ?
- CrumResearch 6y agoHowdy, I came here from http://discord.comma.ai http://discord.comma.ai. littlemtman, what related source code is available? How about more details and the code for connecting to the XBox controller interface, is it open?
- cheschire 6y agoI'm not sure what OP used, but definitely look into FreePIE. http://andersmalmgren.github.io/FreePIE/ http://andersmalmgren.github.io/FreePIE/
- mistymountains 6y agoLet's all give this 12 year old kid some applause. Me water-cooling my xbox 360 (yes, and I wasn't alone in doing so) at 13 felt cool, but this is on another level.
- littlemtman 6y agonot 12. 15
- sixothree 6y agoI'm still jealous. At 15 I had a 10 MB hard drive and 4 color graphics available to me. While I am happy about who I am, I do wish sometimes my childhood had better technology available.
- Causality1 6y agoFrankly I think if I'd had better technology as a child I'd be even more content-addicted and boredom-averse than I am now. At least the way I am I can make it through a wedding without checking my phone.
- bigiain 6y agoThe downside there is that I _had_ to make it through a whole bunch of weddings without a smartphone and social media to escape to...
- ivalm 6y agoI am impressed by this 15 year old.
- gibolt 6y agoThe best part about being a kid is that you have nothing to think about other than a little school work. You are free to scour the web finding all the newest tools and put them to use for 'play'. Not saying it can't be done as an adult, just harder to get that time and unbroken/unlimited focus.
- jgalt212 6y agoThe real challenge of AI vis a vis real world problems is how to create infinite examples, labelled observations, etc at finite cost.
- Qworg 6y agoIf you want to play with ML racing, AWS DeepRacer is free for May (and you can compete against NASCAR drivers). https://aws.amazon.com/blogs/aws/join-the-formula-1-deepracer-proam-special-event/ https://aws.amazon.com/blogs/aws/join-the-formula-1-deeprace...
- permalac 6y agoCan I buy stock of this company?
- littlemtman 6y agoYou can't
- renewiltord 6y agoGreat job, love it. Do commercial solutions use simulations and stuff? Sounds like a great way to bootstrap is to use some of these more realistic games to get some initial work. By the way, I found this part funny: > ...if it doesn’t make sense to you think about it and try to understand it. We had a teacher once who'd say "If you don't understand, try to understand". Haha!
- dmoy 6y ago> Do commercial solutions use simulations and stuff? Yes Here's a random news article on it https://www.theatlantic.com/technology/archive/2017/08/inside-waymos-secret-testing-and-simulation-facilities/537648/ https://www.theatlantic.com/technology/archive/2017/08/insid...
- renewiltord 6y agoTrès cool. Thank you.
- hn_throwaway_99 6y agoThis kid is awesome. I say that even though he's going 25 MPH in the left lane :P
- muffin24 6y agosick
- baron816 6y agoHow about a self riding horse for RDR?
- serf 6y agothis is pretty cool. I used a similar scheme (two computers, one with HID control over the other, and a webcam to observe) as a way to hide a simple aimbot as a proof-of-concept for getting around on-computer anti-cheats that would search through memory on popular FPSs. This was years ago on games with weak anti-cheats, anyway. It worked OKish. It had to rely on computer vision to do the aiming, so it was significantly slower than examples that would rely on memory pointers or straight pixel recognition from the screen itself -- but the point was proven, the computer that was cheating looked fine as far as memory was concerned. modern anti-cheats that look at things like pointer movement and player randomness would definitely flag it now-a-days, but I had fun. The hardest problems were room lighting, which at the time the cheapy webcams had terrible issues with anyway, FPS, and resolution. Major shortcuts had to be taken back then due to the resolution and fps mismatches between the input and output, shortcuts by which reduced the bot performance significantly, but I bet that'd be a lot easier to deal with today. Now, my point : I know that a majority of the memory structs are available, documented, and deconstructed for GTA V, having written cheats for the game in the past. Maybe you can exploit that to emulate some of the functions of an IMU? Lots of vehicle physics variables are fairly easily exposed. "GTA V Memory Structs" google search will get you started on that road, should you care to. Neat project. Good luck.
- littlemtman 6y agoThanks, yes maybe I will add proper sensoring to the project with emulating a can bus.
- jaimex2 6y agoReminds me of the the Mario Kart version someone made 3 years ago: https://github.com/kevinhughes27/TensorKart https://github.com/kevinhughes27/TensorKart
- GekkePrutser 6y agoBut can it do a drive-by???