8 ms·

Europilot: A toolkit for Euro Truck Simulator to develop self-driving algorithms

254 points by daftshady 9y ago

mastazi 9y ago
I'm glad I saw this link, I was looking into building a solution based on parts of the ChosunTruck codebase[1], but Europilot seems more flexible since it's built as a "bridge" rather than a fully fledged self-driving solution. [1] https://github.com/bethesirius/ChosunTruck https://github.com/bethesirius/ChosunTruck
- rakete 9y ago
  Exactly the same for me. I had ChosunTruck bookmarked but Europilot looks much nicer at first glance. Another related thing that I wanted to check out is this: https://github.com/Microsoft/AirSim https://github.com/Microsoft/AirSim although that only gives you the sim and you have to do the machine learning part completely yourself.
crudbug 9y ago
Does this publish CAN bus events ? I was looking into similar solutions to capture CAN bus events for telematics usecase.
- gregschlom 9y ago
  It's a video game so there aren't any CAN bus events.
  - jeroenhd 9y ago
    That sounds like an interesting idea for a mod though...
m3kw9 9y ago
How does training the NN using a limited frame per second affect the training accuracy? Why not go higher or lower than the one you are using?
- imgyuri 9y ago
  In terms of capturing the screen input for training, the fps is capped at 10 fps. This is following the number suggested in NVIDIA's paper[1], which cautions against overfitting. The program captures faster when the steering angle gets larger, because it suggests that curvy road is ahead, which are the more challenging scenarios for the program. In terms of testing, the max fps is determined by the performance of the GPU and the complexity of the model. I couldn't find any definitive studies done on the matter, but anywhere from 10-30 fps seems to work OK. [1] https://arxiv.org/abs/1604.07316 https://arxiv.org/abs/1604.07316
  - bitL 9y ago
    Add a RNN with a 10-frames history on top of NVidia's model and you get the winning Udacity challenge entry (mentioned on selfdrivingcars.mit.edu). I assume you also programmed this model during Udacity's Self-driving car nanodegree.
sosa2k 9y ago
Here's a YouTube playlist where a guy made a self-driving vehicle in GTA V using Python. https://youtu.be/ks4MPfMq8aQ https://youtu.be/ks4MPfMq8aQ
- mastazi 9y ago
  He also streams the self driving vehicle 24/7 on Twitch, it's quite amusing to watch, every now and then the autonomous vehicle causes an accident and ends up being chased by the Police LOL https://www.twitch.tv/sentdex https://www.twitch.tv/sentdex It was also very interesting in the early days because you could see the agent getting much better day after day.
  - mulmen 9y ago
    I just started watching this series independently, it's great to see how far it has come! I had no idea there was a 24/7 twitch stream, this is hilarious.
  - notahacker 9y ago
    haha, I switched just in time to see it meekly surrender to the police. It looks like it still has a lot of improving to do to manage basic stuff like not accelerating into oncoming vehicles, but it's quite neat to see that its mastered the art of stealing another vehicle after writing its own off, which isn't something your average self driving neural network gets trained on.
amelius 9y ago
> With europilot, you can capture the game screen input, and programmatically control the truck inside the simulator. Can't this be easily generalized to any game?
- imgyuri 9y ago
  Yes, and this is by design. However the controlling logic will probably have to be customized for each additional game. Euro Truck Simulator was chosen because it runs on Windows/OS X/Linux, and because the game consists mainly of highway roads, which is an easier target than urban environments. Also I just really like the game itself.
- zpr 9y ago
  This is really cool. Consider introducing this concept to an FPS game, at what point do you consider it to be an aimbot?
  - octalmage 9y ago
    That is exactly what aimbots are. Sometimes you can make bots easier by using an addon system to export information(WoW), or themes to color your targets (CS), but at the root you read the screen and control the mouse.
    - tokenizerrr 9y ago
      No. At root you read and write the game's memory. Most aimbots just read the memory to get locations and call the game's own methods to adjust the aim from their injected dll. Barely any actually bother reading the screen, it's horribly inefficient and limited.
      - octalmage 9y ago
        I've used and written a ton of bots that read the screen, sometimes it's the best option.
amelius 9y ago
An obvious disadvantage is that you don't have LIDAR inputs. I guess it would be rather difficult to hack ETS such that it generates them.
- uhhhouses 9y ago
  Tell that to tesla !
- deleted 9y ago
  [deleted]
- wyager 9y ago
  I'm not sure. LIDAR is just about the simplest thing to render in a simulation: a relatively small number of ray-scene intersections.
  - CardenB 9y ago
    Simulating lidar returns in a way that has noise similar to real devices is difficult
    - sytelus 9y ago
      Indeed. There are tons of things that happen in real world lidar. For ex, if object is too close it may not detect. When you are going at high speed, returned light ray may arrive early or late giving bit of error. Also, think about multiple reflections, noise, cone size and so on.
- rzzzt 9y ago
  The contents of the z-buffer could be a close approximate.
tomdre 9y ago
Do you guys know if it's possible to programmatically alter ETS environment? (such as weather, brightness, traffic, etc). Thinking of using the simulator to generate augmented data.
- scott_karana 9y ago
  Most likely. All those effects are already built-in, and it looks like there's a pretty comprehensive modding community. There might not be an API callable from outside the app, though...
anindha 9y ago
There is quite a lot of interest in a self driving car simulator. I wonder if we could use Kickstarter to raise some money to get the game maker to add simulated radar, lidar, and cameras. You should be able to customize the position of these sensors.
- KGIII 9y ago
  A self-driving car simulator would be interesting and maybe there's some chance at making it open source? A self-driving sim, to be realistic, is also going to need realistic traffic simulation. I helped bring traffic modeling to the computer age, that is what my business did. When asked for a quick description, I like to say that I built the greatest traffic simulator on the planet. Of course, I'm heavily biased. Eventually, we could even render some of that data into graphics. Yeah, so it probably wasn't a thrilling simulator with broad appeal and it required a whole lot of compute power and huge data sets. Either way, it probably would have resulted in a good, albeit very large and computationally expensive, engine for a traffic simulating game. I signed a non-compete that lasts for life. It may not actually be legally binding, but I feel ethically bound to it. But, a game would not be competing in that space - less so if it were FOSS. What I'm trying to say is that I'd be interested in such a project, though I'd want to seek legal council. I can program a bit, but I eventually hired professionals to do that on my behalf. Let's just say that you probably don't want me submitting actual code, but I can almost certainly help with the math involved. I'm just spitballing, really. If someone wants to do something like this, I'd be willing to take a peek and see if there were a way to contribute. I sold and retired, but I still hold great fondness for the art. Modeling traffic isn't easy, it's a chaotic system and you never really know the initial starting state. So, if you decide to go FOSS, let me know. If it is in C, or C++, I might even remember enough to lend a hand coding, though that's probably a horrible idea. My entire code base was rewritten by actual competent people and programmers were different back then. They were quite comfortable expressing their frustrations with working on my existing code. They were, shall we say, very eager to share their knowledge - sometimes with various levels of vulgarity. Anyhow, to model traffic at a high level of accuracy, even things like the reflectivity of lane markings will have a quantifiable impact on traffic. You can observe the effect by monitoring traffic before and after lane markings are painted. Traffic is even influenced by the fonts used in signage. Lots of seemingly small things can be accounted for and the models made more accurate. I'd love to lend a hand, assuming it isn't going to get me sued.
pj_mukh 9y ago
This is brilliant! I built a similar system on top of Flight Simulator for an internship. So useful!

254 points by daftshady 9y ago

I'm glad I saw this link, I was looking into building a solution based on parts of the ChosunTruck codebase[1], but Europilot seems more flexible since it's built as a "bridge" rather than a fully fledged self-driving solution. [1] https://github.com/bethesirius/ChosunTruck https://github.com/bethesirius/ChosunTruck

rakete 9y ago
Exactly the same for me. I had ChosunTruck bookmarked but Europilot looks much nicer at first glance. Another related thing that I wanted to check out is this: https://github.com/Microsoft/AirSim https://github.com/Microsoft/AirSim although that only gives you the sim and you have to do the machine learning part completely yourself.

crudbug 9y ago

Does this publish CAN bus events ? I was looking into similar solutions to capture CAN bus events for telematics usecase.

gregschlom 9y ago
It's a video game so there aren't any CAN bus events.
- jeroenhd 9y ago
  That sounds like an interesting idea for a mod though...

m3kw9 9y ago

How does training the NN using a limited frame per second affect the training accuracy? Why not go higher or lower than the one you are using?

imgyuri 9y ago
In terms of capturing the screen input for training, the fps is capped at 10 fps. This is following the number suggested in NVIDIA's paper[1], which cautions against overfitting. The program captures faster when the steering angle gets larger, because it suggests that curvy road is ahead, which are the more challenging scenarios for the program. In terms of testing, the max fps is determined by the performance of the GPU and the complexity of the model. I couldn't find any definitive studies done on the matter, but anywhere from 10-30 fps seems to work OK. [1] https://arxiv.org/abs/1604.07316 https://arxiv.org/abs/1604.07316
- bitL 9y ago
  Add a RNN with a 10-frames history on top of NVidia's model and you get the winning Udacity challenge entry (mentioned on selfdrivingcars.mit.edu). I assume you also programmed this model during Udacity's Self-driving car nanodegree.

sosa2k 9y ago

Here's a YouTube playlist where a guy made a self-driving vehicle in GTA V using Python. https://youtu.be/ks4MPfMq8aQ https://youtu.be/ks4MPfMq8aQ

mastazi 9y ago
He also streams the self driving vehicle 24/7 on Twitch, it's quite amusing to watch, every now and then the autonomous vehicle causes an accident and ends up being chased by the Police LOL https://www.twitch.tv/sentdex https://www.twitch.tv/sentdex It was also very interesting in the early days because you could see the agent getting much better day after day.
- mulmen 9y ago
  I just started watching this series independently, it's great to see how far it has come! I had no idea there was a 24/7 twitch stream, this is hilarious.
- notahacker 9y ago
  haha, I switched just in time to see it meekly surrender to the police. It looks like it still has a lot of improving to do to manage basic stuff like not accelerating into oncoming vehicles, but it's quite neat to see that its mastered the art of stealing another vehicle after writing its own off, which isn't something your average self driving neural network gets trained on.

amelius 9y ago

> With europilot, you can capture the game screen input, and programmatically control the truck inside the simulator. Can't this be easily generalized to any game?

imgyuri 9y ago
Yes, and this is by design. However the controlling logic will probably have to be customized for each additional game. Euro Truck Simulator was chosen because it runs on Windows/OS X/Linux, and because the game consists mainly of highway roads, which is an easier target than urban environments. Also I just really like the game itself.
zpr 9y ago
This is really cool. Consider introducing this concept to an FPS game, at what point do you consider it to be an aimbot?
- octalmage 9y ago
  That is exactly what aimbots are. Sometimes you can make bots easier by using an addon system to export information(WoW), or themes to color your targets (CS), but at the root you read the screen and control the mouse.
  - tokenizerrr 9y ago
    No. At root you read and write the game's memory. Most aimbots just read the memory to get locations and call the game's own methods to adjust the aim from their injected dll. Barely any actually bother reading the screen, it's horribly inefficient and limited.
    - octalmage 9y ago
      I've used and written a ton of bots that read the screen, sometimes it's the best option.

amelius 9y ago

An obvious disadvantage is that you don't have LIDAR inputs. I guess it would be rather difficult to hack ETS such that it generates them.

uhhhouses 9y ago
Tell that to tesla !
deleted 9y ago
[deleted]
wyager 9y ago
I'm not sure. LIDAR is just about the simplest thing to render in a simulation: a relatively small number of ray-scene intersections.
- CardenB 9y ago
  Simulating lidar returns in a way that has noise similar to real devices is difficult
  - sytelus 9y ago
    Indeed. There are tons of things that happen in real world lidar. For ex, if object is too close it may not detect. When you are going at high speed, returned light ray may arrive early or late giving bit of error. Also, think about multiple reflections, noise, cone size and so on.
rzzzt 9y ago
The contents of the z-buffer could be a close approximate.

tomdre 9y ago

Do you guys know if it's possible to programmatically alter ETS environment? (such as weather, brightness, traffic, etc). Thinking of using the simulator to generate augmented data.

scott_karana 9y ago
Most likely. All those effects are already built-in, and it looks like there's a pretty comprehensive modding community. There might not be an API callable from outside the app, though...

anindha 9y ago

There is quite a lot of interest in a self driving car simulator. I wonder if we could use Kickstarter to raise some money to get the game maker to add simulated radar, lidar, and cameras. You should be able to customize the position of these sensors.

KGIII 9y ago
A self-driving car simulator would be interesting and maybe there's some chance at making it open source? A self-driving sim, to be realistic, is also going to need realistic traffic simulation. I helped bring traffic modeling to the computer age, that is what my business did. When asked for a quick description, I like to say that I built the greatest traffic simulator on the planet. Of course, I'm heavily biased. Eventually, we could even render some of that data into graphics. Yeah, so it probably wasn't a thrilling simulator with broad appeal and it required a whole lot of compute power and huge data sets. Either way, it probably would have resulted in a good, albeit very large and computationally expensive, engine for a traffic simulating game. I signed a non-compete that lasts for life. It may not actually be legally binding, but I feel ethically bound to it. But, a game would not be competing in that space - less so if it were FOSS. What I'm trying to say is that I'd be interested in such a project, though I'd want to seek legal council. I can program a bit, but I eventually hired professionals to do that on my behalf. Let's just say that you probably don't want me submitting actual code, but I can almost certainly help with the math involved. I'm just spitballing, really. If someone wants to do something like this, I'd be willing to take a peek and see if there were a way to contribute. I sold and retired, but I still hold great fondness for the art. Modeling traffic isn't easy, it's a chaotic system and you never really know the initial starting state. So, if you decide to go FOSS, let me know. If it is in C, or C++, I might even remember enough to lend a hand coding, though that's probably a horrible idea. My entire code base was rewritten by actual competent people and programmers were different back then. They were quite comfortable expressing their frustrations with working on my existing code. They were, shall we say, very eager to share their knowledge - sometimes with various levels of vulgarity. Anyhow, to model traffic at a high level of accuracy, even things like the reflectivity of lane markings will have a quantifiable impact on traffic. You can observe the effect by monitoring traffic before and after lane markings are painted. Traffic is even influenced by the fonts used in signage. Lots of seemingly small things can be accounted for and the models made more accurate. I'd love to lend a hand, assuming it isn't going to get me sued.

pj_mukh 9y ago

This is brilliant! I built a similar system on top of Flight Simulator for an internship. So useful!