8 ms·

I work as a lead autonomous vehicle engineer. Here's a few thoughts: * This isn't really open-source. The most important part, the neural network perception p

I work as a lead autonomous vehicle engineer. Here's a few thoughts: * This isn't really open-source. The most important part, the neural network perception pipeline, is closed sourced. Without the training data and the network architectures, the supporting code is not that useful. * This system is non-deterministic, which is a no-no in critical systems like autonomous vehicle. There's no guarantee that vehicle detection or throttle/brake commands will be executed on a acceptably safe timeline. * I think the supporting tools like Panda and Giraffe are great for those wanting to get into car hacking. ------------------------------------------------ By the way, if you want to to become an autonomous vehicle engineer I put together a resource guide here: https://becomeautonomous.com https://becomeautonomous.com

ebg13 7y ago
> The most important part, the neural network perception pipeline, is closed sourced. Knowing what things are is certainly a part, and it may be the least solved part, but surely there is a lot of value in rolling everything else that goes into autonomous driving into a car-pluggable package. Trivial setup for sensing, fusing, actuating, and thrusting is pretty big for most people even if you don't get to see the more fuzzy stuff.
holoduke 7y ago
What do you mean it's non deterministic? Because it is using floating point operations? Can you elaborate?
- pkulak 7y ago
  Not a real-time OS?
  - Lorkki 7y ago
    Yep, it seems built to run on NEOS, which is an Android distribution: https://github.com/commaai/neo https://github.com/commaai/neo
- nialv7 7y ago
  Maybe they are talking about the numerical stability of the algorithms?
- bko 7y ago
  I'm not OP, but I think he means that the final decisions acted upon are not deterministic. So given an input, it won't necessarily produce the same decision every time. For instance, if the model predicts with a 95% certainty to accelerate, then 5% of the time it'll do some action other than accelerating.
  - mgkimsal 7y ago
    might it also be 'non-deterministic' if it accelerates 100% of the time, but accelerates at different speeds 5% of the time? or is that too granular a view?
- taneq 7y ago
  I'd guess it's about guaranteed response times (which is the definition of a realtime system - it must respond within the specified time). Since it's running a non-realtime OS, you could get arbitrarily long gaps in execution, which is not great for something that's driving a car.
- TickleSteve 7y ago
  Deterministic code would be running under a hard real-time kernel such as VxWorks or bare-metal and would be designed such that all timing has fixed upper-bounds. This involves reasoning about the scheduling of tasks to ensure that in worst-case situations, your timing requirements are still being met. All safety-critical systems would be deterministic, you simply would not get certified otherwise. Note: Theres much more to safety-critical application design, but basically, you have to be able to prove the behaviour of you code under all circumstances, which typically means no garbage-collection, dynamic-allocation, etc.
a-dub 7y ago
1) They actually open sourced the vision pipeline code a few months ago. The visiond binary has been replaced by source code. 2) Yeah. The proper way to do these things is with a real RTOS, but at the same time 20hz is pretty slow anyways.
- a-dub 7y ago
  I should also mention that the RT approach isn't the holy grail either for time sensitive robotic control systems. Read up on the Pathfinder/vxWorks priority inversion bug if you'd like to see how hard deadlines can sometimes cause more problems than they solve...
  - MuffinFlavored 7y ago
    What’s the answer then?
  - TickleSteve 7y ago
    priority-inversion is well-known and easily coped with. Hard real-time, deterministic code is simply a prerequisite for safety in these types of systems, you have to be able to reason about how your system will behave in worst-case situations.
- bri3d 7y ago
  The model, training tooling, and data appear to still be closed, no? So it’s still a black box, just now the “load the model binary and run it” part is open source.
taneq 7y ago
I'd love to try it out but I can't bring myself to trust (on any level) something that's essentially a mobile phone running a hacked-together mashup of C and Python. There's no redundancy, no independent components cross-checking each other, nothing as far as I'm aware to try and provide any given level of reliability. (Don't get me wrong, Python is great for research - I wouldn't regard it as highly reliable or good for high-availability systems, though.)
- aphextron 7y ago
  >There's no redundancy, no independent components cross-checking each other, nothing as far as I'm aware to try and provide any given level of reliability. And yet, it works beautifully. The redundancy is you; it's a level 2 system that works the same as cruise control. There have been no accidents with thousands of people using this to drive thousands of miles a day every day. You can continue waiting and hoping for a hypothetical perfect system, or you can have the future right now.
  - mayniac 7y ago
    If anyone has been in a crash while they were driving with this, we won't know about it. Adding a third party, unregulated, quasi-legal modification to your car which can take over steering, brakes, and acceleration will instantly invalidate your insurance. Not to mention it being potentially illegal to use this on the road. Nobody is going to say they were in an accident while being driven by an openpilot.
- jerkstate 7y ago
  I’m not trying to convince you, but the a-ha moment for me was when I was watching George talk about reverse-engineering the firmware for part of the cruise control in his car, and after decompiling it, he found that there was no functionality for the manufacturer to verify that the firmware had been written properly when flashed by the manufacturer. So just because a big, expensive car company made something, doesn’t mean there is any redundancy or cross-checking. By the way, the only cars you can install OpenPilot onto are cars that already have lane keeping assist (i.e. a camera that watches the lines) and adaptive cruise control (i.e. a millimeter wave radar that makes sure you don’t get too close to the car in front of you). So OpenPilot is just a vastly improved version of the software that those guys who didn’t write a firmware checksum routine wrote.
burtonator 7y ago
> Without the training data Big data engineer here... This is what a LOT of open source hackers don't understand about modern systems. It's all about the data. Most "futuristic" technologies that we're going to see in down the road are going to be built using absolutely massive data sets. Data sets that aren't in the public domain. This effectively locks out hackers and open source enthusiasts and means that only companies like Facebook, Google, and Amazon get to play with all the cool toys.
- austinkeller 7y ago
  Open data isn't glorified to the same degree as open source is by the common hacker, but it should be. That's where the future is headed
alphagrep12345 7y ago
I looked at your website. I just want to work on the perception part (CV and Deep Learning pipeline). Do I still need to learn all the robotics stuff you mentioned?

ebg13 7y ago

> The most important part, the neural network perception pipeline, is closed sourced. Knowing what things are is certainly a part, and it may be the least solved part, but surely there is a lot of value in rolling everything else that goes into autonomous driving into a car-pluggable package. Trivial setup for sensing, fusing, actuating, and thrusting is pretty big for most people even if you don't get to see the more fuzzy stuff.

holoduke 7y ago

What do you mean it's non deterministic? Because it is using floating point operations? Can you elaborate?

pkulak 7y ago
Not a real-time OS?
- Lorkki 7y ago
  Yep, it seems built to run on NEOS, which is an Android distribution: https://github.com/commaai/neo https://github.com/commaai/neo
nialv7 7y ago
Maybe they are talking about the numerical stability of the algorithms?
bko 7y ago
I'm not OP, but I think he means that the final decisions acted upon are not deterministic. So given an input, it won't necessarily produce the same decision every time. For instance, if the model predicts with a 95% certainty to accelerate, then 5% of the time it'll do some action other than accelerating.
- mgkimsal 7y ago
  might it also be 'non-deterministic' if it accelerates 100% of the time, but accelerates at different speeds 5% of the time? or is that too granular a view?
taneq 7y ago
I'd guess it's about guaranteed response times (which is the definition of a realtime system - it must respond within the specified time). Since it's running a non-realtime OS, you could get arbitrarily long gaps in execution, which is not great for something that's driving a car.
TickleSteve 7y ago
Deterministic code would be running under a hard real-time kernel such as VxWorks or bare-metal and would be designed such that all timing has fixed upper-bounds. This involves reasoning about the scheduling of tasks to ensure that in worst-case situations, your timing requirements are still being met. All safety-critical systems would be deterministic, you simply would not get certified otherwise. Note: Theres much more to safety-critical application design, but basically, you have to be able to prove the behaviour of you code under all circumstances, which typically means no garbage-collection, dynamic-allocation, etc.

a-dub 7y ago

1) They actually open sourced the vision pipeline code a few months ago. The visiond binary has been replaced by source code. 2) Yeah. The proper way to do these things is with a real RTOS, but at the same time 20hz is pretty slow anyways.

a-dub 7y ago
I should also mention that the RT approach isn't the holy grail either for time sensitive robotic control systems. Read up on the Pathfinder/vxWorks priority inversion bug if you'd like to see how hard deadlines can sometimes cause more problems than they solve...
- MuffinFlavored 7y ago
  What’s the answer then?
- TickleSteve 7y ago
  priority-inversion is well-known and easily coped with. Hard real-time, deterministic code is simply a prerequisite for safety in these types of systems, you have to be able to reason about how your system will behave in worst-case situations.
bri3d 7y ago
The model, training tooling, and data appear to still be closed, no? So it’s still a black box, just now the “load the model binary and run it” part is open source.

taneq 7y ago

I'd love to try it out but I can't bring myself to trust (on any level) something that's essentially a mobile phone running a hacked-together mashup of C and Python. There's no redundancy, no independent components cross-checking each other, nothing as far as I'm aware to try and provide any given level of reliability. (Don't get me wrong, Python is great for research - I wouldn't regard it as highly reliable or good for high-availability systems, though.)

aphextron 7y ago
>There's no redundancy, no independent components cross-checking each other, nothing as far as I'm aware to try and provide any given level of reliability. And yet, it works beautifully. The redundancy is you; it's a level 2 system that works the same as cruise control. There have been no accidents with thousands of people using this to drive thousands of miles a day every day. You can continue waiting and hoping for a hypothetical perfect system, or you can have the future right now.
- mayniac 7y ago
  If anyone has been in a crash while they were driving with this, we won't know about it. Adding a third party, unregulated, quasi-legal modification to your car which can take over steering, brakes, and acceleration will instantly invalidate your insurance. Not to mention it being potentially illegal to use this on the road. Nobody is going to say they were in an accident while being driven by an openpilot.
jerkstate 7y ago
I’m not trying to convince you, but the a-ha moment for me was when I was watching George talk about reverse-engineering the firmware for part of the cruise control in his car, and after decompiling it, he found that there was no functionality for the manufacturer to verify that the firmware had been written properly when flashed by the manufacturer. So just because a big, expensive car company made something, doesn’t mean there is any redundancy or cross-checking. By the way, the only cars you can install OpenPilot onto are cars that already have lane keeping assist (i.e. a camera that watches the lines) and adaptive cruise control (i.e. a millimeter wave radar that makes sure you don’t get too close to the car in front of you). So OpenPilot is just a vastly improved version of the software that those guys who didn’t write a firmware checksum routine wrote.

burtonator 7y ago

> Without the training data Big data engineer here... This is what a LOT of open source hackers don't understand about modern systems. It's all about the data. Most "futuristic" technologies that we're going to see in down the road are going to be built using absolutely massive data sets. Data sets that aren't in the public domain. This effectively locks out hackers and open source enthusiasts and means that only companies like Facebook, Google, and Amazon get to play with all the cool toys.

austinkeller 7y ago
Open data isn't glorified to the same degree as open source is by the common hacker, but it should be. That's where the future is headed

alphagrep12345 7y ago

I looked at your website. I just want to work on the perception part (CV and Deep Learning pipeline). Do I still need to learn all the robotics stuff you mentioned?