9 ms·

Really, a safety critical hard realtime system with control loops on top of Python, a general purpose OS and a CAN-USB adapter? There are reasons why we have r

Really, a safety critical hard realtime system with control loops on top of Python, a general purpose OS and a CAN-USB adapter? There are reasons why we have realtime operating systems, deterministic bus systems and dedicated CPUs for these kinds of applications. That might be an interesting research, prototyping or simulation platform. But nothing you want to have on a real public road.

randomdata 10y ago
> these kinds of applications. As in research projects, which this project expressly states that it is? What are those reasons? It seems like being able to test theories as quickly and cheaply as possible is the top priority. If Python, a general purpose OS, and a CAN-USB adapter is sufficient to conduct the research and satisfies those attributes, it seems like the perfectly logical way forward. I understand why you would want to move to such systems you describe once your research has concluded and you are now building something for production, but that is not the intent of this project, at least at this stage. It specifically says so.
- ryaneager 10y ago
  Why wouldn't you do the research on a platform that could actually be deployed? Build the core using the proper tools and techniques. Any bug in a system like this could be life threatening at worst. Wouldn't you want to use the research phase to help eliminate such bugs? Unless you're not planning on ever releasing then a platform like this makes sense for strictly research, but Comma AI did plan on releasing their product until they got shut down.
  - randomdata 10y ago
    Research implies that you don't know the outcome and are going to be throwing away everything that doesn't pan out. Without knowing the outcome, how can you really be sure of what you need? You might think you need a CPU-based system but learn that GPUs are necessary, for instance. If you can start with cheap off-the-shelf components, at least you don't have much to loose when it is time to throw it away. If you're spinning your own boards, or heavily tied to expensive development kits, that becomes much harder to swallow. Especially for a startup with limited resources. > Wouldn't you want to use the research phase to help eliminate such bugs? I wouldn't think so. Spending your time fixing bugs in something you realize could have been done better another way, which ends up getting thrown out, seems like a waste of time. The words research and development are often paired together because it is a two step process, where development comes after you have learned what can be done.
  - MichaelBurge 10y ago
    As an example of some requirements, any recursion is banned unless accompanied by a termination proof that keeps track of stack size. No calls to malloc() or free() are allowed, and so any library you might use that allocates memory is banned. It's far easier to get the code working in something like Python, and then rewrite it in safety-critical C later. And the kinds of people who can write self-driving cars in Python are using very different skills than the kinds who can write safety-critical C, so it might not even be the same person.
    - JoshTriplett 10y ago
      Exactly; the delta from Python to C is smaller than the delta from C to "safety-critical real-time C". Might as well prototype and simulate in a higher-level language that's obviously not the final one, rather than writing in a language someone might mistake for production and trying to incrementally evolve it into a full safety-critical version.
    - time4tea 10y ago
      Hmm. Clearly this code is miles away from MISRA compliance (although compliance doesn't mean that the software is any good) Even so, sorry to say, the code is miles away from any decent software standard. It is almost exactly what you might call CRAP (classes really a procedure) and mixes responsibilities all over the shop. Little attempt is made to abstract behaviour, and there are differing units, and random multipliers all over the place. Using c++ and python, both of which are OO many classes of error could be avoided if any actual OO features were used, even if MISRA was not a target, sadly the code doesn't do that. This has all the hallmarks of software that doesnt know if it's doing m/s or mph.
- throwaway729 10y ago
  > once your research has concluded and you are now building something for production I think the rub is that this code appears to be intended for "production" (or at least use on public roads), given the fact that it's published by a company that has tested its products on open roads?
  - 56245623456 10y ago
    It explicitly states: "THIS IS ALPHA QUALITY SOFTWARE FOR RESEARCH PURPOSES ONLY. THIS IS NOT A PRODUCT. YOU ARE RESPONSIBLE FOR COMPLYING WITH LOCAL LAWS AND REGULATIONS. NO WARRANTY EXPRESSED OR IMPLIED."
    - throwaway729 10y ago
      And yet comma ai were definitely testing their products on public roads, products which were presumably running this code.
      - mikeash 10y ago
        This would certainly explain why they folded so quickly once the NHTSA started asking questions.
      - __jal 10y ago
        ...Which is why it says "YOU ARE RESPONSIBLE FOR COMPLYING WITH LOCAL LAWS AND REGULATIONS".
        deleted 10y ago
        [deleted]
        deleted 10y ago
        [deleted]
        jakevn 10y ago
        Ah yes, laws are definitely the determinate for safety. This is why texting and driving is illegal everywhere.
- bdrool 10y ago
  > As in research projects, which this project expressly states that it is? This is from comma.ai, who famously tested on public roads with a reporter in the car and very little prior testing ("the first time it worked was this morning").[1] [1] http://www.bloomberg.com/features/2015-george-hotz-self-driving-car/ http://www.bloomberg.com/features/2015-george-hotz-self-driv...
hasbroslasher 10y ago
I'm not really following. If it works then why not use a control loop on top of Python? We are engineers, after all.
- throwaway729 10y ago
  Because by "works" you mean "I tried it out a bunch of times and nothing bad happened, so must be production ready", because that's what you do when you build websites and desktop CRUD apps and nobody ever died cause the web server choked under load or the web page rendered a bit funny or the request took a full quarter second because the GC kicked off as a wave of requests came in. And then one day a one-in-ten-million event that never would've surfaced during testing does happen and the OS crashes or the interpreter hangs or some non-determinism causes a period of non-responsiveness or a bit gets mangled because you don't have any redundancies and solar rays are thing or ... and someone dies. And of course that didn't happen in testing, because all of those things are rare possibilities. And then it happens again. And again. And then you're in court being sued for millions. And a bunch of industry experts who do build their systems with redundancies and do design their systems with a safety-first mindset come to the stand and rip you apart for not taking even the most basic precautions. And when it comes out just how many best practices you completely ignored, you're mostly just hoping beyond hope that all of the legal problems stay on the civil side of the civil/criminal divide. Or, to say it in a sentence, "because the sort of exceptional circumstances that safety-critical software needs to handle with grace are very difficult or impossible to account for with a desktop machine running interpreted code on top of Linux."
  - hasbroslasher 10y ago
    I like that you reduced that to a one-liner, and I understand your point. I sure as hell wouldn't hook my car up to this software. But I also feel like you're being a bit hasty. Obviously a Python script isn't going to turn into a Tesla overnight. But maybe it'll help you find a few bugs before you throw all that time and effort into building the real deal. When I look at a Github repo that claims to be a self-driving car I don't say to myself "yep, looks production ready," I say "Cool prototype, now let's break it." To me it seems entirely reasonable to get a working Python prototype 90% of the way there and then send it off to the OS programmers to design something that actually meets the concept of "production ready".
breck 10y ago
Why not? We entrust humans with operating cars right now. Humans in a sense are running a general purpose OS.
- andars 10y ago
  This is not relevant. If a corporation builds a product that kills people, the corporation is responsible. No corporation is responsible for the behaviour of humans.
- dsr_ 10y ago
  If you read enough security literature, eventually you'll come across the notion of a "trusted system". And your first impression will be that this is a system which is fully debugged and tested and you can trust it to do its job within specifications. But you couldn't be more wrong. A trusted system is one which you have to trust. No representations are made about whether you should or not. Humans are frequently trusted systems.
- tptacek 10y ago
  No hundred humans are running the exact same OS that can deadlock or crash and as a result drive off the road, which is why this trope about humans being worse than computers at driving is entirely irrelevant to a thread about why it's unwise to try to build self-driving cars on top of Python scripts.
  - breck 10y ago
    My take of the OP's point was that the deep general stack was unnecessary and decreased reliability. My read was that they think your self driving car software shouldn't also be able to run arbitrary python scripts. My argument is that it's practically irrelevant to reliability. I would not be surprised if in the future you could download an app to your iPhone, connect your phone to your car, and have the app drive your car. Sure, maybe having a specialized real time o.s. without any higher level interpreters could be more reliable in the early days, but the gains from rapid iteration you'd get from allowing cars to be driven by apps running on higher level abstractions would be worth the trade off, and over time the latter could be iterated upon to achieve practically the same reliability as over-engineered systems.
deleted 10y ago
[deleted]
stevendhansen 10y ago
The big problem in my mind is that Python is the wrong tool for the job. It is 2016, there are more hard-RTOS prototyping platforms than ever before. My favorite is National Instrument's RIO platform, which lets you use C or LabVIEW (imho the best language for prototyping control algorithms by far). Mathworks also has a platform based on Matlab/Simulink, and the list goes on. Why use Python when there are existing tools that are made for this type of application?
- snovv_crash 10y ago
  LabVIEW is great until you need a complicated data structure. I tried writing a tree to do kNN, and it turned out that doing a brute force search was faster even with 100k elements.
- veli_joza 10y ago
  This is (now) open source project expecting community pull requests. How many would they get if they went with LabView? The big emphasis that I like seeing in modern scientists is reproducibility of results. Jupyter Notebook is a huge step in that direction (not really applicable here), but just using open source platform is still great. I agree though that once algorithms are developed sufficiently they should be ported to RTOS platform and this box shouldn't be permitted on open roads.
mrlambchop 10y ago
It seems incredible that this is the source code for the $999 product originally destined to be available this year. I'm extremely familiar with OBD|CAN and car networks - the embedded code alone is an essay in how trivial something can appear (sending commands to a ECU) without considering the million edge cases that make this a safe product to use.
- deleted 10y ago
  [deleted]
- espes 10y ago
  > It seems incredible that this is the source code for the $999 product originally destined to be available this year. It's not
  - mrlambchop 10y ago
    Forgive me but it seemed like this was the case. Is this an earlier version of the code or a special opensource clean up then? http://newatlas.com/geohot-comma-ai-openpilot-open-source/46722/ http://newatlas.com/geohot-comma-ai-openpilot-open-source/46...
    - CmdrSprinkles 10y ago
      The ars technical article covers it better http://arstechnica.com/cars/2016/11/after-mothballing-comma-one-george-hotz-releases-free-autonomous-car-software/ http://arstechnica.com/cars/2016/11/after-mothballing-comma-... But in a nutshell: The company was approached by agencies that had severe concerns over safety and if this would have the required regulations in place. So the company folded and released this instead.
KKKKkkkk1 10y ago
Human drivers don't run an RTOS and a deterministic bus system either. In fact, I would argue that very rarely do human drivers fail because of jitter or nondeterminism.
- savanaly 10y ago
  The more relevant question is not do they but if they could would we want them to?
- tptacek 10y ago
  This is a weird argument. Human drivers don't deadlock or crash either. Are you sure you understand the criticism being made here? Hard realtime systems make guarantees about the frequency with which code will run on a shared system.
  - veli_joza 10y ago
    I agree that this is not production ready tech, but still parent does make a good point. I think those two similar comments are downvoted just because people here don't agree with them. Suppose you make an autopilot in JS running on Electron on WinXP, running on VM on top of Puppy Linux Live CD. And you still manage to prove that your system is 100x more reliable than human driver. Should it be dismissed just because we don't agree with technology stack? The latency of this monstrosity would still be lower than human driver and maybe they would stay lower throughout the operation.
    - terryf 10y ago
      This doesn't have anything to do with latency really. It's about predictability. Given the system you described, there are so many edge cases that can potentially compound, that it is impossible to make any predictability guarantees about the system. As an example, I worked on an embedded system that controlled electrical motors that was hard real time. The fast task time interval was once per millisecond. No matter what, that got called by the RTOS exactly 1000 times per second. When it didn't finish it's job in time, the result could easily wreck real world items or cause harm to people. Nobody even considered using interpreted languages in that project. The fast tasks all had provably run in much less than a millisecond. That means no loops that could be unbounded, no memory allocations, no recursion, no writing to flash, anything that was even slightly unpredictable was out. So, even if you could prove that your system caused less accidents than a human driver when it was running well, it would be impossible to do an analysis that defined under what circumstances the system would be running well. Given that, it would not be allowed in a well-engineered real time or safety critical system.
ma2rten 10y ago
The Python control loop is probably the safest part of this system. The reason that comma.ai was able to get a working prototype quickly (unlike much bigger and well-funded companies) is that they use an end-to-end deep learning model. The model directly learns to issue commands to the car based on raw image data. The problem with this approach is that neural network models are a blackbox. They can behave in completely unpredictable ways.
- Animats 10y ago
  Actually, they didn't. At least not in this code.

randomdata 10y ago

> these kinds of applications. As in research projects, which this project expressly states that it is? What are those reasons? It seems like being able to test theories as quickly and cheaply as possible is the top priority. If Python, a general purpose OS, and a CAN-USB adapter is sufficient to conduct the research and satisfies those attributes, it seems like the perfectly logical way forward. I understand why you would want to move to such systems you describe once your research has concluded and you are now building something for production, but that is not the intent of this project, at least at this stage. It specifically says so.

ryaneager 10y ago
Why wouldn't you do the research on a platform that could actually be deployed? Build the core using the proper tools and techniques. Any bug in a system like this could be life threatening at worst. Wouldn't you want to use the research phase to help eliminate such bugs? Unless you're not planning on ever releasing then a platform like this makes sense for strictly research, but Comma AI did plan on releasing their product until they got shut down.
- randomdata 10y ago
  Research implies that you don't know the outcome and are going to be throwing away everything that doesn't pan out. Without knowing the outcome, how can you really be sure of what you need? You might think you need a CPU-based system but learn that GPUs are necessary, for instance. If you can start with cheap off-the-shelf components, at least you don't have much to loose when it is time to throw it away. If you're spinning your own boards, or heavily tied to expensive development kits, that becomes much harder to swallow. Especially for a startup with limited resources. > Wouldn't you want to use the research phase to help eliminate such bugs? I wouldn't think so. Spending your time fixing bugs in something you realize could have been done better another way, which ends up getting thrown out, seems like a waste of time. The words research and development are often paired together because it is a two step process, where development comes after you have learned what can be done.
- MichaelBurge 10y ago
  As an example of some requirements, any recursion is banned unless accompanied by a termination proof that keeps track of stack size. No calls to malloc() or free() are allowed, and so any library you might use that allocates memory is banned. It's far easier to get the code working in something like Python, and then rewrite it in safety-critical C later. And the kinds of people who can write self-driving cars in Python are using very different skills than the kinds who can write safety-critical C, so it might not even be the same person.
  - JoshTriplett 10y ago
    Exactly; the delta from Python to C is smaller than the delta from C to "safety-critical real-time C". Might as well prototype and simulate in a higher-level language that's obviously not the final one, rather than writing in a language someone might mistake for production and trying to incrementally evolve it into a full safety-critical version.
  - time4tea 10y ago
    Hmm. Clearly this code is miles away from MISRA compliance (although compliance doesn't mean that the software is any good) Even so, sorry to say, the code is miles away from any decent software standard. It is almost exactly what you might call CRAP (classes really a procedure) and mixes responsibilities all over the shop. Little attempt is made to abstract behaviour, and there are differing units, and random multipliers all over the place. Using c++ and python, both of which are OO many classes of error could be avoided if any actual OO features were used, even if MISRA was not a target, sadly the code doesn't do that. This has all the hallmarks of software that doesnt know if it's doing m/s or mph.
throwaway729 10y ago
> once your research has concluded and you are now building something for production I think the rub is that this code appears to be intended for "production" (or at least use on public roads), given the fact that it's published by a company that has tested its products on open roads?
- 56245623456 10y ago
  It explicitly states: "THIS IS ALPHA QUALITY SOFTWARE FOR RESEARCH PURPOSES ONLY. THIS IS NOT A PRODUCT. YOU ARE RESPONSIBLE FOR COMPLYING WITH LOCAL LAWS AND REGULATIONS. NO WARRANTY EXPRESSED OR IMPLIED."
  - throwaway729 10y ago
    And yet comma ai were definitely testing their products on public roads, products which were presumably running this code.
    - mikeash 10y ago
      This would certainly explain why they folded so quickly once the NHTSA started asking questions.
    - __jal 10y ago
      ...Which is why it says "YOU ARE RESPONSIBLE FOR COMPLYING WITH LOCAL LAWS AND REGULATIONS".
      - deleted 10y ago
        [deleted]
      - deleted 10y ago
        [deleted]
      - jakevn 10y ago
        Ah yes, laws are definitely the determinate for safety. This is why texting and driving is illegal everywhere.
bdrool 10y ago
> As in research projects, which this project expressly states that it is? This is from comma.ai, who famously tested on public roads with a reporter in the car and very little prior testing ("the first time it worked was this morning").[1] [1] http://www.bloomberg.com/features/2015-george-hotz-self-driving-car/ http://www.bloomberg.com/features/2015-george-hotz-self-driv...

hasbroslasher 10y ago

I'm not really following. If it works then why not use a control loop on top of Python? We are engineers, after all.

throwaway729 10y ago
Because by "works" you mean "I tried it out a bunch of times and nothing bad happened, so must be production ready", because that's what you do when you build websites and desktop CRUD apps and nobody ever died cause the web server choked under load or the web page rendered a bit funny or the request took a full quarter second because the GC kicked off as a wave of requests came in. And then one day a one-in-ten-million event that never would've surfaced during testing does happen and the OS crashes or the interpreter hangs or some non-determinism causes a period of non-responsiveness or a bit gets mangled because you don't have any redundancies and solar rays are thing or ... and someone dies. And of course that didn't happen in testing, because all of those things are rare possibilities. And then it happens again. And again. And then you're in court being sued for millions. And a bunch of industry experts who do build their systems with redundancies and do design their systems with a safety-first mindset come to the stand and rip you apart for not taking even the most basic precautions. And when it comes out just how many best practices you completely ignored, you're mostly just hoping beyond hope that all of the legal problems stay on the civil side of the civil/criminal divide. Or, to say it in a sentence, "because the sort of exceptional circumstances that safety-critical software needs to handle with grace are very difficult or impossible to account for with a desktop machine running interpreted code on top of Linux."
- hasbroslasher 10y ago
  I like that you reduced that to a one-liner, and I understand your point. I sure as hell wouldn't hook my car up to this software. But I also feel like you're being a bit hasty. Obviously a Python script isn't going to turn into a Tesla overnight. But maybe it'll help you find a few bugs before you throw all that time and effort into building the real deal. When I look at a Github repo that claims to be a self-driving car I don't say to myself "yep, looks production ready," I say "Cool prototype, now let's break it." To me it seems entirely reasonable to get a working Python prototype 90% of the way there and then send it off to the OS programmers to design something that actually meets the concept of "production ready".

breck 10y ago

Why not? We entrust humans with operating cars right now. Humans in a sense are running a general purpose OS.

andars 10y ago
This is not relevant. If a corporation builds a product that kills people, the corporation is responsible. No corporation is responsible for the behaviour of humans.
dsr_ 10y ago
If you read enough security literature, eventually you'll come across the notion of a "trusted system". And your first impression will be that this is a system which is fully debugged and tested and you can trust it to do its job within specifications. But you couldn't be more wrong. A trusted system is one which you have to trust. No representations are made about whether you should or not. Humans are frequently trusted systems.
tptacek 10y ago
No hundred humans are running the exact same OS that can deadlock or crash and as a result drive off the road, which is why this trope about humans being worse than computers at driving is entirely irrelevant to a thread about why it's unwise to try to build self-driving cars on top of Python scripts.
- breck 10y ago
  My take of the OP's point was that the deep general stack was unnecessary and decreased reliability. My read was that they think your self driving car software shouldn't also be able to run arbitrary python scripts. My argument is that it's practically irrelevant to reliability. I would not be surprised if in the future you could download an app to your iPhone, connect your phone to your car, and have the app drive your car. Sure, maybe having a specialized real time o.s. without any higher level interpreters could be more reliable in the early days, but the gains from rapid iteration you'd get from allowing cars to be driven by apps running on higher level abstractions would be worth the trade off, and over time the latter could be iterated upon to achieve practically the same reliability as over-engineered systems.

deleted 10y ago

[deleted]

stevendhansen 10y ago

The big problem in my mind is that Python is the wrong tool for the job. It is 2016, there are more hard-RTOS prototyping platforms than ever before. My favorite is National Instrument's RIO platform, which lets you use C or LabVIEW (imho the best language for prototyping control algorithms by far). Mathworks also has a platform based on Matlab/Simulink, and the list goes on. Why use Python when there are existing tools that are made for this type of application?

snovv_crash 10y ago
LabVIEW is great until you need a complicated data structure. I tried writing a tree to do kNN, and it turned out that doing a brute force search was faster even with 100k elements.
veli_joza 10y ago
This is (now) open source project expecting community pull requests. How many would they get if they went with LabView? The big emphasis that I like seeing in modern scientists is reproducibility of results. Jupyter Notebook is a huge step in that direction (not really applicable here), but just using open source platform is still great. I agree though that once algorithms are developed sufficiently they should be ported to RTOS platform and this box shouldn't be permitted on open roads.

mrlambchop 10y ago

It seems incredible that this is the source code for the $999 product originally destined to be available this year. I'm extremely familiar with OBD|CAN and car networks - the embedded code alone is an essay in how trivial something can appear (sending commands to a ECU) without considering the million edge cases that make this a safe product to use.

deleted 10y ago
[deleted]
espes 10y ago
> It seems incredible that this is the source code for the $999 product originally destined to be available this year. It's not
- mrlambchop 10y ago
  Forgive me but it seemed like this was the case. Is this an earlier version of the code or a special opensource clean up then? http://newatlas.com/geohot-comma-ai-openpilot-open-source/46722/ http://newatlas.com/geohot-comma-ai-openpilot-open-source/46...
  - CmdrSprinkles 10y ago
    The ars technical article covers it better http://arstechnica.com/cars/2016/11/after-mothballing-comma-one-george-hotz-releases-free-autonomous-car-software/ http://arstechnica.com/cars/2016/11/after-mothballing-comma-... But in a nutshell: The company was approached by agencies that had severe concerns over safety and if this would have the required regulations in place. So the company folded and released this instead.

KKKKkkkk1 10y ago

Human drivers don't run an RTOS and a deterministic bus system either. In fact, I would argue that very rarely do human drivers fail because of jitter or nondeterminism.

savanaly 10y ago
The more relevant question is not do they but if they could would we want them to?
tptacek 10y ago
This is a weird argument. Human drivers don't deadlock or crash either. Are you sure you understand the criticism being made here? Hard realtime systems make guarantees about the frequency with which code will run on a shared system.
- veli_joza 10y ago
  I agree that this is not production ready tech, but still parent does make a good point. I think those two similar comments are downvoted just because people here don't agree with them. Suppose you make an autopilot in JS running on Electron on WinXP, running on VM on top of Puppy Linux Live CD. And you still manage to prove that your system is 100x more reliable than human driver. Should it be dismissed just because we don't agree with technology stack? The latency of this monstrosity would still be lower than human driver and maybe they would stay lower throughout the operation.
  - terryf 10y ago
    This doesn't have anything to do with latency really. It's about predictability. Given the system you described, there are so many edge cases that can potentially compound, that it is impossible to make any predictability guarantees about the system. As an example, I worked on an embedded system that controlled electrical motors that was hard real time. The fast task time interval was once per millisecond. No matter what, that got called by the RTOS exactly 1000 times per second. When it didn't finish it's job in time, the result could easily wreck real world items or cause harm to people. Nobody even considered using interpreted languages in that project. The fast tasks all had provably run in much less than a millisecond. That means no loops that could be unbounded, no memory allocations, no recursion, no writing to flash, anything that was even slightly unpredictable was out. So, even if you could prove that your system caused less accidents than a human driver when it was running well, it would be impossible to do an analysis that defined under what circumstances the system would be running well. Given that, it would not be allowed in a well-engineered real time or safety critical system.

ma2rten 10y ago

The Python control loop is probably the safest part of this system. The reason that comma.ai was able to get a working prototype quickly (unlike much bigger and well-funded companies) is that they use an end-to-end deep learning model. The model directly learns to issue commands to the car based on raw image data. The problem with this approach is that neural network models are a blackbox. They can behave in completely unpredictable ways.

Animats 10y ago
Actually, they didn't. At least not in this code.