7 ms·

Are we really ready for self-driving cars? https://www.youtube.com/watch?v=4G1Boh-URIM https://www.youtube.com/watch?v=4G1Boh-URIM

deleted 9y ago
[deleted]
ky738 9y ago
I will take malicious improper analogy for 100
- michaelmcmillan 9y ago
  Please point out the discrepancy. A Tesla has ~ 100.000.000 [1] lines of code. Considering this post, do you think we are sufficiently educated in software security to produce secure self-driving cars? Elon Musk: "I think one of the biggest risks for autonomous vehicles is somebody achieving a fleet wide hack" [2]. [1] https://bit.ly/KIB_linescode https://bit.ly/KIB_linescode [2] https://www.youtube.com/watch?v=4G1Boh-URIM https://www.youtube.com/watch?v=4G1Boh-URIM
  - abestic9 9y ago
    These companies have completely different operating systems, network ACLs, software update policies and subsystems that affect certain mechanical features. By your logic, we should not fly any modern commercial or military aircraft or spacecraft, live within a certain radius of any power or hazardous chemical plant, place any dependency on any first world country's health care network, including life support, or invest in any company or stock. Like most things in life it comes down to a security/convenience risk/benefit compromise.
    - michaelmcmillan 9y ago
      > These companies have completely different operating systems, network ACLs, software update policies and subsystems that affect certain mechanical features. Are you claiming that this could not have happened with Tesla? If so, please explain why. > By your logic, we should not fly any modern commercial or military aircraft or spacecraft, live within a certain radius of any power or hazardous chemical plant, place any dependency on any first world country's health care network, including life support, or invest in any company or stock. Up until now the benefits have clearly outweighed the risks, but that does not mean it will continue to do so.
  - deleted 9y ago
    [deleted]
  - jessriedel 9y ago
    This is a much more interesting comment than your initial quip. Next time, consider leading with this rather than unnecessarily antagonizing people.
  - mikeash 9y ago
    How much of that code is safety critical? I occasionally see misbehavior from my Tesla's center screen, like the network connection failing, or audio glitches, or even the occasional spontaneous reboot. This can be mildly annoying but it doesn't worry me because I know that the center screen is separate from the stuff where bugs can actually get me killed.
    - michaelmcmillan 9y ago
      "On Thursday October 24, 2013, an Oklahoma court ruled against Toyota in a case of unintended acceleration that lead to the death of one the occupants. Central to the trial was the Engine Control Module's (ECM) firmware. Embedded software used to be low-level code we'd bang together using C or assembler. These days, even a relatively straightforward, albeit critical, task like throttle control is likely to use a sophisticated RTOS and tens of thousands of lines of code. " [1] [2] [1] https://www.edn.com/design/automotive/4423428/Toyota-s-killer-firmware--Bad-design-and-its-consequences https://www.edn.com/design/automotive/4423428/Toyota-s-kille... [2] https://www.embedded.com/electronics-blogs/barr-code/4214602/Unintended-acceleration-and-other-embedded-software-bugs https://www.embedded.com/electronics-blogs/barr-code/4214602...
      - mikeash 9y ago
        Sure. My point is just that you can't bring up the total number of lines of code here, because most of those lines aren't in any way related to any safety-critical system. If you want to talk about how much code there is which puts you at risk, you need to look at that particular subset of code.
        michaelmcmillan 9y ago
        I think it is safe to assume that complexity is highly correlated with number of lines of code. And with complexity grows bugs. On the surface, this particular exploit on macOS seems to be caused by a totally unrelated subsystem (security vs. GUI).
        9y ago
- deleted 9y ago
  [deleted]
MentallyRetired 9y ago
As a programmer, the thought terrifies me.
- glhaynes 9y ago
  As someone who tries to do risk analysis, the prospect of sticking with human drivers because of fear of software bugs (which inevitably will kill, just in much smaller numbers) terrifies me.
  - jtbayly 9y ago
    The fear is not of bugs killing people. The fear is of bugs allowing people to kill people. If ISIS was able to hack a major fleet through one such bug, do you think for a single moment they wouldn't make use of it to kill many people?
    - ovao 9y ago
      Which is a legitimate fear, and substantial effort should go into preventing such bugs, but any sufficiently determined person doesn't need to exploit a software bug to be able to kill others. ISIS appears to be quite effective at simply convincing its members to directly and voluntarily engage themselves in such acts.
      - jtbayly 9y ago
        Yeah, but we're dealing with an order of magnitude difference if we give them control over > 1Million mobile bombs (ie 1 large self-driving car network).
        ovao 9y ago
        Unless it was somehow possible to remotely trigger a battery explosion, no, we would not be giving anyone control over “mobile bombs”.
- qrbLPHiKpiux 9y ago
deleted 9y ago
[deleted]
zerostar07 9y ago
technically, after they are hacked they are no longer self-driving.
crispinb 9y ago
Possibly not, but the death & maiming stats everywhere show we're absolutely not ready for human-driven ones.
- carapace 9y ago
  Thank you I imagine a Twilight Zone episode... Go back in time to before cars were invented and imagine some Mephistopheles offering the bargain: "You'll fly like the wind over hills and mountains, making a journey of days in mere hours!" What the catch? "For each mile traveled a certain number of people chosen at random must be put to death or maimed." He would go on about how the chances of someone you love being chosen for sacrifice were infinitesimal, and the benefits to all were so great and so obvious... ("Also, it will poison the air and water, and force you to become dependent on fuel sources that destroy life and engender wars.")
  - crispinb 9y ago
    Well to be fair there are more positives to cars than just cutting journey times. But I agree the negatives are massing hugely as the world's driving population pullulates. As for human readiness to safely control a tonne of speeding metal, my position as a full-time motorcyclist makes me extremely confident that the average alleged 'driver' (actually: daydreamer, snot-picker, instagrammer) isn't even approaching the edge of the competence ballpark.
Oxitendwe 9y ago
No. I am also skeptical of more modern cars with complex computers. The thought of being in one scares me.
ovao 9y ago
I think the question you're fundamentally asking is "are we ready for imperfect systems with potential vulnerabilities?", and the answer to that question has been the same since the advent of software.
therealdrag0 9y ago
They don't need to be perfect. Just better than humans.
naasking 9y ago
> Are we really ready for self-driving cars? Firstly, car automation is machine learning, not programming. Completely incomparable. Finally, we've known how to prove the absence of bugs for decades. It's not a matter of not knowing how, it's about incentives to do it right.
- michaelmcmillan 9y ago
  Is the authentication subsystem in a Tesla machine learning as well?

deleted 9y ago

[deleted]

ky738 9y ago

I will take malicious improper analogy for 100

michaelmcmillan 9y ago
Please point out the discrepancy. A Tesla has ~ 100.000.000 [1] lines of code. Considering this post, do you think we are sufficiently educated in software security to produce secure self-driving cars? Elon Musk: "I think one of the biggest risks for autonomous vehicles is somebody achieving a fleet wide hack" [2]. [1] https://bit.ly/KIB_linescode https://bit.ly/KIB_linescode [2] https://www.youtube.com/watch?v=4G1Boh-URIM https://www.youtube.com/watch?v=4G1Boh-URIM
- abestic9 9y ago
  These companies have completely different operating systems, network ACLs, software update policies and subsystems that affect certain mechanical features. By your logic, we should not fly any modern commercial or military aircraft or spacecraft, live within a certain radius of any power or hazardous chemical plant, place any dependency on any first world country's health care network, including life support, or invest in any company or stock. Like most things in life it comes down to a security/convenience risk/benefit compromise.
  - michaelmcmillan 9y ago
    > These companies have completely different operating systems, network ACLs, software update policies and subsystems that affect certain mechanical features. Are you claiming that this could not have happened with Tesla? If so, please explain why. > By your logic, we should not fly any modern commercial or military aircraft or spacecraft, live within a certain radius of any power or hazardous chemical plant, place any dependency on any first world country's health care network, including life support, or invest in any company or stock. Up until now the benefits have clearly outweighed the risks, but that does not mean it will continue to do so.
- deleted 9y ago
  [deleted]
- jessriedel 9y ago
  This is a much more interesting comment than your initial quip. Next time, consider leading with this rather than unnecessarily antagonizing people.
- mikeash 9y ago
  How much of that code is safety critical? I occasionally see misbehavior from my Tesla's center screen, like the network connection failing, or audio glitches, or even the occasional spontaneous reboot. This can be mildly annoying but it doesn't worry me because I know that the center screen is separate from the stuff where bugs can actually get me killed.
  - michaelmcmillan 9y ago
    "On Thursday October 24, 2013, an Oklahoma court ruled against Toyota in a case of unintended acceleration that lead to the death of one the occupants. Central to the trial was the Engine Control Module's (ECM) firmware. Embedded software used to be low-level code we'd bang together using C or assembler. These days, even a relatively straightforward, albeit critical, task like throttle control is likely to use a sophisticated RTOS and tens of thousands of lines of code. " [1] [2] [1] https://www.edn.com/design/automotive/4423428/Toyota-s-killer-firmware--Bad-design-and-its-consequences https://www.edn.com/design/automotive/4423428/Toyota-s-kille... [2] https://www.embedded.com/electronics-blogs/barr-code/4214602/Unintended-acceleration-and-other-embedded-software-bugs https://www.embedded.com/electronics-blogs/barr-code/4214602...
    - mikeash 9y ago
      Sure. My point is just that you can't bring up the total number of lines of code here, because most of those lines aren't in any way related to any safety-critical system. If you want to talk about how much code there is which puts you at risk, you need to look at that particular subset of code.
      - michaelmcmillan 9y ago
        I think it is safe to assume that complexity is highly correlated with number of lines of code. And with complexity grows bugs. On the surface, this particular exploit on macOS seems to be caused by a totally unrelated subsystem (security vs. GUI).
        9y ago
deleted 9y ago
[deleted]

MentallyRetired 9y ago

As a programmer, the thought terrifies me.

glhaynes 9y ago
As someone who tries to do risk analysis, the prospect of sticking with human drivers because of fear of software bugs (which inevitably will kill, just in much smaller numbers) terrifies me.
- jtbayly 9y ago
  The fear is not of bugs killing people. The fear is of bugs allowing people to kill people. If ISIS was able to hack a major fleet through one such bug, do you think for a single moment they wouldn't make use of it to kill many people?
  - ovao 9y ago
    Which is a legitimate fear, and substantial effort should go into preventing such bugs, but any sufficiently determined person doesn't need to exploit a software bug to be able to kill others. ISIS appears to be quite effective at simply convincing its members to directly and voluntarily engage themselves in such acts.
    - jtbayly 9y ago
      Yeah, but we're dealing with an order of magnitude difference if we give them control over > 1Million mobile bombs (ie 1 large self-driving car network).
      - ovao 9y ago
        Unless it was somehow possible to remotely trigger a battery explosion, no, we would not be giving anyone control over “mobile bombs”.
qrbLPHiKpiux 9y ago

deleted 9y ago

[deleted]

zerostar07 9y ago

technically, after they are hacked they are no longer self-driving.

crispinb 9y ago

Possibly not, but the death & maiming stats everywhere show we're absolutely not ready for human-driven ones.

carapace 9y ago
Thank you I imagine a Twilight Zone episode... Go back in time to before cars were invented and imagine some Mephistopheles offering the bargain: "You'll fly like the wind over hills and mountains, making a journey of days in mere hours!" What the catch? "For each mile traveled a certain number of people chosen at random must be put to death or maimed." He would go on about how the chances of someone you love being chosen for sacrifice were infinitesimal, and the benefits to all were so great and so obvious... ("Also, it will poison the air and water, and force you to become dependent on fuel sources that destroy life and engender wars.")
- crispinb 9y ago
  Well to be fair there are more positives to cars than just cutting journey times. But I agree the negatives are massing hugely as the world's driving population pullulates. As for human readiness to safely control a tonne of speeding metal, my position as a full-time motorcyclist makes me extremely confident that the average alleged 'driver' (actually: daydreamer, snot-picker, instagrammer) isn't even approaching the edge of the competence ballpark.

Oxitendwe 9y ago

No. I am also skeptical of more modern cars with complex computers. The thought of being in one scares me.

ovao 9y ago

I think the question you're fundamentally asking is "are we ready for imperfect systems with potential vulnerabilities?", and the answer to that question has been the same since the advent of software.

therealdrag0 9y ago

They don't need to be perfect. Just better than humans.

naasking 9y ago

> Are we really ready for self-driving cars? Firstly, car automation is machine learning, not programming. Completely incomparable. Finally, we've known how to prove the absence of bugs for decades. It's not a matter of not knowing how, it's about incentives to do it right.

michaelmcmillan 9y ago
Is the authentication subsystem in a Tesla machine learning as well?