7 ms·

Safety Numbers on Autonomous Cars Don't Add Up

46 points by lemiant 9y ago

protoplant 9y ago
One thing I find healthy, is that many companies beyond Tesla are trying to build this technology. Even though the story thinks the data may be suspect at this point, the competition from a variety of sources gives us good chances for some breakthroughs occurring.
- InclinedPlane 9y ago
  In theory, sure, in practice, it's a little disconcerting. I've worked in software since the '90s, I've seen how the sausage is made at all kinds of companies big and small. The theory of autonomous cars is a bunch of really smart folks hand crafting history's finest software. And there may be some cases where that won't be too far from the truth. But the reality on the street is going to be a zillion different competitors cutting every corner, skirting ever regulation they can get away with, and just shitting out the worst "move fast and break things" hackathon bullshit code that "sort of seems to work, most of the time" that they can manage. I know how devs and product managers think about testing and quality in the absence of dedicated and rigorous QA standards and infrastructure, in the context of life critical systems that is frightening. Ask yourself, do you want to put your life in the hands of a code base that had some pimple faced learned-to-code-in-10-days bootcamp graduate who just "fixed a bug" in the drive software by ctrl-c-ctrl-v'ing from stackoverflow and then pushed to master? Because that is going to be the reality, not the ivory tower "well, if they did it the RIGHT way" fantasy that people have in their heads. The only way we'll get the "right" way of autonomous software development is if there is extensive and careful regulation with very rigorous auditing and process requirements. And we are nowhere near that right now.
  - jessriedel 9y ago
    > The only way we'll get the "right" way of autonomous software development is if there is extensive and careful regulation with very rigorous auditing and process requirements. What makes you think the NHTSA has the expertise or resources to carry this out? The proof will be in the empirical results (accidents/fatalities per mile), not super duper code audits.
  - JumpCrisscross 9y ago
    > the reality on the street is going to be a zillion different competitors cutting every corner, skirting ever regulation they can get away with, and just shitting out the worst "move fast and break things" hackathon bullshit code they can get away with that "sort of seems to work, most of the time" Humans are terrible drivers. A half crap autonomous car might still be safer than the status quo. In any case, whether talking about consumer goods or services, this is a space markets work in. Calling for rules to be written before we fully understand the problem is a recipe for overregulation.
    - InclinedPlane 9y ago
      OK. So what? The key word here is "might" be safer than the status quo. We should ensure that we are measuring carefully and confident in our results (which also requires being confident that the systems are well behaved and well understood and thus the testing is valid) so that if they do show they are safer than humans on average it will be beneficial to move to autonomous driving as much as is practicable. And if, on a case by case basis, that is not true then we can avoid killing more people unnecessarily because we want to save some money on development costs.
    - stryk 9y ago
      > Calling for rules to be written before we fully understand the problem is a recipe for overregulation. I think s/he outlined, at least in essence, one of the (many) problems. This is an area where rigorous QA, test, re-testing, and re-re-testing should be paramount above all else -- including profits, at least at first until we get the software down pat. Do you want to trust your children's lives to some half-baked "ship it!" product? I don't.
      - megy 9y ago
        > This is an area where rigorous QA, test, re-testing, and re-re-testing should be paramount above all Great, when are we going to do this for people?
    - erobbins 9y ago
      > Humans are terrible drivers. Compared to what? We have some evidence that computers can do better in tightly controlled scenarios (limited access freeways) but that's always been low hanging fruit. I think humans are pretty good drivers, actually. How many 2 lane undivided roads do people go zipping down all day, every day, with very few accidents? A ton. We're also very adaptable and good at handling outlier situations.
  - gwenzek 9y ago
    I think most big players (like Google) are conscious that 1) there is a lot of money to make 2) a few deaths in a row will scare the public and the regulators So they are not cutting corners. I'll be more concerned about the companies playing catch up, or the cheap clone that will try to cut on the costly sensor like Lidar and on the long and rigorous process required to develop it correctly. .
    - InclinedPlane 9y ago
      Precisely. And once there is the perception of a gold rush and that others (namely google et al) are already way ahead, then what?
      - dchichkov 9y ago
        I'm guessing that Google had deployed Level 4, built with legacy technology. Very large codebase, strong dependency on a well serviced Lidar, dependency on accurate calibration, dependency on real-time high fidelity maps. A solution scalable to well-managed fleets in good neighborhoods. If this is the case, the others still have a chance at getting to Level 5 in the same time frame as Google. If they play the technology in a right way.
  - carlmr 9y ago
    >The only way we'll get the "right" way of autonomous software development is if there is extensive and careful regulation with very rigorous auditing and process requirements. From what I've seen that doesn't really make a difference either. Companies will follow the regulation on paper, but not in spirit. That's already happening in other heavily regulated areas of software development. My experience has actually been that heavy regulation makes for worse software, because the company starts spending more time on lawyers and "quality engineers" driving up meaningless metrics than maintainable clean code.
    - TeMPOraL 9y ago
      So basically, it's too late. Maybe regulators should have been preemptively putting barriers in front of commercializing (then) sci-fi level tech. When there's no competitive pressure and rushing to be first to market, when it's just university people on DARPA funding developing the tech, you can afford to have it done The Right Way. Now that people are excited and build companies around self-driving, we'll only get Worse is Better.
  - TeMPOraL 9y ago
    > Because that is going to be the reality, not the ivory tower "well, if they did it the RIGHT way" fantasy that people have in their heads. And if someone doubts that, remember that's what always happened in computing. That's how Lisp Machines and Smalltalk systems lost to UNIX. That's how we got C and JavaScript. Worse is better. The rule of computing under competitive, market-driven environments. I dread to see it applied to life-critical systems.
Isamu 9y ago
If nothing else, this article provided a link to the California vehicle code that covers autonomous vehicles. That is an interesting read. Back to the article, they point out impatient overrides by the test driver are apparently not being counted as disengagements and do not appear on the reports to the DMV. The author tries to make this into a safety concern, but it just isn't. Maybe it is against the intent of the code as written, but maybe not, since it seems to be concerned with true safety violations. The code: https://www.dmv.ca.gov/portal/wcm/connect/d48f347b-8815-458e-9df2-5ded9f208e9e/adopted_txt.pdf?MOD=AJPERES https://www.dmv.ca.gov/portal/wcm/connect/d48f347b-8815-458e...
- sgillen 9y ago
  I agree with the idea that an overide because the car has been stopped for too long is less dangerous than a more active sort of overide. But I think it still is a safety issue. I’m mostly thinking of when we take the human out of the equation completely, if the car is stopped indefinitely that is going to cause all sorts of problems. If it’s blocking a city street you’re creating traffic as well as potentially slowing down first responders. Being stopped on a highway is very much a safety hazard. Even with a human driver to take control there are places where coming to a stop would be a very bad thing (an on or off ramp for example).
  - asynchronous13 9y ago
    > if the car is stopped indefinitely that is going to cause all sorts of problems. Your comment is true, but consider the context. There was a vehicle already blocking the lane that caused the autonomous car to stop. The autonomous car did not block an unobstructed lane, the lane was already blocked.
    - khc 9y ago
      I've seen Cruise's cars braking for no reason, including once at the intersection (there's nothing in front of the car) I couldn't tell if it figured out it's safe to proceed on its own or if someone took over. It's also possible that the human drivers were doing the braking but if that's the case they clearly need better safety drivers
carapace 9y ago
Do you think self-driving car[0] manufacturers should be allowed to compete on safety, in the sense that if one comes up with some sort of software "seatbelt" of some kind should that company be allowed to prevent others from using it? I should go look up how actual seatbelts and airbags became standard in (almost) all cars... [0] Can we please call them "auto-autos"?
- brainfish 9y ago
  In the case of the 3-point seatbelt, a Volvo engineer invented it and Volvo received the patent, then made it available to other manufacturers at no cost.[0] While that is a great precedent I don't believe it's any kind of requirement. [0] https://en.wikipedia.org/wiki/Seat_belt https://en.wikipedia.org/wiki/Seat_belt
  - Johnny555 9y ago
    Making it a requirement would be a double-edged sword -- on the one hand, companies would share safety features, but on the other hand, they'd have much less incentive to develop new safety features if they had to give them to competitors at no cost. Perhaps some sort of compulsory licensing would be better.
sidibe 9y ago
There is definitely something that doesn't add up between the recent surge in pessimism about the progress on self driving cars and the fact that Waymo is about to start taking passengers without backup drivers. Either they are recklessly competitive or confident that the cars can handle any situations they might run into in the area they're releasing them. If it were some independent startup with less to lose I'd lean more to the former, but theyre not.
- tyingq 9y ago
  Hard to say. Waymo has way more miles than everyone else, but the layman's observation is "yeah, but in environments you control with an iron hand and travel over and over on the same rough, predictable routes, albeit with your very rich suite of data points." A bazillion miles on the same track isn't necessarily as impressive as a magnitude fewer miles on varied uncontrolled/unmapped snowy, icy, narrow, unlit, flooded, unplanned etc, routes. So, how do I compare Google's/Waymo's vast number of sensor rich planned miles versus Tesla's much lower, much less rich (no lidar) but much more diverse and real-world miles? And everyone else's (Uber/Lyft) in-between play? It's pretty unclear, for a layperson, right now, who is ahead of the pack.
  - lambdasquirrel 9y ago
    Do we need driverless to handle all the corner cases? I’m starting to think this is just the wrong way to look at it. Most of the time we commute home it’s just the same dumb stuff. We can make an analogy to Starcraft AI-assisted here. What if instead of having a fully automated AI, instead you could train your car to take you home, by driving your route a couple of times? Kind of like how you might train the computer to go harass your opponents expansion and then just call up that subroutine everytime that it’s apropos (still waiting on Blizzard to make this game). Don’t try so hard to replace the human. Make the human less busy, more powerful.
    - tyingq 9y ago
      Interesting perspective, and your view seems to favor Tesla. Honestly, I just don't know, and expert opinions seem to vary right now. It's hard to remember a time when it was harder to see the ultimate winner in such an important space.
evancox100 9y ago
"Because the Cruise cars were still pondering their next move when the driver took over, these incidents apparently do not constitute failures that must be reported to the DMV. Though a neat piece of legalism, this logic can't help but make one wonder how long a vehicle can remain motionless on public roads without it constituting a failure of the autonomous technology." I know! We'll just add a program that checks whether or not the AI would ever stop and come to a decision.
- ThePadawan 9y ago
  You could say that there is an obvious Halting Problem which needs solving.
- jrodthree24 9y ago
  You could even hire a freelancer to do it. I guarantee you will find someone that can promise to do it at a great price.

46 points by lemiant 9y ago

One thing I find healthy, is that many companies beyond Tesla are trying to build this technology. Even though the story thinks the data may be suspect at this point, the competition from a variety of sources gives us good chances for some breakthroughs occurring.

InclinedPlane 9y ago
In theory, sure, in practice, it's a little disconcerting. I've worked in software since the '90s, I've seen how the sausage is made at all kinds of companies big and small. The theory of autonomous cars is a bunch of really smart folks hand crafting history's finest software. And there may be some cases where that won't be too far from the truth. But the reality on the street is going to be a zillion different competitors cutting every corner, skirting ever regulation they can get away with, and just shitting out the worst "move fast and break things" hackathon bullshit code that "sort of seems to work, most of the time" that they can manage. I know how devs and product managers think about testing and quality in the absence of dedicated and rigorous QA standards and infrastructure, in the context of life critical systems that is frightening. Ask yourself, do you want to put your life in the hands of a code base that had some pimple faced learned-to-code-in-10-days bootcamp graduate who just "fixed a bug" in the drive software by ctrl-c-ctrl-v'ing from stackoverflow and then pushed to master? Because that is going to be the reality, not the ivory tower "well, if they did it the RIGHT way" fantasy that people have in their heads. The only way we'll get the "right" way of autonomous software development is if there is extensive and careful regulation with very rigorous auditing and process requirements. And we are nowhere near that right now.
- jessriedel 9y ago
  > The only way we'll get the "right" way of autonomous software development is if there is extensive and careful regulation with very rigorous auditing and process requirements. What makes you think the NHTSA has the expertise or resources to carry this out? The proof will be in the empirical results (accidents/fatalities per mile), not super duper code audits.
- JumpCrisscross 9y ago
  > the reality on the street is going to be a zillion different competitors cutting every corner, skirting ever regulation they can get away with, and just shitting out the worst "move fast and break things" hackathon bullshit code they can get away with that "sort of seems to work, most of the time" Humans are terrible drivers. A half crap autonomous car might still be safer than the status quo. In any case, whether talking about consumer goods or services, this is a space markets work in. Calling for rules to be written before we fully understand the problem is a recipe for overregulation.
  - InclinedPlane 9y ago
    OK. So what? The key word here is "might" be safer than the status quo. We should ensure that we are measuring carefully and confident in our results (which also requires being confident that the systems are well behaved and well understood and thus the testing is valid) so that if they do show they are safer than humans on average it will be beneficial to move to autonomous driving as much as is practicable. And if, on a case by case basis, that is not true then we can avoid killing more people unnecessarily because we want to save some money on development costs.
  - stryk 9y ago
    > Calling for rules to be written before we fully understand the problem is a recipe for overregulation. I think s/he outlined, at least in essence, one of the (many) problems. This is an area where rigorous QA, test, re-testing, and re-re-testing should be paramount above all else -- including profits, at least at first until we get the software down pat. Do you want to trust your children's lives to some half-baked "ship it!" product? I don't.
    - megy 9y ago
      > This is an area where rigorous QA, test, re-testing, and re-re-testing should be paramount above all Great, when are we going to do this for people?
  - erobbins 9y ago
    > Humans are terrible drivers. Compared to what? We have some evidence that computers can do better in tightly controlled scenarios (limited access freeways) but that's always been low hanging fruit. I think humans are pretty good drivers, actually. How many 2 lane undivided roads do people go zipping down all day, every day, with very few accidents? A ton. We're also very adaptable and good at handling outlier situations.
- gwenzek 9y ago
  I think most big players (like Google) are conscious that 1) there is a lot of money to make 2) a few deaths in a row will scare the public and the regulators So they are not cutting corners. I'll be more concerned about the companies playing catch up, or the cheap clone that will try to cut on the costly sensor like Lidar and on the long and rigorous process required to develop it correctly. .
  - InclinedPlane 9y ago
    Precisely. And once there is the perception of a gold rush and that others (namely google et al) are already way ahead, then what?
    - dchichkov 9y ago
      I'm guessing that Google had deployed Level 4, built with legacy technology. Very large codebase, strong dependency on a well serviced Lidar, dependency on accurate calibration, dependency on real-time high fidelity maps. A solution scalable to well-managed fleets in good neighborhoods. If this is the case, the others still have a chance at getting to Level 5 in the same time frame as Google. If they play the technology in a right way.
- carlmr 9y ago
  >The only way we'll get the "right" way of autonomous software development is if there is extensive and careful regulation with very rigorous auditing and process requirements. From what I've seen that doesn't really make a difference either. Companies will follow the regulation on paper, but not in spirit. That's already happening in other heavily regulated areas of software development. My experience has actually been that heavy regulation makes for worse software, because the company starts spending more time on lawyers and "quality engineers" driving up meaningless metrics than maintainable clean code.
  - TeMPOraL 9y ago
    So basically, it's too late. Maybe regulators should have been preemptively putting barriers in front of commercializing (then) sci-fi level tech. When there's no competitive pressure and rushing to be first to market, when it's just university people on DARPA funding developing the tech, you can afford to have it done The Right Way. Now that people are excited and build companies around self-driving, we'll only get Worse is Better.
- TeMPOraL 9y ago
  > Because that is going to be the reality, not the ivory tower "well, if they did it the RIGHT way" fantasy that people have in their heads. And if someone doubts that, remember that's what always happened in computing. That's how Lisp Machines and Smalltalk systems lost to UNIX. That's how we got C and JavaScript. Worse is better. The rule of computing under competitive, market-driven environments. I dread to see it applied to life-critical systems.

Isamu 9y ago

If nothing else, this article provided a link to the California vehicle code that covers autonomous vehicles. That is an interesting read. Back to the article, they point out impatient overrides by the test driver are apparently not being counted as disengagements and do not appear on the reports to the DMV. The author tries to make this into a safety concern, but it just isn't. Maybe it is against the intent of the code as written, but maybe not, since it seems to be concerned with true safety violations. The code: https://www.dmv.ca.gov/portal/wcm/connect/d48f347b-8815-458e-9df2-5ded9f208e9e/adopted_txt.pdf?MOD=AJPERES https://www.dmv.ca.gov/portal/wcm/connect/d48f347b-8815-458e...

sgillen 9y ago
I agree with the idea that an overide because the car has been stopped for too long is less dangerous than a more active sort of overide. But I think it still is a safety issue. I’m mostly thinking of when we take the human out of the equation completely, if the car is stopped indefinitely that is going to cause all sorts of problems. If it’s blocking a city street you’re creating traffic as well as potentially slowing down first responders. Being stopped on a highway is very much a safety hazard. Even with a human driver to take control there are places where coming to a stop would be a very bad thing (an on or off ramp for example).
- asynchronous13 9y ago
  > if the car is stopped indefinitely that is going to cause all sorts of problems. Your comment is true, but consider the context. There was a vehicle already blocking the lane that caused the autonomous car to stop. The autonomous car did not block an unobstructed lane, the lane was already blocked.
  - khc 9y ago
    I've seen Cruise's cars braking for no reason, including once at the intersection (there's nothing in front of the car) I couldn't tell if it figured out it's safe to proceed on its own or if someone took over. It's also possible that the human drivers were doing the braking but if that's the case they clearly need better safety drivers

carapace 9y ago

Do you think self-driving car[0] manufacturers should be allowed to compete on safety, in the sense that if one comes up with some sort of software "seatbelt" of some kind should that company be allowed to prevent others from using it? I should go look up how actual seatbelts and airbags became standard in (almost) all cars... [0] Can we please call them "auto-autos"?

brainfish 9y ago
In the case of the 3-point seatbelt, a Volvo engineer invented it and Volvo received the patent, then made it available to other manufacturers at no cost.[0] While that is a great precedent I don't believe it's any kind of requirement. [0] https://en.wikipedia.org/wiki/Seat_belt https://en.wikipedia.org/wiki/Seat_belt
- Johnny555 9y ago
  Making it a requirement would be a double-edged sword -- on the one hand, companies would share safety features, but on the other hand, they'd have much less incentive to develop new safety features if they had to give them to competitors at no cost. Perhaps some sort of compulsory licensing would be better.

sidibe 9y ago

There is definitely something that doesn't add up between the recent surge in pessimism about the progress on self driving cars and the fact that Waymo is about to start taking passengers without backup drivers. Either they are recklessly competitive or confident that the cars can handle any situations they might run into in the area they're releasing them. If it were some independent startup with less to lose I'd lean more to the former, but theyre not.

tyingq 9y ago
Hard to say. Waymo has way more miles than everyone else, but the layman's observation is "yeah, but in environments you control with an iron hand and travel over and over on the same rough, predictable routes, albeit with your very rich suite of data points." A bazillion miles on the same track isn't necessarily as impressive as a magnitude fewer miles on varied uncontrolled/unmapped snowy, icy, narrow, unlit, flooded, unplanned etc, routes. So, how do I compare Google's/Waymo's vast number of sensor rich planned miles versus Tesla's much lower, much less rich (no lidar) but much more diverse and real-world miles? And everyone else's (Uber/Lyft) in-between play? It's pretty unclear, for a layperson, right now, who is ahead of the pack.
- lambdasquirrel 9y ago
  Do we need driverless to handle all the corner cases? I’m starting to think this is just the wrong way to look at it. Most of the time we commute home it’s just the same dumb stuff. We can make an analogy to Starcraft AI-assisted here. What if instead of having a fully automated AI, instead you could train your car to take you home, by driving your route a couple of times? Kind of like how you might train the computer to go harass your opponents expansion and then just call up that subroutine everytime that it’s apropos (still waiting on Blizzard to make this game). Don’t try so hard to replace the human. Make the human less busy, more powerful.
  - tyingq 9y ago
    Interesting perspective, and your view seems to favor Tesla. Honestly, I just don't know, and expert opinions seem to vary right now. It's hard to remember a time when it was harder to see the ultimate winner in such an important space.

evancox100 9y ago

"Because the Cruise cars were still pondering their next move when the driver took over, these incidents apparently do not constitute failures that must be reported to the DMV. Though a neat piece of legalism, this logic can't help but make one wonder how long a vehicle can remain motionless on public roads without it constituting a failure of the autonomous technology." I know! We'll just add a program that checks whether or not the AI would ever stop and come to a decision.

ThePadawan 9y ago
You could say that there is an obvious Halting Problem which needs solving.
jrodthree24 9y ago
You could even hire a freelancer to do it. I guarantee you will find someone that can promise to do it at a great price.