6 ms·

Self-driving cars will never be a reality if the standard for success is zero deaths during the early stages of development. It just won't happen. The technolog

Self-driving cars will never be a reality if the standard for success is zero deaths during the early stages of development. It just won't happen. The technology won't ever be good enough if it's not tested in the real world. Furthermore, no self-driving system will be perfect even in a mature stage of development. People will still die, and they may die because of mistakes a human driver would not have made. If society isn't willing to accept that sacrifice, then let's just ban self-driving cars. But let's not try to have it both ways -- let's not allow self-driving while also holding the view that risk of death is unacceptable.

admax88qqq 5y ago
We don't have to ban self driving cars. We just set the success criteria at zero deaths. If companies want to spend money pursuing it believing they can hit that goal that's on them.
- twblalock 5y ago
  > We don't have to ban self driving cars. We just set the success criteria at zero deaths. That's equivalent to a ban.
  - rhizome 5y ago
    Because the technology is a non-starter and will never work unless accepted terms of public safety are redefined in its favor. "Just draw a bigger bullseye!"
  - Karrot_Kream 5y ago
    There's multiple alternatives to driver-driven cars that result in many fewer excess deaths. They've been trialed and deployed in many countries very successfully. You may know of them by common names such as "buses", "streetcars", "trains", "metros", or even "people movers". There's nothing wrong with innovation but we do not need to put self-driving technology on a pedestal when there are alternatives that exist today that move people around that have none of these externalities. If entities want to innovate, let them innovate within safety parameters.
    - thereisnospork 5y ago
      We could also set the speed limit to 5mph and limit cars to 10hp. That would save a lot of lives. The reason we don't is because some deaths are worth the cost. >If entities want to innovate, let them innovate within safety parameters. Sure, but the safety parameters have already been established[0] at 40,000 deaths a year, or less, for transportation equivalently convenient and functional as compared to a manually-driven car. [0]By the crowd-sourced wisdom of democracy.
      - Karrot_Kream 5y ago
        > We could also set the speed limit to 5mph and limit cars to 10hp. That would save a lot of lives. The reason we don't is because some deaths are worth the cost. Are you saying that, despite transit alternatives that do not lead to excess deaths, that deaths due to self-driving cars is worth the cost? We have existing solutions that don't result in these excess deaths, but because of some reason, we don't want those solutions _and_ we want to kill more people just to birth this technology? _Why_? > [0]By the crowd-sourced wisdom of democracy. ... Which bill are you talking about? I don't remember my Rep or Senator voting on allowing me to be a casualty of the self-driving industry. The fact that automobiles in the US result in 40k deaths is itself a travesty. Even Canada is doing better than the US here. Pointing to the worst driving safety record in the G8 and one of the worst in the G20 isn't exactly proof positive that the American automobile system is safe, it's just proof that we have grandfathered an extremely unsafe system here.
        twblalock 5y ago
        It seems like what you really want is to ban cars and force everyone to use public transit. That's just not realistic, not even in Europe.
        6510 5y ago
        That seems like a much to fun thought experiment to just do away with. Why wouldn't it be realistic?
  - kyleee 5y ago
    of course it's not, it just increases expense, but that should be offset by the potential reward of a true breakthrough. it will change the innovation path yes but it's not a ban
    - twblalock 5y ago
      Impossible standards are equivalent to bans, and zero deaths is an impossible standard. All of the companies that are testing self-driving cars have had accidents with other cars and pedestrians. Some of them resulted in death. Others did not, but easily could have, and eventually their luck will run out and they will also cause deaths. Who would invest millions of dollars into developing a technology when a single mistake could flush all that money down the drain?
- standyro 5y ago
  We have around 38,000 deaths a year on US roads. We already accept significant risk to maintain the status quo. Sensible legislation is needed to balance innovation and safety for self driving cars on public roads, but not zero. We don’t even accept zero pedestrian hit and run deaths yet… California, Texas, and Michigan legislators have already bought in to the benefits that these companies will bring.
- josephcsible 5y ago
  Zero deaths is a completely unreasonable standard. I literally can't think of a single thing at all that's never contributed to anyone's death.
marricks 5y ago
You are introducing a ridiculous binary. There are innumerable ways to test this software, even on public roads, without unleashing it on the thousands of Tesla owners. It's not like Tesla has to release new features every year, they just feel they need to to continue to pump up and justify their stock price. EDIT: I misread the parent comment, I’m not opposed to just banning self driving cars as self driving cars likely won't make everyday folks' life better.
- jacquesm 5y ago
  I'm fine with them unleashing it on Tesla owners, they at least have a choice in the matter. I'm not fine with unleashing it on everybody else too.
  - judge2020 5y ago
    Driving on public streets is an exercise in choreography, so they're the same thing.
    - jacquesm 5y ago
      No they are not. Low quality software in control of a two ton chunk of iron moving at 35 mph through traffic including pedestrian and bycicles has nothing to do with choreography, that bike was about to be hit from behind because of a vehicle steering into it.
      - judge2020 5y ago
        Choreography in terms of doing things in concert with other road users without classing, including vulnerable road users (pedestrians, bicyclists). My point is that unleashing it on Tesla owners is obviously going to involve interacting with other road users. Just because this instance was regarding a bicyclist doesn't mean it's only a danger to bicyclists, there are still ways for people to die if the car were only interacting with other Teslas.
andbberger 5y ago
the other companies seem to have no problem following the rules and not killing people. I'm more of the mind that cars should become the second-class citizens they ought to be in cities and that I should be able to bike around the city without constant threat of death, that dream remains far off in american cities. Why is that fire codes and aviation regulations are written in blood, but traffic violence is treated as a normal, necessary cost of modern life? When a biker is killed by a driver who wasn't signaling and cut them off on a right turn (as happens to me on a daily basis in SF), that should trigger immediate redesigns of intersections and streets across the city, for that was an entirely preventable death. But nope. Go ahead and test your fundamentally flawed, unproven death machine on public roads. I gotta say, I am really not that keen on being killed by a white tesla.
- twblalock 5y ago
  > the other companies seem to have no problem following the rules and not killing people. What rules did Tesla break? Also, remember that time Waymo killed a pedestrian?
  - andbberger 5y ago
    that this is not explicitly illegal is the problem. everyone else has safety drivers and is subject to reporting requirements. waymo never killed a pedestrian
    - deleted 5y ago
      [deleted]
    - twblalock 5y ago
      > waymo never killed a pedestrian Yeah, it was Uber, my mistake. But that's an example another self-driving company that killed people. It could have easily been any of them -- they have all had accidents that could have resulted in death, and just got lucky. Tesla is not special here.
      - 5y ago
serioussecurity 5y ago
But there are competent folks in the field... ? Waymo's easily five years ahead of Tesla. Previously: https://news.ycombinator.com/item?id=30182141 https://news.ycombinator.com/item?id=30182141
Veserv 5y ago
The standard for success is not zero deaths, but it should at least be a system no more dangerous than human drivers. Generally speaking, the average human driver causes a “ding” every 100k miles, a insurance worthy event every 250k miles, a police worthy event every 500k miles, and a fatality every 60M miles. Assuming an average driving speed of 60 mph, which is obviously an overestimation, those are 1.6k, 4k, 8k, and 1M hours respectively. Evaluating a L2 autonomy system is somewhat challenging since in truth you are evaluating a human+autonomous system, so you should expect the overall safety to increase unless the autonomy system is actively making things more dangerous. However, for a L4/L5 autonomy system, which is Tesla’s goal, we can reasonably approximate the quality by miles per intervention as an intervention is a failure of the autonomy. As a reasonable baseline based on video evidence of FSD, we can assume that each intervention prevents at least a “ding”. Therefore, we can evaluate the quality of FSD relative to a minimal L4 system as the ratio of time per disengagement versus 1.6k hours. Using those same videos as evidence, we see the FSD system requiring intervention every few minutes, i.e. <10 minutes, on average. That is 1/10,000 of the minimal quality expected from a fully functional L4 system. So, for Tesla to create a minimal acceptable L4 autonomy solution that would actually usher in a age of safe self driving systems, they need to improve the core functionality of their beta product by 10,000x. It is that gap between what they are delivering and what they still need to achieve that is outrageous.
- jquery 5y ago
  Very well said. You’ve put into words what my intuition has been screaming at me but has been unable to properly formulate. Thank you.
- jimmydorry 5y ago
  In this stage of the product development the rate of intervention is not, and likely won't be for a long time, a good measure of failure. We likely won't see full L4 systems for a very long time. In the mean time, we could start to get a better handle on the safety profile of these systems by better classifying the interventions. Something like these: * Intervention due to unreadable conditions (e.g. unpainted lines, a new area with no training data, failure of LIDAR / camera, w/e) * Human intervention due to not feeling safe * Intervention due to unsafe condition (e.g. imminent collision, or just hit something) * Human intervention for other reason (e.g. human is bored, or human changing destination, or human wanting to take another route) --- #1 could certainly lead to #2 and #3, but is more of an indicator that the car is in novel conditions not trained for, or has hardware failure. #1 is the low hanging fruit (relatively speaking), and #2 is where the majority of the work lies. #3 is where the system has actually failed. And #4 need to just be discarded.
heavyset_go 5y ago
Somehow, I don't think the only way to get self-driving cars is if we first crack a few skulls testing them out on the public without their consent.
root_axis 5y ago
Your framing assumes as a given that Tesla's self-driving tech will eventually succeed given a suitable buffer for human collateral damage, but there is no guarantee that this is the case and Elon's decade of falsehoods regarding the timeline and capabilities of FSD technology doesn't inspire confidence.
- twblalock 5y ago
  Right, there is no guarantee it will ever work. However, self driving is definitely guaranteed to fail if it can't be tested on public roads. And that's really the challenge here -- it can only succeed if society is willing to take the risk.
elihu 5y ago
I think a reasonable standard is: if someone dies, is seriously injured, or likely would have died or been seriously injured if the driver hadn't intervened in a situation that a normally competent driver wouldn't have had a problem then the self-driving system in question should be disabled globally until the vendor of the system can reasonably convince the NTSB or other relevant government agency that they've taken adequate precautions that it's very unlikely to happen again. It's to be expected that people will occasionally die in car accidents. Some things are hard to avoid. If someone on a bicycle comes shooting out into a busy street when they don't have the right away and no one could see them coming, they're probably going to get run over. My general expectation for self-driving vehicles is: don't create unsafe situations that weren't there before, don't behave in unpredictable ways, and if you can't avoid running someone over at least do it in a way that the consequences fall on the person doing risky things. In the above example, swerving to miss the biker is fine if it can be done safely, but swerving to miss the biker and instead running over someone on the sidewalk instead is not. (Or, in other words: if you can't avoid an accident, at least be predictable and try to obey the general principles of right-of-way.) Swerving at a bike for no reason as in the video above should result in suspension of self-driving features (assuming the video is real and wasn't staged in some way).
- josephcsible 5y ago
  > or likely would have died or been seriously injured if the driver hadn't intervened I can see this being a reason to ban a system from operating at SAE level 3 or above, but isn't the entire difference between level 2 and level 3 that things like that are expected to happen at level 2 and below, hence the reason for drivers having to always be ready to immediately intervene at those levels?
  - elihu 5y ago
    I think one could reasonably say that a car shouldn't be expected to save the driver from all situations when the self-driving features are just a backup system and the driver is expected to be in control. However, those features shouldn't cause unsafe situations all by itself, like the example video of the car trying to swerve into a bicyclist.

admax88qqq 5y ago

We don't have to ban self driving cars. We just set the success criteria at zero deaths. If companies want to spend money pursuing it believing they can hit that goal that's on them.

twblalock 5y ago
> We don't have to ban self driving cars. We just set the success criteria at zero deaths. That's equivalent to a ban.
- rhizome 5y ago
  Because the technology is a non-starter and will never work unless accepted terms of public safety are redefined in its favor. "Just draw a bigger bullseye!"
- Karrot_Kream 5y ago
  There's multiple alternatives to driver-driven cars that result in many fewer excess deaths. They've been trialed and deployed in many countries very successfully. You may know of them by common names such as "buses", "streetcars", "trains", "metros", or even "people movers". There's nothing wrong with innovation but we do not need to put self-driving technology on a pedestal when there are alternatives that exist today that move people around that have none of these externalities. If entities want to innovate, let them innovate within safety parameters.
  - thereisnospork 5y ago
    We could also set the speed limit to 5mph and limit cars to 10hp. That would save a lot of lives. The reason we don't is because some deaths are worth the cost. >If entities want to innovate, let them innovate within safety parameters. Sure, but the safety parameters have already been established[0] at 40,000 deaths a year, or less, for transportation equivalently convenient and functional as compared to a manually-driven car. [0]By the crowd-sourced wisdom of democracy.
    - Karrot_Kream 5y ago
      > We could also set the speed limit to 5mph and limit cars to 10hp. That would save a lot of lives. The reason we don't is because some deaths are worth the cost. Are you saying that, despite transit alternatives that do not lead to excess deaths, that deaths due to self-driving cars is worth the cost? We have existing solutions that don't result in these excess deaths, but because of some reason, we don't want those solutions _and_ we want to kill more people just to birth this technology? _Why_? > [0]By the crowd-sourced wisdom of democracy. ... Which bill are you talking about? I don't remember my Rep or Senator voting on allowing me to be a casualty of the self-driving industry. The fact that automobiles in the US result in 40k deaths is itself a travesty. Even Canada is doing better than the US here. Pointing to the worst driving safety record in the G8 and one of the worst in the G20 isn't exactly proof positive that the American automobile system is safe, it's just proof that we have grandfathered an extremely unsafe system here.
      - twblalock 5y ago
        It seems like what you really want is to ban cars and force everyone to use public transit. That's just not realistic, not even in Europe.
        6510 5y ago
        That seems like a much to fun thought experiment to just do away with. Why wouldn't it be realistic?
- kyleee 5y ago
  of course it's not, it just increases expense, but that should be offset by the potential reward of a true breakthrough. it will change the innovation path yes but it's not a ban
  - twblalock 5y ago
    Impossible standards are equivalent to bans, and zero deaths is an impossible standard. All of the companies that are testing self-driving cars have had accidents with other cars and pedestrians. Some of them resulted in death. Others did not, but easily could have, and eventually their luck will run out and they will also cause deaths. Who would invest millions of dollars into developing a technology when a single mistake could flush all that money down the drain?
standyro 5y ago
We have around 38,000 deaths a year on US roads. We already accept significant risk to maintain the status quo. Sensible legislation is needed to balance innovation and safety for self driving cars on public roads, but not zero. We don’t even accept zero pedestrian hit and run deaths yet… California, Texas, and Michigan legislators have already bought in to the benefits that these companies will bring.
josephcsible 5y ago
Zero deaths is a completely unreasonable standard. I literally can't think of a single thing at all that's never contributed to anyone's death.

marricks 5y ago

You are introducing a ridiculous binary. There are innumerable ways to test this software, even on public roads, without unleashing it on the thousands of Tesla owners. It's not like Tesla has to release new features every year, they just feel they need to to continue to pump up and justify their stock price. EDIT: I misread the parent comment, I’m not opposed to just banning self driving cars as self driving cars likely won't make everyday folks' life better.

jacquesm 5y ago
I'm fine with them unleashing it on Tesla owners, they at least have a choice in the matter. I'm not fine with unleashing it on everybody else too.
- judge2020 5y ago
  Driving on public streets is an exercise in choreography, so they're the same thing.
  - jacquesm 5y ago
    No they are not. Low quality software in control of a two ton chunk of iron moving at 35 mph through traffic including pedestrian and bycicles has nothing to do with choreography, that bike was about to be hit from behind because of a vehicle steering into it.
    - judge2020 5y ago
      Choreography in terms of doing things in concert with other road users without classing, including vulnerable road users (pedestrians, bicyclists). My point is that unleashing it on Tesla owners is obviously going to involve interacting with other road users. Just because this instance was regarding a bicyclist doesn't mean it's only a danger to bicyclists, there are still ways for people to die if the car were only interacting with other Teslas.

andbberger 5y ago

the other companies seem to have no problem following the rules and not killing people. I'm more of the mind that cars should become the second-class citizens they ought to be in cities and that I should be able to bike around the city without constant threat of death, that dream remains far off in american cities. Why is that fire codes and aviation regulations are written in blood, but traffic violence is treated as a normal, necessary cost of modern life? When a biker is killed by a driver who wasn't signaling and cut them off on a right turn (as happens to me on a daily basis in SF), that should trigger immediate redesigns of intersections and streets across the city, for that was an entirely preventable death. But nope. Go ahead and test your fundamentally flawed, unproven death machine on public roads. I gotta say, I am really not that keen on being killed by a white tesla.

twblalock 5y ago
> the other companies seem to have no problem following the rules and not killing people. What rules did Tesla break? Also, remember that time Waymo killed a pedestrian?
- andbberger 5y ago
  that this is not explicitly illegal is the problem. everyone else has safety drivers and is subject to reporting requirements. waymo never killed a pedestrian
  - deleted 5y ago
    [deleted]
  - twblalock 5y ago
    > waymo never killed a pedestrian Yeah, it was Uber, my mistake. But that's an example another self-driving company that killed people. It could have easily been any of them -- they have all had accidents that could have resulted in death, and just got lucky. Tesla is not special here.
    - 5y ago

serioussecurity 5y ago

But there are competent folks in the field... ? Waymo's easily five years ahead of Tesla. Previously: https://news.ycombinator.com/item?id=30182141 https://news.ycombinator.com/item?id=30182141

Veserv 5y ago

The standard for success is not zero deaths, but it should at least be a system no more dangerous than human drivers. Generally speaking, the average human driver causes a “ding” every 100k miles, a insurance worthy event every 250k miles, a police worthy event every 500k miles, and a fatality every 60M miles. Assuming an average driving speed of 60 mph, which is obviously an overestimation, those are 1.6k, 4k, 8k, and 1M hours respectively. Evaluating a L2 autonomy system is somewhat challenging since in truth you are evaluating a human+autonomous system, so you should expect the overall safety to increase unless the autonomy system is actively making things more dangerous. However, for a L4/L5 autonomy system, which is Tesla’s goal, we can reasonably approximate the quality by miles per intervention as an intervention is a failure of the autonomy. As a reasonable baseline based on video evidence of FSD, we can assume that each intervention prevents at least a “ding”. Therefore, we can evaluate the quality of FSD relative to a minimal L4 system as the ratio of time per disengagement versus 1.6k hours. Using those same videos as evidence, we see the FSD system requiring intervention every few minutes, i.e. <10 minutes, on average. That is 1/10,000 of the minimal quality expected from a fully functional L4 system. So, for Tesla to create a minimal acceptable L4 autonomy solution that would actually usher in a age of safe self driving systems, they need to improve the core functionality of their beta product by 10,000x. It is that gap between what they are delivering and what they still need to achieve that is outrageous.

jquery 5y ago
Very well said. You’ve put into words what my intuition has been screaming at me but has been unable to properly formulate. Thank you.
jimmydorry 5y ago
In this stage of the product development the rate of intervention is not, and likely won't be for a long time, a good measure of failure. We likely won't see full L4 systems for a very long time. In the mean time, we could start to get a better handle on the safety profile of these systems by better classifying the interventions. Something like these: * Intervention due to unreadable conditions (e.g. unpainted lines, a new area with no training data, failure of LIDAR / camera, w/e) * Human intervention due to not feeling safe * Intervention due to unsafe condition (e.g. imminent collision, or just hit something) * Human intervention for other reason (e.g. human is bored, or human changing destination, or human wanting to take another route) --- #1 could certainly lead to #2 and #3, but is more of an indicator that the car is in novel conditions not trained for, or has hardware failure. #1 is the low hanging fruit (relatively speaking), and #2 is where the majority of the work lies. #3 is where the system has actually failed. And #4 need to just be discarded.

heavyset_go 5y ago

Somehow, I don't think the only way to get self-driving cars is if we first crack a few skulls testing them out on the public without their consent.

root_axis 5y ago

Your framing assumes as a given that Tesla's self-driving tech will eventually succeed given a suitable buffer for human collateral damage, but there is no guarantee that this is the case and Elon's decade of falsehoods regarding the timeline and capabilities of FSD technology doesn't inspire confidence.

twblalock 5y ago
Right, there is no guarantee it will ever work. However, self driving is definitely guaranteed to fail if it can't be tested on public roads. And that's really the challenge here -- it can only succeed if society is willing to take the risk.

elihu 5y ago

I think a reasonable standard is: if someone dies, is seriously injured, or likely would have died or been seriously injured if the driver hadn't intervened in a situation that a normally competent driver wouldn't have had a problem then the self-driving system in question should be disabled globally until the vendor of the system can reasonably convince the NTSB or other relevant government agency that they've taken adequate precautions that it's very unlikely to happen again. It's to be expected that people will occasionally die in car accidents. Some things are hard to avoid. If someone on a bicycle comes shooting out into a busy street when they don't have the right away and no one could see them coming, they're probably going to get run over. My general expectation for self-driving vehicles is: don't create unsafe situations that weren't there before, don't behave in unpredictable ways, and if you can't avoid running someone over at least do it in a way that the consequences fall on the person doing risky things. In the above example, swerving to miss the biker is fine if it can be done safely, but swerving to miss the biker and instead running over someone on the sidewalk instead is not. (Or, in other words: if you can't avoid an accident, at least be predictable and try to obey the general principles of right-of-way.) Swerving at a bike for no reason as in the video above should result in suspension of self-driving features (assuming the video is real and wasn't staged in some way).

josephcsible 5y ago
> or likely would have died or been seriously injured if the driver hadn't intervened I can see this being a reason to ban a system from operating at SAE level 3 or above, but isn't the entire difference between level 2 and level 3 that things like that are expected to happen at level 2 and below, hence the reason for drivers having to always be ready to immediately intervene at those levels?
- elihu 5y ago
  I think one could reasonably say that a car shouldn't be expected to save the driver from all situations when the self-driving features are just a backup system and the driver is expected to be in control. However, those features shouldn't cause unsafe situations all by itself, like the example video of the car trying to swerve into a bicyclist.