9 ms·

drivers came to trust it way too quickly. Drivers who were supposed to be closely monitoring the system instead spent their time looking at their phones, puttin

drivers came to trust it way too quickly. Drivers who were supposed to be closely monitoring the system instead spent their time looking at their phones, putting on makeup, and other distractions. Really hope no one dies because of Tesla's autopilot. This is a real risk. Quoting from below: if the car does it right 99% of the time it's just human nature to stop paying attention. The article makes a great point that slowly improving the autopilot with a person behind the wheel is super dangerous because we get lulled into a false sense of security.

breakfastduck 6y ago
It's bound to happen, but also likely to be the result of the driver not paying attention.
- granzymes 6y ago
  Right, but if the car does it right 99% of the time it's just human nature to stop paying attention. The article makes a great point that slowly improving the autopilot with a person behind the wheel is super dangerous because we get lulled into a false sense of security. I hope Tesla just keeps it to highway lane keeping until they're ready to roll out better sensors.
  - mortenjorck 6y ago
    This principle is glaringly obvious in retrospect, and seeing it put so succinctly actually changed my mind. I previously thought Tesla's approach was an achievable long shot, but I'm now convinced it's fundamentally wrong. The safety-driver model pursued by Waymo and every other autonomy startup is the only responsible way to train an autonomy network because the drivers are specifically trained to look out for edge cases, versus consumers who will easily be lulled into complacency once the system looks superficially like it's running at level-5 competency.
    - dragonwriter 6y ago
      While I agree that the Waymo, et al., way is the only responsible way, that is not an argument against Tesla’s way being workable, just that it is grossly unethical.
- ra7 6y ago
  Krafcik talks about this very issue in the article on how driver assistance systems give false sense of security to the driver. > The Waymo team believes its own early experience—when it was the Google self-driving car project—bears that out. In the early 2010s, Google developed a driver-assistance system similar to today's Autopilot and considered selling it to automakers. But when they let Google employees test the software on public roads, they found that drivers came to trust it way too quickly. Drivers who were supposed to be closely monitoring the system instead spent their time looking at their phones, putting on makeup, and other distractions. > The fundamental challenge here is that the better a driver-assistance system gets, the harder it is to get drivers to pay attention, and the less likely they are to be prepared if the software makes a mistake. The Google team didn't see a good solution to this problem, so they completely changed their strategy. They focused on building a self-driving taxi service that would never have customers in the driver's seat, relying on trained, professional safety drivers to oversee the software during testing.
- nightcracker 6y ago
  Can you pay attention to a video game for two hours, and during that time react to some important event within 1 second reaction time? I bet you can. Can you do the same when staring at a wall for two hours?
- mxschumacher 6y ago
  coffee cups say "be careful this is hot", but the regulator allows a product called "Full Self Driving" to be sold - I think ultimately consumer protection will step in here like it has in Germany [0] 0: https://www.reuters.com/article/us-tesla-autopilot-germany/german-court-bans-tesla-ads-claiming-its-cars-are-capable-of-autonomous-driving-idUSKCN24F1T5 https://www.reuters.com/article/us-tesla-autopilot-germany/g...
  - breakfastduck 6y ago
    Yeah it's a fantastic example of technological developments far outpacing legislation
brianwawok 6y ago
I mean people have already died sleeping with it on. But it’s also likely saved lives based on their deaths per mile number.
- ameen 6y ago
  I’m not sure if I feel easy about this “statistic” - deaths per mile?? These are humans, and more importantly to the business - paying customers. If my product has the possibility of killing my customer, I’d probably not be comfortable with making it market ready (without massively improving its safety features).
  - paulryanrogers 6y ago
    Agreed. Where to draw that line is the debate. For kitchen knives there is a similar problem. As cold as it is the data is the best measure.
  - URSpider94 6y ago
    Here’s the thing, every car system will fail at some point and potentially cause a death. There is no such thing as perfect safety. It’s really important to quantify things like fatalities per million miles driven. At some point, you push your system into production because, even though at some point the system will fail and kill someone, it will in the mean time save dozens or hundreds of people from dying in preventable accidents. What is critically important is being confident that your system is safer than the alternatives that your customers would use if your product doesn’t ship.
  - brianwawok 6y ago
    Most products can kill someone. Entire lines of products routinely kill its user. Society deems them worth the risk. A no risk life is sitting alone in a room, where you will die to obesity.
- petters 6y ago
  Are you aware how low the deaths per mile is for humans? I don't think we know if Waymo has a lower rate than a human yet. Tesla I don't know, but wouldn't be surprised if it is higher.
mminer237 6y ago
People already have died because of Autopilot years ago: https://www.nytimes.com/2018/03/31/business/tesla-crash-autopilot-musk.html https://www.nytimes.com/2018/03/31/business/tesla-crash-auto... It is an inevitability that people will die because of any autopilot system if it lasts long enough.
- judge2020 6y ago
  Better source, also linked elsewhere in this thread https://www.tesladeaths.com/ https://www.tesladeaths.com/
deleted 6y ago
[deleted]
cwhiz 6y ago
People will die. It will also save lives. It’s probably already prevented thousands of accidents, some of which would have been fatal. It just doesn’t make national news when your car moves you out of the way of danger.
- smnrchrds 6y ago
  I am not so sure about this. Modern cars come with a lot of safety features, like automatic braking and adaptive cruise control. Sure, if you replace all 1995 Honda Civics with Teslas, you will save many lives. I don't think you could save any lives by replacing all modern 2020 cars from other manufacturers with Teslas.
  - diebeforei485 6y ago
    All of the systems in other vehicles have the same pitfalls as any driver-assistance system and have the same driver behavior risks as Tesla's system.
    - ashtonkem 6y ago
      They’re also designed and marketed very differently. The lane keeping in my Subaru isn’t good enough for me to fully trust it, so I monitor it carefully. I suspect that there’s probably a local minimum for safety, where the autopilot is good enough to lull users into complacency, but not good enough to actually be safe.
      - SkyPuncher 6y ago
        > The lane keeping in my Subaru isn’t good enough for me to fully trust it, so I monitor it carefully. We have a Hyundai Palisade with Adaptive Cruise Control and Lane Keep assist. Amazing for reducing fatigue (the small little corrections). Not good enough to let you get distracted. I always have to be mentally ready to hop in. I think it's a perfect solution until we have true FSD that can handle absolutely all conditions and situations.
      - 6y ago
mrtksn 6y ago
People die all the time, there are even higher profile cases like the Apple Engineer who slammed in the barriers at full speed: https://www.kqed.org/news/11801138/apple-engineer-killed-in-tesla-crash-had-previously-complained-about-autopilot https://www.kqed.org/news/11801138/apple-engineer-killed-in-... There are some spectacular failures that are very scary because the car does something that a person would never do, unless unconscious, like this one: https://www.youtube.com/watch?v=LfmAG4dk-rU https://www.youtube.com/watch?v=LfmAG4dk-rU You don't hear much about it probably because Tesla fanboys are plenty and rabid, so people avoid talking about it online. The defence is usually stats about human drivers crashing more often and it makes sense until you dive deeper into the numbers because these stats are usually oranges v.s. Apples. It feels like they have some playbook with statistics to slap when someone says something negative. If that doesn't cut, they say that the victim should have followed the manual that says "your attention should always be on the road" then proceed posting a video about how thanks to the latest update they can sleep drive to work and attach a banana to the driving wheel to disable the attention safeguards. If someone asks how is this an autopilot, there are usually two ways to handle it: 1) Autopilot is just a brand name, the self driving software is in beta, so the victim should pay attention all the times. 2) Autopilot is like on the planes, so only fools think that it is autonomous, therefore it was working as intended but they should have been using it like an airline autopilot. Crash due to user error. I'm actually a fan of Musk and Tesla but I feel like the community engagement is very unhealthy and lacks scrutiny due to the "online army" of his.
- granzymes 6y ago
  Wow that second one is terrifying. Particularly since that was only 7 months ago!
  - deleted 6y ago
    [deleted]
  - dehrmann 6y ago
    That was the article, but the crash was in March 2018.
qeternity 6y ago
I think the scrutiny should be whether fewer people die than otherwise would have.
trianglem 6y ago
People can die. Honestly in my opinion it’s worth it. Will it happen to me? No, because I’m extra cautious and won’t slack off but to make progress quickly these things must be done. If you really want to be hyper cautious, you can always make the case that it’s saving lives that would otherwise be lost.
- perryizgr8 6y ago
  > No, because I’m extra cautious and won’t slack off A big issue is that some random guy in the Tesla behind you may not be so careful. So you cold still get into an accident because of Tesla's recklessness.
dehrmann 6y ago
> This is a real risk It's even a risk to Waymo because, despite being responsible about it, Tesla made people either scared of autonomous cars or forced out legislation restricting it.
mung 6y ago
There's two levels of risk, Tesla owners taking themselves out, and then there is innocent bystanders being taken out. It's pretty infuriating if you consider that we're all put at risk.
- jiofih 6y ago
  You are at a much higher risk from all kinds of humans, with varying levels of skill, in different states of mind manually driving these vehicles.
  - s17n 6y ago
    Humans are much, much better drivers than autopilot.
    - jiofih 6y ago
      For now, that’s kind of the goal here...
      - mung 6y ago
        Road users aren't beta-testers. Especially the ones who didn't sign up for it and buy a Tesla.
  - mung 6y ago
    You've always been at risk of other road users, but that is well understood when you get in a car and drive somewhere. What's new, and on top of that risk, and not well understood is the risk due to drivers mis-using a half-baked feature named "autopilot" as if it's actually an autopilot, because Musk insists that it works when it clearly does not always work.
    - deleted 6y ago
      [deleted]

breakfastduck 6y ago

It's bound to happen, but also likely to be the result of the driver not paying attention.

granzymes 6y ago
Right, but if the car does it right 99% of the time it's just human nature to stop paying attention. The article makes a great point that slowly improving the autopilot with a person behind the wheel is super dangerous because we get lulled into a false sense of security. I hope Tesla just keeps it to highway lane keeping until they're ready to roll out better sensors.
- mortenjorck 6y ago
  This principle is glaringly obvious in retrospect, and seeing it put so succinctly actually changed my mind. I previously thought Tesla's approach was an achievable long shot, but I'm now convinced it's fundamentally wrong. The safety-driver model pursued by Waymo and every other autonomy startup is the only responsible way to train an autonomy network because the drivers are specifically trained to look out for edge cases, versus consumers who will easily be lulled into complacency once the system looks superficially like it's running at level-5 competency.
  - dragonwriter 6y ago
    While I agree that the Waymo, et al., way is the only responsible way, that is not an argument against Tesla’s way being workable, just that it is grossly unethical.
ra7 6y ago
Krafcik talks about this very issue in the article on how driver assistance systems give false sense of security to the driver. > The Waymo team believes its own early experience—when it was the Google self-driving car project—bears that out. In the early 2010s, Google developed a driver-assistance system similar to today's Autopilot and considered selling it to automakers. But when they let Google employees test the software on public roads, they found that drivers came to trust it way too quickly. Drivers who were supposed to be closely monitoring the system instead spent their time looking at their phones, putting on makeup, and other distractions. > The fundamental challenge here is that the better a driver-assistance system gets, the harder it is to get drivers to pay attention, and the less likely they are to be prepared if the software makes a mistake. The Google team didn't see a good solution to this problem, so they completely changed their strategy. They focused on building a self-driving taxi service that would never have customers in the driver's seat, relying on trained, professional safety drivers to oversee the software during testing.
nightcracker 6y ago
Can you pay attention to a video game for two hours, and during that time react to some important event within 1 second reaction time? I bet you can. Can you do the same when staring at a wall for two hours?
mxschumacher 6y ago
coffee cups say "be careful this is hot", but the regulator allows a product called "Full Self Driving" to be sold - I think ultimately consumer protection will step in here like it has in Germany [0] 0: https://www.reuters.com/article/us-tesla-autopilot-germany/german-court-bans-tesla-ads-claiming-its-cars-are-capable-of-autonomous-driving-idUSKCN24F1T5 https://www.reuters.com/article/us-tesla-autopilot-germany/g...
- breakfastduck 6y ago
  Yeah it's a fantastic example of technological developments far outpacing legislation

brianwawok 6y ago

I mean people have already died sleeping with it on. But it’s also likely saved lives based on their deaths per mile number.

ameen 6y ago
I’m not sure if I feel easy about this “statistic” - deaths per mile?? These are humans, and more importantly to the business - paying customers. If my product has the possibility of killing my customer, I’d probably not be comfortable with making it market ready (without massively improving its safety features).
- paulryanrogers 6y ago
  Agreed. Where to draw that line is the debate. For kitchen knives there is a similar problem. As cold as it is the data is the best measure.
- URSpider94 6y ago
  Here’s the thing, every car system will fail at some point and potentially cause a death. There is no such thing as perfect safety. It’s really important to quantify things like fatalities per million miles driven. At some point, you push your system into production because, even though at some point the system will fail and kill someone, it will in the mean time save dozens or hundreds of people from dying in preventable accidents. What is critically important is being confident that your system is safer than the alternatives that your customers would use if your product doesn’t ship.
- brianwawok 6y ago
  Most products can kill someone. Entire lines of products routinely kill its user. Society deems them worth the risk. A no risk life is sitting alone in a room, where you will die to obesity.
petters 6y ago
Are you aware how low the deaths per mile is for humans? I don't think we know if Waymo has a lower rate than a human yet. Tesla I don't know, but wouldn't be surprised if it is higher.

mminer237 6y ago

People already have died because of Autopilot years ago: https://www.nytimes.com/2018/03/31/business/tesla-crash-autopilot-musk.html https://www.nytimes.com/2018/03/31/business/tesla-crash-auto... It is an inevitability that people will die because of any autopilot system if it lasts long enough.

judge2020 6y ago
Better source, also linked elsewhere in this thread https://www.tesladeaths.com/ https://www.tesladeaths.com/

deleted 6y ago

[deleted]

cwhiz 6y ago

People will die. It will also save lives. It’s probably already prevented thousands of accidents, some of which would have been fatal. It just doesn’t make national news when your car moves you out of the way of danger.

smnrchrds 6y ago
I am not so sure about this. Modern cars come with a lot of safety features, like automatic braking and adaptive cruise control. Sure, if you replace all 1995 Honda Civics with Teslas, you will save many lives. I don't think you could save any lives by replacing all modern 2020 cars from other manufacturers with Teslas.
- diebeforei485 6y ago
  All of the systems in other vehicles have the same pitfalls as any driver-assistance system and have the same driver behavior risks as Tesla's system.
  - ashtonkem 6y ago
    They’re also designed and marketed very differently. The lane keeping in my Subaru isn’t good enough for me to fully trust it, so I monitor it carefully. I suspect that there’s probably a local minimum for safety, where the autopilot is good enough to lull users into complacency, but not good enough to actually be safe.
    - SkyPuncher 6y ago
      > The lane keeping in my Subaru isn’t good enough for me to fully trust it, so I monitor it carefully. We have a Hyundai Palisade with Adaptive Cruise Control and Lane Keep assist. Amazing for reducing fatigue (the small little corrections). Not good enough to let you get distracted. I always have to be mentally ready to hop in. I think it's a perfect solution until we have true FSD that can handle absolutely all conditions and situations.
    - 6y ago

mrtksn 6y ago

People die all the time, there are even higher profile cases like the Apple Engineer who slammed in the barriers at full speed: https://www.kqed.org/news/11801138/apple-engineer-killed-in-tesla-crash-had-previously-complained-about-autopilot https://www.kqed.org/news/11801138/apple-engineer-killed-in-... There are some spectacular failures that are very scary because the car does something that a person would never do, unless unconscious, like this one: https://www.youtube.com/watch?v=LfmAG4dk-rU https://www.youtube.com/watch?v=LfmAG4dk-rU You don't hear much about it probably because Tesla fanboys are plenty and rabid, so people avoid talking about it online. The defence is usually stats about human drivers crashing more often and it makes sense until you dive deeper into the numbers because these stats are usually oranges v.s. Apples. It feels like they have some playbook with statistics to slap when someone says something negative. If that doesn't cut, they say that the victim should have followed the manual that says "your attention should always be on the road" then proceed posting a video about how thanks to the latest update they can sleep drive to work and attach a banana to the driving wheel to disable the attention safeguards. If someone asks how is this an autopilot, there are usually two ways to handle it: 1) Autopilot is just a brand name, the self driving software is in beta, so the victim should pay attention all the times. 2) Autopilot is like on the planes, so only fools think that it is autonomous, therefore it was working as intended but they should have been using it like an airline autopilot. Crash due to user error. I'm actually a fan of Musk and Tesla but I feel like the community engagement is very unhealthy and lacks scrutiny due to the "online army" of his.

granzymes 6y ago
Wow that second one is terrifying. Particularly since that was only 7 months ago!
- deleted 6y ago
  [deleted]
- dehrmann 6y ago
  That was the article, but the crash was in March 2018.

qeternity 6y ago

I think the scrutiny should be whether fewer people die than otherwise would have.

trianglem 6y ago

People can die. Honestly in my opinion it’s worth it. Will it happen to me? No, because I’m extra cautious and won’t slack off but to make progress quickly these things must be done. If you really want to be hyper cautious, you can always make the case that it’s saving lives that would otherwise be lost.

perryizgr8 6y ago
> No, because I’m extra cautious and won’t slack off A big issue is that some random guy in the Tesla behind you may not be so careful. So you cold still get into an accident because of Tesla's recklessness.

dehrmann 6y ago

> This is a real risk It's even a risk to Waymo because, despite being responsible about it, Tesla made people either scared of autonomous cars or forced out legislation restricting it.

mung 6y ago

There's two levels of risk, Tesla owners taking themselves out, and then there is innocent bystanders being taken out. It's pretty infuriating if you consider that we're all put at risk.

jiofih 6y ago
You are at a much higher risk from all kinds of humans, with varying levels of skill, in different states of mind manually driving these vehicles.
- s17n 6y ago
  Humans are much, much better drivers than autopilot.
  - jiofih 6y ago
    For now, that’s kind of the goal here...
    - mung 6y ago
      Road users aren't beta-testers. Especially the ones who didn't sign up for it and buy a Tesla.
- mung 6y ago
  You've always been at risk of other road users, but that is well understood when you get in a car and drive somewhere. What's new, and on top of that risk, and not well understood is the risk due to drivers mis-using a half-baked feature named "autopilot" as if it's actually an autopilot, because Musk insists that it works when it clearly does not always work.
  - deleted 6y ago
    [deleted]