12 ms·

What is happening was clear to many since the start: Tesla incarnates the behavior of his founder, exaggerating the technology that is really available right no

What is happening was clear to many since the start: Tesla incarnates the behavior of his founder, exaggerating the technology that is really available right now, and selling it as a product without the premise for it to be safe. A product that kinda drives your car but sometimes fails, and requires you to pay attention, is simply a crash that we are waiting to happen. And don't get trapped by the "data driven" analysis, like "yeah but it's a lot safer than humans" because there are at least three problems with this statement: 1. Ethical. One thing is that you do a stupid thing and die. Another is that a technology fails in trivial circumstances that in theory are avoidable. A doctor can make also errors, but a medical device is required to be very safe and reliable in what it does. 2. Of wrong comparisons: you should check the autopilot against a rested and concentrated driver that drives slowly and with a lot of care. Since in the statistics otherwise you do not account for the fact that when you drive, you are in control, and you can decide your level of safety. Such a diligent driver that goes to crash because of an immature software is terrible. 3. And lack of data: AFAIK there is not even enough data publicly available that tells us what is the crash rate of Teslas with auto-pilot enabled vs Teslas without auto pilot enabled, per kilometer in the same street conditions. That is, you need to compare only in the same streets where the auto pilot is able to drive. Autopilot will rule the world eventually and we will be free of using the time differently (even if this was possible 100 years ago already stressing on public transportation instead of spending the money to have a car each one... this is sad. Fortunately in central Europe this happened to a degree, check for instance North Italy, France, ...). But till it is not ready, to plug it as a feature in a so immature way just to have an advantage over competitors, is terrible business practice.

kalal 9y ago
4. This statistics must be conditioned. Not all drivers are the same. Some are careful, some are not. So in fact, the careful drivers will make their life miserable by using autopilot.
sschueller 9y ago
How can anyone take a company serios that sells features such as "Bio-Weapon Defense Mode". It almost sounds like snake oil and not far from something like the ADE 651. [1] https://en.wikipedia.org/wiki/ADE_651 https://en.wikipedia.org/wiki/ADE_651
- buvanshak 9y ago
  Had to look it up [1] >"We’re trying to be a leader in apocalyptic defense scenarios," Musk continued. Is this guy serious? [1] https://www.theverge.com/2015/9/30/9421719/tesla-model-x-bioweapon-defense-mode-button https://www.theverge.com/2015/9/30/9421719/tesla-model-x-bio...
  - Piskvorrr 9y ago
    Scenarios, indeed. Hollywood-grade threat models - riveting yet improbable. As opposed to such mundane threats such as "not driving into massive stationary objects".
- rcMgD2BwE72F 9y ago
  "Not only did the vehicle system completely scrub the cabin air, but in the ensuing minutes, it began to vacuum the air outside the car as well, reducing PM2.5 levels by 40%. In other words, Bioweapon Defense Mode is not a marketing statement, it is real. You can literally survive a military grade bio attack by sitting in your car." https://www.tesla.com/blog/putting-tesla-hepa-filter-and-bioweapon-defense-mode-to-the-test https://www.tesla.com/blog/putting-tesla-hepa-filter-and-bio...
  - sschueller 9y ago
    "extreme levels of pollution" is not the same as a bio-chemical weapon.
    - josefresco 9y ago
      Can you expand on this and provide evidence to support your claim? I would imagine Telsa would have throughly vetted this statement however I'm curious to hear how bio-chemical weapons differ from extreme pollution (they do also mention viruses)
      - deleted 9y ago
        [deleted]
simias 9y ago
I agree with most of your points however I'm not convinced by your problem number 2: >you should check the autopilot against a rested and concentrated driver that drives slowly and with a lot of care. Since in the statistics otherwise you do not account for the fact that when you drive, you are in control, and you can decide your level of safety. Such a diligent driver that goes to crash because of an immature software is terrible. I think you overestimate the rationality of human beings. I commute to work by motorcycle every day and I've noticed that I tend to ride more dangerously (faster, "dirtier" passes etc...) when I'm tired, which rationally is stupid. I know that but I still instinctively do it, probably because I'm in a rush to get home or because I feel like the adrenaline keeps me awake. This is an advantage of autonomous vehicles, they can be reasonable when I'm not. I expect that few drivers (and especially everyday commuters like me) constantly drive "slowly and with a lot of care". A good enough autonomous vehicle would.
Nokinside 9y ago
Designing for safety means that you take into account human behavior at every level and engineer the product to avoid those mistakes. We already know that there is an area between drive assistant and automatic driving where people just can't keep their attention level where it needs to be. Driving is activity that maintains attention. People can't watch the road hand on the wheel when nothing happens and keep their level of attention up when the car is driving. The way I see it, the biggest safety sin from Tesla is having experimental beta feature that that is known weak point for humans. Adding potentially dangerous experimental feature, warning about it, then washing your hands about it is not good safety engineering. The news story points out how other car manufacturers have cameras that watch the driver for the signs of inattention. Driving assistant should watch both the driver and the road. You can't have a driving assistant that can be used as an autopilot.
smoyer 9y ago
4. There is another completely unknown variable - how many times would the autopilot have crashed if the human hadn't taken over. So Tesla's statistics are actually stating how safe the autopilot and human are when combined, not how safe the autopilot is by itself.
- melling 9y ago
  I thought the autopilot was always supposed to be combined with the human. It’s not a self-driving car. It’s really an advanced cruise control.
  - teppix 9y ago
    The statistics also includes all the cases where drivers are not paying attention as they should, and it's still safer than the average car (at least according to Tesla).
    - mozumder 9y ago
      > it's still safer than the average car (at least according to Tesla). This is Tesla's big lie. In all their marketing, Tesla is comparing crash rates of their passenger cars to ALL vehicles, including trucks and motorcycles, which have higher fatality rates. Motorcycles are about 10x-50x more dangerous than cars. Not only that, they aren't controlling for variances in driver demographics - younger and old people have higher accident rates than middle-aged Tesla drivers - as well as environmental factors - rural roads have higher fatalities than highways. Never-mind the obvious "accidents in cars with Autopilot" vs "accidents in cars with Autopilot on". If you do a proper comparison, Tesla's Autopilot is probably 100x more fatal than other cars. It's a god-dammed death trap. And remember, there are several extremely normal cars with ZERO fatalities: https://www.nbcnews.com/business/autos/record-9-models-have-zero-driver-deaths-iihs-says-n296036 https://www.nbcnews.com/business/autos/record-9-models-have-... This is not a problem that will be solved without fundamental infrastructure changes in the roads themselves. Anyone that believes in self-driving cars should never be employed, since they don't know what they're talking about.
donkeyd 9y ago
What makes the "self driving" functionally that all the other brands are marketing different though? Is it only that Tesla was first? Volvo literally calls their technology Autopilot too.
epanchin 9y ago
Saying “When you drive you are in control” is fine, but you’re not always the only person impacted when you crash. As we have no control over how others drive, statistics are more relevant. As a pedistrian, I care about cars being on average less likely to kill me. If it means I’m safer I would rather the driver wasn’t in control of their own safety. The ethical solution is the one with the least overall harm. Of course being safer overall while taking control from the driver is unlikely to drive sales.
kachurovskiy 9y ago
I'm in central Europe, switched from a train to a car. Reduced commute time, increased comfort and reliability. Costs several times more though.
- antirez 9y ago
  What about the fact you cannot do anything useful other than podcasts / music while driving? Btw your point sounds to me just that public transport needs to get better, not that in general is not a good idea to move towards this model.
  - kachurovskiy 9y ago
    > What about the fact you cannot do anything useful other than podcasts / music while driving? To be fair I rarely was doing anything useful on my 40 min train commute either :) Mostly reading Hacked News on my phone. Now I'm at least looking into the distance, taking some strain from the eyes. I totally agree that public transport has to get better, it's just that there always has to be a mixture of transportation options.
- gkya 9y ago
  > Reduced commute time, increased comfort and reliability. Can the reason for these be that many other people use public transport instead and thus the roads are way less busier?
  - kachurovskiy 9y ago
    I'm sure that's a big contributor. I myself relied solely on public transport for years we lived close to the subway. Now that we moved outside of the city driving makes more sense.
LeifCarrotson 9y ago
> 1. Ethical. One thing is that you do a stupid thing and die. Another is that a technology fails in trivial circumstances that in theory are avoidable. A doctor can make also errors, but a medical device is required to be very safe and reliable in what it does. I'm not sure I follow you here. Are you saying that because humans are known to be falible, but technology can be made nigh-infalible, that technology should be held to a higher ethical standard? Those two statements don't connect for me. I suppose that's partly because I am an automation engineer, and I deal a lot with inspecting operator-assembled and machine-assembled parts. If the machine builds parts that also pass the requirements, it's good. It's nice if it's faster or more consistent, and sure we can build machines that produce parts with unnecessarily tight tolerances, but not meeting those possibilities doesn't feel like an ethical failing to me.
- antirez 9y ago
  > Are you saying that because humans are known to be falible, but technology can be made nigh-infalible, that technology should be held to a higher ethical standard? Yes, not infallible, but I believe that to take our lives in the hands of machines, the technology must be at least on par with the best drivers. To be better than average, but for instance to be more fallible than a good driver, is IMHO still not a good standard to be ethically ok to sell self driving cars, even if you have 5% less death per year compared to everybody driving while, like, writing text messages in their phones. If instead the machine will be even a bit better than an high standard driver, driving with care, at that point it makes sense because you are not going to care about the distribution of deaths based on the behaviors.
- gkya 9y ago
  If a survey is conducted in almost any part of the world, I'd guess that most people (including me) would prefer to be hit by a human rather than an autonomous car. I'm not sure why this is, but having someone to blame and the fact that being a victim of an emotionless lifeless machine is subjectively way more horrifying than being a victim of a person are some that I can think of.
emiliobumachar 9y ago
A lot of human-driven car accident victims have done nothing wrong at all. Almost every driver thinks they're better than average. Even when it's a stupid person dying from their stupidity, it's still a tragedy. I really think data driven analysys is the way to go. If we can get US car fatalities from 30000 a year to 29000 a year by adopting self-driving cars, that's 1000 lives saved a year. Agree with your point #3. If Tesla autopilot only works in some conditions, its numbers are only comparable to human drivers in these same conditions.
- ccalf 9y ago
  OK but if by saving 1000 lives a year required as a side-effect that you personally be among the fatalities, would that be OK for you? I hope not. Think of this as a technical corner case; so, the question is the soundness of the analysis—for example, the distribution of deaths and what that means for safety—and not letting various facile logic get in the way of that work.
  - fsloth 9y ago
    That's not really a very good argument. If you change parameters in a complex system then the odds are that you are going to find pathologies in new places. People claim seatbelts have caused lots of deaths, and I'm sure at least a percentage of these claims are fair ([0]). I still think it's safer to drive a car with a seatbelt rather than without. [0] http://www.againstseatbeltcompulsion.org/victims.htm http://www.againstseatbeltcompulsion.org/victims.htm
  - emiliobumachar 9y ago
    The downside of a laissez-faire policy towards self-driving cars could fall on anyone, but so does the upside. Again, a lot of human driven car crash victims have done nothing wrong and were victimized pretty much at random. Run over, rear-ended, T-crashed at an intersection, and could not reasonably have done anything to prevent it.
    - YeGoblynQueenne 9y ago
brownbat 9y ago
> you should check the autopilot against a rested and concentrated driver that drives slowly and with a lot of care You raise an interesting point, accidents are not normally distributed throughout the driver's day, or even the population. Your likelihood of having a crash with injuries is highly correlated with whether or not you've had one before. A substantial number involve alcohol, consumed by drivers previously cited for DUIs. We keep using average crash statistics for humans as a baseline, but that may be misleading. Some drivers may actually be increasing their risk by moving to first gen self driving technology, even while others reduce their risk. On the other hand, we do face a real Hindenburg threat here. Zeppelins flew thousands of miles safely, and even on that disaster, many survived. But all people could think of when boarding an airship after that was Herbert Morrison's voice and flames. I have already heard people who don't work in technology mumbling about how they think computers are far more dangerous than humans (not because of your nuanced distinction, but simply ignoring or unaware of any statistics). I worry we're only a few high profile accidents away from total bans on driverless cars, at least in some states. Especially if every one of these is going to make great press, while almost no pure human accidents will. The availability heuristic alone will confuse people.
mannykannot 9y ago
Everyone talking about statistical evidence should take a look at this NHTSA report [1]. For example, "Figure 11. Crash Rates in MY 2014-16 Tesla Model S and 2016 Model X vehicles Before and After Autosteer Installation", where they are 1.3 and 0.8 per million miles respectively. Unfortunately, this report seems to have shot itself in the foot by apparently using 'Autopilot' and 'Autosteer' interchangeably, so it leaves open the possibility that the Autopilot software improves or adds features to the performance of fairly basic mitigation methods, such as in emergency braking, while having unacceptable flaws in the area of steering and obstacle detection at speed. In addition, no information is given on the distribution of accident severity. It is unfortunate that this report leaves these questions open. Even if these claims are sustained, there are two specific matters in which I think the NHTSA has been too passive: As this is beta software, it should not be on the on the public roads in what amounts to an unsupervised test that potentially puts other road users at risk. Secondly, as this system requires constant supervision by an alert driver, hands-on-the-wheel is not an acceptable test that this constraint is being satisfied. [1] https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.PDF https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.PDF
- danso 9y ago
  I mentioned this in a reply to another comment, but the NHTSA findings have apparently raised suspicions in other researchers. Currently, the DOT/NHTSA are facing a FOIA lawsuit for not releasing data (they assert that the data reveals Tesla trade secrets) that can be used to independently verify the study's conclusion: http://www.safetyresearch.net/blog/articles/quality-control-systems-corp-sues-dot-tesla-data http://www.safetyresearch.net/blog/articles/quality-control-... https://jalopnik.com/feds-cant-say-why-they-claim-teslas-autosteer-cuts-cras-1819229456 https://jalopnik.com/feds-cant-say-why-they-claim-teslas-aut...
- wilun 9y ago
  You can add safety automated features without requiring completely insane and dangerous mode of operations (like pretending the car can drive itself except that it can not and pretending that you did not pretend in the first place and reminding that the driver should have its hand on the wheel and be alert and what is even the point then?????) They could be more useful with -- for now -- less features. They probably won't do it because they want some sacrificial beta testers to collect some more data for their marginally less crappy next version. But given the car does not even have the necessary hardware to become a real self driving car (and that some analysts even think Tesla is gonna close soon) the people taking the risks of being sacrificed will probably not even reap the benefits of the risks they have taken, paying enormous amount of money to effectively work for that company (among other things).
hyperbole 9y ago
While the marketing fuzz might have the tone of self driving car, I doubt the legal material in a Tesla model S as it relates to 'autopilot' has such language - its advanced steering assist and cruise control, its not an entirely autonomous car - but its marketed as an autonomous car. Functionality that can _most_ of the time drive itself without human intervention and occasionally drives itself into a divider on the highway seems like a callous disregard for human life & how such functionality will be used. Sure, everything is avoidable, if there's some expectation it needs to be avoided. The whole point of autopilot is to avoid focusing on the road all of the time. So setting up circumstances under which humans perceive the functionality (autopilot) behaving as expected most of the time, then its highly likely they'll treat it as such & will succumb to a false sense of security. My point: when a feature is life threatening your marketing fluff shouldn't deviate significantly from your legal language.
ajross 9y ago
> 1. Ethical. One thing is that you do a stupid thing and die. Another is that a technology fails in trivial circumstances that in theory are avoidable. A doctor can make also errors, but a medical device is required to be very safe and reliable in what it does. This needs elaboration or citation. You're one of the only folks I've seen come straight out and make this point: that somehow a technology or practice that is objectively safer can still be "unethically safe" and should be shunned in favor of less safe stuff. I don't buy it. This isn't individual action here, there's no utilitarian philosophy rejection that's going to help you. In fact, medical devices seem actually counter to your argument. No one says an automatic defibrillator's decisions as to when to fire are going to be as good as a human doctor, and in fact these things have non-zero (often fatal) false positive rates. But they save lives, so we use them anyway. Bottom line, that point just doesn't work.
- Ajedi32 9y ago
  I think the point he's making is that it's worse if a person dies because of something they have no control over (self-driving car malfunction) than if a person dies because of their own stupid choices (driving drunk, driving too fast for conditions, running red lights, etc). This, of course, ignores the fact that stupid choices drivers make tend to affect other people on the road who did nothing wrong, so the introduction of a self-driving car which makes less stupid decisions would reduce deaths from both categories of people here.
  - matte_black 9y ago
    No, that conclusion completely ignores the fact that the kind of people who buy Teslas are generally not the kind of people who drive unsafely anyway and die in terrible wrecks. If you give every shitty driver a self driving Tesla maybe you would do something to make roads safer, but if you’re just giving it to higher net worth individuals who place greater value on their own life, you haven’t even made a dent in traffic safety. In fact in some cases all you’re doing is making drivers unsafer because the autopilot encourages them to not pay attention to the road no matter how much you think they are watching carefully. The men killed in Teslas could have all avoided their deaths if only they had been paying attention. If I see a Tesla on the road I stay the hell away lest it do something irrational from an error and kill us both.
jsight 9y ago
That is a long description of a problem that is much deeper than just marketing, IMO. The biggest issue that I see is that the AutoPilot system does not have particularly strong alertness monitoring.
- deleted 9y ago
  [deleted]
pkulak 9y ago
> A product that kinda drives your car but sometimes fails, and requires you to pay attention So, cruise control? If people got confused and thought that cruise control was more than it really was in the 80s, or whenever it came out, what would we have done?
deegles 9y ago
Also, it's my belief that the "miles driven" statistic they always point at (wrt safety) should be reset with each software release.
Splines 9y ago
> Of wrong comparisons: you should check the autopilot against a rested and concentrated driver that drives slowly and with a lot of care. Since in the statistics otherwise you do not account for the fact that when you drive, you are in control, and you can decide your level of safety. Such a diligent driver that goes to crash because of an immature software is terrible. Do these car companies test their software by just leaving it on all the time in a passive mode and measuring the deviance from the human driver? I'd think that at least in MV case, you'd see a higher incidence of lane following difference at this spot and it would warrant an investigation. Something like this isn't easy but for a company as ambitious as Tesla it doesn't sound unreasonable. Such a crash with such an easy repro should have been known about a long long time ago.

kalal 9y ago

4. This statistics must be conditioned. Not all drivers are the same. Some are careful, some are not. So in fact, the careful drivers will make their life miserable by using autopilot.

sschueller 9y ago

How can anyone take a company serios that sells features such as "Bio-Weapon Defense Mode". It almost sounds like snake oil and not far from something like the ADE 651. [1] https://en.wikipedia.org/wiki/ADE_651 https://en.wikipedia.org/wiki/ADE_651

buvanshak 9y ago
Had to look it up [1] >"We’re trying to be a leader in apocalyptic defense scenarios," Musk continued. Is this guy serious? [1] https://www.theverge.com/2015/9/30/9421719/tesla-model-x-bioweapon-defense-mode-button https://www.theverge.com/2015/9/30/9421719/tesla-model-x-bio...
- Piskvorrr 9y ago
  Scenarios, indeed. Hollywood-grade threat models - riveting yet improbable. As opposed to such mundane threats such as "not driving into massive stationary objects".
rcMgD2BwE72F 9y ago
"Not only did the vehicle system completely scrub the cabin air, but in the ensuing minutes, it began to vacuum the air outside the car as well, reducing PM2.5 levels by 40%. In other words, Bioweapon Defense Mode is not a marketing statement, it is real. You can literally survive a military grade bio attack by sitting in your car." https://www.tesla.com/blog/putting-tesla-hepa-filter-and-bioweapon-defense-mode-to-the-test https://www.tesla.com/blog/putting-tesla-hepa-filter-and-bio...
- sschueller 9y ago
  "extreme levels of pollution" is not the same as a bio-chemical weapon.
  - josefresco 9y ago
    Can you expand on this and provide evidence to support your claim? I would imagine Telsa would have throughly vetted this statement however I'm curious to hear how bio-chemical weapons differ from extreme pollution (they do also mention viruses)
    - deleted 9y ago
      [deleted]

simias 9y ago

I agree with most of your points however I'm not convinced by your problem number 2: >you should check the autopilot against a rested and concentrated driver that drives slowly and with a lot of care. Since in the statistics otherwise you do not account for the fact that when you drive, you are in control, and you can decide your level of safety. Such a diligent driver that goes to crash because of an immature software is terrible. I think you overestimate the rationality of human beings. I commute to work by motorcycle every day and I've noticed that I tend to ride more dangerously (faster, "dirtier" passes etc...) when I'm tired, which rationally is stupid. I know that but I still instinctively do it, probably because I'm in a rush to get home or because I feel like the adrenaline keeps me awake. This is an advantage of autonomous vehicles, they can be reasonable when I'm not. I expect that few drivers (and especially everyday commuters like me) constantly drive "slowly and with a lot of care". A good enough autonomous vehicle would.

Nokinside 9y ago

Designing for safety means that you take into account human behavior at every level and engineer the product to avoid those mistakes. We already know that there is an area between drive assistant and automatic driving where people just can't keep their attention level where it needs to be. Driving is activity that maintains attention. People can't watch the road hand on the wheel when nothing happens and keep their level of attention up when the car is driving. The way I see it, the biggest safety sin from Tesla is having experimental beta feature that that is known weak point for humans. Adding potentially dangerous experimental feature, warning about it, then washing your hands about it is not good safety engineering. The news story points out how other car manufacturers have cameras that watch the driver for the signs of inattention. Driving assistant should watch both the driver and the road. You can't have a driving assistant that can be used as an autopilot.

smoyer 9y ago

4. There is another completely unknown variable - how many times would the autopilot have crashed if the human hadn't taken over. So Tesla's statistics are actually stating how safe the autopilot and human are when combined, not how safe the autopilot is by itself.

melling 9y ago
I thought the autopilot was always supposed to be combined with the human. It’s not a self-driving car. It’s really an advanced cruise control.
- teppix 9y ago
  The statistics also includes all the cases where drivers are not paying attention as they should, and it's still safer than the average car (at least according to Tesla).
  - mozumder 9y ago
    > it's still safer than the average car (at least according to Tesla). This is Tesla's big lie. In all their marketing, Tesla is comparing crash rates of their passenger cars to ALL vehicles, including trucks and motorcycles, which have higher fatality rates. Motorcycles are about 10x-50x more dangerous than cars. Not only that, they aren't controlling for variances in driver demographics - younger and old people have higher accident rates than middle-aged Tesla drivers - as well as environmental factors - rural roads have higher fatalities than highways. Never-mind the obvious "accidents in cars with Autopilot" vs "accidents in cars with Autopilot on". If you do a proper comparison, Tesla's Autopilot is probably 100x more fatal than other cars. It's a god-dammed death trap. And remember, there are several extremely normal cars with ZERO fatalities: https://www.nbcnews.com/business/autos/record-9-models-have-zero-driver-deaths-iihs-says-n296036 https://www.nbcnews.com/business/autos/record-9-models-have-... This is not a problem that will be solved without fundamental infrastructure changes in the roads themselves. Anyone that believes in self-driving cars should never be employed, since they don't know what they're talking about.

donkeyd 9y ago

What makes the "self driving" functionally that all the other brands are marketing different though? Is it only that Tesla was first? Volvo literally calls their technology Autopilot too.

epanchin 9y ago

Saying “When you drive you are in control” is fine, but you’re not always the only person impacted when you crash. As we have no control over how others drive, statistics are more relevant. As a pedistrian, I care about cars being on average less likely to kill me. If it means I’m safer I would rather the driver wasn’t in control of their own safety. The ethical solution is the one with the least overall harm. Of course being safer overall while taking control from the driver is unlikely to drive sales.

kachurovskiy 9y ago

I'm in central Europe, switched from a train to a car. Reduced commute time, increased comfort and reliability. Costs several times more though.

antirez 9y ago
What about the fact you cannot do anything useful other than podcasts / music while driving? Btw your point sounds to me just that public transport needs to get better, not that in general is not a good idea to move towards this model.
- kachurovskiy 9y ago
  > What about the fact you cannot do anything useful other than podcasts / music while driving? To be fair I rarely was doing anything useful on my 40 min train commute either :) Mostly reading Hacked News on my phone. Now I'm at least looking into the distance, taking some strain from the eyes. I totally agree that public transport has to get better, it's just that there always has to be a mixture of transportation options.
gkya 9y ago
> Reduced commute time, increased comfort and reliability. Can the reason for these be that many other people use public transport instead and thus the roads are way less busier?
- kachurovskiy 9y ago
  I'm sure that's a big contributor. I myself relied solely on public transport for years we lived close to the subway. Now that we moved outside of the city driving makes more sense.

LeifCarrotson 9y ago

> 1. Ethical. One thing is that you do a stupid thing and die. Another is that a technology fails in trivial circumstances that in theory are avoidable. A doctor can make also errors, but a medical device is required to be very safe and reliable in what it does. I'm not sure I follow you here. Are you saying that because humans are known to be falible, but technology can be made nigh-infalible, that technology should be held to a higher ethical standard? Those two statements don't connect for me. I suppose that's partly because I am an automation engineer, and I deal a lot with inspecting operator-assembled and machine-assembled parts. If the machine builds parts that also pass the requirements, it's good. It's nice if it's faster or more consistent, and sure we can build machines that produce parts with unnecessarily tight tolerances, but not meeting those possibilities doesn't feel like an ethical failing to me.

antirez 9y ago
> Are you saying that because humans are known to be falible, but technology can be made nigh-infalible, that technology should be held to a higher ethical standard? Yes, not infallible, but I believe that to take our lives in the hands of machines, the technology must be at least on par with the best drivers. To be better than average, but for instance to be more fallible than a good driver, is IMHO still not a good standard to be ethically ok to sell self driving cars, even if you have 5% less death per year compared to everybody driving while, like, writing text messages in their phones. If instead the machine will be even a bit better than an high standard driver, driving with care, at that point it makes sense because you are not going to care about the distribution of deaths based on the behaviors.
gkya 9y ago
If a survey is conducted in almost any part of the world, I'd guess that most people (including me) would prefer to be hit by a human rather than an autonomous car. I'm not sure why this is, but having someone to blame and the fact that being a victim of an emotionless lifeless machine is subjectively way more horrifying than being a victim of a person are some that I can think of.

emiliobumachar 9y ago

A lot of human-driven car accident victims have done nothing wrong at all. Almost every driver thinks they're better than average. Even when it's a stupid person dying from their stupidity, it's still a tragedy. I really think data driven analysys is the way to go. If we can get US car fatalities from 30000 a year to 29000 a year by adopting self-driving cars, that's 1000 lives saved a year. Agree with your point #3. If Tesla autopilot only works in some conditions, its numbers are only comparable to human drivers in these same conditions.

ccalf 9y ago
OK but if by saving 1000 lives a year required as a side-effect that you personally be among the fatalities, would that be OK for you? I hope not. Think of this as a technical corner case; so, the question is the soundness of the analysis—for example, the distribution of deaths and what that means for safety—and not letting various facile logic get in the way of that work.
- fsloth 9y ago
  That's not really a very good argument. If you change parameters in a complex system then the odds are that you are going to find pathologies in new places. People claim seatbelts have caused lots of deaths, and I'm sure at least a percentage of these claims are fair ([0]). I still think it's safer to drive a car with a seatbelt rather than without. [0] http://www.againstseatbeltcompulsion.org/victims.htm http://www.againstseatbeltcompulsion.org/victims.htm
- emiliobumachar 9y ago
  The downside of a laissez-faire policy towards self-driving cars could fall on anyone, but so does the upside. Again, a lot of human driven car crash victims have done nothing wrong and were victimized pretty much at random. Run over, rear-ended, T-crashed at an intersection, and could not reasonably have done anything to prevent it.
  - YeGoblynQueenne 9y ago

brownbat 9y ago

> you should check the autopilot against a rested and concentrated driver that drives slowly and with a lot of care You raise an interesting point, accidents are not normally distributed throughout the driver's day, or even the population. Your likelihood of having a crash with injuries is highly correlated with whether or not you've had one before. A substantial number involve alcohol, consumed by drivers previously cited for DUIs. We keep using average crash statistics for humans as a baseline, but that may be misleading. Some drivers may actually be increasing their risk by moving to first gen self driving technology, even while others reduce their risk. On the other hand, we do face a real Hindenburg threat here. Zeppelins flew thousands of miles safely, and even on that disaster, many survived. But all people could think of when boarding an airship after that was Herbert Morrison's voice and flames. I have already heard people who don't work in technology mumbling about how they think computers are far more dangerous than humans (not because of your nuanced distinction, but simply ignoring or unaware of any statistics). I worry we're only a few high profile accidents away from total bans on driverless cars, at least in some states. Especially if every one of these is going to make great press, while almost no pure human accidents will. The availability heuristic alone will confuse people.

mannykannot 9y ago

Everyone talking about statistical evidence should take a look at this NHTSA report [1]. For example, "Figure 11. Crash Rates in MY 2014-16 Tesla Model S and 2016 Model X vehicles Before and After Autosteer Installation", where they are 1.3 and 0.8 per million miles respectively. Unfortunately, this report seems to have shot itself in the foot by apparently using 'Autopilot' and 'Autosteer' interchangeably, so it leaves open the possibility that the Autopilot software improves or adds features to the performance of fairly basic mitigation methods, such as in emergency braking, while having unacceptable flaws in the area of steering and obstacle detection at speed. In addition, no information is given on the distribution of accident severity. It is unfortunate that this report leaves these questions open. Even if these claims are sustained, there are two specific matters in which I think the NHTSA has been too passive: As this is beta software, it should not be on the on the public roads in what amounts to an unsupervised test that potentially puts other road users at risk. Secondly, as this system requires constant supervision by an alert driver, hands-on-the-wheel is not an acceptable test that this constraint is being satisfied. [1] https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.PDF https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.PDF

danso 9y ago
I mentioned this in a reply to another comment, but the NHTSA findings have apparently raised suspicions in other researchers. Currently, the DOT/NHTSA are facing a FOIA lawsuit for not releasing data (they assert that the data reveals Tesla trade secrets) that can be used to independently verify the study's conclusion: http://www.safetyresearch.net/blog/articles/quality-control-systems-corp-sues-dot-tesla-data http://www.safetyresearch.net/blog/articles/quality-control-... https://jalopnik.com/feds-cant-say-why-they-claim-teslas-autosteer-cuts-cras-1819229456 https://jalopnik.com/feds-cant-say-why-they-claim-teslas-aut...
wilun 9y ago
You can add safety automated features without requiring completely insane and dangerous mode of operations (like pretending the car can drive itself except that it can not and pretending that you did not pretend in the first place and reminding that the driver should have its hand on the wheel and be alert and what is even the point then?????) They could be more useful with -- for now -- less features. They probably won't do it because they want some sacrificial beta testers to collect some more data for their marginally less crappy next version. But given the car does not even have the necessary hardware to become a real self driving car (and that some analysts even think Tesla is gonna close soon) the people taking the risks of being sacrificed will probably not even reap the benefits of the risks they have taken, paying enormous amount of money to effectively work for that company (among other things).

hyperbole 9y ago

While the marketing fuzz might have the tone of self driving car, I doubt the legal material in a Tesla model S as it relates to 'autopilot' has such language - its advanced steering assist and cruise control, its not an entirely autonomous car - but its marketed as an autonomous car. Functionality that can _most_ of the time drive itself without human intervention and occasionally drives itself into a divider on the highway seems like a callous disregard for human life & how such functionality will be used. Sure, everything is avoidable, if there's some expectation it needs to be avoided. The whole point of autopilot is to avoid focusing on the road all of the time. So setting up circumstances under which humans perceive the functionality (autopilot) behaving as expected most of the time, then its highly likely they'll treat it as such & will succumb to a false sense of security. My point: when a feature is life threatening your marketing fluff shouldn't deviate significantly from your legal language.

ajross 9y ago

> 1. Ethical. One thing is that you do a stupid thing and die. Another is that a technology fails in trivial circumstances that in theory are avoidable. A doctor can make also errors, but a medical device is required to be very safe and reliable in what it does. This needs elaboration or citation. You're one of the only folks I've seen come straight out and make this point: that somehow a technology or practice that is objectively safer can still be "unethically safe" and should be shunned in favor of less safe stuff. I don't buy it. This isn't individual action here, there's no utilitarian philosophy rejection that's going to help you. In fact, medical devices seem actually counter to your argument. No one says an automatic defibrillator's decisions as to when to fire are going to be as good as a human doctor, and in fact these things have non-zero (often fatal) false positive rates. But they save lives, so we use them anyway. Bottom line, that point just doesn't work.

Ajedi32 9y ago
I think the point he's making is that it's worse if a person dies because of something they have no control over (self-driving car malfunction) than if a person dies because of their own stupid choices (driving drunk, driving too fast for conditions, running red lights, etc). This, of course, ignores the fact that stupid choices drivers make tend to affect other people on the road who did nothing wrong, so the introduction of a self-driving car which makes less stupid decisions would reduce deaths from both categories of people here.
- matte_black 9y ago
  No, that conclusion completely ignores the fact that the kind of people who buy Teslas are generally not the kind of people who drive unsafely anyway and die in terrible wrecks. If you give every shitty driver a self driving Tesla maybe you would do something to make roads safer, but if you’re just giving it to higher net worth individuals who place greater value on their own life, you haven’t even made a dent in traffic safety. In fact in some cases all you’re doing is making drivers unsafer because the autopilot encourages them to not pay attention to the road no matter how much you think they are watching carefully. The men killed in Teslas could have all avoided their deaths if only they had been paying attention. If I see a Tesla on the road I stay the hell away lest it do something irrational from an error and kill us both.

jsight 9y ago

That is a long description of a problem that is much deeper than just marketing, IMO. The biggest issue that I see is that the AutoPilot system does not have particularly strong alertness monitoring.

deleted 9y ago
[deleted]

pkulak 9y ago

> A product that kinda drives your car but sometimes fails, and requires you to pay attention So, cruise control? If people got confused and thought that cruise control was more than it really was in the 80s, or whenever it came out, what would we have done?

deegles 9y ago

Also, it's my belief that the "miles driven" statistic they always point at (wrt safety) should be reset with each software release.

Splines 9y ago

> Of wrong comparisons: you should check the autopilot against a rested and concentrated driver that drives slowly and with a lot of care. Since in the statistics otherwise you do not account for the fact that when you drive, you are in control, and you can decide your level of safety. Such a diligent driver that goes to crash because of an immature software is terrible. Do these car companies test their software by just leaving it on all the time in a passive mode and measuring the deviance from the human driver? I'd think that at least in MV case, you'd see a higher incidence of lane following difference at this spot and it would warrant an investigation. Something like this isn't easy but for a company as ambitious as Tesla it doesn't sound unreasonable. Such a crash with such an easy repro should have been known about a long long time ago.