5 ms·

The “autopilot” concept isn’t going to work at scale with cars. It’s either autonomous or it’s not. As these accidents demonstrate drivers can’t be relied upon

The “autopilot” concept isn’t going to work at scale with cars. It’s either autonomous or it’s not. As these accidents demonstrate drivers can’t be relied upon to ‘monitor’ the autopilot and react if something goes wrong. It’s a deeply flawed design concept from an otherwise very innovative company.

crististm 8y ago
You touched a nerve on some people. And rightly so. Perception and context awareness is not instantaneous. Here's a quick improvement on testing for the learning agent: watch the driver while he is driving, watch the road, AI predicts what do in the same situation, diff that with the driver; diff it offline if necessary. Bang! massive improvement in learning data, time and pinpointing blunders. If they are not already doing this, they should.
- martin_bech 8y ago
  Tesla is doing this. Comparing driver vs autopilot, all information from the fleet, is sent back to Tesla and used as learning data. Thats one of the reasons why all the cars have free internet access.
  - Someone1234 8y ago
    So Tesla is using its drivers as guinea pigs in a high steaks game that could result in death? A lot of people want to treat Tesla like they're just a tech company, while ignoring that the "A/B test" conducted here could literally get people killed (and has). For example, in the most recent death some owners are claiming that an over-the-air software update caused the issue (lane centering), it made their cars drive differently. In tech regular updates are common, but normally the biggest consequence for getting one wrong is angry customers, not dead customers.
    - JoeAltmaier 8y ago
      This is done everywhere. Testing tires or medicines or road surface treatments. We all risk our lives every day to some degree. Testing how best to preserve them with A/B testing is a long respected approach. And is it fair to put the blame solely on a cherrypicked action by Tesla engineering? A tragic coincidence is a compilation of events.
      - jstarfish 8y ago
        Tires, medicines and road surface treatments are all tested in controlled environments before being released to the public, not tested on the public. Conducting A/B tests with live human subjects and in a context where failures will kill people (including those not even associated with the experiment!) is the level of scientific rigor I expect from Josef Mengele.
        Piskvorrr 8y ago
        Inb4 "don't buy a Tesla then" - this is not limited to Tesla customers, but extends to anyone in the vicinity.
        jccooper 8y ago
        I drive on an experimental highway surface daily. It's quieter, but uses an unorthodox drainage system (that at least feels less controlled in wet conditions). Only real use statistics will say for sure if it's safety-equivalent to standard surfaces. The state DOT is testing this on me and tens of thousands of other people daily at high speed. Sure, I suspect that they did smaller scale testing before deploying it, but so has Tesla. So tell me: what's the difference?
        deleted 8y ago
        [deleted]
        gamblor956 8y ago
        That road surface was first tested in non-public situations to verify basic safety and functionality concerns. If it's reached the public-stage of testing, that means safety concerns were addressed and they're now testing durability.
    - nnf 8y ago
      I read the parent comment "Tesla is doing this" as confirming that Tesla vehicles alway have Autopilot running, even when it's not controlling the car, and comparing what Autopilot thinks the driver should do with what the driver is actually doing. If true, having that data would give Tesla an enormous amount of data on situations in which Autopilot might have failed if it was in control of the car, such as if Autopilot (when not engaged) didn't see an obstruction but the human driver swerved suddenly. This is also an advantage of Tesla installing full Autopilot hardware on every car regardless of whether the Autopilot upgrade has been purchased.
fastball 8y ago
What're your thoughts on cruise control?
- code4tee 8y ago
  Cruise control is very different. It doesn’t allow people to take their attention off the road. In the Tesla deaths it appears quite clear that the driver was not paying attention to the road.
jacksmith21006 8y ago
Great post and completely agree. You can NOT be kind of driving. Plus who does not work with a limited group of users while you are testing. That is true for tech that does not kill you and more so when it can. What Tesla is doing is honestly close to criminal. Waymo approach makes far more sense.
- matz1 8y ago
  I disagree, some kind of driving still can be useful. Of course like anything there always be some risk involved. Even when the death do happen, you can still learn from it.
  - jacksmith21006 8y ago
    The problem is you can NOT be kind of driving. The Uber video of the driver is so telling. You can see he is reading his phone when the car hit the lady. Google has indicated this for years. There really should NOT be safety drivers behind the wheel. What Tesla is doing is almost criminal. Having a autopilot button when the car really can not do auto pilot is about as wrong as it gets. I do not believe you can do SDC directly to the consumer while it is really in beta.
    - matz1 8y ago
      Obviously you CAN. Of course it wont be perfect but most of time it should be fine. In some situation its still better than a driver using phone while driving a car that has no self driving capability whatsover, which many people do that right now. Waiting until the technology is perfect will take a long time.
      - jacksmith21006 8y ago
        Could not disagree more. When I say "CAN" I mean safely.
NickM 8y ago
As these accidents demonstrate drivers can’t be relied upon to ‘monitor’ the autopilot and react if something goes wrong. We don't know how many other accidents have been prevented by autopilot. A safety system should not be judged by whether it's perfect, but by whether having it is better than not having it. The NHTSA did a study on exactly this issue by looking at accident rates in cars with autopilot hardware before and after autopilot was actually enabled via a software update. They found that accident rates went down by about 40% after autosteer was enabled. This seems to suggest that either people are fine with monitoring and taking over when needed, or else autosteer was already so good in 2016 that it made fewer mistakes than humans. https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.pdf https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.pdf
- frgtpsswrdlame 8y ago
  >else autosteer was already so good in 2016 that it made fewer mistakes than humans. No no no no no! The NHTSA report says nothing about whether Tesla's system is safer than humans. In that study there are two buckets, one which is total Tesla miles in TACC-enabled cars and then after the update total Tesla miles in cars with TACC + Autosteer and they calculated on airbag deployments. Human driven miles are going to dominate both of those buckets and there's a reason the NHTSA report makes zero claims about Tesla's safety relative to human drivers. It's totally outside the scope of the study. Then add in that some researchers who are skeptical of the methodology and have been asking for the raw data from NHTSA/Tesla have yet to receive it.
  - NickM 8y ago
    I didn't say it was safer than humans. I said that the study results suggest that either humans are fine with taking over and driving manually when required or if humans are not taking over when they should then autosteer might just be safer than humans. I don't claim to know exactly why autosteer would reduce accident rates, and neither does the study, but there must be some explanation and the last bit of my comment was simply speculating as to what that explanation might be.
    - 8y ago

crististm 8y ago

You touched a nerve on some people. And rightly so. Perception and context awareness is not instantaneous. Here's a quick improvement on testing for the learning agent: watch the driver while he is driving, watch the road, AI predicts what do in the same situation, diff that with the driver; diff it offline if necessary. Bang! massive improvement in learning data, time and pinpointing blunders. If they are not already doing this, they should.

martin_bech 8y ago
Tesla is doing this. Comparing driver vs autopilot, all information from the fleet, is sent back to Tesla and used as learning data. Thats one of the reasons why all the cars have free internet access.
- Someone1234 8y ago
  So Tesla is using its drivers as guinea pigs in a high steaks game that could result in death? A lot of people want to treat Tesla like they're just a tech company, while ignoring that the "A/B test" conducted here could literally get people killed (and has). For example, in the most recent death some owners are claiming that an over-the-air software update caused the issue (lane centering), it made their cars drive differently. In tech regular updates are common, but normally the biggest consequence for getting one wrong is angry customers, not dead customers.
  - JoeAltmaier 8y ago
    This is done everywhere. Testing tires or medicines or road surface treatments. We all risk our lives every day to some degree. Testing how best to preserve them with A/B testing is a long respected approach. And is it fair to put the blame solely on a cherrypicked action by Tesla engineering? A tragic coincidence is a compilation of events.
    - jstarfish 8y ago
      Tires, medicines and road surface treatments are all tested in controlled environments before being released to the public, not tested on the public. Conducting A/B tests with live human subjects and in a context where failures will kill people (including those not even associated with the experiment!) is the level of scientific rigor I expect from Josef Mengele.
      - Piskvorrr 8y ago
        Inb4 "don't buy a Tesla then" - this is not limited to Tesla customers, but extends to anyone in the vicinity.
      - jccooper 8y ago
        I drive on an experimental highway surface daily. It's quieter, but uses an unorthodox drainage system (that at least feels less controlled in wet conditions). Only real use statistics will say for sure if it's safety-equivalent to standard surfaces. The state DOT is testing this on me and tens of thousands of other people daily at high speed. Sure, I suspect that they did smaller scale testing before deploying it, but so has Tesla. So tell me: what's the difference?
        deleted 8y ago
        [deleted]
        gamblor956 8y ago
        That road surface was first tested in non-public situations to verify basic safety and functionality concerns. If it's reached the public-stage of testing, that means safety concerns were addressed and they're now testing durability.
  - nnf 8y ago
    I read the parent comment "Tesla is doing this" as confirming that Tesla vehicles alway have Autopilot running, even when it's not controlling the car, and comparing what Autopilot thinks the driver should do with what the driver is actually doing. If true, having that data would give Tesla an enormous amount of data on situations in which Autopilot might have failed if it was in control of the car, such as if Autopilot (when not engaged) didn't see an obstruction but the human driver swerved suddenly. This is also an advantage of Tesla installing full Autopilot hardware on every car regardless of whether the Autopilot upgrade has been purchased.

fastball 8y ago

What're your thoughts on cruise control?

code4tee 8y ago
Cruise control is very different. It doesn’t allow people to take their attention off the road. In the Tesla deaths it appears quite clear that the driver was not paying attention to the road.

jacksmith21006 8y ago

Great post and completely agree. You can NOT be kind of driving. Plus who does not work with a limited group of users while you are testing. That is true for tech that does not kill you and more so when it can. What Tesla is doing is honestly close to criminal. Waymo approach makes far more sense.

matz1 8y ago
I disagree, some kind of driving still can be useful. Of course like anything there always be some risk involved. Even when the death do happen, you can still learn from it.
- jacksmith21006 8y ago
  The problem is you can NOT be kind of driving. The Uber video of the driver is so telling. You can see he is reading his phone when the car hit the lady. Google has indicated this for years. There really should NOT be safety drivers behind the wheel. What Tesla is doing is almost criminal. Having a autopilot button when the car really can not do auto pilot is about as wrong as it gets. I do not believe you can do SDC directly to the consumer while it is really in beta.
  - matz1 8y ago
    Obviously you CAN. Of course it wont be perfect but most of time it should be fine. In some situation its still better than a driver using phone while driving a car that has no self driving capability whatsover, which many people do that right now. Waiting until the technology is perfect will take a long time.
    - jacksmith21006 8y ago
      Could not disagree more. When I say "CAN" I mean safely.

NickM 8y ago

As these accidents demonstrate drivers can’t be relied upon to ‘monitor’ the autopilot and react if something goes wrong. We don't know how many other accidents have been prevented by autopilot. A safety system should not be judged by whether it's perfect, but by whether having it is better than not having it. The NHTSA did a study on exactly this issue by looking at accident rates in cars with autopilot hardware before and after autopilot was actually enabled via a software update. They found that accident rates went down by about 40% after autosteer was enabled. This seems to suggest that either people are fine with monitoring and taking over when needed, or else autosteer was already so good in 2016 that it made fewer mistakes than humans. https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.pdf https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.pdf

frgtpsswrdlame 8y ago
>else autosteer was already so good in 2016 that it made fewer mistakes than humans. No no no no no! The NHTSA report says nothing about whether Tesla's system is safer than humans. In that study there are two buckets, one which is total Tesla miles in TACC-enabled cars and then after the update total Tesla miles in cars with TACC + Autosteer and they calculated on airbag deployments. Human driven miles are going to dominate both of those buckets and there's a reason the NHTSA report makes zero claims about Tesla's safety relative to human drivers. It's totally outside the scope of the study. Then add in that some researchers who are skeptical of the methodology and have been asking for the raw data from NHTSA/Tesla have yet to receive it.
- NickM 8y ago
  I didn't say it was safer than humans. I said that the study results suggest that either humans are fine with taking over and driving manually when required or if humans are not taking over when they should then autosteer might just be safer than humans. I don't claim to know exactly why autosteer would reduce accident rates, and neither does the study, but there must be some explanation and the last bit of my comment was simply speculating as to what that explanation might be.
  - 8y ago