28 ms·

The whole premise of "almost working" autopilot is the logical thing to do from Tesla's business perspective but deeply flawed from a safety one. The allure of

The whole premise of "almost working" autopilot is the logical thing to do from Tesla's business perspective but deeply flawed from a safety one. The allure of having a car that drives itself is to relax the mind from paying attention to the road, handling 99% of the situations correctly through the software. But the 1% that require human input are made much more dangerous because the driver not only needs to handle the situation, they need to instantly catch up and familiarize themselves with the situation they've found themselves in. What was the car trying to do? Why is it suddenly stopping? What do my blind spots look like? Is there anybody behind me? Was anybody in the process of passing me? Unfortunately the time it takes to refamiliarize with the situation could exceed the total time budget to respond. It's similar to not paying attention to the road and being in the passenger seat when a junior driver is driving instead. But there are 2 exceptions: - Most people increase the attention they pay to the road when a learner is behind the wheel. Preparing to respond and offer guidance to correct. - Self-driving software is a lot more indeterministic than a new driver. And software updates can easily change the behavior in non trivial ways. As if the driving student had Multiple Personality Disorder, one day being over cautious in certain situation and other days over-confident. Though I contend that MPD is actually more predictable than an opaque algorithm, as there are other human clues about the behavior mode of the moment that could help guide the response of the trainer.

amelius 5y ago
The standard argument in favor of FSD vehicles is that they perform better than the average human driver. Except it doesn't take into account the amount of adrenaline involved.
- lotsofpulp 5y ago
  I do not see the point of conversing about full self driving when it is nowhere close to existing yet.
  - simion314 5y ago
    Tesla use this names unforunetly "FSD" and "Autopilot", instead of lane assist or driver assist... we need everyone including fanboys to demand Tesla stop push fake capabilities, maybe name it MFFSD = Maybe in future Full Self Driving
    - lotsofpulp 5y ago
      Yes, that was the point of my comment. Tesla does not unfortunately use the words “full self driving”, they intentionally use the words “full self driving” for marketing purposes to mislead people. How it is not a clear case of false advertising for regulators to go after, I do not know. I could even envision the plausible deniability/marketing bullshit behind “self driving”, but to go out of your way and declare FULL self driving, and not get smacked for false advertising is beyond me.
- rootusrootus 5y ago
  It also doesn't take account how much the average is skewed by a minority of really terrible drivers.
- simion314 5y ago
  And the issue is the numbers are fake: different population, different car types are compared. Dis-engagements are not noted as "20K dis-engagements = 20K times the human saved himself from the bad AI. Would people get operated by an AI doctor that is better then the medicine students and the drunk doctors but only when a good doctor is paying attention and intervenes in case the AI shits itself?
  - buran77 5y ago
    While AP is sometimes touted as "FSD safer than a human driver" in reality it's just a set of driver assists that can help a driver drive safer. Since AP doesn't drive itself except for very limited definitions of driving, it can't be itself safer than a human driver. Like when a pair of sneakers help a runner bit can't logically be called "better than a human runner". Let's not forget that the human does most of the work in either example. As long as any "self-driving" car operates only under human supervision it's not relevant to compare that system taken independently to the human. Compare the FSD alone with the human alone and then we have stats. Everything else is candy for people with more money than sense.
jazzyjackson 5y ago
I'm also of the opinion that automation is not compatible with safe driving. The issue is AI is able to handle the simple tasks first, trusting humans to takeover in outlier scenarios. But then, where does a new driver gain experience to handle the outliers? We already went through this with airlines and decided to require pilots fly a certain number of hours without the automation, so when automation fails they are able to control the aircraft with some confidence. As it is self driving cars are a luxury item for experienced drivers but I fear a future where a plurality of people panic when control is handed over to them once the computer determines the situation is too risky to continue on autopilot. Some great backstory on automation in airlines is 99% Invisible's "Children of the Magenta": https://99percentinvisible.org/episode/children-of-the-magenta-automation-paradox-pt-1/ https://99percentinvisible.org/episode/children-of-the-magen...
- gitfan86 5y ago
  We don't let 80 year old pilots with bad eyesight fly planes. We don't let pilots fly planes 15/hours a day for multiple days in a row. Pilots must pass tests every year. If we are going to hold car drivers to the same training and restrictions that pilots are held to we will have far fewer drivers and much safer drivers. But it seems very unrealistic to think that will happen in a democracy. Realistically it seems that technology is the most realistic way to reduce deaths.
  - throwaway0a5e 5y ago
    The overwhelming majority of mayhem on the roads is caused by the long tail of bad decisions made by humans who would be able to pass whatever dumb requirements you concoct. Kicking the elderly off the road, throwing the book at the one guy who gets caught doing NYC-DC and back in a day and onerous regulations on equipment is fundamentally a waste of resources chasing a long tail and is going to do nearly nothing to actually affect things relative to the expense. Most of the problems are caused by people doing things they knew were risky and stupid at the time and did them anyway, breaking rules in an irresponsible manner, distractions, intoxication, etc. Being applicable to a broad cross section of the actual problem is why low hanging fruit type stuff like AEB has such a big impact.
FireBeyond 5y ago
> The allure of having a car that drives itself is to relax the mind from paying attention to the road, handling 99% of the situations correctly through the software. But the 1% that require human input are made much more dangerous because the driver not only needs to handle the situation, they need to instantly catch up and familiarize themselves with the situation they've found themselves in. There's also the separate recognition that no matter Musk's bombast on Twitter, Tesla's FSD stack is _nowhere near 99%_ correct handling. It's still absolutely regularly confused by bike lanes, only recently resolved roundabouts, and I could go on. But bottom line, not 99%. I'd be thinking closer to 60-75%, at best.
SkyPuncher 5y ago
> The allure of having a car that drives itself is to relax the mind from paying attention to the road, handling 99% of the situations correctly through the software. I have this with a Hyundai Palisade with L2. I still have to watch the road. I still need to identify emergency situations and navigation. I still need to _assess_ many things - just like I would without any support. However, I don't need to take action on everything. This is mentally and physically freeing. On well marked highways, in particular, it means I can just cruise along without much worry. The problem with Tesla is they set the expectation that you don't have to pay attention. They've set an expectation that the car is capable of _assessing_ every situation when it's clearly not. They further re-enforce this with the car's UI and behaviors such that drivers get lulled into not paying attention. To me, it almost sounds like Tesla would have better UX and safety if they reframed the capabilities of the self-driving to make it clear that the driver still needs to be assessing.
- mariusor 5y ago
  > They've set an expectation that the car is capable of _assessing_ every situation when it's clearly not. How did they do that ? The materials that new Tesla owners receive are specifying that the driver needs to pay attention in all circumstances. From the https://www.tesla.com/support/autopilot https://www.tesla.com/support/autopilot page: > Autopilot and Full Self-Driving Capability are intended for use with a fully attentive driver, who has their hands on the wheel and is prepared to take over at any moment. While these features are designed to become more capable over time, the currently enabled features do not make the vehicle autonomous.
  - jjulius 5y ago
    >The materials that new Tesla owners receive are specifying that the driver needs to pay attention in all circumstances. Can you please provide an example of material that owners actually receive that states as much? It's safe to assume that we can't expect everyone to read a support page on a website, just like how it's safe to assume that most people don't read the owner's manual when they buy a new ICE vehicle. >How did they do that? It's in how they promote it publicly. It's in how Musk constantly says it's "a year away". It's in the fucking name - "autopilot" has certain connotations within society writ large that Tesla can't just handwave away with legalese.
antattack 5y ago
Most likely scenarios: -Driver stopped himself due to medical reason (driver was 74). -Car's power-train issue caused it to stop. Less likely in dual-motor configuration. Even less likely to happen suddenly. -Autopilot was on and driver fell asleep - Autopilot is programmed to come to a stop if driver input is not detected. Because it's Tesla there's great deal of attention given to Autopilot but in this circumstance, where car stopped completely, driver most likely have been unable to make the car go (for medical or technical reasons).
- DennisP 5y ago
  > driver fell asleep - Autopilot is programmed to come to a stop if driver input is not detected Hopefully it slows gently, signals a lane shift, and parks on the shoulder with hazards on, rather than slamming on brakes in the middle of the highway with cars close behind.
  - antattack 5y ago
    Of course, Autopilot gives driver visual and audio warnings before stopping, and it takes a while to activate. It does so because it's a Level 2 system and requires driver's supervision. In absence of supervision it has to stop as it's not autonomous.
    - DennisP 5y ago
      GM Supercruise pulls over as I described: https://gmauthority.com/blog/2016/11/gm-super-cruise-technology-will-pull-the-vehicle-over-if-the-driver-is-unresponsive/ https://gmauthority.com/blog/2016/11/gm-super-cruise-technol... Tesla brakes gradually and turns on hazard lights: https://www.youtube.com/watch?v=oBIKikBmdN8 https://www.youtube.com/watch?v=oBIKikBmdN8 As common sense suggests, a sleeping driver does not cause a Tesla to slam on its brakes in the middle of high-speed traffic. That is not the cause of this accident.
- whiplash451 5y ago
  What makes you rule out an issue with the AI?
decodebytes 5y ago
Do we know this was the result of FSD and not another malfunction, such as a seized component, power failure etc? Seems we should not speculate if we don't know the RCA?
- robbedpeter 5y ago
  No, the article says police speculate it was mechanical failure and loss of power, but I'm sure we'll see plenty of rigorous, unbiased followup in the following weeks. It's funny to see the pro and anti Tesla crowd arrive at truth through verbal combat, at least.
whiplash451 5y ago
Let's make sure we wait for the investigation to be done before accusing the AI so quickly.

amelius 5y ago

The standard argument in favor of FSD vehicles is that they perform better than the average human driver. Except it doesn't take into account the amount of adrenaline involved.

lotsofpulp 5y ago
I do not see the point of conversing about full self driving when it is nowhere close to existing yet.
- simion314 5y ago
  Tesla use this names unforunetly "FSD" and "Autopilot", instead of lane assist or driver assist... we need everyone including fanboys to demand Tesla stop push fake capabilities, maybe name it MFFSD = Maybe in future Full Self Driving
  - lotsofpulp 5y ago
    Yes, that was the point of my comment. Tesla does not unfortunately use the words “full self driving”, they intentionally use the words “full self driving” for marketing purposes to mislead people. How it is not a clear case of false advertising for regulators to go after, I do not know. I could even envision the plausible deniability/marketing bullshit behind “self driving”, but to go out of your way and declare FULL self driving, and not get smacked for false advertising is beyond me.
rootusrootus 5y ago
It also doesn't take account how much the average is skewed by a minority of really terrible drivers.
simion314 5y ago
And the issue is the numbers are fake: different population, different car types are compared. Dis-engagements are not noted as "20K dis-engagements = 20K times the human saved himself from the bad AI. Would people get operated by an AI doctor that is better then the medicine students and the drunk doctors but only when a good doctor is paying attention and intervenes in case the AI shits itself?
- buran77 5y ago
  While AP is sometimes touted as "FSD safer than a human driver" in reality it's just a set of driver assists that can help a driver drive safer. Since AP doesn't drive itself except for very limited definitions of driving, it can't be itself safer than a human driver. Like when a pair of sneakers help a runner bit can't logically be called "better than a human runner". Let's not forget that the human does most of the work in either example. As long as any "self-driving" car operates only under human supervision it's not relevant to compare that system taken independently to the human. Compare the FSD alone with the human alone and then we have stats. Everything else is candy for people with more money than sense.

jazzyjackson 5y ago

I'm also of the opinion that automation is not compatible with safe driving. The issue is AI is able to handle the simple tasks first, trusting humans to takeover in outlier scenarios. But then, where does a new driver gain experience to handle the outliers? We already went through this with airlines and decided to require pilots fly a certain number of hours without the automation, so when automation fails they are able to control the aircraft with some confidence. As it is self driving cars are a luxury item for experienced drivers but I fear a future where a plurality of people panic when control is handed over to them once the computer determines the situation is too risky to continue on autopilot. Some great backstory on automation in airlines is 99% Invisible's "Children of the Magenta": https://99percentinvisible.org/episode/children-of-the-magenta-automation-paradox-pt-1/ https://99percentinvisible.org/episode/children-of-the-magen...

gitfan86 5y ago
We don't let 80 year old pilots with bad eyesight fly planes. We don't let pilots fly planes 15/hours a day for multiple days in a row. Pilots must pass tests every year. If we are going to hold car drivers to the same training and restrictions that pilots are held to we will have far fewer drivers and much safer drivers. But it seems very unrealistic to think that will happen in a democracy. Realistically it seems that technology is the most realistic way to reduce deaths.
- throwaway0a5e 5y ago
  The overwhelming majority of mayhem on the roads is caused by the long tail of bad decisions made by humans who would be able to pass whatever dumb requirements you concoct. Kicking the elderly off the road, throwing the book at the one guy who gets caught doing NYC-DC and back in a day and onerous regulations on equipment is fundamentally a waste of resources chasing a long tail and is going to do nearly nothing to actually affect things relative to the expense. Most of the problems are caused by people doing things they knew were risky and stupid at the time and did them anyway, breaking rules in an irresponsible manner, distractions, intoxication, etc. Being applicable to a broad cross section of the actual problem is why low hanging fruit type stuff like AEB has such a big impact.

FireBeyond 5y ago

> The allure of having a car that drives itself is to relax the mind from paying attention to the road, handling 99% of the situations correctly through the software. But the 1% that require human input are made much more dangerous because the driver not only needs to handle the situation, they need to instantly catch up and familiarize themselves with the situation they've found themselves in. There's also the separate recognition that no matter Musk's bombast on Twitter, Tesla's FSD stack is _nowhere near 99%_ correct handling. It's still absolutely regularly confused by bike lanes, only recently resolved roundabouts, and I could go on. But bottom line, not 99%. I'd be thinking closer to 60-75%, at best.

SkyPuncher 5y ago

> The allure of having a car that drives itself is to relax the mind from paying attention to the road, handling 99% of the situations correctly through the software. I have this with a Hyundai Palisade with L2. I still have to watch the road. I still need to identify emergency situations and navigation. I still need to _assess_ many things - just like I would without any support. However, I don't need to take action on everything. This is mentally and physically freeing. On well marked highways, in particular, it means I can just cruise along without much worry. The problem with Tesla is they set the expectation that you don't have to pay attention. They've set an expectation that the car is capable of _assessing_ every situation when it's clearly not. They further re-enforce this with the car's UI and behaviors such that drivers get lulled into not paying attention. To me, it almost sounds like Tesla would have better UX and safety if they reframed the capabilities of the self-driving to make it clear that the driver still needs to be assessing.

mariusor 5y ago
> They've set an expectation that the car is capable of _assessing_ every situation when it's clearly not. How did they do that ? The materials that new Tesla owners receive are specifying that the driver needs to pay attention in all circumstances. From the https://www.tesla.com/support/autopilot https://www.tesla.com/support/autopilot page: > Autopilot and Full Self-Driving Capability are intended for use with a fully attentive driver, who has their hands on the wheel and is prepared to take over at any moment. While these features are designed to become more capable over time, the currently enabled features do not make the vehicle autonomous.
- jjulius 5y ago
  >The materials that new Tesla owners receive are specifying that the driver needs to pay attention in all circumstances. Can you please provide an example of material that owners actually receive that states as much? It's safe to assume that we can't expect everyone to read a support page on a website, just like how it's safe to assume that most people don't read the owner's manual when they buy a new ICE vehicle. >How did they do that? It's in how they promote it publicly. It's in how Musk constantly says it's "a year away". It's in the fucking name - "autopilot" has certain connotations within society writ large that Tesla can't just handwave away with legalese.

antattack 5y ago

Most likely scenarios: -Driver stopped himself due to medical reason (driver was 74). -Car's power-train issue caused it to stop. Less likely in dual-motor configuration. Even less likely to happen suddenly. -Autopilot was on and driver fell asleep - Autopilot is programmed to come to a stop if driver input is not detected. Because it's Tesla there's great deal of attention given to Autopilot but in this circumstance, where car stopped completely, driver most likely have been unable to make the car go (for medical or technical reasons).

DennisP 5y ago
> driver fell asleep - Autopilot is programmed to come to a stop if driver input is not detected Hopefully it slows gently, signals a lane shift, and parks on the shoulder with hazards on, rather than slamming on brakes in the middle of the highway with cars close behind.
- antattack 5y ago
  Of course, Autopilot gives driver visual and audio warnings before stopping, and it takes a while to activate. It does so because it's a Level 2 system and requires driver's supervision. In absence of supervision it has to stop as it's not autonomous.
  - DennisP 5y ago
    GM Supercruise pulls over as I described: https://gmauthority.com/blog/2016/11/gm-super-cruise-technology-will-pull-the-vehicle-over-if-the-driver-is-unresponsive/ https://gmauthority.com/blog/2016/11/gm-super-cruise-technol... Tesla brakes gradually and turns on hazard lights: https://www.youtube.com/watch?v=oBIKikBmdN8 https://www.youtube.com/watch?v=oBIKikBmdN8 As common sense suggests, a sleeping driver does not cause a Tesla to slam on its brakes in the middle of high-speed traffic. That is not the cause of this accident.
whiplash451 5y ago
What makes you rule out an issue with the AI?

decodebytes 5y ago

Do we know this was the result of FSD and not another malfunction, such as a seized component, power failure etc? Seems we should not speculate if we don't know the RCA?

robbedpeter 5y ago
No, the article says police speculate it was mechanical failure and loss of power, but I'm sure we'll see plenty of rigorous, unbiased followup in the following weeks. It's funny to see the pro and anti Tesla crowd arrive at truth through verbal combat, at least.

whiplash451 5y ago

Let's make sure we wait for the investigation to be done before accusing the AI so quickly.