8 ms·

> What we know is that the vehicle was on a divided highway with Autopilot engaged when a tractor trailer drove across the highway perpendicular to the Model S.

> What we know is that the vehicle was on a divided highway with Autopilot engaged when a tractor trailer drove across the highway perpendicular to the Model S. Neither Autopilot nor the driver noticed the white side of the tractor trailer against a brightly lit sky, so the brake was not applied. Tragic no doubt, but I'm relieved that this was not a "Autopilot did something very very wrong" story. Autopilot has the potential to save a large number of lives (I'm sure Tesla execs are thinking about touting "estimated lives saved by autopilot" if the numbers work out, after a few billion miles), so I hope incidents like this don't hamper public perception and therefore research.

Pxtl 10y ago
That doesn't sound like what happened. "The driver didn't notice" well, that's why they're using autopilot. No matter how many messages people throw at users, any autopilot that doesn't detect that the driver is awake and operating the vehicle is going to be used by sleeping or otherwise unattentive drivers. It's just human nature. So what likely happened was the driver was screwing around and the autopilot completely failed to notice a giant white truck crossing the road and blocking their way until it was too late. edit: reply is quite right, shouldn't speak ill of the dead that way. "Screwing around" is unfair. I just mean that "driver falls asleep/paying attention to other things" is the expected outcome of any autopilot system, and so it's unreasonable of autopilots to have any expectation of drivers for attentiveness.
- mikeash 10y ago
  From the information given, we don't know if the truck pulled out right in front of the car giving the driver no time to react, or if the truck pulled out a minute beforehand and the driver was just oblivious. In the absence of any information either way, it's nasty to just assume the driver was inattentive.
  - krschultz 10y ago
    I don't think it's nasty, I think it's realistic. I also think that it aligns with the research that shows humans take a significant amount of time to adjust to taking back over control from autopilot. This is one of the primary design considerations behind the Google self driving car having absolutely no steering wheel or brake. The illusion of humans being able to be the fallback when they're not physically capable of it is a dangerous situation.
    - mikeash 10y ago
      I don't think it's realistic to rampantly speculate about a situation where so little information is available. And I absolutely think it is nasty to talk ill of the dead when there is zero evidence for what's being said. For all we know the driver had half a second to react before he died. Or maybe he had five minutes. The point is, we don't know. To start from that position and say it's "likely... the driver was screwing around" is pretty crappy.
      - krschultz 10y ago
        Fair, I wasn't reading it as a knock on the driver. I read it as a mistake for Tesla to hide behind the driver having ultimate responsibility when the reality is no driver is going to actually be attentive or even capable of taking over.
      - Pxtl 10y ago
        I'll admit that the word "screwing around" was unfair and judgemental, but I'm gonna be honest: I would never want to own a car with this sort of "autopilot with driver fallback" feature because I'd totally fall asleep at the wheel. Completely. So I didn't mean any judgment by it. I could have made the same mistake. In fact, I'm judging the software, if anything. Any autopilot software that doesn't force the driver to be attentive is effectively taking the responsibility for driving onto themselves, and so cannot blame the driver for being inattentive. You can't say "I got this" and then at the last second say "oh shit, I need you driver!" even if you put warnings that you don't really got this, because you're dealing with human beings who do normal human things. Yes, I shouldn't have said the driver was screwing around, you're right. But if the driver fell asleep, and a conscious driver would have avoided this accident? I still think it's Tesla's fault, because he'd be awake and alive in a manual car.
- gjulianm 10y ago
  From the article, it seems that this autopilot checks that the driver is attentive. > When drivers activate Autopilot, the acknowledgment box explains, among other things, that Autopilot “is an assist feature that requires you to keep your hands on the steering wheel at all times," and that "you need to maintain control and responsibility for your vehicle” while using it. Additionally, every time that Autopilot is engaged, the car reminds the driver to “Always keep your hands on the wheel. Be prepared to take over at any time.” The system also makes frequent checks to ensure that the driver's hands remain on the wheel and provides visual and audible alerts if hands-on is not detected. It then gradually slows down the car until hands-on is detected again.
  - BinaryIdiot 10y ago
    It does but it's not terribly frequent and it's easy to bypass. When you give people a feature they're going to use it however they want regardless of what you tell them. Users are funny like that :)
  - gohrt 10y ago
    What if you get into a collision while you are reading all the text in that acknowledgement box?
    - mikeash 10y ago
      I don't think the setting can be changed unless the car is in Park.
  - mikeash 10y ago
    Those "frequent checks" are pretty rare. I've gone 15-20 minutes without seeing one. For a while it was every 3-5 minutes, but they've dialed it back up in newer software updates.
- deleted 10y ago
  [deleted]
- hueving 10y ago
  >detect that the driver is awake and operating the vehicle RTFA. It mentions explicitly that autopilot checks for hands on the wheel.
  - hluska 10y ago
    This is a very weak response.
  - mikeash 10y ago
    And yet it really doesn't. Every so often it'll beep at you to make your presence known, but it can easily go many minutes with hands off the wheel and no checks.
- diamondlovesyou 10y ago
  I'd say most accidents are the result of the driver "not noticing", especially in the current majority of cars which lack autopilot, so even if this edge case exists in the future, having an autopilot is still an improvement.
- kennysmoothx 10y ago
  > That doesn't sound like what happened. Everything is speculation at this point so I am not sure how you can assume that's not what happened. > "The driver didn't notice" well, that's why they're using autopilot. If Tesla builds and designs a system to be used as an assistant, explicitly tells and warns the user of it's use case and potential danger of using it as anything other than an assistant, then the excuse that "it's human nature" is null, I don't think it's human nature to leave your safety and well being at the hands of something designed to be nothing more than an assistant. Perhaps the name autopilot exerts to much confidence in its capabilities.
dvcc 10y ago
How would we ever know that the autopilot acted correctly? Should companies be required to release logs after every crash when a car is in autopilot? Seems fair.
- nomercy400 10y ago
  Black box? But logs must be useful for the insurance claim (by either the manufacturer or the driver).
georgecmu 10y ago
Autopilot engaged when a tractor trailer drove across the highway perpendicular to the Model S. Neither Autopilot nor the driver noticed the white side of the tractor trailer against a brightly lit sky, so the brake was not applied. How is this not an "Autopilot did something very very wrong" story? Tesla autopilot drove the vehicle full-speed into a tractor-trailer, which wasn't registered by its sensor systems. It's great that Tesla already has an explanation for the failure, but the situation as described is far from an improbable edge case. What they are saying is that the autopilot can't detect white objects when a bright light is present -- that's a pretty serious limitation to overlook.
- mikeash 10y ago
  Human drivers do this kind of thing with great frequency. As long as the system is at least as good as a human driver at this stuff, I'm happy with it. Actually, scratch that: as long as the system combined with me is at least as good as a human driver, I'm happy with it.
- nomercy400 10y ago
  How do tractor trailers drive perpendicular on a highway in a probable case? Still, I agree it is strange that a white/glossy object cannot be detected with radar. Is it a stealth truck?
- simonsarris 10y ago
  Where I live some highways have a sign that says: > CAUTION: SOLAR GLARE IN THE A.M. It's serious enough to warrant its own road sign, but its not as much of a realistic issue on a divided highway, so maybe this road didn't have such a sign. This type of danger is probably what the article is describing.
- joemi 10y ago
  If the driver didn't notice it either, why is it Autopilot's fault?
  - ceejayoz 10y ago
    The autopilot may reduce driver attentiveness, making the edge cases in its automation more deadly than they might've been without it.
rspeer 10y ago
The only situation I can imagine a driver not noticing a huge truck perpendicular to them on the highway, regardless of what color it is, is that they're not paying any attention at all. This peculiar wording is making it sound like the computer made the same mistake as the driver, but it seems much more likely that they made two different mistakes. The computer's mistake was in image recognition, and the person's mistake was trusting the autopilot fully and not paying attention to the road.
- deleted 10y ago
  [deleted]
- stcredzero 10y ago
  The only situation I can imagine a driver not noticing a huge truck perpendicular to them on the highway, regardless of what color it is, is that they're not paying any attention at all. The optical distortions that cause road surface mirages could well entirely hide another vehicle. In this case, since the trailer was painted white, only the wheels and lower chassis would need to be hidden for the image to become unrecognizable. Not saying that's what happened in this case, but there are situations when your forward vision could be compromised. (There is a theory that the colder ocean may have caused a similar optical phenomenon and hidden the iceberg that sunk the Titanic.)
dismal2 10y ago
This seems like a pretty big oversight, large object that car is too tall to clear safely seems like it could cause a lot of problems. Would the car try to make a lane change while next to a trailer?
- mikeash 10y ago
  Tesla's Autopilot only changes lanes when commanded to by the driver. This is done by putting the turn signal on. If you put the turn signal on while next to a trailer that doesn't have stuff coming down near the ground, then yes, the car will likely change lanes right into it. But it's easy to avoid: just look before you signal.
- mdorazio 10y ago
  Unless it's changed recently, drivers have to tell the car specifically to change lanes (using the turn signal) or else it won't do so. And yes, this is the same issue as that other fairly recent incident in which a Model S drove itself into a parked trailer when the owner summoned it (supposedly inadvertently) due to the height of the trailer bed above the ground. Very concerning to me given the prevalence of trucks on the road.
krschultz 10y ago
I'll give you that it's not a story of Autopilot driving off a cliff, but it may be a case of someone getting killed due to a problem with Autopilot. While it's factually true that the driver did not notice, what we can't know is if the driver would have noticed if they were more aware without autopilot engaged. Anyone that thinks drivers pay as much attention to the road when Autopilot is enabled is being naive.
orthoganol 10y ago
If autopilot fails when it's sunny and the obstacle is white, I have doubts about autopilot.
- andrewchambers 10y ago
  Humans fail for even more stupid reasons.
sawthat 10y ago
Technologies like autopilot are going to save lives. This was one of around 90 fatal traffic accidents in the US on the same day (based on historical averages). It's going to be hard to make a foolproof system, however anything that makes it less likely for an inattentive drive to crash into something is a win.

Pxtl 10y ago

That doesn't sound like what happened. "The driver didn't notice" well, that's why they're using autopilot. No matter how many messages people throw at users, any autopilot that doesn't detect that the driver is awake and operating the vehicle is going to be used by sleeping or otherwise unattentive drivers. It's just human nature. So what likely happened was the driver was screwing around and the autopilot completely failed to notice a giant white truck crossing the road and blocking their way until it was too late. edit: reply is quite right, shouldn't speak ill of the dead that way. "Screwing around" is unfair. I just mean that "driver falls asleep/paying attention to other things" is the expected outcome of any autopilot system, and so it's unreasonable of autopilots to have any expectation of drivers for attentiveness.

mikeash 10y ago
From the information given, we don't know if the truck pulled out right in front of the car giving the driver no time to react, or if the truck pulled out a minute beforehand and the driver was just oblivious. In the absence of any information either way, it's nasty to just assume the driver was inattentive.
- krschultz 10y ago
  I don't think it's nasty, I think it's realistic. I also think that it aligns with the research that shows humans take a significant amount of time to adjust to taking back over control from autopilot. This is one of the primary design considerations behind the Google self driving car having absolutely no steering wheel or brake. The illusion of humans being able to be the fallback when they're not physically capable of it is a dangerous situation.
  - mikeash 10y ago
    I don't think it's realistic to rampantly speculate about a situation where so little information is available. And I absolutely think it is nasty to talk ill of the dead when there is zero evidence for what's being said. For all we know the driver had half a second to react before he died. Or maybe he had five minutes. The point is, we don't know. To start from that position and say it's "likely... the driver was screwing around" is pretty crappy.
    - krschultz 10y ago
      Fair, I wasn't reading it as a knock on the driver. I read it as a mistake for Tesla to hide behind the driver having ultimate responsibility when the reality is no driver is going to actually be attentive or even capable of taking over.
    - Pxtl 10y ago
      I'll admit that the word "screwing around" was unfair and judgemental, but I'm gonna be honest: I would never want to own a car with this sort of "autopilot with driver fallback" feature because I'd totally fall asleep at the wheel. Completely. So I didn't mean any judgment by it. I could have made the same mistake. In fact, I'm judging the software, if anything. Any autopilot software that doesn't force the driver to be attentive is effectively taking the responsibility for driving onto themselves, and so cannot blame the driver for being inattentive. You can't say "I got this" and then at the last second say "oh shit, I need you driver!" even if you put warnings that you don't really got this, because you're dealing with human beings who do normal human things. Yes, I shouldn't have said the driver was screwing around, you're right. But if the driver fell asleep, and a conscious driver would have avoided this accident? I still think it's Tesla's fault, because he'd be awake and alive in a manual car.
gjulianm 10y ago
From the article, it seems that this autopilot checks that the driver is attentive. > When drivers activate Autopilot, the acknowledgment box explains, among other things, that Autopilot “is an assist feature that requires you to keep your hands on the steering wheel at all times," and that "you need to maintain control and responsibility for your vehicle” while using it. Additionally, every time that Autopilot is engaged, the car reminds the driver to “Always keep your hands on the wheel. Be prepared to take over at any time.” The system also makes frequent checks to ensure that the driver's hands remain on the wheel and provides visual and audible alerts if hands-on is not detected. It then gradually slows down the car until hands-on is detected again.
- BinaryIdiot 10y ago
  It does but it's not terribly frequent and it's easy to bypass. When you give people a feature they're going to use it however they want regardless of what you tell them. Users are funny like that :)
- gohrt 10y ago
  What if you get into a collision while you are reading all the text in that acknowledgement box?
  - mikeash 10y ago
    I don't think the setting can be changed unless the car is in Park.
- mikeash 10y ago
  Those "frequent checks" are pretty rare. I've gone 15-20 minutes without seeing one. For a while it was every 3-5 minutes, but they've dialed it back up in newer software updates.
deleted 10y ago
[deleted]
hueving 10y ago
>detect that the driver is awake and operating the vehicle RTFA. It mentions explicitly that autopilot checks for hands on the wheel.
- hluska 10y ago
  This is a very weak response.
- mikeash 10y ago
  And yet it really doesn't. Every so often it'll beep at you to make your presence known, but it can easily go many minutes with hands off the wheel and no checks.
diamondlovesyou 10y ago
I'd say most accidents are the result of the driver "not noticing", especially in the current majority of cars which lack autopilot, so even if this edge case exists in the future, having an autopilot is still an improvement.
kennysmoothx 10y ago
> That doesn't sound like what happened. Everything is speculation at this point so I am not sure how you can assume that's not what happened. > "The driver didn't notice" well, that's why they're using autopilot. If Tesla builds and designs a system to be used as an assistant, explicitly tells and warns the user of it's use case and potential danger of using it as anything other than an assistant, then the excuse that "it's human nature" is null, I don't think it's human nature to leave your safety and well being at the hands of something designed to be nothing more than an assistant. Perhaps the name autopilot exerts to much confidence in its capabilities.

dvcc 10y ago

How would we ever know that the autopilot acted correctly? Should companies be required to release logs after every crash when a car is in autopilot? Seems fair.

nomercy400 10y ago
Black box? But logs must be useful for the insurance claim (by either the manufacturer or the driver).

georgecmu 10y ago

Autopilot engaged when a tractor trailer drove across the highway perpendicular to the Model S. Neither Autopilot nor the driver noticed the white side of the tractor trailer against a brightly lit sky, so the brake was not applied. How is this not an "Autopilot did something very very wrong" story? Tesla autopilot drove the vehicle full-speed into a tractor-trailer, which wasn't registered by its sensor systems. It's great that Tesla already has an explanation for the failure, but the situation as described is far from an improbable edge case. What they are saying is that the autopilot can't detect white objects when a bright light is present -- that's a pretty serious limitation to overlook.

mikeash 10y ago
Human drivers do this kind of thing with great frequency. As long as the system is at least as good as a human driver at this stuff, I'm happy with it. Actually, scratch that: as long as the system combined with me is at least as good as a human driver, I'm happy with it.
nomercy400 10y ago
How do tractor trailers drive perpendicular on a highway in a probable case? Still, I agree it is strange that a white/glossy object cannot be detected with radar. Is it a stealth truck?
simonsarris 10y ago
Where I live some highways have a sign that says: > CAUTION: SOLAR GLARE IN THE A.M. It's serious enough to warrant its own road sign, but its not as much of a realistic issue on a divided highway, so maybe this road didn't have such a sign. This type of danger is probably what the article is describing.
joemi 10y ago
If the driver didn't notice it either, why is it Autopilot's fault?
- ceejayoz 10y ago
  The autopilot may reduce driver attentiveness, making the edge cases in its automation more deadly than they might've been without it.

rspeer 10y ago

The only situation I can imagine a driver not noticing a huge truck perpendicular to them on the highway, regardless of what color it is, is that they're not paying any attention at all. This peculiar wording is making it sound like the computer made the same mistake as the driver, but it seems much more likely that they made two different mistakes. The computer's mistake was in image recognition, and the person's mistake was trusting the autopilot fully and not paying attention to the road.

deleted 10y ago
[deleted]
stcredzero 10y ago
The only situation I can imagine a driver not noticing a huge truck perpendicular to them on the highway, regardless of what color it is, is that they're not paying any attention at all. The optical distortions that cause road surface mirages could well entirely hide another vehicle. In this case, since the trailer was painted white, only the wheels and lower chassis would need to be hidden for the image to become unrecognizable. Not saying that's what happened in this case, but there are situations when your forward vision could be compromised. (There is a theory that the colder ocean may have caused a similar optical phenomenon and hidden the iceberg that sunk the Titanic.)

dismal2 10y ago

This seems like a pretty big oversight, large object that car is too tall to clear safely seems like it could cause a lot of problems. Would the car try to make a lane change while next to a trailer?

mikeash 10y ago
Tesla's Autopilot only changes lanes when commanded to by the driver. This is done by putting the turn signal on. If you put the turn signal on while next to a trailer that doesn't have stuff coming down near the ground, then yes, the car will likely change lanes right into it. But it's easy to avoid: just look before you signal.
mdorazio 10y ago
Unless it's changed recently, drivers have to tell the car specifically to change lanes (using the turn signal) or else it won't do so. And yes, this is the same issue as that other fairly recent incident in which a Model S drove itself into a parked trailer when the owner summoned it (supposedly inadvertently) due to the height of the trailer bed above the ground. Very concerning to me given the prevalence of trucks on the road.

krschultz 10y ago

I'll give you that it's not a story of Autopilot driving off a cliff, but it may be a case of someone getting killed due to a problem with Autopilot. While it's factually true that the driver did not notice, what we can't know is if the driver would have noticed if they were more aware without autopilot engaged. Anyone that thinks drivers pay as much attention to the road when Autopilot is enabled is being naive.

orthoganol 10y ago

If autopilot fails when it's sunny and the obstacle is white, I have doubts about autopilot.

andrewchambers 10y ago
Humans fail for even more stupid reasons.

sawthat 10y ago

Technologies like autopilot are going to save lives. This was one of around 90 fatal traffic accidents in the US on the same day (based on historical averages). It's going to be hard to make a foolproof system, however anything that makes it less likely for an inattentive drive to crash into something is a win.