6 ms·

Autopilot engaged when a tractor trailer drove across the highway perpendicular to the Model S. Neither Autopilot nor the driver noticed the white side of the t

Autopilot engaged when a tractor trailer drove across the highway perpendicular to the Model S. Neither Autopilot nor the driver noticed the white side of the tractor trailer against a brightly lit sky, so the brake was not applied. How is this not an "Autopilot did something very very wrong" story? Tesla autopilot drove the vehicle full-speed into a tractor-trailer, which wasn't registered by its sensor systems. It's great that Tesla already has an explanation for the failure, but the situation as described is far from an improbable edge case. What they are saying is that the autopilot can't detect white objects when a bright light is present -- that's a pretty serious limitation to overlook.

mikeash 10y ago
Human drivers do this kind of thing with great frequency. As long as the system is at least as good as a human driver at this stuff, I'm happy with it. Actually, scratch that: as long as the system combined with me is at least as good as a human driver, I'm happy with it.
nomercy400 10y ago
How do tractor trailers drive perpendicular on a highway in a probable case? Still, I agree it is strange that a white/glossy object cannot be detected with radar. Is it a stealth truck?
simonsarris 10y ago
Where I live some highways have a sign that says: > CAUTION: SOLAR GLARE IN THE A.M. It's serious enough to warrant its own road sign, but its not as much of a realistic issue on a divided highway, so maybe this road didn't have such a sign. This type of danger is probably what the article is describing.
joemi 10y ago
If the driver didn't notice it either, why is it Autopilot's fault?
- ceejayoz 10y ago
  The autopilot may reduce driver attentiveness, making the edge cases in its automation more deadly than they might've been without it.
SapphireSun 10y ago
I guess the question is, was the driver able to see it? I can see how maybe the contrast was so low that maybe it was impossible to see anything except for the wheels which you might not notice. And if the part of the tractor trailer in the path is elevated... sensors sweeping forward see just road. I guess I'd like to see an NHTSB simulation of the situation in manual driving to see how humans do unassisted.
- stcredzero 10y ago
  And if the part of the tractor trailer in the path is elevated... sensors sweeping forward see just road. It could be even worse. If there was a "road surface mirage" then it's possible that both the truck and the road underneath it were not visible at all. Since the forward radar doesn't have the range or coverage to see a truck at a distance, the only sensor that could have detected the problem was the camera. Because of the phenomena I've mentioned above and the parent post has mentioned, we know this can't possibly work. Even worse, the circumstances in which the car's sensor could fail are precisely the ones in which the Mark 1 Eyeball would also fail.
- tim333 10y ago
  A google search shows several white trailers against blue skies. It's hard to see how a human could miss them unless there were weird conditions or they weren't looking that way. https://www.google.co.uk/search?q=semi-trailer&num=30&source=lnms&tbm=isch&sa=X&ved=0ahUKEwj4_vmf8tDNAhWIJMAKHSRTBnoQ_AUICCgB&biw=1093&bih=479 https://www.google.co.uk/search?q=semi-trailer&num=30&source...
derekp7 10y ago
Replace "autopilot" with "cruise control" -- would you have the same situation? Autopilot is basically advanced cruise control plus steering assist. From what I make of this story, this same accident would have happened with any other car with cruise control turned on.
- alkonaut 10y ago
  A lot of modern cars have cruise control with radar for keeping distance to the car in front. Some use the same sensor to auto brake for obstacles at city speeds. I'm sure some will also alert for obstacles at highway speeds, but likely don't auto break then, for safety reasons.
- mdorazio 10y ago
  I disagree. In a car with normal or even advanced cruise control, the driver would still be paying attention to the road and ostensibly would have seen the truck ahead and applied the brakes. Autopilot makes it very easy to not pay attention at all and let the car drive itself for minutes at a time. I'm sure you'll say, "yes but that's a driver attention problem, not a technology problem", but to me they're the same thing. If you give drivers technology that encourages them to not pay attention then accidents caused as a result of that are partially your fault.
  - deleted 10y ago
    [deleted]
  - lsaferite 10y ago
    That's the same as blaming your car company for getting speeding tickets when you used cruise control. How many of those letters have you sent out?
    - mdorazio 10y ago
      No, it's not the same at all. With cruise control you are still fully in control of your car and necessarily aware of how fast you are going and what is going on in your surroundings. With Autopilot that is simply not the case - the technology virtually begs you to set it and then play with your phone or the touchscreen instead of paying attention to the road, text warnings be damned. If you want an actually applicable analogy, I'll pull one I've used before. Say you buy an oven that has this great auto-baking feature that lets you just put some cookie dough inside it and then come back whenever you want, even hours later, to have fresh baked cookies waiting. That's how it's marketed - the AutoBake Oven. Except the manufacturer says that you're still supposed to watch the oven at all times in case something goes wrong. So you make some cookies with it and they turn out great, and nothing goes wrong. And hey, it's the AutoBake Oven, right? It says it will bake things automatically so you don't have to worry about them for hours at a time. So you logically decide to put some dough in there and then go the grocery store for a few minutes to get some items for dinner. And you come back and your house burned down because the AutoBake Oven lit itself on fire due to having subpar sensors and beta software in it. Now tell me you wouldn't file a lawsuit against the manufacturer.
alkonaut 10y ago
The autopilot doesn't drive the car, the driver does. The autopilot is a help just like cruise control. Tesla are very clear (apart from the naming of the feature) that the driver must still drive the car. The autopilot will avoid obstacles it can detect and the driver is expected to avoid others (since he has eyes on the wheel and eyes on the road the whole time). I think they misleading thing about Teslas Autopilot I its current state is that they call it "Autopilot" while other manufacturers call this kind of assist features e.g "Lane Assist". A distance-sensing cruise control can't be trusted not to crash into the back of the vehicle in front either, not least because it can't be sure it can "see" it. I suspect motorcycles and other small vehicles may fool radars just like this case seems to have been about contrast and colors in a camera sensor.
- twinkletwinkle 10y ago
  >The autopilot will avoid obstacles it can detect and the driver is expected to avoid others I'm sorry, how is that supposed to work, exactly? How does the driver know? Sounds like the driver has to avoid every obstacle - to avoid the case where the autopilot fucks you, like this one - and in that case, what is the autopilot doing?
  - alkonaut 10y ago
    The driver has to be aware of every obstacle and observe the autopilot avoid them, or intervene should it fail. Yes. Have you driven a car with a distance-keeping cruise control? It's very comfortable to let it do that and just watch it and take over when it fails (it's limited in the braking and acceleration it is allowed to do). I can't take my eyes of the road or my hands off the wheel for even a second but it's still useful because it removes all the micromanagement of throttle and brake and just lets me supervise.
  - Nadya 10y ago
    That is a massive assumption to be making. That the driver can see and react to every obstacle. Not to mention there are videos where the Tesla reacted before and on behalf of the driver, saving them from an accident. The "lane assist" (what it should be and is called by other manufacturers) spots any number of obstacles that the driver failed to see - takes over - and saves the driver from an accident. Or reacts faster than the driver could react - thus saving the driver. That is what it is doing. It is not driving your vehicle. It is assisting in driving your vehicle. It is still possible to drown with a life vest on. That doesn't mean you shouldn't wear one because "if I'm expected to stay above water - what is the life vest doing?"
sago 10y ago
> Tesla autopilot drove the vehicle full-speed into a tractor-trailer The tractor trailer drove across the divided highway without seeing an oncoming car. I've seen this accident happen in front of me with a fatality, no auto-pilot involved. A vehicle moves in front of you when you're sufficiently close and you've no time to react. Anything (like glare) that makes such a vehicle harder to spot reduces your available reaction time. Making it sound like there was a static obstacle in front of the car, and it drove into it at high speed is very tendentious, and not warranted by the report, I feel. It is exactly the kind of 'omg AI kills driver' fearbait that we don't need. If self-driving systems were ten-times safer than people, then there'd still be a lot of people being killed in them. We need to get a sane perspective on their risk.
- cameldrv 10y ago
  It is essentially AI kills driver. The truck was making a left turn in front of the Tesla, which was in the right lane of a 4 lane divided highway. The truck would have had to accelerate in the left turn lane, cross the divider, cross the left lane, then drive further so that the middle of the trailer was in the right lane. This takes at least 5-6 seconds for a semi, and if the semi were detected, there would have been ample time to apply the brakes to give the truck enough time to clear the intersection. If the autopilot system can't detect and respond to this situation appropriately, it shouldn't be able to be activated on a non-controlled access highway.
  - amalag 10y ago
    Yeah i can't believe this is not bigger news. The Tesla AI was almost good enough to replace an attentive driver. But almost is not good enough.
    - fdsaaf 10y ago
      > replace an attentive driver It's unrealistic to assume that human drivers are attentive all the time. Autopilot never gets tired, never eats, never applies makeup, never sneezes, and never gets distracted by screaming children. You have to compare autopilot to realistic humans, not your platonic ideal of what a driver should be. I'm sick and tired of people making statements about road transit predicated on the idea that real humans pay attention to the road with 100% fidelity at all times.
milcron 10y ago
Autopilot doesn't need to be perfect, it just needs to be better than the average human driver. On average, humans are rather unsafe car drivers.

mikeash 10y ago

Human drivers do this kind of thing with great frequency. As long as the system is at least as good as a human driver at this stuff, I'm happy with it. Actually, scratch that: as long as the system combined with me is at least as good as a human driver, I'm happy with it.

nomercy400 10y ago

How do tractor trailers drive perpendicular on a highway in a probable case? Still, I agree it is strange that a white/glossy object cannot be detected with radar. Is it a stealth truck?

simonsarris 10y ago

Where I live some highways have a sign that says: > CAUTION: SOLAR GLARE IN THE A.M. It's serious enough to warrant its own road sign, but its not as much of a realistic issue on a divided highway, so maybe this road didn't have such a sign. This type of danger is probably what the article is describing.

joemi 10y ago

If the driver didn't notice it either, why is it Autopilot's fault?

ceejayoz 10y ago
The autopilot may reduce driver attentiveness, making the edge cases in its automation more deadly than they might've been without it.

SapphireSun 10y ago

I guess the question is, was the driver able to see it? I can see how maybe the contrast was so low that maybe it was impossible to see anything except for the wheels which you might not notice. And if the part of the tractor trailer in the path is elevated... sensors sweeping forward see just road. I guess I'd like to see an NHTSB simulation of the situation in manual driving to see how humans do unassisted.

stcredzero 10y ago
And if the part of the tractor trailer in the path is elevated... sensors sweeping forward see just road. It could be even worse. If there was a "road surface mirage" then it's possible that both the truck and the road underneath it were not visible at all. Since the forward radar doesn't have the range or coverage to see a truck at a distance, the only sensor that could have detected the problem was the camera. Because of the phenomena I've mentioned above and the parent post has mentioned, we know this can't possibly work. Even worse, the circumstances in which the car's sensor could fail are precisely the ones in which the Mark 1 Eyeball would also fail.
tim333 10y ago
A google search shows several white trailers against blue skies. It's hard to see how a human could miss them unless there were weird conditions or they weren't looking that way. https://www.google.co.uk/search?q=semi-trailer&num=30&source=lnms&tbm=isch&sa=X&ved=0ahUKEwj4_vmf8tDNAhWIJMAKHSRTBnoQ_AUICCgB&biw=1093&bih=479 https://www.google.co.uk/search?q=semi-trailer&num=30&source...

derekp7 10y ago

Replace "autopilot" with "cruise control" -- would you have the same situation? Autopilot is basically advanced cruise control plus steering assist. From what I make of this story, this same accident would have happened with any other car with cruise control turned on.

alkonaut 10y ago
A lot of modern cars have cruise control with radar for keeping distance to the car in front. Some use the same sensor to auto brake for obstacles at city speeds. I'm sure some will also alert for obstacles at highway speeds, but likely don't auto break then, for safety reasons.
mdorazio 10y ago
I disagree. In a car with normal or even advanced cruise control, the driver would still be paying attention to the road and ostensibly would have seen the truck ahead and applied the brakes. Autopilot makes it very easy to not pay attention at all and let the car drive itself for minutes at a time. I'm sure you'll say, "yes but that's a driver attention problem, not a technology problem", but to me they're the same thing. If you give drivers technology that encourages them to not pay attention then accidents caused as a result of that are partially your fault.
- deleted 10y ago
  [deleted]
- lsaferite 10y ago
  That's the same as blaming your car company for getting speeding tickets when you used cruise control. How many of those letters have you sent out?
  - mdorazio 10y ago
    No, it's not the same at all. With cruise control you are still fully in control of your car and necessarily aware of how fast you are going and what is going on in your surroundings. With Autopilot that is simply not the case - the technology virtually begs you to set it and then play with your phone or the touchscreen instead of paying attention to the road, text warnings be damned. If you want an actually applicable analogy, I'll pull one I've used before. Say you buy an oven that has this great auto-baking feature that lets you just put some cookie dough inside it and then come back whenever you want, even hours later, to have fresh baked cookies waiting. That's how it's marketed - the AutoBake Oven. Except the manufacturer says that you're still supposed to watch the oven at all times in case something goes wrong. So you make some cookies with it and they turn out great, and nothing goes wrong. And hey, it's the AutoBake Oven, right? It says it will bake things automatically so you don't have to worry about them for hours at a time. So you logically decide to put some dough in there and then go the grocery store for a few minutes to get some items for dinner. And you come back and your house burned down because the AutoBake Oven lit itself on fire due to having subpar sensors and beta software in it. Now tell me you wouldn't file a lawsuit against the manufacturer.

alkonaut 10y ago

The autopilot doesn't drive the car, the driver does. The autopilot is a help just like cruise control. Tesla are very clear (apart from the naming of the feature) that the driver must still drive the car. The autopilot will avoid obstacles it can detect and the driver is expected to avoid others (since he has eyes on the wheel and eyes on the road the whole time). I think they misleading thing about Teslas Autopilot I its current state is that they call it "Autopilot" while other manufacturers call this kind of assist features e.g "Lane Assist". A distance-sensing cruise control can't be trusted not to crash into the back of the vehicle in front either, not least because it can't be sure it can "see" it. I suspect motorcycles and other small vehicles may fool radars just like this case seems to have been about contrast and colors in a camera sensor.

twinkletwinkle 10y ago
>The autopilot will avoid obstacles it can detect and the driver is expected to avoid others I'm sorry, how is that supposed to work, exactly? How does the driver know? Sounds like the driver has to avoid every obstacle - to avoid the case where the autopilot fucks you, like this one - and in that case, what is the autopilot doing?
- alkonaut 10y ago
  The driver has to be aware of every obstacle and observe the autopilot avoid them, or intervene should it fail. Yes. Have you driven a car with a distance-keeping cruise control? It's very comfortable to let it do that and just watch it and take over when it fails (it's limited in the braking and acceleration it is allowed to do). I can't take my eyes of the road or my hands off the wheel for even a second but it's still useful because it removes all the micromanagement of throttle and brake and just lets me supervise.
- Nadya 10y ago
  That is a massive assumption to be making. That the driver can see and react to every obstacle. Not to mention there are videos where the Tesla reacted before and on behalf of the driver, saving them from an accident. The "lane assist" (what it should be and is called by other manufacturers) spots any number of obstacles that the driver failed to see - takes over - and saves the driver from an accident. Or reacts faster than the driver could react - thus saving the driver. That is what it is doing. It is not driving your vehicle. It is assisting in driving your vehicle. It is still possible to drown with a life vest on. That doesn't mean you shouldn't wear one because "if I'm expected to stay above water - what is the life vest doing?"

sago 10y ago

> Tesla autopilot drove the vehicle full-speed into a tractor-trailer The tractor trailer drove across the divided highway without seeing an oncoming car. I've seen this accident happen in front of me with a fatality, no auto-pilot involved. A vehicle moves in front of you when you're sufficiently close and you've no time to react. Anything (like glare) that makes such a vehicle harder to spot reduces your available reaction time. Making it sound like there was a static obstacle in front of the car, and it drove into it at high speed is very tendentious, and not warranted by the report, I feel. It is exactly the kind of 'omg AI kills driver' fearbait that we don't need. If self-driving systems were ten-times safer than people, then there'd still be a lot of people being killed in them. We need to get a sane perspective on their risk.

cameldrv 10y ago
It is essentially AI kills driver. The truck was making a left turn in front of the Tesla, which was in the right lane of a 4 lane divided highway. The truck would have had to accelerate in the left turn lane, cross the divider, cross the left lane, then drive further so that the middle of the trailer was in the right lane. This takes at least 5-6 seconds for a semi, and if the semi were detected, there would have been ample time to apply the brakes to give the truck enough time to clear the intersection. If the autopilot system can't detect and respond to this situation appropriately, it shouldn't be able to be activated on a non-controlled access highway.
- amalag 10y ago
  Yeah i can't believe this is not bigger news. The Tesla AI was almost good enough to replace an attentive driver. But almost is not good enough.
  - fdsaaf 10y ago
    > replace an attentive driver It's unrealistic to assume that human drivers are attentive all the time. Autopilot never gets tired, never eats, never applies makeup, never sneezes, and never gets distracted by screaming children. You have to compare autopilot to realistic humans, not your platonic ideal of what a driver should be. I'm sick and tired of people making statements about road transit predicated on the idea that real humans pay attention to the road with 100% fidelity at all times.

milcron 10y ago

Autopilot doesn't need to be perfect, it just needs to be better than the average human driver. On average, humans are rather unsafe car drivers.