4 ms·

>It appears to have been something of a freak accident -- white trailer riding high against the bright sky, so that the autopilot didn’t detect the truck in its

>It appears to have been something of a freak accident -- white trailer riding high against the bright sky, so that the autopilot didn’t detect the truck in its path. It wasn't a freak accident. The AI failed. The human driver was distracted and failed to respond in time. Two failures Tesla has to answer for. IMO the second one is the gravest. They shouldn't have deployed the technology, primitive as it is.

ricardobeat 10y ago
How come Tesla has to answer for human failure?
- benkarst 10y ago
  Tesla calls the technology "Autopilot". What does that mean to you?
  - deleted 10y ago
    [deleted]
- diggan 10y ago
  It depends on what kind of failure. If there is a red button next to an slightly darker red button and you get the instruction from your boss to press the red one, you have a 50% of failing. But maybe the designer of said buttons should have thought about giving the buttons different colors instead of similar ones. "The Design of Everyday Things" talks about this, that sometimes it's the designer of something's fault, rather than the user of the something. Also called "Human Error" by statements. Search for "https://archive.org/stream/DesignOfEverydayThings/DesignOfEverydayThings_djvu.txt" https://archive.org/stream/DesignOfEverydayThings/DesignOfEv... here https://archive.org/stream/DesignOfEverydayThings/DesignOfEverydayThings_djvu.txt https://archive.org/stream/DesignOfEverydayThings/DesignOfEv... Point being, unless Tesla makes it safe to be used as autopilot, you probably shouldn't say it's autopilot. Otherwise it's not good enough and you would fault Tesla for the design rather than the users of the autopilot.
  - ricardobeat 10y ago
    I agree that the naming is poor - that doesn't make them liable for human error though, the driver has to opt in and is made well aware of the operation modes. Incidentally I'm carrying that exact book in my backpack right now. I think the problem here is much deeper, and even opposite to what Norman discusses in the book. An autopilot is the epitome of "don't make me think" - I want to go from A to B, no need to worry about the driving. The problem at hand is that the technology is not mature enough to fully take over from humans, and we are not good at staying alert in this half-focused state - I don't own a Tesla but it must be incredibly boring. It's an interesting conflict of tech vs human nature.
- synpatcal 10y ago
  This is a easily repeatable failure. Camera looks up from bumper, see a overhead sign, continues on. Later it looks up from the bumper, and a trailer is down about the same area, continues on. Nope... Need radar. Give me a break.
TillE 10y ago
Yes, "freak accident" generally implies some wildly unlikely confluence of events. White trailers are really common. The sun shining in your face is not uncommon. As far as I can tell, the only rare circumstance here is that Tesla's autopilot would be enabled on a road where there's cross traffic.
- deleted 10y ago
  [deleted]
- mickronome 10y ago
  I agree that it's not much of a freak accident, and you got me thinking that there might be some additional aspects of this case that might be worth considering. For most of my life I have lived somewhere with lots of snow, and where the sun hangs out close to the horizon during the entire day during the winter, and I have never understood why white cars and tractor trailers are legal on public roads. In some conditions you are lucky if you see them at 1/2 the range of a vehicle of just about any other color. Even compared to ligt grays/silver metallic it's a huge difference. Maybe something with the faint silhouette not looking very vehicle-like making the brain ignore it, I don't know? While white vehicles can be harder to spot only with the sun being close to the horizon, add some drifting snow and they essentially becomes ghosts. So while the accident clearly show Teslas autopilot is not good enough for hands free driving - of which it has made no claim to be - it might also be telling us a very simple fact: That white is a particularly bad colour for tractor trailers regardless if it is a human or a computer interpreting the visual field ?
trhway 10y ago
>white trailer riding high against the bright sky the point of this tech is that it can be better than human, specifically wider viewfield at wider wavelength band. Humans have only narrow wavelength band ("visible light") only in reflected/passive mode in a 120 by 60 viewfield. The tech can do 360 by 360 viewfield in human visible - cameras(passive) and lidars(active), infrared - cameras and lidars, centimeter - radar, ultrasound radar. Most of this tech (except may be non-stereo low quality camera) would see the trailer. Tesla's main failure here is that they did viewfield smaller than even the human's one.
benkarst 10y ago
Agreed. This article seems like Tesla propaganda designed to protect their multi-billion dollar investment. I know I'm not going to be one of Tesla's beta users. Fuck that. Any system that controls the fate of human lives should have several fail-safes. i.e They shouldn't only have one way of detecting objects. Maybe video + sonar + laser. I don't know the details of their architecture but even a Microsoft Kinect could have detected a white trailer against the bright sky using it's infrared + video camera. Gravely disappointed in Tesla and even more for articles like this who don't hold them accountable for inferior technology.
- tptacek 10y ago
  The author of this piece is a market libertarian generally and an admirer of tech companies specifically. McArdle is not a vector for Tesla propaganda. I disagree with the piece --- as I disagree with almost everything McArdle writes --- but it's useful to get an intellectually coherent perspective from a different worldview than my own. We're very rarely well served by phrases like "this article seems like propaganda".
  - icebraining 10y ago
    To the point, here's a previous post by the same author about the topic, and which touches on Level 2 systems like Tesla: "Self-Driving Cars Will Thrive With More Regulation" Level 2 automation is not really a self-driving car, and from the driver’s perspective, sitting there staring at the road and waiting for something to happen is probably worse than just driving the car. Moreover, since people don’t actually pay as much attention as they’re supposed to when using these features, these systems will probably also cause some accidents, even as they prevent others. And another Getting to level-three automation, where the car mostly drives itself most of the time, seems imminent. But the actual advantages of level-three automation over cruise control and early warning systems seem dubious. If people are distracted when the car needs them to take over, then level-three automation could end up being more dangerous than the older systems. And if people aren’t distracted, which is to say they are grimly staring at the road with nothing to do for hours at a stretch, then it seems worse than actually driving the car. http://www.bloomberg.com/view/articles/2016-01-15/self-driving-cars-will-thrive-with-more-regulation http://www.bloomberg.com/view/articles/2016-01-15/self-drivi... http://www.bloomberg.com/view/articles/2015-12-01/winning-the-driverless-car-wars http://www.bloomberg.com/view/articles/2015-12-01/winning-th...
ams6110 10y ago
The analogy in aircraft circles is "lining up the holes in the swiss cheese" Between the systems, the SOPs, and the training, there are layers of redundancy (slices of cheese) in almost everything. All have weaknesses, but not the same weaknesses. Still, occasionally the holes in the cheese line up and you have a crash. Usually after the investigation another layer of cheese is added. In this case the Tesla had only two layers: the autopilot, and the driver. The driver wasn't paying attention so when the autopilot made a mistake there was nothing to save the situation.

ricardobeat 10y ago

How come Tesla has to answer for human failure?

benkarst 10y ago
Tesla calls the technology "Autopilot". What does that mean to you?
- deleted 10y ago
  [deleted]
diggan 10y ago
It depends on what kind of failure. If there is a red button next to an slightly darker red button and you get the instruction from your boss to press the red one, you have a 50% of failing. But maybe the designer of said buttons should have thought about giving the buttons different colors instead of similar ones. "The Design of Everyday Things" talks about this, that sometimes it's the designer of something's fault, rather than the user of the something. Also called "Human Error" by statements. Search for "https://archive.org/stream/DesignOfEverydayThings/DesignOfEverydayThings_djvu.txt" https://archive.org/stream/DesignOfEverydayThings/DesignOfEv... here https://archive.org/stream/DesignOfEverydayThings/DesignOfEverydayThings_djvu.txt https://archive.org/stream/DesignOfEverydayThings/DesignOfEv... Point being, unless Tesla makes it safe to be used as autopilot, you probably shouldn't say it's autopilot. Otherwise it's not good enough and you would fault Tesla for the design rather than the users of the autopilot.
- ricardobeat 10y ago
  I agree that the naming is poor - that doesn't make them liable for human error though, the driver has to opt in and is made well aware of the operation modes. Incidentally I'm carrying that exact book in my backpack right now. I think the problem here is much deeper, and even opposite to what Norman discusses in the book. An autopilot is the epitome of "don't make me think" - I want to go from A to B, no need to worry about the driving. The problem at hand is that the technology is not mature enough to fully take over from humans, and we are not good at staying alert in this half-focused state - I don't own a Tesla but it must be incredibly boring. It's an interesting conflict of tech vs human nature.
synpatcal 10y ago
This is a easily repeatable failure. Camera looks up from bumper, see a overhead sign, continues on. Later it looks up from the bumper, and a trailer is down about the same area, continues on. Nope... Need radar. Give me a break.

TillE 10y ago

Yes, "freak accident" generally implies some wildly unlikely confluence of events. White trailers are really common. The sun shining in your face is not uncommon. As far as I can tell, the only rare circumstance here is that Tesla's autopilot would be enabled on a road where there's cross traffic.

deleted 10y ago
[deleted]
mickronome 10y ago
I agree that it's not much of a freak accident, and you got me thinking that there might be some additional aspects of this case that might be worth considering. For most of my life I have lived somewhere with lots of snow, and where the sun hangs out close to the horizon during the entire day during the winter, and I have never understood why white cars and tractor trailers are legal on public roads. In some conditions you are lucky if you see them at 1/2 the range of a vehicle of just about any other color. Even compared to ligt grays/silver metallic it's a huge difference. Maybe something with the faint silhouette not looking very vehicle-like making the brain ignore it, I don't know? While white vehicles can be harder to spot only with the sun being close to the horizon, add some drifting snow and they essentially becomes ghosts. So while the accident clearly show Teslas autopilot is not good enough for hands free driving - of which it has made no claim to be - it might also be telling us a very simple fact: That white is a particularly bad colour for tractor trailers regardless if it is a human or a computer interpreting the visual field ?

trhway 10y ago

>white trailer riding high against the bright sky the point of this tech is that it can be better than human, specifically wider viewfield at wider wavelength band. Humans have only narrow wavelength band ("visible light") only in reflected/passive mode in a 120 by 60 viewfield. The tech can do 360 by 360 viewfield in human visible - cameras(passive) and lidars(active), infrared - cameras and lidars, centimeter - radar, ultrasound radar. Most of this tech (except may be non-stereo low quality camera) would see the trailer. Tesla's main failure here is that they did viewfield smaller than even the human's one.

benkarst 10y ago

Agreed. This article seems like Tesla propaganda designed to protect their multi-billion dollar investment. I know I'm not going to be one of Tesla's beta users. Fuck that. Any system that controls the fate of human lives should have several fail-safes. i.e They shouldn't only have one way of detecting objects. Maybe video + sonar + laser. I don't know the details of their architecture but even a Microsoft Kinect could have detected a white trailer against the bright sky using it's infrared + video camera. Gravely disappointed in Tesla and even more for articles like this who don't hold them accountable for inferior technology.

tptacek 10y ago
The author of this piece is a market libertarian generally and an admirer of tech companies specifically. McArdle is not a vector for Tesla propaganda. I disagree with the piece --- as I disagree with almost everything McArdle writes --- but it's useful to get an intellectually coherent perspective from a different worldview than my own. We're very rarely well served by phrases like "this article seems like propaganda".
- icebraining 10y ago
  To the point, here's a previous post by the same author about the topic, and which touches on Level 2 systems like Tesla: "Self-Driving Cars Will Thrive With More Regulation" Level 2 automation is not really a self-driving car, and from the driver’s perspective, sitting there staring at the road and waiting for something to happen is probably worse than just driving the car. Moreover, since people don’t actually pay as much attention as they’re supposed to when using these features, these systems will probably also cause some accidents, even as they prevent others. And another Getting to level-three automation, where the car mostly drives itself most of the time, seems imminent. But the actual advantages of level-three automation over cruise control and early warning systems seem dubious. If people are distracted when the car needs them to take over, then level-three automation could end up being more dangerous than the older systems. And if people aren’t distracted, which is to say they are grimly staring at the road with nothing to do for hours at a stretch, then it seems worse than actually driving the car. http://www.bloomberg.com/view/articles/2016-01-15/self-driving-cars-will-thrive-with-more-regulation http://www.bloomberg.com/view/articles/2016-01-15/self-drivi... http://www.bloomberg.com/view/articles/2015-12-01/winning-the-driverless-car-wars http://www.bloomberg.com/view/articles/2015-12-01/winning-th...

ams6110 10y ago

The analogy in aircraft circles is "lining up the holes in the swiss cheese" Between the systems, the SOPs, and the training, there are layers of redundancy (slices of cheese) in almost everything. All have weaknesses, but not the same weaknesses. Still, occasionally the holes in the cheese line up and you have a crash. Usually after the investigation another layer of cheese is added. In this case the Tesla had only two layers: the autopilot, and the driver. The driver wasn't paying attention so when the autopilot made a mistake there was nothing to save the situation.