5 ms·

This article and many others like it are written as if one ought to be ashamed for thinking self driving cars should not routinely smash into stationary objects

by nanis 8y ago

This article and many others like it are written as if one ought to be ashamed for thinking self driving cars should not routinely smash into stationary objects on their paths or kill people: This isn't the only recent case where Autopilot steered a Tesla vehicle directly into a stationary object—though thankfully the others didn't get anyone killed. Back in January, firefighters in Culver City, California, said that a Tesla with Autopilot engaged had plowed into the back of a fire truck at 65mph. In an eerily similar incident last month, a Tesla Model S with Autopilot active crashed into a fire truck at 60mph in the suburbs of Salt Lake City. A natural reaction to these incidents is to assume that there must be something seriously wrong with Tesla's Autopilot system. After all, you might expect that avoiding collisions with large, stationary objects like fire engines and concrete lane dividers would be one of the most basic functions of a car's automatic emergency braking technology. Regardless of how they came to be, yes there is something seriously wrong with these systems.

marcell 8y ago
I didn’t get that at all. It was written in a dispassionate tone but it didn’t blame the driver at any point.
tracer4201 8y ago
Didn't hit me that way. Sounds more like we as consumers misinterpret what the technology is and trust it too much... of course, there's a feedback loop where companies like Tesla call everything autonomous and hype the marketing to mislead consumers.
- vkou 8y ago
  It's not companies like Tesla. As far as I can tell, it's just Tesla.
- stevenwoo 8y ago
  It seems like the radar/camera system Toyota uses for its Pre Collision System has the same weaknesses as Tesla's sensor suite but the list of exceptions in the Toyota manual is pretty brutally honest about the issues. Just for pedestrians it warns: Some pedestrians such as the following may not be detected by the radar sensor and camera sensor, preventing the system from operating properly: • Pedestrians shorter than approximately 3.2 ft. (1 m) or taller than approx- imately 6.5 ft. (2 m) • Pedestrians wearing oversized clothing (a rain coat, long skirt, etc.), mak- ing their silhouette obscure • Pedestrians who are carrying large baggage, holding an umbrella, etc., hiding part of their body • Pedestrians who are bending forward or squatting • Pedestrians who are pushing a stroller, wheelchair, bicycle or other vehi- cle • Groups of pedestrians which are close together • Pedestrians who are wearing white and look extremely bright • Pedestrians in the dark, such as at night or while in a tunnel • Pedestrians whose clothing appears to be nearly the same color or brightness as their surroundings • Pedestrians near walls, fences, guardrails, or large objects • Pedestrians who are on a metal object (manhole cover, steel plate, etc.) on the road • Pedestrians who are walking fast • Pedestrians who are changing speed abruptly • Pedestrians running out from behind a vehicle or a large object • Pedestrians who are extremely close to the side of the vehicle (outside rear view mirror, etc.)
  - sib 8y ago
    I'm trying to figure out whether there are actually a meaningful fraction of pedestrians who would be detected...
    - rootusrootus 8y ago
      I imagine that list was drafted with the direction of the legal department and they wanted to exhaustive list that might very well exclude basically all pedestrians. They don't really expect many people to read it unless they find themselves in court, when it will be too late.
  - octothorp3 8y ago
    Huh, most of these I can understand with my layman's understanding of the sensors involved, but why would it miss the pedestrians who are on a metal manhole cover?
    - TheSpiceIsLife 8y ago
      Not a Toyota engineer familiar with the problem, but my guess is a metal object produces a radar reflection significantly different from a pedestrian.
  - taneq 8y ago
    It still somewhat baffles me that these systems are trying to identify certain classes of object to avoid colliding with, rather than simply avoiding hitting anything solid over a certain size. Do I care whether the thing in front of me is a hatchback or a group of pedestrians? No! I just want the car not to hit it!
- Spooky23 8y ago
  Why be so defensive of Tesla? This isn’t fluffing up phone features, it’s presenting a system in such a way that key, known deficiencies that maim or kill are buried. The people impacted by this trusted the system based on manufacturer claims, which unfortunately were and remain horseshit.
- rhizome 8y ago
  Isn't it designed and marketed to be trusted?
rcthompson 8y ago
This is an odd claim to make, since the article doesn't say anything at all about self-driving cars. Moreover, I don't think it's trying to shame anyone. It's making the point that there is a difference between crashing due to an error in the driver assistance systems and crashing due to encountering a situation that the system is incapable of handling. The tail end of the article states unambiguously that this is not an acceptable state of affairs, and that the next generation of driver assistance systems needs to take a different approach. As to whether this article is really a stealth advertisement for GM's Super Cruise technology, well... yeah, it probably is. But I still found it informative as to the current state of driver assistance systems. I was not aware that the cruise control, lane keeping, and automatic braking systems were generally completely independent of each other.
- joe_the_user 8y ago
  Moreover, I don't think it's trying to shame anyone. It's making the point that there is a difference between crashing due to an error in the driver assistance systems and crashing due to encountering a situation that the system is incapable of handling. Well, that point is most likely lost on someone who dies in a crash of these cars. IE, the difference is big from some viewpoints but irrelevant from others. I think we can see the point where objections to self-driving cars start to become strong. It's not some cooked-up version of the "trolley car problem" but a situation where a self-driving car does something a human observer naturally interprets as "wrong" and "dumb", does it in a way that kills someone and the designers give the answer "sorry, not a mistake, just a necessary design compromise. This is still, on-average safer than a human". I think this would produce a strong emotional effect regardless of any "generally safer" assertion.
  - rcthompson 8y ago
    Knowing the difference is important because in one case, you need to fix a bug, and in the other case, you need to design a new system (or decide that the problem is unfixable and give up).
    - 8y ago
ambicapter 8y ago
Sounds more like sarcasm than anything else.
rcthompson 8y ago
Regarding your edit, I agree that, in isolation, the passage you quoted can easily be misinterpreted to imply that the current functioning of these systems is acceptable, and the only reason I didn't read it that way is because the rest of the article explicitly rejects that interpretation.
dredmorbius 8y ago
I see this as far more a caution to the public for presuming that automated systems have a singular worldview as humans ... think ... we ourselves do[1], rather than being comprised of multiple and independent systems. And a caution to both the technologists who fail to consider this and real-world use consequences, as well as those who fail to see divorced systems as problematic. The real world is where all your independent worldviews are tested, simultaneously and integrated. And where metaphor shear may well prove fatal. ________________________________ Notes: 1. A not-necessarily accurate worldview itself, though I'll leave that to the cog-psych types to explore in depth.
YeGoblynQueenne 8y ago
I think Timothy B. Lee, the author of the article, was caught up in the early hype about "self-driving" cars, obviously before he (or many others) had a good understanding of the technology. He's covered the field extensively. Now that the limitations of the technology are becoming more clear, it seems to me he's trying to negotiate the very narrow border between keeping his original enthusiasm up, and not spreading misinformation. I interpret this article as an attempt to keep some sort of balance, between reporting on a promising technology without over-promising and without veering too far into accusing his own readers of ignorance. But, hey. I'm not in the guy's head :)
- binarybits 8y ago
  Timothy B. Lee here. You are correct, except that I think it's a mistake to treat ADAS systems as "self-driving cars." I'm bullish on true self-driving car projects like Waymo and Cruise. I think ADAS systems have a lot of problems that have become more clear to me as I've covered them in more depth.
  - YeGoblynQueenne 8y ago
    Ohi Timothy! Great to see you here :) I think that full autonomy (level 4 and 5) is an AI-complete problem and a true solution is many, many years away. On the other hand, I'm not an expert (I'm doing an AI PhD but on a different subject) so I may be overestimating the difficulties involved. One problem with reporting on autonomous driving is that a lot of the technology is proprietary and the research takes place behind closed doors, so it's very hard to understand exactly where the state of the art is. We're left with the announcements from the companies who actually sell it, that will inevitably tend to be overinflated. As the technology becomes more common, I guess we'll all end up adjusting our expectations, one way or another. I'm looking forward to your future articles :)

by nanis 8y ago

marcell 8y ago

I didn’t get that at all. It was written in a dispassionate tone but it didn’t blame the driver at any point.

tracer4201 8y ago

Didn't hit me that way. Sounds more like we as consumers misinterpret what the technology is and trust it too much... of course, there's a feedback loop where companies like Tesla call everything autonomous and hype the marketing to mislead consumers.

vkou 8y ago
It's not companies like Tesla. As far as I can tell, it's just Tesla.
stevenwoo 8y ago
It seems like the radar/camera system Toyota uses for its Pre Collision System has the same weaknesses as Tesla's sensor suite but the list of exceptions in the Toyota manual is pretty brutally honest about the issues. Just for pedestrians it warns: Some pedestrians such as the following may not be detected by the radar sensor and camera sensor, preventing the system from operating properly: • Pedestrians shorter than approximately 3.2 ft. (1 m) or taller than approx- imately 6.5 ft. (2 m) • Pedestrians wearing oversized clothing (a rain coat, long skirt, etc.), mak- ing their silhouette obscure • Pedestrians who are carrying large baggage, holding an umbrella, etc., hiding part of their body • Pedestrians who are bending forward or squatting • Pedestrians who are pushing a stroller, wheelchair, bicycle or other vehi- cle • Groups of pedestrians which are close together • Pedestrians who are wearing white and look extremely bright • Pedestrians in the dark, such as at night or while in a tunnel • Pedestrians whose clothing appears to be nearly the same color or brightness as their surroundings • Pedestrians near walls, fences, guardrails, or large objects • Pedestrians who are on a metal object (manhole cover, steel plate, etc.) on the road • Pedestrians who are walking fast • Pedestrians who are changing speed abruptly • Pedestrians running out from behind a vehicle or a large object • Pedestrians who are extremely close to the side of the vehicle (outside rear view mirror, etc.)
- sib 8y ago
  I'm trying to figure out whether there are actually a meaningful fraction of pedestrians who would be detected...
  - rootusrootus 8y ago
    I imagine that list was drafted with the direction of the legal department and they wanted to exhaustive list that might very well exclude basically all pedestrians. They don't really expect many people to read it unless they find themselves in court, when it will be too late.
- octothorp3 8y ago
  Huh, most of these I can understand with my layman's understanding of the sensors involved, but why would it miss the pedestrians who are on a metal manhole cover?
  - TheSpiceIsLife 8y ago
    Not a Toyota engineer familiar with the problem, but my guess is a metal object produces a radar reflection significantly different from a pedestrian.
- taneq 8y ago
  It still somewhat baffles me that these systems are trying to identify certain classes of object to avoid colliding with, rather than simply avoiding hitting anything solid over a certain size. Do I care whether the thing in front of me is a hatchback or a group of pedestrians? No! I just want the car not to hit it!
Spooky23 8y ago
Why be so defensive of Tesla? This isn’t fluffing up phone features, it’s presenting a system in such a way that key, known deficiencies that maim or kill are buried. The people impacted by this trusted the system based on manufacturer claims, which unfortunately were and remain horseshit.
rhizome 8y ago
Isn't it designed and marketed to be trusted?

rcthompson 8y ago

This is an odd claim to make, since the article doesn't say anything at all about self-driving cars. Moreover, I don't think it's trying to shame anyone. It's making the point that there is a difference between crashing due to an error in the driver assistance systems and crashing due to encountering a situation that the system is incapable of handling. The tail end of the article states unambiguously that this is not an acceptable state of affairs, and that the next generation of driver assistance systems needs to take a different approach. As to whether this article is really a stealth advertisement for GM's Super Cruise technology, well... yeah, it probably is. But I still found it informative as to the current state of driver assistance systems. I was not aware that the cruise control, lane keeping, and automatic braking systems were generally completely independent of each other.

joe_the_user 8y ago
Moreover, I don't think it's trying to shame anyone. It's making the point that there is a difference between crashing due to an error in the driver assistance systems and crashing due to encountering a situation that the system is incapable of handling. Well, that point is most likely lost on someone who dies in a crash of these cars. IE, the difference is big from some viewpoints but irrelevant from others. I think we can see the point where objections to self-driving cars start to become strong. It's not some cooked-up version of the "trolley car problem" but a situation where a self-driving car does something a human observer naturally interprets as "wrong" and "dumb", does it in a way that kills someone and the designers give the answer "sorry, not a mistake, just a necessary design compromise. This is still, on-average safer than a human". I think this would produce a strong emotional effect regardless of any "generally safer" assertion.
- rcthompson 8y ago
  Knowing the difference is important because in one case, you need to fix a bug, and in the other case, you need to design a new system (or decide that the problem is unfixable and give up).
  - 8y ago

ambicapter 8y ago

Sounds more like sarcasm than anything else.

rcthompson 8y ago

Regarding your edit, I agree that, in isolation, the passage you quoted can easily be misinterpreted to imply that the current functioning of these systems is acceptable, and the only reason I didn't read it that way is because the rest of the article explicitly rejects that interpretation.

dredmorbius 8y ago

I see this as far more a caution to the public for presuming that automated systems have a singular worldview as humans ... think ... we ourselves do[1], rather than being comprised of multiple and independent systems. And a caution to both the technologists who fail to consider this and real-world use consequences, as well as those who fail to see divorced systems as problematic. The real world is where all your independent worldviews are tested, simultaneously and integrated. And where metaphor shear may well prove fatal. ________________________________ Notes: 1. A not-necessarily accurate worldview itself, though I'll leave that to the cog-psych types to explore in depth.

YeGoblynQueenne 8y ago

I think Timothy B. Lee, the author of the article, was caught up in the early hype about "self-driving" cars, obviously before he (or many others) had a good understanding of the technology. He's covered the field extensively. Now that the limitations of the technology are becoming more clear, it seems to me he's trying to negotiate the very narrow border between keeping his original enthusiasm up, and not spreading misinformation. I interpret this article as an attempt to keep some sort of balance, between reporting on a promising technology without over-promising and without veering too far into accusing his own readers of ignorance. But, hey. I'm not in the guy's head :)

binarybits 8y ago
Timothy B. Lee here. You are correct, except that I think it's a mistake to treat ADAS systems as "self-driving cars." I'm bullish on true self-driving car projects like Waymo and Cruise. I think ADAS systems have a lot of problems that have become more clear to me as I've covered them in more depth.
- YeGoblynQueenne 8y ago
  Ohi Timothy! Great to see you here :) I think that full autonomy (level 4 and 5) is an AI-complete problem and a true solution is many, many years away. On the other hand, I'm not an expert (I'm doing an AI PhD but on a different subject) so I may be overestimating the difficulties involved. One problem with reporting on autonomous driving is that a lot of the technology is proprietary and the research takes place behind closed doors, so it's very hard to understand exactly where the state of the art is. We're left with the announcements from the companies who actually sell it, that will inevitably tend to be overinflated. As the technology becomes more common, I guess we'll all end up adjusting our expectations, one way or another. I'm looking forward to your future articles :)