10 ms·

Tests show all driver assist systems can be fooled

46 points by heshiebee 5y ago

IntemerateApe 5y ago
Isn't this obvious? "tests show human drivers can be fooled too"
ajross 5y ago
I still don't understand the entire premise here. Who cares if a driver can deliberately defeat these things? That's not the threat model! A driver nag is an attentiveness reminder. You're driving a car. You're expected to be paying attention. No one else is responsible for safety. So the car wants (for both safety and liability reasons) to make sure you're doing what you're supposed to do. And it works. The car nags me and I tug the wheel to prove I'm there. Most of the time I'm paying attention anyway, but sometimes it's because I got distracted. It works. If a driver deliberately wants to behave unsafely, there's no way to stop that, because there's an infinite number of ways drivers can behave unsafely. Do we want a system to detect when drivers speed up on yellow? Tesla could absolutely implement that. Why aren't we demanding it? Prevent drivers from approaching cars in adjacent lanes with higher than a 30mph speed delta? Could do that too. Prevent accelerating in the wrong direction on a road? Possible. But no, all that stuff would be dumb, because it doesn't correspond to an actual safety problem with the hardware. We recognize it's driver behavior at fault, and we don't want our cars regulating behavior. Why is this different? I submit it's different because "Actually Elon Musk is an Asshole and TSLA went to the moon". Which isn't a very good premise for automobile safety designs.
- mattmcknight 5y ago
  > I still don't understand the entire premise here. It's because the premise is so weak. I can put a brick on the gas pedal, so no car is safe!
  - thereisnospork 5y ago
    Don't say that too loudly, carpets became just such a Boogeyman not long ago
  - haberman 5y ago
    Not to mention cruise control has the same "vulnerability": https://www.youtube.com/watch?v=f1q0FihkTfk https://www.youtube.com/watch?v=f1q0FihkTfk Is there any evidence that drivers are abusing Autopilot etc. any more than cruise control and causing a higher rate of accidents due to inattentiveness? I was especially disappointed in Consumer Reports, which insinuated that these methods of "fooling" Autopilot were somehow connected to a fatal crash in Texas. https://www.consumerreports.org/autonomous-driving/cr-engineers-show-tesla-will-drive-with-no-one-in-drivers-seat/ https://www.consumerreports.org/autonomous-driving/cr-engine... You have to get 11 paragraphs in before you get to this: > Our evaluation does not provide specific insight into the Texas crash, but safety advocates and researchers at CR say that it does show that driver monitoring systems need to work harder to keep drivers from using the systems in foreseeably dangerous ways. By this standard, cruise control should never have launched in any vehicle until technology was capable of verifying driver attentiveness with eye tracking.
- throw0101a 5y ago
  > Who cares if a driver can deliberately defeat these things? That's not the threat model! Faking a "malfunction" and then suing could be part of the threat model to the car maker and/or the insurance companies that provide liability coverage.
  - theshrike79 5y ago
    Priuses had a bunch of fake "stuck accelerator" incidents a while ago.
- HWR_14 5y ago
  > Why is this different? Because the big shiny text that people pay $10,000 for says "Autopilot". Between the name and the large price tag, it conveys that it's higher than L2 autonomy. Putting in the fine print that it's not, doesn't change things.
  - lostsock 5y ago
    You don't pay an extra $10,000 for autopilot, that's the amount for Full Self Driving
    - HWR_14 5y ago
      Error in what I called it. The point not only holds up replacing Autopilot with Full Self Driving, but is actually strengthened.
  - frumper 5y ago
    Who buys Full Self Driving for 10k and thinks you can’t pay attention? The only people I ever see say that are people that don’t have it.
    - ajross 5y ago
      Which is the most ridiculously meta argument. I have people all over the internet (amusingly never in real life) tell me to my virtual face that I clearly don't understand the product that I paid ten thousand dollars to buy, and that it doesn't do what I know damn well it does. But I do, I did, and it does. So there.
      - froh 5y ago
        I dont get it. do you confirm the argument that buyers of tesla autopilot understand its an advanced driver assist system which at this time requires drivers attention? or you do not? asking becuase the message yours responds to says exactly that: people whoy actually buy autopilot do know very well that it is a system that requests and sometimes even requires driver's attention. so if you confirm that, then how is a meta argument, ridiculously so?
- sorokod 5y ago
  Who cares if a driver can deliberately defeat these things? This is really to do with the expectations. Consider the safety belt, there is an expectation that with the safety belt on, the driver will not go through the windscreen in case of an accident. It is not the case that the driver has defeated the seat belt feature by not using the seat belt. A reasonable person will not have the expectation of having the safety provided by seatbelt without put it on. Things are not as clear cut for assisted driving as can be seen in this thread.
darkerside 5y ago
What author fails to, or pretends not to, understand is that the responsibility clearly lies with the driver. If you "cheat" or ignore the system and there is any type of negative outcome, it is the driver's fault. That's all the incentive you need, and it's the same incentive we have for people driving today. Level 2 is not intended to be autopilot. It's intended to be safer than driver unassisted.
- babesh 5y ago
  Here is what darkerside said 2 years ago. ----- darkerside on June 8, 2018 [–] This comparison just sold me on how morally wrong it is what Tesla is doing. Intentionally misleading and marketing to customers a feature called Autopilot that is only a marginal improvement on what other cars already offer. What if Volvo started calling their (clearly not independent) feature Autopilot and saying it was the future of hands-free driving? Seems inexcusable.
  - deleted 5y ago
    [deleted]
  - revscat 5y ago
    … ok? How is this relevant?
    - babesh 5y ago
      Because the comment is exactly what Tesla argues you have to do but then the unnamed called it morally wrong. Yet, when other systems are found vulnerable to driver hacking, the unnamed falls back to the same argument that Tesla uses, namely driver responsibility. Go ahead and flag this comment as well if you want Hacker News. Tesla “While using Autopilot, it is your responsibility to stay alert, keep your hands on the steering wheel at all times and maintain control of your car.” Unnamed regarding Tesla Autopilot “Morally wrong”, “intentionally misleading” Unnamed regarding this report about other car makers “ the responsibility clearly lies with the driver. If you "cheat" or ignore the system and there is any type of negative outcome, it is the driver's fault” Furthermore, when another comment points out that other automakers have similar naming to Autopilot, said unnamed says it is just marketing pressure. 1+1=2, not 3 as per guidelines
fnord77 5y ago
original non blog spam article: https://www.caranddriver.com/news/a37260363/driver-assist-systems-tested/ https://www.caranddriver.com/news/a37260363/driver-assist-sy...
wtfrmyinitials 5y ago
No mention of comma.ai
lsh123 5y ago
So the summary is that humans can still outsmart current technology that monitors drivers attention. Not bad for humans. But declaring Level 2 as garbage is stretching it. I am driving Level 2 (ACC plus lane assists plus speed sings recognition plus a few other things) for a few years now and I definitely get less tired on long drives. I recently drove coast to coast and it was more boring than tired because of the Level 2.
- Texaner 5y ago
  How do you cope with the temptation to distract yourself?
  - lsh123 5y ago
    I plead the Fifth ;)
  - suction 5y ago
    As another "level 2 driver" with the same features as OP has, I don't understand how someone could even begin to think it would be safe to "distract themselves" at this early level of automation. The temptation to stay alive wins over any temptation to distract myself. When I hear reports of accidents in Teslas because the driver "took a nap" or watched a movie or whatever, it's hard to feel empathy, because that's just reckless driving at this level. Of course Tesla is to blame when they marketed, and maybe are still marketing, level 2 capabilities as being on par with an "Auto Pilot".
    - orangepanda 5y ago
      From Waymo's tests, people stop paying attention even when they know they're being monitored.
  - theshrike79 5y ago
    Because I know it's just an "assist". Basically the camera sees the white lines in the road and tries to bounce between them while keeping a safe distance to the car in front with a radar. My car also requires me to put actual pressure on the wheel at all times, I think a 10-15 second lapse is allowed, then it starts to beep. We do have the anecdotal stories of people buying RVs with "Cruise control", enabling it on a freeway and going to the back to make a cup of coffee. Then being all surprised_pikachu.jpg when they're being dug out from the wreckage.
Causality1 5y ago
I mean obviously, that's why people buy them. Everybody thinks they're an incredibly safe driver and most of them privately think the safety aspect is the smallest reason they buy these systems. They get bought because people want to text and watch Netflix on the way to work. So I believe there's a tradeoff here. People drive to work all the time staring at their phones and only glancing up at the road every few seconds. They always have the option of turning the assist systems off and going back to distracted driving the old fashioned way. Obviously companies have no financial incentive to do this, but I posit that not making it impossible to use assist systems while completely distracted is actually a net benefit for road safety. I'd rather have Honda Sense driving a car down the interstate by itself than some dude who's only looking up when the road curves.
- jasonhansel 5y ago
  > People drive to work all the time staring at their phones and only glancing up at the road every few seconds. Do you know people who drive like this? Driving this way is (a) very illegal and (b) recklessly endangering other people. I doubt it's especially common, at least in most of the US.
  - revscat 5y ago
    I question this as well. A citation would certainly be helpful.
  - throaway46546 5y ago
    Are we even living in the same world? I see it every day.
  - rotifer 5y ago
    I don't know how common it is, but people do drive like this. From a 2012 news article [1] about Rob Ford, the mayor of Toronto at the time: Reporter: "Sir, there's a picture that went out on Twitter this morning of you reading while still driving on the Gardiner [Expressway]." Ford: "Yeah, probably. I'm busy." Reporter: "So you read while driving?" Ford: "Yeah, probably, yeah. I'm try[ing] to catch up on my work and you know I keep my eyes on the road, but I'm a busy man." Later in the article: In October, Ford was accused of illegally dialing numbers on his cellphone and talking on it as he steered his gold minivan westbound along Dundas Street West near Spadina Avenue. [1] https://www.cbc.ca/news/canada/toronto/busy-rob-ford-admits-to-reading-while-driving-1.1278580 https://www.cbc.ca/news/canada/toronto/busy-rob-ford-admits-...
NoblePublius 5y ago
So what? This is the most absurd line of inquiry. Did you know that seat belts don’t work if you don’t put them on? The question is not can something be circumvented. The question is if the system is safe. There is no evidence presented here to suggest driver assist systems are a net hazard.
dosshell 5y ago
The truth is that some car manufacturers actually request it to be somewhat fool proof, but very few. It costs hardware to run and cost to develop/test, and car manufacturing is all about price. But also: Today the algorithm knows the 3D geometry of your head, and knows that it does not makes sense with a flat face. But it is more important to get it to work good on everyone than to catch foolers. Disclaimer: i'm one of all the developers behind camera monitoring.
chrischen 5y ago
"It's currently the only system that allows hands-free driving for an extended period of time." This is false. Comma.ai/OpenPilot allows this. Video seems under-researched.

46 points by heshiebee 5y ago

Isn't this obvious? "tests show human drivers can be fooled too"

I still don't understand the entire premise here. Who cares if a driver can deliberately defeat these things? That's not the threat model! A driver nag is an attentiveness reminder. You're driving a car. You're expected to be paying attention. No one else is responsible for safety. So the car wants (for both safety and liability reasons) to make sure you're doing what you're supposed to do. And it works. The car nags me and I tug the wheel to prove I'm there. Most of the time I'm paying attention anyway, but sometimes it's because I got distracted. It works. If a driver deliberately wants to behave unsafely, there's no way to stop that, because there's an infinite number of ways drivers can behave unsafely. Do we want a system to detect when drivers speed up on yellow? Tesla could absolutely implement that. Why aren't we demanding it? Prevent drivers from approaching cars in adjacent lanes with higher than a 30mph speed delta? Could do that too. Prevent accelerating in the wrong direction on a road? Possible. But no, all that stuff would be dumb, because it doesn't correspond to an actual safety problem with the hardware. We recognize it's driver behavior at fault, and we don't want our cars regulating behavior. Why is this different? I submit it's different because "Actually Elon Musk is an Asshole and TSLA went to the moon". Which isn't a very good premise for automobile safety designs.

mattmcknight 5y ago
> I still don't understand the entire premise here. It's because the premise is so weak. I can put a brick on the gas pedal, so no car is safe!
- thereisnospork 5y ago
  Don't say that too loudly, carpets became just such a Boogeyman not long ago
- haberman 5y ago
  Not to mention cruise control has the same "vulnerability": https://www.youtube.com/watch?v=f1q0FihkTfk https://www.youtube.com/watch?v=f1q0FihkTfk Is there any evidence that drivers are abusing Autopilot etc. any more than cruise control and causing a higher rate of accidents due to inattentiveness? I was especially disappointed in Consumer Reports, which insinuated that these methods of "fooling" Autopilot were somehow connected to a fatal crash in Texas. https://www.consumerreports.org/autonomous-driving/cr-engineers-show-tesla-will-drive-with-no-one-in-drivers-seat/ https://www.consumerreports.org/autonomous-driving/cr-engine... You have to get 11 paragraphs in before you get to this: > Our evaluation does not provide specific insight into the Texas crash, but safety advocates and researchers at CR say that it does show that driver monitoring systems need to work harder to keep drivers from using the systems in foreseeably dangerous ways. By this standard, cruise control should never have launched in any vehicle until technology was capable of verifying driver attentiveness with eye tracking.
throw0101a 5y ago
> Who cares if a driver can deliberately defeat these things? That's not the threat model! Faking a "malfunction" and then suing could be part of the threat model to the car maker and/or the insurance companies that provide liability coverage.
- theshrike79 5y ago
  Priuses had a bunch of fake "stuck accelerator" incidents a while ago.
HWR_14 5y ago
> Why is this different? Because the big shiny text that people pay $10,000 for says "Autopilot". Between the name and the large price tag, it conveys that it's higher than L2 autonomy. Putting in the fine print that it's not, doesn't change things.
- lostsock 5y ago
  You don't pay an extra $10,000 for autopilot, that's the amount for Full Self Driving
  - HWR_14 5y ago
    Error in what I called it. The point not only holds up replacing Autopilot with Full Self Driving, but is actually strengthened.
- frumper 5y ago
  Who buys Full Self Driving for 10k and thinks you can’t pay attention? The only people I ever see say that are people that don’t have it.
  - ajross 5y ago
    Which is the most ridiculously meta argument. I have people all over the internet (amusingly never in real life) tell me to my virtual face that I clearly don't understand the product that I paid ten thousand dollars to buy, and that it doesn't do what I know damn well it does. But I do, I did, and it does. So there.
    - froh 5y ago
      I dont get it. do you confirm the argument that buyers of tesla autopilot understand its an advanced driver assist system which at this time requires drivers attention? or you do not? asking becuase the message yours responds to says exactly that: people whoy actually buy autopilot do know very well that it is a system that requests and sometimes even requires driver's attention. so if you confirm that, then how is a meta argument, ridiculously so?
sorokod 5y ago
Who cares if a driver can deliberately defeat these things? This is really to do with the expectations. Consider the safety belt, there is an expectation that with the safety belt on, the driver will not go through the windscreen in case of an accident. It is not the case that the driver has defeated the seat belt feature by not using the seat belt. A reasonable person will not have the expectation of having the safety provided by seatbelt without put it on. Things are not as clear cut for assisted driving as can be seen in this thread.

darkerside 5y ago

What author fails to, or pretends not to, understand is that the responsibility clearly lies with the driver. If you "cheat" or ignore the system and there is any type of negative outcome, it is the driver's fault. That's all the incentive you need, and it's the same incentive we have for people driving today. Level 2 is not intended to be autopilot. It's intended to be safer than driver unassisted.

babesh 5y ago
Here is what darkerside said 2 years ago. ----- darkerside on June 8, 2018 [–] This comparison just sold me on how morally wrong it is what Tesla is doing. Intentionally misleading and marketing to customers a feature called Autopilot that is only a marginal improvement on what other cars already offer. What if Volvo started calling their (clearly not independent) feature Autopilot and saying it was the future of hands-free driving? Seems inexcusable.
- deleted 5y ago
  [deleted]
- revscat 5y ago
  … ok? How is this relevant?
  - babesh 5y ago
    Because the comment is exactly what Tesla argues you have to do but then the unnamed called it morally wrong. Yet, when other systems are found vulnerable to driver hacking, the unnamed falls back to the same argument that Tesla uses, namely driver responsibility. Go ahead and flag this comment as well if you want Hacker News. Tesla “While using Autopilot, it is your responsibility to stay alert, keep your hands on the steering wheel at all times and maintain control of your car.” Unnamed regarding Tesla Autopilot “Morally wrong”, “intentionally misleading” Unnamed regarding this report about other car makers “ the responsibility clearly lies with the driver. If you "cheat" or ignore the system and there is any type of negative outcome, it is the driver's fault” Furthermore, when another comment points out that other automakers have similar naming to Autopilot, said unnamed says it is just marketing pressure. 1+1=2, not 3 as per guidelines

fnord77 5y ago

original non blog spam article: https://www.caranddriver.com/news/a37260363/driver-assist-systems-tested/ https://www.caranddriver.com/news/a37260363/driver-assist-sy...

wtfrmyinitials 5y ago

No mention of comma.ai

lsh123 5y ago

So the summary is that humans can still outsmart current technology that monitors drivers attention. Not bad for humans. But declaring Level 2 as garbage is stretching it. I am driving Level 2 (ACC plus lane assists plus speed sings recognition plus a few other things) for a few years now and I definitely get less tired on long drives. I recently drove coast to coast and it was more boring than tired because of the Level 2.

Texaner 5y ago
How do you cope with the temptation to distract yourself?
- lsh123 5y ago
  I plead the Fifth ;)
- suction 5y ago
  As another "level 2 driver" with the same features as OP has, I don't understand how someone could even begin to think it would be safe to "distract themselves" at this early level of automation. The temptation to stay alive wins over any temptation to distract myself. When I hear reports of accidents in Teslas because the driver "took a nap" or watched a movie or whatever, it's hard to feel empathy, because that's just reckless driving at this level. Of course Tesla is to blame when they marketed, and maybe are still marketing, level 2 capabilities as being on par with an "Auto Pilot".
  - orangepanda 5y ago
    From Waymo's tests, people stop paying attention even when they know they're being monitored.
- theshrike79 5y ago
  Because I know it's just an "assist". Basically the camera sees the white lines in the road and tries to bounce between them while keeping a safe distance to the car in front with a radar. My car also requires me to put actual pressure on the wheel at all times, I think a 10-15 second lapse is allowed, then it starts to beep. We do have the anecdotal stories of people buying RVs with "Cruise control", enabling it on a freeway and going to the back to make a cup of coffee. Then being all surprised_pikachu.jpg when they're being dug out from the wreckage.

Causality1 5y ago

I mean obviously, that's why people buy them. Everybody thinks they're an incredibly safe driver and most of them privately think the safety aspect is the smallest reason they buy these systems. They get bought because people want to text and watch Netflix on the way to work. So I believe there's a tradeoff here. People drive to work all the time staring at their phones and only glancing up at the road every few seconds. They always have the option of turning the assist systems off and going back to distracted driving the old fashioned way. Obviously companies have no financial incentive to do this, but I posit that not making it impossible to use assist systems while completely distracted is actually a net benefit for road safety. I'd rather have Honda Sense driving a car down the interstate by itself than some dude who's only looking up when the road curves.

jasonhansel 5y ago
> People drive to work all the time staring at their phones and only glancing up at the road every few seconds. Do you know people who drive like this? Driving this way is (a) very illegal and (b) recklessly endangering other people. I doubt it's especially common, at least in most of the US.
- revscat 5y ago
  I question this as well. A citation would certainly be helpful.
- throaway46546 5y ago
  Are we even living in the same world? I see it every day.
- rotifer 5y ago
  I don't know how common it is, but people do drive like this. From a 2012 news article [1] about Rob Ford, the mayor of Toronto at the time: Reporter: "Sir, there's a picture that went out on Twitter this morning of you reading while still driving on the Gardiner [Expressway]." Ford: "Yeah, probably. I'm busy." Reporter: "So you read while driving?" Ford: "Yeah, probably, yeah. I'm try[ing] to catch up on my work and you know I keep my eyes on the road, but I'm a busy man." Later in the article: In October, Ford was accused of illegally dialing numbers on his cellphone and talking on it as he steered his gold minivan westbound along Dundas Street West near Spadina Avenue. [1] https://www.cbc.ca/news/canada/toronto/busy-rob-ford-admits-to-reading-while-driving-1.1278580 https://www.cbc.ca/news/canada/toronto/busy-rob-ford-admits-...

NoblePublius 5y ago

So what? This is the most absurd line of inquiry. Did you know that seat belts don’t work if you don’t put them on? The question is not can something be circumvented. The question is if the system is safe. There is no evidence presented here to suggest driver assist systems are a net hazard.

dosshell 5y ago

The truth is that some car manufacturers actually request it to be somewhat fool proof, but very few. It costs hardware to run and cost to develop/test, and car manufacturing is all about price. But also: Today the algorithm knows the 3D geometry of your head, and knows that it does not makes sense with a flat face. But it is more important to get it to work good on everyone than to catch foolers. Disclaimer: i'm one of all the developers behind camera monitoring.

chrischen 5y ago

"It's currently the only system that allows hands-free driving for an extended period of time." This is false. Comma.ai/OpenPilot allows this. Video seems under-researched.