4 ms·

Make a sudden turn? It's just lane following, so I really doubt that. And if your hands are on the wheel (as they should be) you'll very quickly become aware th

Make a sudden turn? It's just lane following, so I really doubt that. And if your hands are on the wheel (as they should be) you'll very quickly become aware that the car is not staying in the lane as it's supposed to, and be in the best position to deliver the right input. Even looking at this video, the car didn't seem to "turn" suddenly and unexpectedly; if anything, it failed to physically take a turn even though it should have. Which is quite bad, but it's not startling to the point where a driver would be in trouble.

lisper 8y ago
> Make a sudden turn? It's just lane following You're begging the question [1]. The autopilot has full control authority over the steering wheel, so if it fails, nothing constrains it from making a sudden turn. If it is "just lane following" then it hasn't failed (yet). [1] https://en.wikipedia.org/wiki/Begging_the_question https://en.wikipedia.org/wiki/Begging_the_question
- Karunamon 8y ago
  Your hands. If you're just resting them on the wheel, that's not enough to prevent the attention monitor from working, it actually has to measure some physical resistance. If you resist the automatic inputs too much, the AP cuts out. You can test this yourself in a tesla by engaging cruise control, then hitting a turn signal. This would normally initiate an automatic lane change - but keep your hands tightly on the wheel as if you wanted to stay in the lane you're in. The wheel will attempt to turn, fail as you're preventing it from turning - and the AP disables.
  - lisper 8y ago
    > Your hands. Sure, if you react fast enough. Let's not lose the plot here. The original claim was: "The AP can only kill you if you're distracted/asleep at the wheel and not paying attention." (Emphasis added.) And that's not true. It can kill you, quite simply, by producing the wrong control input in a situation where the available recovery time is less than your reaction time. If you doubt this, then I challenge you to drive a car where the autopilot is under my control. (It will have to be remote control because no fucking way am I willing to be in the car with you when we do this experiment.)
    - Domenic_S 8y ago
      This seems like a meaningless argument - we use fly-by-wire systems all the time and your point is true for most of them. Should we be suspicious of electronic throttle because it could theoretically hit the gas at a crosswalk when you tried to stop?
      - lisper 8y ago
        Should we be suspicious of any automation that can cause death and injury if it doesn't work right? Yes. Absolutely. These systems need to be very carefully vetted. When you don't do that, you end up with cars driving into cyclists [1] and barriers [2], and airplanes driving themselves into the ground [3]. Automation that puts civilian lives in danger is not something where you can "move fast and break things" because the things you end up breaking are innocent human beings. [1] https://www.nytimes.com/2018/03/19/technology/uber-driverless-fatality.html https://www.nytimes.com/2018/03/19/technology/uber-driverles... [2] https://www.popularmechanics.com/technology/infrastructure/a21205127/ntsb-teslas-autopilot-steered-into-barrier-in-fatal-crash/ https://www.popularmechanics.com/technology/infrastructure/a... [3] https://www.aviationcv.com/aviation-blog/2019/shocking-facts-boeing-737max-crash https://www.aviationcv.com/aviation-blog/2019/shocking-facts...
    - Karunamon 8y ago
      What I'm trying to imply is that "not paying attention" is exactly what happened here. The "attention checking" has a delay of a few seconds on it before it'll start warning you to grip the wheel. If your hands were on the wheel, and you were paying attention, there's no reaction time, since the erroneous control input would be overridden by your hands keeping you in your lane. Put simply, if your grip on the wheel was loose enough to where the computer-generated move could physically move the wheel, you weren't in control of the vehicle and would be generating warnings.
      - lisper 8y ago
        No, it wouldn't. Having your hands are on the wheel doesn't insure that you're paying attention. And if you need to be paying attention 100% of the time anyway, what's the point of having an autopilot?
madmax96 8y ago
Consider how widely affected contemporary machine learning models are by adversarial examples [1]. I don't know the specific approach used by Tesla or their release process, but I would not be surprised at all if their software has similar shortcomings. Granted, a ton of active research is going on trying to prevent these sorts of problems, but it definitely isn't a solved problem. Basically, ask yourself this: how robust is this software? How are you measuring that? [1] https://arxiv.org/pdf/1712.07107.pdf https://arxiv.org/pdf/1712.07107.pdf
MertsA 8y ago
Here's autopilot swerving into oncoming traffic. https://www.youtube.com/watch?v=ZBaolsFyD9I https://www.youtube.com/watch?v=ZBaolsFyD9I Here's autopilot following a lane straight into a concrete barrier. https://www.youtube.com/watch?v=-2ml6sjk_8c https://www.youtube.com/watch?v=-2ml6sjk_8c You can't assume that autopilot won't screw up lane following and swerve into a large obstacle. In those situations it's not as simple as making sure the lane ahead of you is clear, you might only have a split second warning between Autopilot going into "casual murder mode" and a collision.

lisper 8y ago

> Make a sudden turn? It's just lane following You're begging the question [1]. The autopilot has full control authority over the steering wheel, so if it fails, nothing constrains it from making a sudden turn. If it is "just lane following" then it hasn't failed (yet). [1] https://en.wikipedia.org/wiki/Begging_the_question https://en.wikipedia.org/wiki/Begging_the_question

Karunamon 8y ago
Your hands. If you're just resting them on the wheel, that's not enough to prevent the attention monitor from working, it actually has to measure some physical resistance. If you resist the automatic inputs too much, the AP cuts out. You can test this yourself in a tesla by engaging cruise control, then hitting a turn signal. This would normally initiate an automatic lane change - but keep your hands tightly on the wheel as if you wanted to stay in the lane you're in. The wheel will attempt to turn, fail as you're preventing it from turning - and the AP disables.
- lisper 8y ago
  > Your hands. Sure, if you react fast enough. Let's not lose the plot here. The original claim was: "The AP can only kill you if you're distracted/asleep at the wheel and not paying attention." (Emphasis added.) And that's not true. It can kill you, quite simply, by producing the wrong control input in a situation where the available recovery time is less than your reaction time. If you doubt this, then I challenge you to drive a car where the autopilot is under my control. (It will have to be remote control because no fucking way am I willing to be in the car with you when we do this experiment.)
  - Domenic_S 8y ago
    This seems like a meaningless argument - we use fly-by-wire systems all the time and your point is true for most of them. Should we be suspicious of electronic throttle because it could theoretically hit the gas at a crosswalk when you tried to stop?
    - lisper 8y ago
      Should we be suspicious of any automation that can cause death and injury if it doesn't work right? Yes. Absolutely. These systems need to be very carefully vetted. When you don't do that, you end up with cars driving into cyclists [1] and barriers [2], and airplanes driving themselves into the ground [3]. Automation that puts civilian lives in danger is not something where you can "move fast and break things" because the things you end up breaking are innocent human beings. [1] https://www.nytimes.com/2018/03/19/technology/uber-driverless-fatality.html https://www.nytimes.com/2018/03/19/technology/uber-driverles... [2] https://www.popularmechanics.com/technology/infrastructure/a21205127/ntsb-teslas-autopilot-steered-into-barrier-in-fatal-crash/ https://www.popularmechanics.com/technology/infrastructure/a... [3] https://www.aviationcv.com/aviation-blog/2019/shocking-facts-boeing-737max-crash https://www.aviationcv.com/aviation-blog/2019/shocking-facts...
  - Karunamon 8y ago
    What I'm trying to imply is that "not paying attention" is exactly what happened here. The "attention checking" has a delay of a few seconds on it before it'll start warning you to grip the wheel. If your hands were on the wheel, and you were paying attention, there's no reaction time, since the erroneous control input would be overridden by your hands keeping you in your lane. Put simply, if your grip on the wheel was loose enough to where the computer-generated move could physically move the wheel, you weren't in control of the vehicle and would be generating warnings.
    - lisper 8y ago
      No, it wouldn't. Having your hands are on the wheel doesn't insure that you're paying attention. And if you need to be paying attention 100% of the time anyway, what's the point of having an autopilot?

madmax96 8y ago

Consider how widely affected contemporary machine learning models are by adversarial examples [1]. I don't know the specific approach used by Tesla or their release process, but I would not be surprised at all if their software has similar shortcomings. Granted, a ton of active research is going on trying to prevent these sorts of problems, but it definitely isn't a solved problem. Basically, ask yourself this: how robust is this software? How are you measuring that? [1] https://arxiv.org/pdf/1712.07107.pdf https://arxiv.org/pdf/1712.07107.pdf

MertsA 8y ago

Here's autopilot swerving into oncoming traffic. https://www.youtube.com/watch?v=ZBaolsFyD9I https://www.youtube.com/watch?v=ZBaolsFyD9I Here's autopilot following a lane straight into a concrete barrier. https://www.youtube.com/watch?v=-2ml6sjk_8c https://www.youtube.com/watch?v=-2ml6sjk_8c You can't assume that autopilot won't screw up lane following and swerve into a large obstacle. In those situations it's not as simple as making sure the lane ahead of you is clear, you might only have a split second warning between Autopilot going into "casual murder mode" and a collision.