10 ms·

To other commenters: this is a technical audit of autopilot software, not Tesla FSD. And the actual context is much less of a big deal than it seems: the bigge

by epgui 4y ago

To other commenters: this is a technical audit of autopilot software, not Tesla FSD. And the actual context is much less of a big deal than it seems: the biggest plausible consequence would be forcing Tesla to push an over-the-air update with better driver attention monitoring or alerting. I encourage reading the actual report.

dangrossman 4y ago
The worst case outcome would require Tesla to disable the autopilot suite entirely, for an indeterminate amount of time, perhaps permanently on the existing fleet of vehicles. The NHTSA is tired of Tesla's hand-waving away their safety investigations into Autopilot by pushing stealth updates that fix specific scenarios in specific places being investigated. NHTSA wisened up to that and independently purchased their own Tesla vehicles, and disabled software updates, so that they can reproduce those scenarios themselves. If NHTSA asks Tesla to provide system validation tests showing that an updated version of their software meets the design intent of the system, Tesla would not be able to do so. If they can't prove the new Autopilot software corrects the safety-related defects identified in the current version, then it's not a valid recall remedy. All evidence from their own AI/AP team and presentations is that there is no real design and system validation going on over there. They're flying by the seat of their pants, introducing potentially lethal regressions in every update.
- AzzieElbab 4y ago
  If disabling FSD makes teslas less safe then what is the point? Are they saying fsd can potentially go berserk? Are we into future crime prevention?
  - hedora 4y ago
    Yes, in the same way that taking down an active gunman with a loaded weapon is future crime prevention.
    - AzzieElbab 4y ago
      What about possible future software update to any hospital system? Should we preemptively stop all of those?
      - hedora 4y ago
        If the manufacturer isn't adequately testing for regressions that kill people, then yes, we should block those updates, and use the software the device was certified with.
        AzzieElbab 4y ago
        gl getting anything done with a government agency defining adequacy of your software testing. here in canada government employees cannot get software that pays them salaries to work https://en.wikipedia.org/wiki/Phoenix_pay_system https://en.wikipedia.org/wiki/Phoenix_pay_system
- epgui 4y ago
  If you read the report, you will realize that NHTSA is considering requiring Tesla to do better driver attention monitoring, or to improve alerting. They are not considering banning autopilot.
  - dangrossman 4y ago
    I assure you, I'd already read the report before it was shared here. I also assure you, there's more to the investigation than that.
    - epgui 4y ago
      Perhaps, but that’s speculation at best.
      - deleted 4y ago
        [deleted]
      - sumy23 4y ago
        But you’ve been assured with absolutely no evidence /s.
      - moralestapia 4y ago
        99% of everything is speculation at best, even this site is pretty much (high value) speculation as a service. The NHTSA has a reputation of not f*king around so I would definitely side with @dangrossman on this thing. As of today, Autopilot IS dangerous software and it is not something that should be tested live on the streets.
        GekkePrutser 4y ago
        > The NHTSA has a reputation of not f*king around so I would definitely side with @dangrossman on this thing. So did the FAA and then they let Boeing self-validate the 737 MAX. Just saying..
        4y ago
- wolverine876 4y ago
  > The NHTSA is tired of Tesla's hand-waving away their safety investigations into Autopilot by pushing stealth updates that fix specific scenarios in specific places being investigated. Why isn't Tesla prosecuted for that? It's lawless!
  - jquery 4y ago
    It really is insane. It’s one thing to have flaws, it’s quite another to stealth cover-them-up like it’s a game of hide and seek.
  - extheat 4y ago
    No, that’s typical software development. Find a bug in circulation, fix and deploy a fix. There are probably hundreds of internal issues that get fixed per normal protocol, as with any piece of software. Putting out a “recall” or alert for every modification to the code is pointless. What regulators need to do is keep up with the times. They need to have their own safety test suite which manufacturers can test against, and be independently audited
    - rtlfe 4y ago
      > No, that’s typical software development. Software that controls multi-thousand pound machines at 70+mph isn't typical, and typical practices don't necessarily apply.
      - codebolt 4y ago
        Yes, those practices absolutely shouldn't apply for self driving cars. Good luck regression testing how a system change impacts the AI handling of every edge case of traffic.
        sidibe 4y ago
        Waymo does this. Their infrastructure costs would be astronomical though if not attached to a company with its own cloud
- ajross 4y ago
  > introducing potentially lethal regressions in every update. Meh. I mean, I understand the emotional content of that argument, and the propriety angle is real enough. I really do get what you're saying. And if your prior is "Tesla is bad", that's going to be really convincing. But if it's not... The bottom line is that they're getting close to 3M of these vehicles on the roads now. You can't spit without hitting one in the south bay. And the accident statistics just aren't there. They're not. There's a small handful of verifiable accidents, all on significantly older versions of the software. Bugs are real. They've happened before and they'll no doubt happen again, just like they do with every other product. But the Simple Truth remains that these are very safe cars. They are. So... what exactly are people getting upset about? Because it doesn't seem to be what people claim they're getting upset about.
  - ken47 4y ago
    Total straw man. The question isn't whether Tesla's are safe to drive. The question is whether "autopilot" is safe to auto pilot.
    - Retric 4y ago
      Actual real world data says autopilot is on average at least as safe as the average driver in the average car. Of course that’s on average, in many specific situations it’s much worse but conversely it means it’s better in other situations. How regulators deal with this frankly tricky as the same will likely apply to all self driving systems.
      - ken47 4y ago
        Sources for claims would be appreciated.
      - usrusr 4y ago
        Those real world autopilot averages happen exclusively in the most trivial driving situations. Fair weather and almost exclusively on limited access roads. No real world average driver dataset exists that is similarly restricted to the subset of least error-prone driving situations.
- deleted 4y ago
  [deleted]
- baryphonic 4y ago
  > All evidence from their own AI/AP team and presentations is that there is no real design and system validation going on over there. They're flying by the seat of their pants, introducing potentially lethal regressions in every update. What is this evidence? I've seen a few talks from Andrej Karpathy that indicate to me a more deliberate approach.[0] "Software 2.0" itself seems like an approach meant to systematize the development, validation & testing of AI systems, hardly a seat-of-your-pants approach to releases. I have my own criticisms of their approach, but it seems there is pretty deliberate care taken when developing models. [0] https://youtu.be/hx7BXih7zx8 https://youtu.be/hx7BXih7zx8
  - refulgentis 4y ago
    I see your point, to OP's point, I know a couple people who were horrified at what they saw and it did not match this public talk. Both started at least 6 months after this video, and both left Tesla within 8 months, of their own volition. Unfortunately, off the record. Not to disparage Andrej, sometimes (frequently, even) what executive leadership thinks is going is not the day-to-day reality of the team.
    - plankers 4y ago
      can confirm, a former coworker had just come from Tesla 5 years ago and he had serious ethical problems with his work over there. Tesla is killing people through negligence and greed, it's pretty disgusting, but par for the course
  - justapassenger 4y ago
    I’ve been working few years ago at a very big tech company, focusing on validation of the AI systems. It’s all smoke and mirrors. You cannot perform proper validation of AI systems. Rollbacks of new versions of ML models are very common in production, and even after very extensive validation you can see that real life results are nothing like what tests have shown.
    - urthor 4y ago
Isinlor 4y ago
Do you have the link to the report?
- dangrossman 4y ago
  It's in the article: https://static.nhtsa.gov/odi/inv/2022/INOA-EA22002-3184.PDF https://static.nhtsa.gov/odi/inv/2022/INOA-EA22002-3184.PDF
deleted 4y ago
[deleted]
dragonwriter 4y ago
> the biggest plausible consequence would be forcing Tesla to push an over-the-air update with better driver attention monitoring or alerting. I see no basis for this conclusion, which appears to be pure speculation about what NHTSA might decide is necessary and sufficient to address the potential problems if confirmed by the deeper analysis. I encourage reading the actual report. > I encourage reading the actual report. I did, and the conclusion do which you appeal to it does not appear to be well-supported by it.
justapassenger 4y ago
Tesla has no hardware for proper driver monitoring. Most of model S have no internal camera. And model 3 internal camera wasn’t designed for it (doesn’t work in the dark, cannot see through sunglasses, cannot see where your eyes are actually pointing, etc). You cannot OTA HW deficiencies. Now, will they be forced to have monitoring like that, to be on par with their competitors? That’s a different story, and given how weak USA regulatory agencies are, and how reckless Tesla is at disregarding them - I’m pretty sure Tesla won’t be hurt by it.
- zionic 4y ago
  Because that’s a stupid system. I will never accept being under camera surveillance constantly by the car, no matter what stupid name people come up for it (“attention monitoring”)
  - alluro2 4y ago
    Playing the devil's advocate if I may - why would you oppose e.g. an internal, offline system that monitors your eyes for being closed and can save your spouse and kids from dying in a car crash if you doze off during a long night drive, by alerting you / safely stopping the vehicle? I saw a similar reaction to Volvo's announced plans for such tech (it was a YouTube video) - it seemed to be a completely offline system, but people still read "camera" and reacted that they don't want "someone watching them" in the car.
    - lstodd 4y ago
      Because you can never be sure it's 'completely offline' or that it actually can help. If you don't want to kill a couple dozen of people by driving into a fuel truck, just don't drive while about to pass out. It's that simple, but no electronics can ever replace responsibility.
      - ollifi 4y ago
        Also if you don’t want to kill people don’t have a seizure or a brain infarct or a stroke or such while driving.
deleted 4y ago
[deleted]
joe_the_user 4y ago
"I encourage reading the actual report." Wow. It seems like now on HN, this sort of turn-of-phrase has sadly become a boiler plate way to dismiss a host of comments. IE, what "actual report" are you referring to?? The headline article is about the investigation that has opened but not yet closed, as the linked text shows (it's a PDF but it's essentially a press release showing discoveries so far - why they've escalated. It's not closed and not friendly to Tesla). "Accordingly, PE21-020 is upgraded to an Engineering Analysis to extend the existing crash analysis, evaluate additional data sets, perform vehicle evaluations, and to explore the degree to which Autopilot and associated Tesla systems may exacerbate human factors or behavioral safety risks by undermining the effectiveness of the driver’s supervision" https://static.nhtsa.gov/odi/inv/2022/INOA-EA22002-3184.PDF https://static.nhtsa.gov/odi/inv/2022/INOA-EA22002-3184.PDF
ActorNightly 4y ago
It really should be clear to anyone that any of this political attention on Tesla is just theater to win elections/reelections. Its pretty much not worth even paying attention to.
- dangrossman 4y ago
  NHTSA formally investigated Tesla Autopilot after incidents in 2016, 2017 and 2018. There was also a formal NHTSA investigation into Tesla battery fires in 2013. These investigations span 3 presidencies under both parties.
  - ActorNightly 4y ago
    NTHSA also is the type of agency that got lobbied by auto makers into instituting the minimum 25 year old import rule under the pretense of safety, because it ate into profits of foreign manufactures selling cars in US.
    - dangrossman 4y ago
      So is this clearly political theater for an election, or is it clearly lobbying by foreign auto manufacturers?

by epgui 4y ago

dangrossman 4y ago

The worst case outcome would require Tesla to disable the autopilot suite entirely, for an indeterminate amount of time, perhaps permanently on the existing fleet of vehicles. The NHTSA is tired of Tesla's hand-waving away their safety investigations into Autopilot by pushing stealth updates that fix specific scenarios in specific places being investigated. NHTSA wisened up to that and independently purchased their own Tesla vehicles, and disabled software updates, so that they can reproduce those scenarios themselves. If NHTSA asks Tesla to provide system validation tests showing that an updated version of their software meets the design intent of the system, Tesla would not be able to do so. If they can't prove the new Autopilot software corrects the safety-related defects identified in the current version, then it's not a valid recall remedy. All evidence from their own AI/AP team and presentations is that there is no real design and system validation going on over there. They're flying by the seat of their pants, introducing potentially lethal regressions in every update.

AzzieElbab 4y ago
If disabling FSD makes teslas less safe then what is the point? Are they saying fsd can potentially go berserk? Are we into future crime prevention?
- hedora 4y ago
  Yes, in the same way that taking down an active gunman with a loaded weapon is future crime prevention.
  - AzzieElbab 4y ago
    What about possible future software update to any hospital system? Should we preemptively stop all of those?
    - hedora 4y ago
      If the manufacturer isn't adequately testing for regressions that kill people, then yes, we should block those updates, and use the software the device was certified with.
      - AzzieElbab 4y ago
        gl getting anything done with a government agency defining adequacy of your software testing. here in canada government employees cannot get software that pays them salaries to work https://en.wikipedia.org/wiki/Phoenix_pay_system https://en.wikipedia.org/wiki/Phoenix_pay_system
epgui 4y ago
If you read the report, you will realize that NHTSA is considering requiring Tesla to do better driver attention monitoring, or to improve alerting. They are not considering banning autopilot.
- dangrossman 4y ago
  I assure you, I'd already read the report before it was shared here. I also assure you, there's more to the investigation than that.
  - epgui 4y ago
    Perhaps, but that’s speculation at best.
    - deleted 4y ago
      [deleted]
    - sumy23 4y ago
      But you’ve been assured with absolutely no evidence /s.
    - moralestapia 4y ago
      99% of everything is speculation at best, even this site is pretty much (high value) speculation as a service. The NHTSA has a reputation of not f*king around so I would definitely side with @dangrossman on this thing. As of today, Autopilot IS dangerous software and it is not something that should be tested live on the streets.
      - GekkePrutser 4y ago
        > The NHTSA has a reputation of not f*king around so I would definitely side with @dangrossman on this thing. So did the FAA and then they let Boeing self-validate the 737 MAX. Just saying..
        4y ago
wolverine876 4y ago
> The NHTSA is tired of Tesla's hand-waving away their safety investigations into Autopilot by pushing stealth updates that fix specific scenarios in specific places being investigated. Why isn't Tesla prosecuted for that? It's lawless!
- jquery 4y ago
  It really is insane. It’s one thing to have flaws, it’s quite another to stealth cover-them-up like it’s a game of hide and seek.
- extheat 4y ago
  No, that’s typical software development. Find a bug in circulation, fix and deploy a fix. There are probably hundreds of internal issues that get fixed per normal protocol, as with any piece of software. Putting out a “recall” or alert for every modification to the code is pointless. What regulators need to do is keep up with the times. They need to have their own safety test suite which manufacturers can test against, and be independently audited
  - rtlfe 4y ago
    > No, that’s typical software development. Software that controls multi-thousand pound machines at 70+mph isn't typical, and typical practices don't necessarily apply.
    - codebolt 4y ago
      Yes, those practices absolutely shouldn't apply for self driving cars. Good luck regression testing how a system change impacts the AI handling of every edge case of traffic.
      - sidibe 4y ago
        Waymo does this. Their infrastructure costs would be astronomical though if not attached to a company with its own cloud
ajross 4y ago
> introducing potentially lethal regressions in every update. Meh. I mean, I understand the emotional content of that argument, and the propriety angle is real enough. I really do get what you're saying. And if your prior is "Tesla is bad", that's going to be really convincing. But if it's not... The bottom line is that they're getting close to 3M of these vehicles on the roads now. You can't spit without hitting one in the south bay. And the accident statistics just aren't there. They're not. There's a small handful of verifiable accidents, all on significantly older versions of the software. Bugs are real. They've happened before and they'll no doubt happen again, just like they do with every other product. But the Simple Truth remains that these are very safe cars. They are. So... what exactly are people getting upset about? Because it doesn't seem to be what people claim they're getting upset about.
- ken47 4y ago
  Total straw man. The question isn't whether Tesla's are safe to drive. The question is whether "autopilot" is safe to auto pilot.
  - Retric 4y ago
    Actual real world data says autopilot is on average at least as safe as the average driver in the average car. Of course that’s on average, in many specific situations it’s much worse but conversely it means it’s better in other situations. How regulators deal with this frankly tricky as the same will likely apply to all self driving systems.
    - ken47 4y ago
      Sources for claims would be appreciated.
    - usrusr 4y ago
      Those real world autopilot averages happen exclusively in the most trivial driving situations. Fair weather and almost exclusively on limited access roads. No real world average driver dataset exists that is similarly restricted to the subset of least error-prone driving situations.
deleted 4y ago
[deleted]
baryphonic 4y ago
> All evidence from their own AI/AP team and presentations is that there is no real design and system validation going on over there. They're flying by the seat of their pants, introducing potentially lethal regressions in every update. What is this evidence? I've seen a few talks from Andrej Karpathy that indicate to me a more deliberate approach.[0] "Software 2.0" itself seems like an approach meant to systematize the development, validation & testing of AI systems, hardly a seat-of-your-pants approach to releases. I have my own criticisms of their approach, but it seems there is pretty deliberate care taken when developing models. [0] https://youtu.be/hx7BXih7zx8 https://youtu.be/hx7BXih7zx8
- refulgentis 4y ago
  I see your point, to OP's point, I know a couple people who were horrified at what they saw and it did not match this public talk. Both started at least 6 months after this video, and both left Tesla within 8 months, of their own volition. Unfortunately, off the record. Not to disparage Andrej, sometimes (frequently, even) what executive leadership thinks is going is not the day-to-day reality of the team.
  - plankers 4y ago
    can confirm, a former coworker had just come from Tesla 5 years ago and he had serious ethical problems with his work over there. Tesla is killing people through negligence and greed, it's pretty disgusting, but par for the course
- justapassenger 4y ago
  I’ve been working few years ago at a very big tech company, focusing on validation of the AI systems. It’s all smoke and mirrors. You cannot perform proper validation of AI systems. Rollbacks of new versions of ML models are very common in production, and even after very extensive validation you can see that real life results are nothing like what tests have shown.
  - urthor 4y ago

Isinlor 4y ago

Do you have the link to the report?

dangrossman 4y ago
It's in the article: https://static.nhtsa.gov/odi/inv/2022/INOA-EA22002-3184.PDF https://static.nhtsa.gov/odi/inv/2022/INOA-EA22002-3184.PDF

deleted 4y ago

[deleted]

dragonwriter 4y ago

> the biggest plausible consequence would be forcing Tesla to push an over-the-air update with better driver attention monitoring or alerting. I see no basis for this conclusion, which appears to be pure speculation about what NHTSA might decide is necessary and sufficient to address the potential problems if confirmed by the deeper analysis. I encourage reading the actual report. > I encourage reading the actual report. I did, and the conclusion do which you appeal to it does not appear to be well-supported by it.

justapassenger 4y ago

Tesla has no hardware for proper driver monitoring. Most of model S have no internal camera. And model 3 internal camera wasn’t designed for it (doesn’t work in the dark, cannot see through sunglasses, cannot see where your eyes are actually pointing, etc). You cannot OTA HW deficiencies. Now, will they be forced to have monitoring like that, to be on par with their competitors? That’s a different story, and given how weak USA regulatory agencies are, and how reckless Tesla is at disregarding them - I’m pretty sure Tesla won’t be hurt by it.

zionic 4y ago
Because that’s a stupid system. I will never accept being under camera surveillance constantly by the car, no matter what stupid name people come up for it (“attention monitoring”)
- alluro2 4y ago
  Playing the devil's advocate if I may - why would you oppose e.g. an internal, offline system that monitors your eyes for being closed and can save your spouse and kids from dying in a car crash if you doze off during a long night drive, by alerting you / safely stopping the vehicle? I saw a similar reaction to Volvo's announced plans for such tech (it was a YouTube video) - it seemed to be a completely offline system, but people still read "camera" and reacted that they don't want "someone watching them" in the car.
  - lstodd 4y ago
    Because you can never be sure it's 'completely offline' or that it actually can help. If you don't want to kill a couple dozen of people by driving into a fuel truck, just don't drive while about to pass out. It's that simple, but no electronics can ever replace responsibility.
    - ollifi 4y ago
      Also if you don’t want to kill people don’t have a seizure or a brain infarct or a stroke or such while driving.

deleted 4y ago

[deleted]

joe_the_user 4y ago

"I encourage reading the actual report." Wow. It seems like now on HN, this sort of turn-of-phrase has sadly become a boiler plate way to dismiss a host of comments. IE, what "actual report" are you referring to?? The headline article is about the investigation that has opened but not yet closed, as the linked text shows (it's a PDF but it's essentially a press release showing discoveries so far - why they've escalated. It's not closed and not friendly to Tesla). "Accordingly, PE21-020 is upgraded to an Engineering Analysis to extend the existing crash analysis, evaluate additional data sets, perform vehicle evaluations, and to explore the degree to which Autopilot and associated Tesla systems may exacerbate human factors or behavioral safety risks by undermining the effectiveness of the driver’s supervision" https://static.nhtsa.gov/odi/inv/2022/INOA-EA22002-3184.PDF https://static.nhtsa.gov/odi/inv/2022/INOA-EA22002-3184.PDF

ActorNightly 4y ago

It really should be clear to anyone that any of this political attention on Tesla is just theater to win elections/reelections. Its pretty much not worth even paying attention to.

dangrossman 4y ago
NHTSA formally investigated Tesla Autopilot after incidents in 2016, 2017 and 2018. There was also a formal NHTSA investigation into Tesla battery fires in 2013. These investigations span 3 presidencies under both parties.
- ActorNightly 4y ago
  NTHSA also is the type of agency that got lobbied by auto makers into instituting the minimum 25 year old import rule under the pretense of safety, because it ate into profits of foreign manufactures selling cars in US.
  - dangrossman 4y ago
    So is this clearly political theater for an election, or is it clearly lobbying by foreign auto manufacturers?