7 ms·

The worst case outcome would require Tesla to disable the autopilot suite entirely, for an indeterminate amount of time, perhaps permanently on the existing fle

The worst case outcome would require Tesla to disable the autopilot suite entirely, for an indeterminate amount of time, perhaps permanently on the existing fleet of vehicles. The NHTSA is tired of Tesla's hand-waving away their safety investigations into Autopilot by pushing stealth updates that fix specific scenarios in specific places being investigated. NHTSA wisened up to that and independently purchased their own Tesla vehicles, and disabled software updates, so that they can reproduce those scenarios themselves. If NHTSA asks Tesla to provide system validation tests showing that an updated version of their software meets the design intent of the system, Tesla would not be able to do so. If they can't prove the new Autopilot software corrects the safety-related defects identified in the current version, then it's not a valid recall remedy. All evidence from their own AI/AP team and presentations is that there is no real design and system validation going on over there. They're flying by the seat of their pants, introducing potentially lethal regressions in every update.

AzzieElbab 4y ago
If disabling FSD makes teslas less safe then what is the point? Are they saying fsd can potentially go berserk? Are we into future crime prevention?
- hedora 4y ago
  Yes, in the same way that taking down an active gunman with a loaded weapon is future crime prevention.
  - AzzieElbab 4y ago
    What about possible future software update to any hospital system? Should we preemptively stop all of those?
    - hedora 4y ago
      If the manufacturer isn't adequately testing for regressions that kill people, then yes, we should block those updates, and use the software the device was certified with.
      - AzzieElbab 4y ago
        gl getting anything done with a government agency defining adequacy of your software testing. here in canada government employees cannot get software that pays them salaries to work https://en.wikipedia.org/wiki/Phoenix_pay_system https://en.wikipedia.org/wiki/Phoenix_pay_system
epgui 4y ago
If you read the report, you will realize that NHTSA is considering requiring Tesla to do better driver attention monitoring, or to improve alerting. They are not considering banning autopilot.
- dangrossman 4y ago
  I assure you, I'd already read the report before it was shared here. I also assure you, there's more to the investigation than that.
  - epgui 4y ago
    Perhaps, but that’s speculation at best.
    - deleted 4y ago
      [deleted]
    - sumy23 4y ago
      But you’ve been assured with absolutely no evidence /s.
    - moralestapia 4y ago
      99% of everything is speculation at best, even this site is pretty much (high value) speculation as a service. The NHTSA has a reputation of not f*king around so I would definitely side with @dangrossman on this thing. As of today, Autopilot IS dangerous software and it is not something that should be tested live on the streets.
      - GekkePrutser 4y ago
        > The NHTSA has a reputation of not f*king around so I would definitely side with @dangrossman on this thing. So did the FAA and then they let Boeing self-validate the 737 MAX. Just saying..
        4y ago
wolverine876 4y ago
> The NHTSA is tired of Tesla's hand-waving away their safety investigations into Autopilot by pushing stealth updates that fix specific scenarios in specific places being investigated. Why isn't Tesla prosecuted for that? It's lawless!
- jquery 4y ago
  It really is insane. It’s one thing to have flaws, it’s quite another to stealth cover-them-up like it’s a game of hide and seek.
- extheat 4y ago
  No, that’s typical software development. Find a bug in circulation, fix and deploy a fix. There are probably hundreds of internal issues that get fixed per normal protocol, as with any piece of software. Putting out a “recall” or alert for every modification to the code is pointless. What regulators need to do is keep up with the times. They need to have their own safety test suite which manufacturers can test against, and be independently audited
  - rtlfe 4y ago
    > No, that’s typical software development. Software that controls multi-thousand pound machines at 70+mph isn't typical, and typical practices don't necessarily apply.
    - codebolt 4y ago
      Yes, those practices absolutely shouldn't apply for self driving cars. Good luck regression testing how a system change impacts the AI handling of every edge case of traffic.
      - sidibe 4y ago
        Waymo does this. Their infrastructure costs would be astronomical though if not attached to a company with its own cloud
ajross 4y ago
> introducing potentially lethal regressions in every update. Meh. I mean, I understand the emotional content of that argument, and the propriety angle is real enough. I really do get what you're saying. And if your prior is "Tesla is bad", that's going to be really convincing. But if it's not... The bottom line is that they're getting close to 3M of these vehicles on the roads now. You can't spit without hitting one in the south bay. And the accident statistics just aren't there. They're not. There's a small handful of verifiable accidents, all on significantly older versions of the software. Bugs are real. They've happened before and they'll no doubt happen again, just like they do with every other product. But the Simple Truth remains that these are very safe cars. They are. So... what exactly are people getting upset about? Because it doesn't seem to be what people claim they're getting upset about.
- ken47 4y ago
  Total straw man. The question isn't whether Tesla's are safe to drive. The question is whether "autopilot" is safe to auto pilot.
  - Retric 4y ago
    Actual real world data says autopilot is on average at least as safe as the average driver in the average car. Of course that’s on average, in many specific situations it’s much worse but conversely it means it’s better in other situations. How regulators deal with this frankly tricky as the same will likely apply to all self driving systems.
    - ken47 4y ago
      Sources for claims would be appreciated.
    - usrusr 4y ago
      Those real world autopilot averages happen exclusively in the most trivial driving situations. Fair weather and almost exclusively on limited access roads. No real world average driver dataset exists that is similarly restricted to the subset of least error-prone driving situations.
deleted 4y ago
[deleted]
baryphonic 4y ago
> All evidence from their own AI/AP team and presentations is that there is no real design and system validation going on over there. They're flying by the seat of their pants, introducing potentially lethal regressions in every update. What is this evidence? I've seen a few talks from Andrej Karpathy that indicate to me a more deliberate approach.[0] "Software 2.0" itself seems like an approach meant to systematize the development, validation & testing of AI systems, hardly a seat-of-your-pants approach to releases. I have my own criticisms of their approach, but it seems there is pretty deliberate care taken when developing models. [0] https://youtu.be/hx7BXih7zx8 https://youtu.be/hx7BXih7zx8
- refulgentis 4y ago
  I see your point, to OP's point, I know a couple people who were horrified at what they saw and it did not match this public talk. Both started at least 6 months after this video, and both left Tesla within 8 months, of their own volition. Unfortunately, off the record. Not to disparage Andrej, sometimes (frequently, even) what executive leadership thinks is going is not the day-to-day reality of the team.
  - plankers 4y ago
    can confirm, a former coworker had just come from Tesla 5 years ago and he had serious ethical problems with his work over there. Tesla is killing people through negligence and greed, it's pretty disgusting, but par for the course
- justapassenger 4y ago
  I’ve been working few years ago at a very big tech company, focusing on validation of the AI systems. It’s all smoke and mirrors. You cannot perform proper validation of AI systems. Rollbacks of new versions of ML models are very common in production, and even after very extensive validation you can see that real life results are nothing like what tests have shown.
  - urthor 4y ago

AzzieElbab 4y ago

If disabling FSD makes teslas less safe then what is the point? Are they saying fsd can potentially go berserk? Are we into future crime prevention?

hedora 4y ago
Yes, in the same way that taking down an active gunman with a loaded weapon is future crime prevention.
- AzzieElbab 4y ago
  What about possible future software update to any hospital system? Should we preemptively stop all of those?
  - hedora 4y ago
    If the manufacturer isn't adequately testing for regressions that kill people, then yes, we should block those updates, and use the software the device was certified with.
    - AzzieElbab 4y ago
      gl getting anything done with a government agency defining adequacy of your software testing. here in canada government employees cannot get software that pays them salaries to work https://en.wikipedia.org/wiki/Phoenix_pay_system https://en.wikipedia.org/wiki/Phoenix_pay_system

epgui 4y ago

If you read the report, you will realize that NHTSA is considering requiring Tesla to do better driver attention monitoring, or to improve alerting. They are not considering banning autopilot.

dangrossman 4y ago
I assure you, I'd already read the report before it was shared here. I also assure you, there's more to the investigation than that.
- epgui 4y ago
  Perhaps, but that’s speculation at best.
  - deleted 4y ago
    [deleted]
  - sumy23 4y ago
    But you’ve been assured with absolutely no evidence /s.
  - moralestapia 4y ago
    99% of everything is speculation at best, even this site is pretty much (high value) speculation as a service. The NHTSA has a reputation of not f*king around so I would definitely side with @dangrossman on this thing. As of today, Autopilot IS dangerous software and it is not something that should be tested live on the streets.
    - GekkePrutser 4y ago
      > The NHTSA has a reputation of not f*king around so I would definitely side with @dangrossman on this thing. So did the FAA and then they let Boeing self-validate the 737 MAX. Just saying..
      - 4y ago

wolverine876 4y ago

> The NHTSA is tired of Tesla's hand-waving away their safety investigations into Autopilot by pushing stealth updates that fix specific scenarios in specific places being investigated. Why isn't Tesla prosecuted for that? It's lawless!

jquery 4y ago
It really is insane. It’s one thing to have flaws, it’s quite another to stealth cover-them-up like it’s a game of hide and seek.
extheat 4y ago
No, that’s typical software development. Find a bug in circulation, fix and deploy a fix. There are probably hundreds of internal issues that get fixed per normal protocol, as with any piece of software. Putting out a “recall” or alert for every modification to the code is pointless. What regulators need to do is keep up with the times. They need to have their own safety test suite which manufacturers can test against, and be independently audited
- rtlfe 4y ago
  > No, that’s typical software development. Software that controls multi-thousand pound machines at 70+mph isn't typical, and typical practices don't necessarily apply.
  - codebolt 4y ago
    Yes, those practices absolutely shouldn't apply for self driving cars. Good luck regression testing how a system change impacts the AI handling of every edge case of traffic.
    - sidibe 4y ago
      Waymo does this. Their infrastructure costs would be astronomical though if not attached to a company with its own cloud

ajross 4y ago

> introducing potentially lethal regressions in every update. Meh. I mean, I understand the emotional content of that argument, and the propriety angle is real enough. I really do get what you're saying. And if your prior is "Tesla is bad", that's going to be really convincing. But if it's not... The bottom line is that they're getting close to 3M of these vehicles on the roads now. You can't spit without hitting one in the south bay. And the accident statistics just aren't there. They're not. There's a small handful of verifiable accidents, all on significantly older versions of the software. Bugs are real. They've happened before and they'll no doubt happen again, just like they do with every other product. But the Simple Truth remains that these are very safe cars. They are. So... what exactly are people getting upset about? Because it doesn't seem to be what people claim they're getting upset about.

ken47 4y ago
Total straw man. The question isn't whether Tesla's are safe to drive. The question is whether "autopilot" is safe to auto pilot.
- Retric 4y ago
  Actual real world data says autopilot is on average at least as safe as the average driver in the average car. Of course that’s on average, in many specific situations it’s much worse but conversely it means it’s better in other situations. How regulators deal with this frankly tricky as the same will likely apply to all self driving systems.
  - ken47 4y ago
    Sources for claims would be appreciated.
  - usrusr 4y ago
    Those real world autopilot averages happen exclusively in the most trivial driving situations. Fair weather and almost exclusively on limited access roads. No real world average driver dataset exists that is similarly restricted to the subset of least error-prone driving situations.

deleted 4y ago

[deleted]

baryphonic 4y ago

> All evidence from their own AI/AP team and presentations is that there is no real design and system validation going on over there. They're flying by the seat of their pants, introducing potentially lethal regressions in every update. What is this evidence? I've seen a few talks from Andrej Karpathy that indicate to me a more deliberate approach.[0] "Software 2.0" itself seems like an approach meant to systematize the development, validation & testing of AI systems, hardly a seat-of-your-pants approach to releases. I have my own criticisms of their approach, but it seems there is pretty deliberate care taken when developing models. [0] https://youtu.be/hx7BXih7zx8 https://youtu.be/hx7BXih7zx8

refulgentis 4y ago
I see your point, to OP's point, I know a couple people who were horrified at what they saw and it did not match this public talk. Both started at least 6 months after this video, and both left Tesla within 8 months, of their own volition. Unfortunately, off the record. Not to disparage Andrej, sometimes (frequently, even) what executive leadership thinks is going is not the day-to-day reality of the team.
- plankers 4y ago
  can confirm, a former coworker had just come from Tesla 5 years ago and he had serious ethical problems with his work over there. Tesla is killing people through negligence and greed, it's pretty disgusting, but par for the course
justapassenger 4y ago
I’ve been working few years ago at a very big tech company, focusing on validation of the AI systems. It’s all smoke and mirrors. You cannot perform proper validation of AI systems. Rollbacks of new versions of ML models are very common in production, and even after very extensive validation you can see that real life results are nothing like what tests have shown.
- urthor 4y ago