15 ms·

Tesla is ordered to turn over Autopilot data to a federal safety agency

128 points by donohoe 5y ago

meepmorp 5y ago
> The safety agency informed Tesla weeks ago that it was looking into the spate of crashes in which Tesla vehicles operating on Autopilot failed to detect stopped emergency vehicles with flashing lights. The regulator originally said it was looking into 11 such crashes. A 12th occurred on Saturday, when a Model 3 hit a police cruiser that had stopped behind a car that had broken down on an interstate in Orlando, Fla. 12 crashes into vehicles with flashing lights on them doesn't sound like great success from Autopilot, gonna be honest.
- brianwawok 5y ago
  The driver is responsible for stopping the car. You literally agree to it when you enable AP. People are too comfortable and not paying attention.
  - mhermher 5y ago
    The fact that it occurs so often should suggest to you that autopilot itself makes people too comfortable and not pay attention.
    - jerf 5y ago
      I, and many others, take the position that there really isn't any practical gradation between more-or-less conventional driving, maybe with light lane switching assistance or something, and full on "Level 5" autonomous driving, because the idea that human brains can watch something for hours at a time only to take instant, sub-second actions every few hours is an insane model of human cognition. I don't disagree with the idea of having a gradient to measure our progress up to full autonomy, but I haven't seen much to say that we should deploy the intermediate stages. Until autonomy can be reasonably guaranteed at at least human levels of safety under the assumption of no human control, it really shouldn't be deployed. "But the human had 673 milliseconds to take control and avert the accident! It's their fault!" is simply not a reasonable defense.
      - joelwilliamson 5y ago
        Shouldn't the objection be to anything less than Level 4, rather than Level 5? Level 4 requires that the vehicle can safely stop itself and wait for a human to take over when it is faced with a situation it can't handle. (There's an argument that if you take away the routine parts of driving, humans won't be able to deal with anomalous situations, but that's a different argument than the attentiveness one.)
        jerf 5y ago
        If that is possible, yes. I'm not convinced it is possible to create an autonomous vehicle that only panics in situations where there is a safe pull-over, or, if such a vehicle exists, that it'll be considered "autonomous" when it can't go past 25 or 35 mph or something. There's plenty of driving situations that may be confusing where pulling over and stopping isn't possible. (And, to be clear, I'm not asking for perfect safety, I'm asking for "human or better", and also understand that there may be some interesting fuzziness near the border where the two have different strengths and weakenesses but come out statistically the same. However, I don't feel like we're there yet.)
    - galangalalgol 5y ago
      Mixed control between automated and human for tasks that require persistent attention is bad. We have known this for decades. The example often given in engineering textbooks is 3 mile island. I suspect that the engineers involved in self driving systems know this too, but the liability is too great to make the switch all at once. It seems better to take baby steps into self driving, but history teaches us it should be all or nothing.
      - fallingknife 5y ago
        You could say the same about airplane autopilots, but they have an amazing safety record.
        kaibee 5y ago
        Airplanes also have 2 pilots and nothing to crash into.
        Piskvorrr 5y ago
        Far fewer objects to hit in the air: disengagement means "keep going this way," rarely "and crash into an eventual obstacle". Plus, their early safety record is written in blood like everything else is.
        fallingknife 5y ago
        > Plus, their early safety record is written in blood like everything else is. Yeah, and that's how it's going to be with car autopilots. Which is why it pisses me off that people want to stop progress on it.
        Piskvorrr 5y ago
        Interesting typoes in "be unwilling test subjects". Feel free to volunteer, though. Elaine Herzberg sure didn't.
        meepmorp 5y ago
  - deleted 5y ago
    [deleted]
  - meepmorp 5y ago
    > The driver is responsible for stopping the car. You literally agree to it when you enable AP. People are too comfortable and not paying attention. Autopilot encourages people to not pay as much attention while driving, until a critical situation comes up at which point they're expected to be fully in control. That's a design flaw that isn't fixed with a liability shifting "Agree" button.
    - tyfon 5y ago
      It will nag you every 17 seconds (if not attentive/holding the wheel with a torque) and abort auto pilot completely if you do not comply. If you do this three times in a drive you have to power off and on the car again to be able to use autopilot again. I've done maybe 40.000 km on autopilot and it's very aggressive compared to my other car (ioniq electric) when it comes to attention. The fact that some people purposely defeat this system with weights on the wheel is another matter. But saying that tesla encourages you to not paying attention is not correct.
      - jsight 5y ago
        To be fair, the person you are responding to may not like those other systems either. All the articles with industry analysts insisting that other driver monitoring systems are nearly perfect have sometimes blinded me to that fact. A lot of people don't like any of these systems for a lot of the same reasons that people didn't like basic cruise control in the early days.
        ghaff 5y ago
        I haven't used basic cruise control in years. It's actually convenient not to have to pay attention to the speed but I find, on roads with even moderate traffic, it comes at the cost of prioritizing maintaining a constant speed rather than flowing with traffic and generally staying in a lane.
  - avereveard 5y ago
    You have no telemetry and no idea what actually happened, car might have veered without enough warning. Which is the whole point of having an investigation, really.
- capableweb 5y ago
  > 12 crashes into vehicles with flashing lights on them doesn't sound like great success from Autopilot, gonna be honest. I'm sorry, but how can you form an opinion so quickly? 12 crashes out of what? Any number? 12 crashes out of 10000000 scenarios sound like a pretty good record, while 12 crashes out of 15 scenarios indeed doesn't sound like a great success. But what information do you have that the rest of us doesn't?
  - swiley 5y ago
    yes. 12 crashes out of the maximum possible number of encounters.
  - AshamedCaptain 5y ago
    I still remember the days when the SciFi of the day sold robotic drivers as "they don't get tired, they don't make mistakes, they are relentless in their pursuit". Instead of that, we got robots that are apparently unable to avoid crashing into large vehicles with flashing lights. Sometimes. As usual, the future never ceases to disappoint me.
    - leesec 5y ago
      Well if you're basing your expectations of reality on SciFi books than I expect you will be dissapointed quite often.
      - Gravityloss 5y ago
        There's always dystopian scifi as well.
        sslayer 5y ago
        Yes! - This has not failed to disappoint, rather it has exceeded all expectations!
    - DennisP 5y ago
      Maybe we just haven't reached that future yet.
- brk 5y ago
  We just did a 1,500 mile round-trip journey to visit family. Because of the Tesla stuff, I was casually aware of roadside emergency vehicles moreso than usual. We passed 28 police/fire/emergency vehicles that were next to our travel lane. I saw a number of Tesla vehicles on the road as well, each passing these same emergency vehicles, but not always in the same lane. By this point, there must be 1M+ incidents of Tesla vehicles passing roadside emergency vehicles in an adjacent lane. 12 crashes is 12 crashes too many, statistically speaking, but it's also a tiny tiny percentage of the total number of cases. IMO, it is good that there is federal investigation into this. My main concern is that is seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things.
  - fallingknife 5y ago
    > IMO, it is good that there is federal investigation into this > seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things. Not sure how you think it's good then.
  - tw04 5y ago
    >I saw a number of Tesla vehicles on the road as well, each passing these same emergency vehicles, but not always in the same lane. How do you know all of them were on autopilot? >12 crashes is 12 crashes too many, statistically speaking, but it's also a tiny tiny percentage of the total number of cases. It's also 12 more than any normal attentive human has gotten into, and an average attentive human driver is the absolute floor for self-driving on public roads unless Tesla is going to go to the lengths GM has to ensure the person behind the wheel is remaining attentive. >IMO, it is good that there is federal investigation into this. My main concern is that is seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things. I think they're good at understanding if Tesla loses complete public confidence in self driving that is a net-loss for humanity and technology, as well as a prime source of political shitstorm when people ask why something wasn't done sooner.
- nashashmi 5y ago
  The fact that Tesla and Elon have been fighting these autopilot errors in completely the wrong way is the primary reason the feds should be involved. I have been using autopilot in Nissan. If the car were to crash into an ambulance light going off I would not be able to predict it and stop it from happening quickly enough. I just have a whole lot of trust in some basic things the car does. And this is one of those cases where they should have been able to get this right after the first crash but they didn’t and now the feds have to get involved. A lesson on why we can’t have nice things.
  - CultMember001 5y ago
    We should not allow the Fed to stifle innovation and technological progress. If we are not willing to take risks, then we will never be able to reach out true potential of becoming a truly advanced space faring civilization and a multi-planetary species. The NHTSA does not have the technological capability to understand advanced AI used in Autopilot and Full Self Driving and they should stop their interference immediately. I have Tesla and I bought the Full Self Driving package. I believe it is far more valuable than the current price. I am willing to pay 30 times more for Full Self Driving.
- noslenwerdna 5y ago
  Thanks for your honesty. Hopefully the NHTSA is also going to investigate the many other cars with traffic aware cruise control, which have the same problem with stationary vehicles. I was driving my Rav4 with TACC enabled, and it would have crashed into a column of stopped cars if I hadn't noticed in time and intervened.
  - ahahahahah 5y ago
    Tesla has something like 1000x greater rate of hitting stationary emergency vehicles. The fact that all the other cars have the same issue with tacc actually points even more strongly to something being wrong with Tesla.
    - kwhitefoot 5y ago
      > 1000x greater rate What's your denominator? Is this per year or per kilometre driven with TACC active?
    - noslenwerdna 5y ago
      citation, please.
- tablespoon 5y ago
  >> The safety agency informed Tesla weeks ago that it was looking into the spate of crashes in which Tesla vehicles operating on Autopilot failed to detect stopped emergency vehicles with flashing lights. The regulator originally said it was looking into 11 such crashes. A 12th occurred on Saturday, when a Model 3 hit a police cruiser that had stopped behind a car that had broken down on an interstate in Orlando, Fla. > 12 crashes into vehicles with flashing lights on them doesn't sound like great success from Autopilot, gonna be honest. It's really surprising what people will put up with. IIRC, recently "autopilot" had some regressions in the situations it could handle, and Elon Musk tweeted something to the effect of "Yeah, this version sucks since we're retraining all our NNs." Like how did something that half-baked ever make it to production in a safety critical system?
fallingknife 5y ago
> The regulator originally said it was looking into 11 such crashes. A 12th occurred on Saturday But there are 4500 crashes every year, and that's only including ambulances (https://www.ems.gov/pdf/GroundAmbulanceCrashesPresentation.pdf https://www.ems.gov/pdf/GroundAmbulanceCrashesPresentation.p... pg 22). So why is Tesla being investigated for being involved in 0.25% of these crashes when they made up 3% of car sales in 2020? Also, radar ignores stationary objects on any car. Why is only Tesla being investigated when this exact same flaw exists in every other car manufacturer that uses radar based cruise control?
- roystonvassey 5y ago
  I think the right stat to measure is - of all the Tesla cars out there, how many had crashes, and compare this against the average for other car brands/models.
  - SketchySeaBeast 5y ago
    I'd hope that "automated driving" has standards much much higher than the average.
- asddubs 5y ago
  They're looking specifically into crashes happening on autopilot though, so using that 3% figure assumes people are using the tesla in autopilot mode 100% of the time
- okl 5y ago
  No offense but that is whataboutism. I think every crash, where an investigation could prevent another crash in the future, should be investigated. Shouldn't Tesla "celebrate their mistakes" (yes, morbid, sorry) and see this as an opportunity to improve their product?
  - DennisP 5y ago
    Tesla is already putting a lot of resources into improving Autopilot and FSD. I'm not sure how regulators are going to help that process, and I think there are a couple possible pitfalls: (1) Will they take relative risk into account? If the car sometimes crashes but less often than humans, will that be sufficient, or considered a flaw because it's not perfect and regulators hate looking bad? (2) Will they take progress into account? If the car drives worse than humans but its accident rate drops 10% per year, then after some number of years we'll accumulate fewer total accidents than if we hadn't allowed progress to continue. That's important because progress won't continue without lots of cars on the road using Autopilot and FSD betas.
pomian 5y ago
Maybe the driving data has nothing to do with crash, or to put it another way, the log of the driver's route and actions up to the point of the crash, is the point.
mchusma 5y ago
Hopefully the TSA doesn't try to kill as many people as the FDA has. The big lesson from the pandemic was how poorly these agencies are run, and their complete willingness to see thousands die by implementing restrictions.
Traster 5y ago
Let's ignore the larger point (which is that this is just one of a million possible edge cases where autopilot could systematically fail), I could absolutely see Tesla just bodging in a "If you see emergency vehicle lights flashing disengage" - it'd perfectly reasonable to say that you shouldn't be using autopilot in that situation because you should be paying attention and ready to react to the emergency vehicle.
- Cthulhu_ 5y ago
  It should warn and turn off in a LOT of situations imo; there was another accident in a construction zone where the car hit a barrier, for example. And there's the shared videos of people literally sleeping or sitting in the back seat of a Tesla on autopilot; the car should detect that and disable autopilot for an X amount of time. Of course, at the same time the car should not play police; if someone is caught doing that, their license should be taken away from them.
  - adrr 5y ago
    It should warn and force the driver to confirm to keep autopilot going at set speed. Same way it handles traffic lights and autopilot.
- mikejb 5y ago
  As much as it would be a solution to Tesla's problem, it would impact Tesla's overall plan and strategy. Automatically disengaging when emergency vehicles are detected is a huge barrier in their selling strategy ("It can drive itself, it's just not allowed yet to for legal reasons, but soon").
- duxup 5y ago
  I don't think you're wrong policy wise specifically. I do think that this 'auto pilot ... but you have to manage it' system is not workable for most humans. Paying attention when you sorta don't have to ... is just not paying attention for most people. It's a space I don't think we manage well outside of say a real pilot in a plane trained to do his thing (and there's actually a lot of cases with trained airline pilots losing situational awareness and auto pilot does the wrong thing and the pilots don't notice soon enough / auto pilot makes bad choices for a surprisingly long time before anyone notices). I think this might be a space where it is all or nothing auto pilot wise. Ultimately a human problem.
gubnerpoo 5y ago
Biden, when he's not eating ice cream and awake, hates Tesla because they aren't a union shop. This and other things are just a shakedown. You can see by the echo chamber here who's the bot army and who's not. Remember, the primary purpose of Tesla is to save the planet. But I guess the unions trump that. Fuckers.
emptybits 5y ago
I assume (?) Teslas have interior microphones. Maybe not in this case but how easily will cabin interior data be turned over to authorities to paint a picture of what goes on inside and outside a vehicle or nearby vehicles leading up to an accident? It may be easier to just ask for and release everything. In the name of safety, of course.
longcommonname 5y ago
I wonder if the NHTSA could require all self driving cars to use lidar if the safety stats show it is a significant safety improvement.
- panick21 5y ago
  Then the waste majority of cars simply wouldn't come with lane assist. Deploying lidar on such scale is simply not practical at the moment. Check out how long it took for even simple safety features to make it into all cars.
  - tablespoon 5y ago
    >> I wonder if the NHTSA could require all self driving cars to use lidar if the safety stats show it is a significant safety improvement. > Then the waste majority of cars simply wouldn't come with lane assist. Deploying lidar on such scale is simply not practical at the moment. IMHO, lane assist isn't "self driving" any more than cruise control is. I think you get into self driving territory when the car makes lane changes or turns at intersections.
yawaworht1978 5y ago
The absolving argument that it's still the drivers responsibility to intervene has flown out of the window with this request. The federal safety agency is not having any of that excuse anymore. They might find that Tesla did no wrong here, further enabling Tesla to get away with this. But this looks a lot like they want to regulate the feature a bit more. If self regulation via companies and humans does not work, the government will step in. Curious to see the outcome, they haven't ordered the immediate disablement of the feature, they might have a neutral view.
- duxup 5y ago
  I suspect humans are really bad when they only have to sorta pay attention and it results in just not paying attention a great deal of the time. I'm not sure that's fixable on the human end.
  - matt123456789 5y ago
    The question is, how much of that is Tesla’s fault? They’re an EV company, not a psychology consultancy. We don’t hold gun manufacturers or alcohol companies liable when people do stupid things with their products, despite the best efforts of their marketing divisions. Granted “pull trigger get bullet” is much simpler than the Autopilot response.
    - duxup 5y ago
      We ask airplane makers to make adjustments to better indicate problems / auto pilot status when humans don't do the right thing all the time. And that's for professionals who get tons of training. We have all sorts of car / road safety features because humans are bad at driving. We can't reprogram the humans, can change the environment they operate in.
    - SerLava 5y ago
      It's not "doing a stupid thing" though Fundamentally any life form has a slower reaction when they have to notice an issue and start to take motor control. Versus already being in control and simply not causing the issue. They are not just "an EV company" they are making a guidance system for thousands of high speed metal death machines. They BETTER consult some psychologists. This is a really bizarre thing to say.
Dumblydorr 5y ago
Tesla should never have named this Autopilot, and Elon should've remained silent about way overoptimistic promises. Sometimes, "coming soon" is a great answer, vagueness is a powerful tool to help with time demands of clients.
- tenpies 5y ago
  FTC should have obliterated Tesla from orbit the minute they called cruise control "Autopilot", and the Tesla beta opt-in fund "Full Self Driving", and any time Elon made any claim that the vehicle was going to be self-driving by X date. This would have saved several lives, untold injuries, and millions in damages. It would have been disastrous for Elon's wealth and the market cap of Tesla, but then we could at least pretend there's still rule of law in the US. Thankfully other jurisdictions have actual regulators and are telling Tesla that you cannot use fraudulent names, but who knows how much longer it will take for the US to join the rest of the world in this.
- nilstycho 5y ago
  This was definitely a poor name in retrospect, but I can see why they chose the name. The current functionality of autopilot is in fact pretty similar to aviation autopilots. Unfortunately most of the public thinks that an autopilot can fly a plane unassisted, so they naturally expect "Autopilot" to drive their car unassisted.
deleted 5y ago
[deleted]
CultMember001 5y ago
Is this only for Autopilot data or also for Full Self Driving data? Hasn't Full Self Driving already passed all approvals and perfectly safe? Why would they try to regulate something that is already perfectly safe? I have never heard of Full Self Driving crashing. I think Tesla should charge more for this advanced tech. I think $5000/month/car is a fairer price.
dqpb 5y ago
I'm curious what the data requirement will be. Is it just the segmented map? Is it the actual model plus all inputs for n-seconds leading up to the crash? Is it all training data that meets some threshold of similarity to the incident? It would be nice if the investigation of these incidents resulted in an open standard for analysis and quality control of self-driving systems.
yawaworht1978 5y ago
Another point is, legal human operator liability or not, the agency has been alerted of 12 such incidents which could never, ever occur unless a human driver is asleep, fully drunk/under the influence of drugs, some sort of fight happened in the car or was distracted from heavy sunlight to the eyes or had to dodge a hazard or animal on the road. Their way of thought is simply that if the software can't deal with what is trivial for humans, maybe a recall or better look is in order. I know, this is not trivial for software, it just looks to not technical people like something that should never happen, and if it does , they loose all the trust in it. Obviously, there are many bright engineers working at Tesla, not sure how this was deployed and how this slipped QA repeatedly. This, the authority asks Tesla, before the authority has to answer to the voters , congress and media shows.
panick21 5y ago
This not a feature that Autopilot or any other of the lane keeping systems have at the moment. If we are going to say that this is an issue then pretty much all lane keeping systems would have to be outlawed. If it is really impossible for people to pay attention without their hands on the wheel at all the time then all lane keeping must be abolished or a standard needs to be established what 'paying attention' means. Crashing into emergency vehicles happens often even without lane keeping system as well, so its a univeral problem when you have big stationary objects standing around on driving lanes. What do people on HN actually want or expect in terms or regulation? Here are some suggestions: - Only allow any kind of lane keeping if a comprehensive testing that they can detect any obstetrical of any kind and have emergency breaking. This of course would simply mean less such system would be deployed as most companies don't have practical scalable solution for this as of yet. - Only allow any kind of lane keeping if a comprehensive driver monitoring system is implemented that makes sure driver is alert. Additionally you could require wheel touches every so often in addition to that, if 'hands free' driving should not be allowed. Tesla is in a good spot here, they have the internal cameras and already have a driver monitoring system yet despite that they have not allowed hands free driving. Tesla new Vision Autopilot or the more extensive FSD stack (that will do highway driving with v10) should solve the stationary object problem. I'm sure that if there was any kind of regulation, Tesla would simply comply and move on. In the worst case if regulation would somehow insist on having things named in a specific way, Tesla would just rename their features. However the point here seems to be non of this is technological problem with Tesla, is simply a situation with no regulation and so every manufacturer picks their own solution. Tesla is the most prominent and most talked about so they are the whipping boy of anybody that wants more regulation.
- asddubs 5y ago
  >- Only allow any kind of lane keeping if a comprehensive testing that they can detect any obstetrical of any kind and have emergency breaking. This of course would simply mean less such system would be deployed as most companies don't have practical scalable solution for this as of yet. probably this. it's a nice feature but it lulls you into a false sense of security and just doesn't seem to mesh well with human nature. It works well enough most of the time that people start trusting it too much over time
CultMember002 5y ago
We should not allow the Fed to stifle innovation and technological progress. If we are not willing to take risks, then we will never be able to reach out true potential of becoming a truly advanced space faring civilization and a multi-planetary species. The NHTSA does not have the technological capability to understand advanced AI used in Autopilot and Full Self Driving and they should stop their interference immediately. I have a Tesla and I bought the Full Self Driving package. I believe it is far more valuable than the current price. I am willing to pay 30 times more for Full Self Driving.

128 points by donohoe 5y ago

> The safety agency informed Tesla weeks ago that it was looking into the spate of crashes in which Tesla vehicles operating on Autopilot failed to detect stopped emergency vehicles with flashing lights. The regulator originally said it was looking into 11 such crashes. A 12th occurred on Saturday, when a Model 3 hit a police cruiser that had stopped behind a car that had broken down on an interstate in Orlando, Fla. 12 crashes into vehicles with flashing lights on them doesn't sound like great success from Autopilot, gonna be honest.

brianwawok 5y ago
The driver is responsible for stopping the car. You literally agree to it when you enable AP. People are too comfortable and not paying attention.
- mhermher 5y ago
  The fact that it occurs so often should suggest to you that autopilot itself makes people too comfortable and not pay attention.
  - jerf 5y ago
    I, and many others, take the position that there really isn't any practical gradation between more-or-less conventional driving, maybe with light lane switching assistance or something, and full on "Level 5" autonomous driving, because the idea that human brains can watch something for hours at a time only to take instant, sub-second actions every few hours is an insane model of human cognition. I don't disagree with the idea of having a gradient to measure our progress up to full autonomy, but I haven't seen much to say that we should deploy the intermediate stages. Until autonomy can be reasonably guaranteed at at least human levels of safety under the assumption of no human control, it really shouldn't be deployed. "But the human had 673 milliseconds to take control and avert the accident! It's their fault!" is simply not a reasonable defense.
    - joelwilliamson 5y ago
      Shouldn't the objection be to anything less than Level 4, rather than Level 5? Level 4 requires that the vehicle can safely stop itself and wait for a human to take over when it is faced with a situation it can't handle. (There's an argument that if you take away the routine parts of driving, humans won't be able to deal with anomalous situations, but that's a different argument than the attentiveness one.)
      - jerf 5y ago
        If that is possible, yes. I'm not convinced it is possible to create an autonomous vehicle that only panics in situations where there is a safe pull-over, or, if such a vehicle exists, that it'll be considered "autonomous" when it can't go past 25 or 35 mph or something. There's plenty of driving situations that may be confusing where pulling over and stopping isn't possible. (And, to be clear, I'm not asking for perfect safety, I'm asking for "human or better", and also understand that there may be some interesting fuzziness near the border where the two have different strengths and weakenesses but come out statistically the same. However, I don't feel like we're there yet.)
  - galangalalgol 5y ago
    Mixed control between automated and human for tasks that require persistent attention is bad. We have known this for decades. The example often given in engineering textbooks is 3 mile island. I suspect that the engineers involved in self driving systems know this too, but the liability is too great to make the switch all at once. It seems better to take baby steps into self driving, but history teaches us it should be all or nothing.
    - fallingknife 5y ago
      You could say the same about airplane autopilots, but they have an amazing safety record.
      - kaibee 5y ago
        Airplanes also have 2 pilots and nothing to crash into.
      - Piskvorrr 5y ago
        Far fewer objects to hit in the air: disengagement means "keep going this way," rarely "and crash into an eventual obstacle". Plus, their early safety record is written in blood like everything else is.
        fallingknife 5y ago
        > Plus, their early safety record is written in blood like everything else is. Yeah, and that's how it's going to be with car autopilots. Which is why it pisses me off that people want to stop progress on it.
        Piskvorrr 5y ago
        Interesting typoes in "be unwilling test subjects". Feel free to volunteer, though. Elaine Herzberg sure didn't.
      - meepmorp 5y ago
- deleted 5y ago
  [deleted]
- meepmorp 5y ago
  > The driver is responsible for stopping the car. You literally agree to it when you enable AP. People are too comfortable and not paying attention. Autopilot encourages people to not pay as much attention while driving, until a critical situation comes up at which point they're expected to be fully in control. That's a design flaw that isn't fixed with a liability shifting "Agree" button.
  - tyfon 5y ago
    It will nag you every 17 seconds (if not attentive/holding the wheel with a torque) and abort auto pilot completely if you do not comply. If you do this three times in a drive you have to power off and on the car again to be able to use autopilot again. I've done maybe 40.000 km on autopilot and it's very aggressive compared to my other car (ioniq electric) when it comes to attention. The fact that some people purposely defeat this system with weights on the wheel is another matter. But saying that tesla encourages you to not paying attention is not correct.
    - jsight 5y ago
      To be fair, the person you are responding to may not like those other systems either. All the articles with industry analysts insisting that other driver monitoring systems are nearly perfect have sometimes blinded me to that fact. A lot of people don't like any of these systems for a lot of the same reasons that people didn't like basic cruise control in the early days.
      - ghaff 5y ago
        I haven't used basic cruise control in years. It's actually convenient not to have to pay attention to the speed but I find, on roads with even moderate traffic, it comes at the cost of prioritizing maintaining a constant speed rather than flowing with traffic and generally staying in a lane.
- avereveard 5y ago
  You have no telemetry and no idea what actually happened, car might have veered without enough warning. Which is the whole point of having an investigation, really.
capableweb 5y ago
> 12 crashes into vehicles with flashing lights on them doesn't sound like great success from Autopilot, gonna be honest. I'm sorry, but how can you form an opinion so quickly? 12 crashes out of what? Any number? 12 crashes out of 10000000 scenarios sound like a pretty good record, while 12 crashes out of 15 scenarios indeed doesn't sound like a great success. But what information do you have that the rest of us doesn't?
- swiley 5y ago
  yes. 12 crashes out of the maximum possible number of encounters.
- AshamedCaptain 5y ago
  I still remember the days when the SciFi of the day sold robotic drivers as "they don't get tired, they don't make mistakes, they are relentless in their pursuit". Instead of that, we got robots that are apparently unable to avoid crashing into large vehicles with flashing lights. Sometimes. As usual, the future never ceases to disappoint me.
  - leesec 5y ago
    Well if you're basing your expectations of reality on SciFi books than I expect you will be dissapointed quite often.
    - Gravityloss 5y ago
      There's always dystopian scifi as well.
      - sslayer 5y ago
        Yes! - This has not failed to disappoint, rather it has exceeded all expectations!
  - DennisP 5y ago
    Maybe we just haven't reached that future yet.
brk 5y ago
We just did a 1,500 mile round-trip journey to visit family. Because of the Tesla stuff, I was casually aware of roadside emergency vehicles moreso than usual. We passed 28 police/fire/emergency vehicles that were next to our travel lane. I saw a number of Tesla vehicles on the road as well, each passing these same emergency vehicles, but not always in the same lane. By this point, there must be 1M+ incidents of Tesla vehicles passing roadside emergency vehicles in an adjacent lane. 12 crashes is 12 crashes too many, statistically speaking, but it's also a tiny tiny percentage of the total number of cases. IMO, it is good that there is federal investigation into this. My main concern is that is seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things.
- fallingknife 5y ago
  > IMO, it is good that there is federal investigation into this > seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things. Not sure how you think it's good then.
- tw04 5y ago
  >I saw a number of Tesla vehicles on the road as well, each passing these same emergency vehicles, but not always in the same lane. How do you know all of them were on autopilot? >12 crashes is 12 crashes too many, statistically speaking, but it's also a tiny tiny percentage of the total number of cases. It's also 12 more than any normal attentive human has gotten into, and an average attentive human driver is the absolute floor for self-driving on public roads unless Tesla is going to go to the lengths GM has to ensure the person behind the wheel is remaining attentive. >IMO, it is good that there is federal investigation into this. My main concern is that is seems our government agencies at large are not very good when it comes to properly understanding anything technology related, or making relevant decisions/rulings/etc. on these kinds of things. I think they're good at understanding if Tesla loses complete public confidence in self driving that is a net-loss for humanity and technology, as well as a prime source of political shitstorm when people ask why something wasn't done sooner.
nashashmi 5y ago
The fact that Tesla and Elon have been fighting these autopilot errors in completely the wrong way is the primary reason the feds should be involved. I have been using autopilot in Nissan. If the car were to crash into an ambulance light going off I would not be able to predict it and stop it from happening quickly enough. I just have a whole lot of trust in some basic things the car does. And this is one of those cases where they should have been able to get this right after the first crash but they didn’t and now the feds have to get involved. A lesson on why we can’t have nice things.
- CultMember001 5y ago
  We should not allow the Fed to stifle innovation and technological progress. If we are not willing to take risks, then we will never be able to reach out true potential of becoming a truly advanced space faring civilization and a multi-planetary species. The NHTSA does not have the technological capability to understand advanced AI used in Autopilot and Full Self Driving and they should stop their interference immediately. I have Tesla and I bought the Full Self Driving package. I believe it is far more valuable than the current price. I am willing to pay 30 times more for Full Self Driving.
noslenwerdna 5y ago
Thanks for your honesty. Hopefully the NHTSA is also going to investigate the many other cars with traffic aware cruise control, which have the same problem with stationary vehicles. I was driving my Rav4 with TACC enabled, and it would have crashed into a column of stopped cars if I hadn't noticed in time and intervened.
- ahahahahah 5y ago
  Tesla has something like 1000x greater rate of hitting stationary emergency vehicles. The fact that all the other cars have the same issue with tacc actually points even more strongly to something being wrong with Tesla.
  - kwhitefoot 5y ago
    > 1000x greater rate What's your denominator? Is this per year or per kilometre driven with TACC active?
  - noslenwerdna 5y ago
    citation, please.
tablespoon 5y ago
>> The safety agency informed Tesla weeks ago that it was looking into the spate of crashes in which Tesla vehicles operating on Autopilot failed to detect stopped emergency vehicles with flashing lights. The regulator originally said it was looking into 11 such crashes. A 12th occurred on Saturday, when a Model 3 hit a police cruiser that had stopped behind a car that had broken down on an interstate in Orlando, Fla. > 12 crashes into vehicles with flashing lights on them doesn't sound like great success from Autopilot, gonna be honest. It's really surprising what people will put up with. IIRC, recently "autopilot" had some regressions in the situations it could handle, and Elon Musk tweeted something to the effect of "Yeah, this version sucks since we're retraining all our NNs." Like how did something that half-baked ever make it to production in a safety critical system?

fallingknife 5y ago

> The regulator originally said it was looking into 11 such crashes. A 12th occurred on Saturday But there are 4500 crashes every year, and that's only including ambulances (https://www.ems.gov/pdf/GroundAmbulanceCrashesPresentation.pdf https://www.ems.gov/pdf/GroundAmbulanceCrashesPresentation.p... pg 22). So why is Tesla being investigated for being involved in 0.25% of these crashes when they made up 3% of car sales in 2020? Also, radar ignores stationary objects on any car. Why is only Tesla being investigated when this exact same flaw exists in every other car manufacturer that uses radar based cruise control?

roystonvassey 5y ago
I think the right stat to measure is - of all the Tesla cars out there, how many had crashes, and compare this against the average for other car brands/models.
- SketchySeaBeast 5y ago
  I'd hope that "automated driving" has standards much much higher than the average.
asddubs 5y ago
They're looking specifically into crashes happening on autopilot though, so using that 3% figure assumes people are using the tesla in autopilot mode 100% of the time
okl 5y ago
No offense but that is whataboutism. I think every crash, where an investigation could prevent another crash in the future, should be investigated. Shouldn't Tesla "celebrate their mistakes" (yes, morbid, sorry) and see this as an opportunity to improve their product?
- DennisP 5y ago
  Tesla is already putting a lot of resources into improving Autopilot and FSD. I'm not sure how regulators are going to help that process, and I think there are a couple possible pitfalls: (1) Will they take relative risk into account? If the car sometimes crashes but less often than humans, will that be sufficient, or considered a flaw because it's not perfect and regulators hate looking bad? (2) Will they take progress into account? If the car drives worse than humans but its accident rate drops 10% per year, then after some number of years we'll accumulate fewer total accidents than if we hadn't allowed progress to continue. That's important because progress won't continue without lots of cars on the road using Autopilot and FSD betas.

pomian 5y ago

Maybe the driving data has nothing to do with crash, or to put it another way, the log of the driver's route and actions up to the point of the crash, is the point.

mchusma 5y ago

Hopefully the TSA doesn't try to kill as many people as the FDA has. The big lesson from the pandemic was how poorly these agencies are run, and their complete willingness to see thousands die by implementing restrictions.

Traster 5y ago

Let's ignore the larger point (which is that this is just one of a million possible edge cases where autopilot could systematically fail), I could absolutely see Tesla just bodging in a "If you see emergency vehicle lights flashing disengage" - it'd perfectly reasonable to say that you shouldn't be using autopilot in that situation because you should be paying attention and ready to react to the emergency vehicle.

Cthulhu_ 5y ago
It should warn and turn off in a LOT of situations imo; there was another accident in a construction zone where the car hit a barrier, for example. And there's the shared videos of people literally sleeping or sitting in the back seat of a Tesla on autopilot; the car should detect that and disable autopilot for an X amount of time. Of course, at the same time the car should not play police; if someone is caught doing that, their license should be taken away from them.
- adrr 5y ago
  It should warn and force the driver to confirm to keep autopilot going at set speed. Same way it handles traffic lights and autopilot.
mikejb 5y ago
As much as it would be a solution to Tesla's problem, it would impact Tesla's overall plan and strategy. Automatically disengaging when emergency vehicles are detected is a huge barrier in their selling strategy ("It can drive itself, it's just not allowed yet to for legal reasons, but soon").
duxup 5y ago
I don't think you're wrong policy wise specifically. I do think that this 'auto pilot ... but you have to manage it' system is not workable for most humans. Paying attention when you sorta don't have to ... is just not paying attention for most people. It's a space I don't think we manage well outside of say a real pilot in a plane trained to do his thing (and there's actually a lot of cases with trained airline pilots losing situational awareness and auto pilot does the wrong thing and the pilots don't notice soon enough / auto pilot makes bad choices for a surprisingly long time before anyone notices). I think this might be a space where it is all or nothing auto pilot wise. Ultimately a human problem.

gubnerpoo 5y ago

Biden, when he's not eating ice cream and awake, hates Tesla because they aren't a union shop. This and other things are just a shakedown. You can see by the echo chamber here who's the bot army and who's not. Remember, the primary purpose of Tesla is to save the planet. But I guess the unions trump that. Fuckers.

emptybits 5y ago

I assume (?) Teslas have interior microphones. Maybe not in this case but how easily will cabin interior data be turned over to authorities to paint a picture of what goes on inside and outside a vehicle or nearby vehicles leading up to an accident? It may be easier to just ask for and release everything. In the name of safety, of course.

longcommonname 5y ago

I wonder if the NHTSA could require all self driving cars to use lidar if the safety stats show it is a significant safety improvement.

panick21 5y ago
Then the waste majority of cars simply wouldn't come with lane assist. Deploying lidar on such scale is simply not practical at the moment. Check out how long it took for even simple safety features to make it into all cars.
- tablespoon 5y ago
  >> I wonder if the NHTSA could require all self driving cars to use lidar if the safety stats show it is a significant safety improvement. > Then the waste majority of cars simply wouldn't come with lane assist. Deploying lidar on such scale is simply not practical at the moment. IMHO, lane assist isn't "self driving" any more than cruise control is. I think you get into self driving territory when the car makes lane changes or turns at intersections.

yawaworht1978 5y ago

The absolving argument that it's still the drivers responsibility to intervene has flown out of the window with this request. The federal safety agency is not having any of that excuse anymore. They might find that Tesla did no wrong here, further enabling Tesla to get away with this. But this looks a lot like they want to regulate the feature a bit more. If self regulation via companies and humans does not work, the government will step in. Curious to see the outcome, they haven't ordered the immediate disablement of the feature, they might have a neutral view.

duxup 5y ago
I suspect humans are really bad when they only have to sorta pay attention and it results in just not paying attention a great deal of the time. I'm not sure that's fixable on the human end.
- matt123456789 5y ago
  The question is, how much of that is Tesla’s fault? They’re an EV company, not a psychology consultancy. We don’t hold gun manufacturers or alcohol companies liable when people do stupid things with their products, despite the best efforts of their marketing divisions. Granted “pull trigger get bullet” is much simpler than the Autopilot response.
  - duxup 5y ago
    We ask airplane makers to make adjustments to better indicate problems / auto pilot status when humans don't do the right thing all the time. And that's for professionals who get tons of training. We have all sorts of car / road safety features because humans are bad at driving. We can't reprogram the humans, can change the environment they operate in.
  - SerLava 5y ago
    It's not "doing a stupid thing" though Fundamentally any life form has a slower reaction when they have to notice an issue and start to take motor control. Versus already being in control and simply not causing the issue. They are not just "an EV company" they are making a guidance system for thousands of high speed metal death machines. They BETTER consult some psychologists. This is a really bizarre thing to say.

Dumblydorr 5y ago

Tesla should never have named this Autopilot, and Elon should've remained silent about way overoptimistic promises. Sometimes, "coming soon" is a great answer, vagueness is a powerful tool to help with time demands of clients.

tenpies 5y ago
FTC should have obliterated Tesla from orbit the minute they called cruise control "Autopilot", and the Tesla beta opt-in fund "Full Self Driving", and any time Elon made any claim that the vehicle was going to be self-driving by X date. This would have saved several lives, untold injuries, and millions in damages. It would have been disastrous for Elon's wealth and the market cap of Tesla, but then we could at least pretend there's still rule of law in the US. Thankfully other jurisdictions have actual regulators and are telling Tesla that you cannot use fraudulent names, but who knows how much longer it will take for the US to join the rest of the world in this.
nilstycho 5y ago
This was definitely a poor name in retrospect, but I can see why they chose the name. The current functionality of autopilot is in fact pretty similar to aviation autopilots. Unfortunately most of the public thinks that an autopilot can fly a plane unassisted, so they naturally expect "Autopilot" to drive their car unassisted.

deleted 5y ago

[deleted]

CultMember001 5y ago

Is this only for Autopilot data or also for Full Self Driving data? Hasn't Full Self Driving already passed all approvals and perfectly safe? Why would they try to regulate something that is already perfectly safe? I have never heard of Full Self Driving crashing. I think Tesla should charge more for this advanced tech. I think $5000/month/car is a fairer price.

dqpb 5y ago

I'm curious what the data requirement will be. Is it just the segmented map? Is it the actual model plus all inputs for n-seconds leading up to the crash? Is it all training data that meets some threshold of similarity to the incident? It would be nice if the investigation of these incidents resulted in an open standard for analysis and quality control of self-driving systems.

yawaworht1978 5y ago

Another point is, legal human operator liability or not, the agency has been alerted of 12 such incidents which could never, ever occur unless a human driver is asleep, fully drunk/under the influence of drugs, some sort of fight happened in the car or was distracted from heavy sunlight to the eyes or had to dodge a hazard or animal on the road. Their way of thought is simply that if the software can't deal with what is trivial for humans, maybe a recall or better look is in order. I know, this is not trivial for software, it just looks to not technical people like something that should never happen, and if it does , they loose all the trust in it. Obviously, there are many bright engineers working at Tesla, not sure how this was deployed and how this slipped QA repeatedly. This, the authority asks Tesla, before the authority has to answer to the voters , congress and media shows.

panick21 5y ago

This not a feature that Autopilot or any other of the lane keeping systems have at the moment. If we are going to say that this is an issue then pretty much all lane keeping systems would have to be outlawed. If it is really impossible for people to pay attention without their hands on the wheel at all the time then all lane keeping must be abolished or a standard needs to be established what 'paying attention' means. Crashing into emergency vehicles happens often even without lane keeping system as well, so its a univeral problem when you have big stationary objects standing around on driving lanes. What do people on HN actually want or expect in terms or regulation? Here are some suggestions: - Only allow any kind of lane keeping if a comprehensive testing that they can detect any obstetrical of any kind and have emergency breaking. This of course would simply mean less such system would be deployed as most companies don't have practical scalable solution for this as of yet. - Only allow any kind of lane keeping if a comprehensive driver monitoring system is implemented that makes sure driver is alert. Additionally you could require wheel touches every so often in addition to that, if 'hands free' driving should not be allowed. Tesla is in a good spot here, they have the internal cameras and already have a driver monitoring system yet despite that they have not allowed hands free driving. Tesla new Vision Autopilot or the more extensive FSD stack (that will do highway driving with v10) should solve the stationary object problem. I'm sure that if there was any kind of regulation, Tesla would simply comply and move on. In the worst case if regulation would somehow insist on having things named in a specific way, Tesla would just rename their features. However the point here seems to be non of this is technological problem with Tesla, is simply a situation with no regulation and so every manufacturer picks their own solution. Tesla is the most prominent and most talked about so they are the whipping boy of anybody that wants more regulation.

asddubs 5y ago
>- Only allow any kind of lane keeping if a comprehensive testing that they can detect any obstetrical of any kind and have emergency breaking. This of course would simply mean less such system would be deployed as most companies don't have practical scalable solution for this as of yet. probably this. it's a nice feature but it lulls you into a false sense of security and just doesn't seem to mesh well with human nature. It works well enough most of the time that people start trusting it too much over time

CultMember002 5y ago

We should not allow the Fed to stifle innovation and technological progress. If we are not willing to take risks, then we will never be able to reach out true potential of becoming a truly advanced space faring civilization and a multi-planetary species. The NHTSA does not have the technological capability to understand advanced AI used in Autopilot and Full Self Driving and they should stop their interference immediately. I have a Tesla and I bought the Full Self Driving package. I believe it is far more valuable than the current price. I am willing to pay 30 times more for Full Self Driving.