11 ms·

Felony charges are first in a fatal crash involving Autopilot

57 points by tiahura 5y ago

MBCook 5y ago
I think the suit against Tesla will be the more interesting of the two but I guess there’s no news on that yet.
- Kon-Peki 5y ago
  It definitely has the potential. This quote is in the article: > Lopez’s family, in court documents, alleges that the car “suddenly and unintentionally accelerated to an excessive, unsafe and uncontrollable speed.” Nieves-Lopez’s family further asserts that Riad was an unsafe driver, with multiple moving infractions on his record, and couldn’t handle the high-performance Tesla. Are they suggesting that Tesla should have known and therefore should not have allowed the vehicle to be on autopilot or to drive at such high speeds? That's a HUGE claim. Tesla collects a lot of data, and they analyze the data. I don't think that this is disputed, but have they now accepted liability for allowing the car to operate at full capability with the knowledge that he's a bad driver? That would be a major blow, and would reverberate throughout the entire industry. You are very right - this could be really interesting.
  - MBCook 5y ago
    Whatever happens it could end up being the major precedent over assisted services/self driving; depending on how it’s structured.
petee 5y ago
Not clear is why it took so long for charges? Crash in Dec 29, 2019, charges in October 2021, almost two years...is this Tesla holding up data?
- killingtime74 5y ago
  I think it’s the prosecution gathering evidence and deciding on whether to charge and what with. Justice system is slow
- Tuna-Fish 5y ago
  There is nothing uncommon about this. There are many jurisdictions in the USA where charges for most crimes other than very serious stuff like murder are usually raised only just before the statute of limitations runs out.
- plorkyeran 5y ago
  The "speedy" part of the sixth amendment is interpreted very loosely.
  - Sohcahtoa82 5y ago
    The "speedy" part was to prevent the government from locking people up indefinitely while awaiting a trial that was never going to happen.
marstall 5y ago
will be fascinating to follow this one and see how responsibility for the death is apportioned. I can see the argument for indicting the driver. He could be seen as negligent for choosing to unleash this car, known to be buggy, on a public street, leading to two deaths. His counterargument is obvious too, that the manufacturer assured him of its safety, and its therefore the manufacturer's fault. Which then begs the question - if it is Tesla's fault, does that mean manslaughter charges will be brought against a corporation? What would that even mean?
- deleted 5y ago
  [deleted]
- quickthrowman 5y ago
  IANAL, but corporations have been successfully charged and convicted of felonies in the US: https://en.m.wikipedia.org/wiki/List_of_companies_convicted_of_felony_offenses_in_the_United_States https://en.m.wikipedia.org/wiki/List_of_companies_convicted_...
- bryanlarsen 5y ago
  At the time, Autopilot did not stop at red lights. It was (and mostly still is) a lane-following cruise control. How could it be Tesla's fault?
- Johnny555 5y ago
  Tesla does not promote FSD as being autonomous, the driver is going to have a hard time blaming Tesla: https://www.tesla.com/support/full-self-driving-subscriptions https://www.tesla.com/support/full-self-driving-subscription... Note: These features are designed to become more capable over time; however the currently enabled features do not make the vehicle autonomous. The currently enabled features require a fully attentive driver, who has their hands on the wheel and is prepared to take over at any moment.
  - modzu 5y ago
    if the car automatically “suddenly and unintentionally accelerated to an excessive, unsafe and uncontrollable speed", even an attentive driver could find themselves unable to prevent the crash. it strikes me as more akin to a car with faulty brakes. such a car would not be certified as fit for the road. perhaps self driving should not be either.
alfalfasprout 5y ago
I'm probably in the minority in the HN audience that thinks this... but we are a long way off from fully self driving cars that can be used anywhere across the US. Be it Waymo, Tesla, or Cruise the problem is that these systems have way too many edge cases. For example, what happens when the sun is "blinding" you and prevents identification of the traffic light? What happens when there's construction and a construction worker gives confusing instructions to follow? This isn't unique to Tesla-- I've seen waymo cars nearly force a cyclist into a car door in broad daylight. In Tesla's case they've also been particularly aggressive with releasing self-driving tech to the end user without much stipulation on how safe it really is. I wonder how many more easily preventable deaths will have to follow before people crack down on such lax rollout of autonomous vehicle tech.
- giraffe_lady 5y ago
  I don't think the self-driving technology is good enough, agreed. But the legal and cultural practices they develop to fit these into already existing frameworks and expectations is also technology. I think they're likely to have enough expertise at these other skills to enable a national rollout of self-driving cars even if they aren't safe enough. And after all, we have a lot of practice ignoring road deaths and a lot of legal precedent placing that burden anywhere but on the car manufacturers. I don't think a handful of preventable deaths will create any real obstacles. Like I said we already have a lot of those. Plus as usual the people most likely to be hurt are going to be non-car users in pedestrian-hostile places, which are also harms we don't really prioritize addressing. I also think, and someone is very likely to bring it up here, the "WELL BY THAT LOGIC" argument does a lot of rhetorical favors for the self driving cars. Any argument you can make about self-driving cars being dangerous is also likely an argument about cars being dangerous. Which is true, but if you acknowledge it you get discredited as a universal car hater. Anyway there are a lot of good reasons not to allow automated cars on public roads right now. Unfortunately I don't think any of those arguments will stop them.
- ModernMech 5y ago
  > What happens when there's construction and a construction worker gives confusing instructions to follow? Indeed and I believe this is one of the primary failings of the nascent self-driving car industry; driving is seen as an engineering challenge, rather than an engineering and social challenge. When viewed solely as an engineering challenge, the task is daunting enough, but maybe tractable if you squint and target only a subset of driving conditions. But add in the social component of driving, which is largely being ignored by this new industry, and it's really impossible with today's technology.
megaman821 5y ago
I don't know if it is purposeful or the name is genuinely confusing, but Autopilot is basically adaptive cruise control and lane assist. Updates that happened after this crash took place always stopping at stop signs and red lights for properly equipped cars. FSD or Full Self Driving is the thing that drives the car all by itself. If you are not in the beta, it can only be activated on highways. So basically this guy plowed a red light and cruise control and killed someone.
- paxys 5y ago
  Some agency really needs to enforce consistent naming for all these features. Vague marketing terms like "autopilot" and "self-driving" cause enough confusion to be outright dangerous. A car manufacturer can't advertise ABS, airbags, tire pressure monitoring, traction control etc. and then say oh we define these terms differently. Autopilot should be no different.
  - dheera 5y ago
    But if it was named something different the the stock would crash. :-/
  - criddell 5y ago
    Is Tesla's definition of autopilot way different than the definition used in aircraft?
    - avalys 5y ago
      Does the ordinary consumer have any idea what an aircraft autopilot actually does?
    - gamblor956 5y ago
      Yes. An autopilot in a commercial passenger aircraft is fully capable of controlling the plane after takeoff, landing, and taxiing up to the destination gate. Or in other words, after takeoff, it is fully capable of controlling the plane without any input from the pilot. So, when Tesla advertises their cruise control as "autopilot" they are overstating the abilities of autopilot mode if they put in the fine print that a human actually is responsible for controlling the vehicle and that autopilot is just a glorified cruise control.
JCM9 5y ago
Fully self driving cars are far further off in the future than most in the sector would publicly admit. In controlled environments there’s a lot of potential today and things like self driving trams at airports are nothing new. But self driving in the “real world” mixed with humans and all the crazy one-off things that happen in such scenarios is not something that will happen at scale anytime soon. There are still far too many edge cases to work out and for all the amazing things that “AI and ML” can do, handling edge cases is something this technology is actually quite terrible at relative to humans.
- paxys 5y ago
  I'll go a step further and say they aren't just further off but impossible to achieve until someone can reasonably prove otherwise. We simply aren't going to see cars without a steering wheel and manual controls (i.e. Level 5 automation) for general purpose use on public streets in our lifetime.
  - somewhereoutth 5y ago
    Yep. I would argue that driving is AI complete - in the sense that only true Artificial General Intelligence can do it, and so do everything else to. Perhaps this is because it requires cooperating with other General Intelligences (i.e. us) in an open world.
    - smorgusofborg 5y ago
      I'm still mystified as to why no country has switched over to ultra-light rail. The costs of asphalt projects seem to dwarf the cost of doing anything reasonable and leave AI as an exercise for the over confident.
  - kobalsky 5y ago
    https://www.youtube.com/watch?v=tBJ0GvsQeak https://www.youtube.com/watch?v=tBJ0GvsQeak can't you get a ride in phoenix like today with waymo? or is not open to the public? I mean I understand that their AI is not perfect but it doesn't seem like a pipe dream
jlv2 5y ago
There was no "fully self driving" in play with this accident. The driver has the responsibility to control the car. He let the car drive through a red light. He's wholly at fault. (for background: I've driven tens of thousands of miles using Autopilot in both a Model S and Model X, on both local roads and highways)
- vanusa 5y ago
  As long at Tesla continues to willfully and fraudulently push a product called "Full Self Driving" -- onto a customer base they know to be largely naive and unsophisticated -- they share responsibility for whatever death and carnage inevitably results from this cynical and manipulative course of action.
  - JPKab 5y ago
    I'm always fascinated by this urge to push individual lapses and judgment onto large and wealthy entities. I own a Tesla Model 3 and I have chosen not to purchase the full self-driving package nor do I intend to rent it. I don't think the technology is worth it at this point to me personally. When you own the vehicle, it is very clear that the autopilot mode is simply a lane assist system. They have messages flashing all over the screen stating that it will not stop at stop signs or red lights. It's not like it's hidden from anybody and instead is blatantly displayed with no way to stop it from displaying every time it is activated. This driver was abusing a system that is meant to reduce fatigue. He should bear the consequences. Those of us who enjoy the reduction fatigue when driving from Denver to Omaha like I've had to do recently, should not have the feature disabled because of his irresponsibility.
    - vanusa 5y ago
      Do you think the product name "Full Self Driving" has anything to do with reality?
      - dml2135 5y ago
        Tesla bears some responsibility, but it's also on an individual to sniff out bullshit when necessary, and not be gullible to the point of committing negligent homicide. When you operate heavy machinery, it's your responsibility to read the owner's manual, and not just listen to the marketing.
verytrivial 5y ago
It is expert defence against charges like this that will undo the whole nearly-but-not-quite autonomous driving industry. And rightly so. I am stunned the industry thinks effectively an [OK] click through is going to save them here. (And before anyone quotes million of miles per fatality stats, this is a legal question on a single case, generalities don't cleanly apply. Edit: I think Tesla and co. will need to show that it is reasonable to assume a driver can quickly notice and respond to a hazard particularly after being lulled into passivity for what could be hours by the drive system. Good luck.)
- rich_sasha 5y ago
  What are some analogous legal outcomes where companies got users to agree to take liability for the company’s product outcomes and it failed in court? In aviation, I’m pretty sure that so long as you can disable an autopilot, anything that happens is the human’s fault. Guns - nothing to do with the makers and everything to do with the shooter. I can’t think of any examples where it worked.
  - paulryanrogers 5y ago
    Most guns aren't self aiming and self firing.
  - verytrivial 5y ago
    Well, in aviation, they do go to great lengths to let the pilot know which mode they're flying in and transition into and out of that mode has it's own protocol, training and feedback. Indeed the whole surepticious engagement of the MCAS system on the 787-MAX aircraft, and in particular not including updates for pilots in training to deal with what to do when that system fails -- i.e. awareness -- is probably going to end a few careers at Boeing. That seems to be an example very close to the Tesla situation. The AF447 crash in 2009 has a spookily similar pathology -- confusion about automated flight control modes. Both incidents were awareness of automation issues, one with it being on, one with it being off (more or less). Tesla is banking on a beep and/or input control shake covering them for mode change awareness for a completely novice, untrained and uncertified driver. Is that a reasonably assumption? Edit: There were other (tragic) factors in the AF447 crash, but in that case, if they'd re-enabled autopilot and let go of the controls, they would have very likely survived.
sschueller 5y ago
Elon and his sales team should be charged. They are selling this FUD. You can't sell something and then blame the user that they should have read the fine print.
- universenz 5y ago
  Are you sure you meant to use “FUD” in this sentence? It doesn’t make sense. “Elon and his sales team (“they”) are selling this fear uncertainty and doubt”.
alistairSH 5y ago
Does Tesla Autopilot normally stop at traffic signals? If not, seems pretty cut and dry - the driver is at fault. If Autopilot does stop at signals, then it's a bit more muddied to me.
- bryanlarsen 5y ago
  When the accident happened, it didn't. That was added in 2020.
  - dogma1138 5y ago
    Didn’t Gen 1 “autopilot” had this? Mobileye has had traffic light and even speed limit sign detection for its cruise control for years..
- wlesieutre 5y ago
  It does not, "autopilot" is what Tesla calls adaptive cruise control and lanekeeping assist. Attempting to obey stop signs and traffic lights is part of "full self driving." https://www.tesla.com/support/autopilot https://www.tesla.com/support/autopilot Only defense I can imagine is that he was misled by Tesla's "autopilot" naming and and thought the car would drive itself. Outside of that the story is basically "driver puts on cruise control, blows through red light without braking, kills two people." EDIT: since the crash was in 2019, here's an older version of that page http://web.archive.org/web/20190906125836/https://www.tesla.com/support/autopilot http://web.archive.org/web/20190906125836/https://www.tesla.... > Autopilot is an advanced driver assistance system that enhances safety and convenience behind the wheel. When used properly, Autopilot reduces your overall workload as a driver. 8 external cameras, a radar, 12 ultrasonic sensors and a powerful onboard computer provide an additional layer of safety to guide you on your journey. Cars built between September 2014 and October 2016 include one camera and less-powerful radar and ultrasonic sensors. > Autopilot is intended for use with a fully attentive driver, who has their hands on the wheel and is prepared to take over at any time. While Autopilot is designed to become more capable over time, in its current form, it is not a self-driving system. There are five levels of automation and Autopilot is currently classified as a Level 2 automated system according to SAE J3016, which is endorsed by the National Highway Traffic Safety Administration. I can't say that 100% for certain since Tesla's features come and go with software updates, and the above excerpt doesn't get very specific on capabilities. But I don't believe it knew how to stop for stoplights at the time.
olliej 5y ago
The result of "autopilot" is entirely predictable. The NTSB, NHTSA, CSB, and pretty much every other safety board have repeatedly found that any system that requires someone (anyone) pay complete attention, while also not actually doing anything, will fail in the second part. It is wholly unreasonable to expect a person "in control" of a partially self driving car to have instantaneous response, it isn't even reasonable to expect 90% attention to what is going on around the car. Add to that Tesla's insistence on intentionally mislabeling things - autopilot has a clear implication, especially given their demos of it, the "full self driving" mode is clearly outright false labeling - and I think Tesla does share liability. The current model of "self driving" with the expectation the a driver will be able to immediately respond to anything the driving system misses, is inherently unsafe.
- pxmpxm 5y ago
  There are countless HCI studies showing the problem that you mentioned - any interface that requires full and immediate context 1/10000 times while being hands off otherwise will fail in real world use. This issue is so pronounced in mission critical systems that the problem is intentionally inverted, where the machine intervenes in the 1/10k scenarios and humans are in control the rest of the time - think flight control laws in aero or auto scram in nuclear engineering etc.
kornhole 5y ago
When I read the headline, I thought the operators of the FSD tech were being charged: Tesla or Uber. To literally slam the brakes on this, hold the companies accountable rather than individuals who believed that the products would work as advertised. If the supposed FSD still requires an individual to fully man the vehicle, what is the point of having it? It seems like bloatware surveilling you with little benefit to the user similar to what modern phones do.
- varelse 5y ago
  To give your car a devil tempting you to do something stupid like let down your guard and trust it to do the right thing which it will usually do right up to the point it doesn't?
- azinman2 5y ago
  Exactly, what is the point? It’s also clearly misnamed. Level 5 or bust.
- dml2135 5y ago
  Unfortunately, I think that holding the driver responsible is more legally feasible route to take. If you are successful with that, consumer confidence in these features will drop to 0 and the companies will be forced to put up or shut up with their advertising claims.
  - deleted 5y ago
    [deleted]
  - aiisjustanif 5y ago
    That’s ridiculous. Ruin multiple lives for a product that failed. But now that I say that. Was it Auto-pilot which legal is cruise control? Or was it Full-Self Driving and the car made a decision? Legally it’s a matter of man vs machine of it is FSD. Is man wrong if there was even the slightest attempt to correct the machine’s decision after the fact? If it is in fact legal to allow said machine to act while supervised. It could also be a matter of arguing current training and regulation for FSD, is regulation of drivers sufficient for what now is a new mode of private transportation? Should autonomous vehicles be allowed to be owned yet, do they really fit under the same umbrella as normal license? Or should they be special like a CDL or MJ licenses?
datavirtue 5y ago
"I hate this car. It is so complicated and it doesn't even take gas!"
bryan_w 5y ago
Was there a title edit or something? Why are people talking about FSD? This seems to be about autopilot
- lupire 5y ago
  On Tesla, commenters don't read before commenting.
boublepop 5y ago
> The misuse of Autopilot, which can control steering, speed and braking, has occurred on numerous occasions It feels wrong to call it misuse when you’re literally using it the way Tesla has previously, is currently, and will continue to market it.
gorpy7 5y ago
I have autopilot, fsd and now fsd beta. i’ve read all the comments up till now. most of the opinions seem to come from a good place but not experience. before i say “ ask me anything” let me start out by saying, within 20 seconds of activating any of these features, you need to shake the wheel to prove you’re there. there are plenty of clues before that that indicate this is not a hands off affair, but this alone is where you know it’s not going to be a ‘set it and forget it’ . ok, ask me anything.
deleted 5y ago
[deleted]

57 points by tiahura 5y ago

MBCook 5y ago

I think the suit against Tesla will be the more interesting of the two but I guess there’s no news on that yet.

Kon-Peki 5y ago
It definitely has the potential. This quote is in the article: > Lopez’s family, in court documents, alleges that the car “suddenly and unintentionally accelerated to an excessive, unsafe and uncontrollable speed.” Nieves-Lopez’s family further asserts that Riad was an unsafe driver, with multiple moving infractions on his record, and couldn’t handle the high-performance Tesla. Are they suggesting that Tesla should have known and therefore should not have allowed the vehicle to be on autopilot or to drive at such high speeds? That's a HUGE claim. Tesla collects a lot of data, and they analyze the data. I don't think that this is disputed, but have they now accepted liability for allowing the car to operate at full capability with the knowledge that he's a bad driver? That would be a major blow, and would reverberate throughout the entire industry. You are very right - this could be really interesting.
- MBCook 5y ago
  Whatever happens it could end up being the major precedent over assisted services/self driving; depending on how it’s structured.

petee 5y ago

Not clear is why it took so long for charges? Crash in Dec 29, 2019, charges in October 2021, almost two years...is this Tesla holding up data?

killingtime74 5y ago
I think it’s the prosecution gathering evidence and deciding on whether to charge and what with. Justice system is slow
Tuna-Fish 5y ago
There is nothing uncommon about this. There are many jurisdictions in the USA where charges for most crimes other than very serious stuff like murder are usually raised only just before the statute of limitations runs out.
plorkyeran 5y ago
The "speedy" part of the sixth amendment is interpreted very loosely.
- Sohcahtoa82 5y ago
  The "speedy" part was to prevent the government from locking people up indefinitely while awaiting a trial that was never going to happen.

marstall 5y ago

will be fascinating to follow this one and see how responsibility for the death is apportioned. I can see the argument for indicting the driver. He could be seen as negligent for choosing to unleash this car, known to be buggy, on a public street, leading to two deaths. His counterargument is obvious too, that the manufacturer assured him of its safety, and its therefore the manufacturer's fault. Which then begs the question - if it is Tesla's fault, does that mean manslaughter charges will be brought against a corporation? What would that even mean?

deleted 5y ago
[deleted]
quickthrowman 5y ago
IANAL, but corporations have been successfully charged and convicted of felonies in the US: https://en.m.wikipedia.org/wiki/List_of_companies_convicted_of_felony_offenses_in_the_United_States https://en.m.wikipedia.org/wiki/List_of_companies_convicted_...
bryanlarsen 5y ago
At the time, Autopilot did not stop at red lights. It was (and mostly still is) a lane-following cruise control. How could it be Tesla's fault?
Johnny555 5y ago
Tesla does not promote FSD as being autonomous, the driver is going to have a hard time blaming Tesla: https://www.tesla.com/support/full-self-driving-subscriptions https://www.tesla.com/support/full-self-driving-subscription... Note: These features are designed to become more capable over time; however the currently enabled features do not make the vehicle autonomous. The currently enabled features require a fully attentive driver, who has their hands on the wheel and is prepared to take over at any moment.
- modzu 5y ago
  if the car automatically “suddenly and unintentionally accelerated to an excessive, unsafe and uncontrollable speed", even an attentive driver could find themselves unable to prevent the crash. it strikes me as more akin to a car with faulty brakes. such a car would not be certified as fit for the road. perhaps self driving should not be either.

alfalfasprout 5y ago

I'm probably in the minority in the HN audience that thinks this... but we are a long way off from fully self driving cars that can be used anywhere across the US. Be it Waymo, Tesla, or Cruise the problem is that these systems have way too many edge cases. For example, what happens when the sun is "blinding" you and prevents identification of the traffic light? What happens when there's construction and a construction worker gives confusing instructions to follow? This isn't unique to Tesla-- I've seen waymo cars nearly force a cyclist into a car door in broad daylight. In Tesla's case they've also been particularly aggressive with releasing self-driving tech to the end user without much stipulation on how safe it really is. I wonder how many more easily preventable deaths will have to follow before people crack down on such lax rollout of autonomous vehicle tech.

giraffe_lady 5y ago
I don't think the self-driving technology is good enough, agreed. But the legal and cultural practices they develop to fit these into already existing frameworks and expectations is also technology. I think they're likely to have enough expertise at these other skills to enable a national rollout of self-driving cars even if they aren't safe enough. And after all, we have a lot of practice ignoring road deaths and a lot of legal precedent placing that burden anywhere but on the car manufacturers. I don't think a handful of preventable deaths will create any real obstacles. Like I said we already have a lot of those. Plus as usual the people most likely to be hurt are going to be non-car users in pedestrian-hostile places, which are also harms we don't really prioritize addressing. I also think, and someone is very likely to bring it up here, the "WELL BY THAT LOGIC" argument does a lot of rhetorical favors for the self driving cars. Any argument you can make about self-driving cars being dangerous is also likely an argument about cars being dangerous. Which is true, but if you acknowledge it you get discredited as a universal car hater. Anyway there are a lot of good reasons not to allow automated cars on public roads right now. Unfortunately I don't think any of those arguments will stop them.
ModernMech 5y ago
> What happens when there's construction and a construction worker gives confusing instructions to follow? Indeed and I believe this is one of the primary failings of the nascent self-driving car industry; driving is seen as an engineering challenge, rather than an engineering and social challenge. When viewed solely as an engineering challenge, the task is daunting enough, but maybe tractable if you squint and target only a subset of driving conditions. But add in the social component of driving, which is largely being ignored by this new industry, and it's really impossible with today's technology.

megaman821 5y ago

I don't know if it is purposeful or the name is genuinely confusing, but Autopilot is basically adaptive cruise control and lane assist. Updates that happened after this crash took place always stopping at stop signs and red lights for properly equipped cars. FSD or Full Self Driving is the thing that drives the car all by itself. If you are not in the beta, it can only be activated on highways. So basically this guy plowed a red light and cruise control and killed someone.

paxys 5y ago
Some agency really needs to enforce consistent naming for all these features. Vague marketing terms like "autopilot" and "self-driving" cause enough confusion to be outright dangerous. A car manufacturer can't advertise ABS, airbags, tire pressure monitoring, traction control etc. and then say oh we define these terms differently. Autopilot should be no different.
- dheera 5y ago
  But if it was named something different the the stock would crash. :-/
- criddell 5y ago
  Is Tesla's definition of autopilot way different than the definition used in aircraft?
  - avalys 5y ago
    Does the ordinary consumer have any idea what an aircraft autopilot actually does?
  - gamblor956 5y ago
    Yes. An autopilot in a commercial passenger aircraft is fully capable of controlling the plane after takeoff, landing, and taxiing up to the destination gate. Or in other words, after takeoff, it is fully capable of controlling the plane without any input from the pilot. So, when Tesla advertises their cruise control as "autopilot" they are overstating the abilities of autopilot mode if they put in the fine print that a human actually is responsible for controlling the vehicle and that autopilot is just a glorified cruise control.

JCM9 5y ago

Fully self driving cars are far further off in the future than most in the sector would publicly admit. In controlled environments there’s a lot of potential today and things like self driving trams at airports are nothing new. But self driving in the “real world” mixed with humans and all the crazy one-off things that happen in such scenarios is not something that will happen at scale anytime soon. There are still far too many edge cases to work out and for all the amazing things that “AI and ML” can do, handling edge cases is something this technology is actually quite terrible at relative to humans.

paxys 5y ago
I'll go a step further and say they aren't just further off but impossible to achieve until someone can reasonably prove otherwise. We simply aren't going to see cars without a steering wheel and manual controls (i.e. Level 5 automation) for general purpose use on public streets in our lifetime.
- somewhereoutth 5y ago
  Yep. I would argue that driving is AI complete - in the sense that only true Artificial General Intelligence can do it, and so do everything else to. Perhaps this is because it requires cooperating with other General Intelligences (i.e. us) in an open world.
  - smorgusofborg 5y ago
    I'm still mystified as to why no country has switched over to ultra-light rail. The costs of asphalt projects seem to dwarf the cost of doing anything reasonable and leave AI as an exercise for the over confident.
- kobalsky 5y ago
  https://www.youtube.com/watch?v=tBJ0GvsQeak https://www.youtube.com/watch?v=tBJ0GvsQeak can't you get a ride in phoenix like today with waymo? or is not open to the public? I mean I understand that their AI is not perfect but it doesn't seem like a pipe dream

jlv2 5y ago

There was no "fully self driving" in play with this accident. The driver has the responsibility to control the car. He let the car drive through a red light. He's wholly at fault. (for background: I've driven tens of thousands of miles using Autopilot in both a Model S and Model X, on both local roads and highways)

vanusa 5y ago
As long at Tesla continues to willfully and fraudulently push a product called "Full Self Driving" -- onto a customer base they know to be largely naive and unsophisticated -- they share responsibility for whatever death and carnage inevitably results from this cynical and manipulative course of action.
- JPKab 5y ago
  I'm always fascinated by this urge to push individual lapses and judgment onto large and wealthy entities. I own a Tesla Model 3 and I have chosen not to purchase the full self-driving package nor do I intend to rent it. I don't think the technology is worth it at this point to me personally. When you own the vehicle, it is very clear that the autopilot mode is simply a lane assist system. They have messages flashing all over the screen stating that it will not stop at stop signs or red lights. It's not like it's hidden from anybody and instead is blatantly displayed with no way to stop it from displaying every time it is activated. This driver was abusing a system that is meant to reduce fatigue. He should bear the consequences. Those of us who enjoy the reduction fatigue when driving from Denver to Omaha like I've had to do recently, should not have the feature disabled because of his irresponsibility.
  - vanusa 5y ago
    Do you think the product name "Full Self Driving" has anything to do with reality?
    - dml2135 5y ago
      Tesla bears some responsibility, but it's also on an individual to sniff out bullshit when necessary, and not be gullible to the point of committing negligent homicide. When you operate heavy machinery, it's your responsibility to read the owner's manual, and not just listen to the marketing.

verytrivial 5y ago

It is expert defence against charges like this that will undo the whole nearly-but-not-quite autonomous driving industry. And rightly so. I am stunned the industry thinks effectively an [OK] click through is going to save them here. (And before anyone quotes million of miles per fatality stats, this is a legal question on a single case, generalities don't cleanly apply. Edit: I think Tesla and co. will need to show that it is reasonable to assume a driver can quickly notice and respond to a hazard particularly after being lulled into passivity for what could be hours by the drive system. Good luck.)

rich_sasha 5y ago
What are some analogous legal outcomes where companies got users to agree to take liability for the company’s product outcomes and it failed in court? In aviation, I’m pretty sure that so long as you can disable an autopilot, anything that happens is the human’s fault. Guns - nothing to do with the makers and everything to do with the shooter. I can’t think of any examples where it worked.
- paulryanrogers 5y ago
  Most guns aren't self aiming and self firing.
- verytrivial 5y ago
  Well, in aviation, they do go to great lengths to let the pilot know which mode they're flying in and transition into and out of that mode has it's own protocol, training and feedback. Indeed the whole surepticious engagement of the MCAS system on the 787-MAX aircraft, and in particular not including updates for pilots in training to deal with what to do when that system fails -- i.e. awareness -- is probably going to end a few careers at Boeing. That seems to be an example very close to the Tesla situation. The AF447 crash in 2009 has a spookily similar pathology -- confusion about automated flight control modes. Both incidents were awareness of automation issues, one with it being on, one with it being off (more or less). Tesla is banking on a beep and/or input control shake covering them for mode change awareness for a completely novice, untrained and uncertified driver. Is that a reasonably assumption? Edit: There were other (tragic) factors in the AF447 crash, but in that case, if they'd re-enabled autopilot and let go of the controls, they would have very likely survived.

sschueller 5y ago

Elon and his sales team should be charged. They are selling this FUD. You can't sell something and then blame the user that they should have read the fine print.

universenz 5y ago
Are you sure you meant to use “FUD” in this sentence? It doesn’t make sense. “Elon and his sales team (“they”) are selling this fear uncertainty and doubt”.

alistairSH 5y ago

Does Tesla Autopilot normally stop at traffic signals? If not, seems pretty cut and dry - the driver is at fault. If Autopilot does stop at signals, then it's a bit more muddied to me.

bryanlarsen 5y ago
When the accident happened, it didn't. That was added in 2020.
- dogma1138 5y ago
  Didn’t Gen 1 “autopilot” had this? Mobileye has had traffic light and even speed limit sign detection for its cruise control for years..
wlesieutre 5y ago
It does not, "autopilot" is what Tesla calls adaptive cruise control and lanekeeping assist. Attempting to obey stop signs and traffic lights is part of "full self driving." https://www.tesla.com/support/autopilot https://www.tesla.com/support/autopilot Only defense I can imagine is that he was misled by Tesla's "autopilot" naming and and thought the car would drive itself. Outside of that the story is basically "driver puts on cruise control, blows through red light without braking, kills two people." EDIT: since the crash was in 2019, here's an older version of that page http://web.archive.org/web/20190906125836/https://www.tesla.com/support/autopilot http://web.archive.org/web/20190906125836/https://www.tesla.... > Autopilot is an advanced driver assistance system that enhances safety and convenience behind the wheel. When used properly, Autopilot reduces your overall workload as a driver. 8 external cameras, a radar, 12 ultrasonic sensors and a powerful onboard computer provide an additional layer of safety to guide you on your journey. Cars built between September 2014 and October 2016 include one camera and less-powerful radar and ultrasonic sensors. > Autopilot is intended for use with a fully attentive driver, who has their hands on the wheel and is prepared to take over at any time. While Autopilot is designed to become more capable over time, in its current form, it is not a self-driving system. There are five levels of automation and Autopilot is currently classified as a Level 2 automated system according to SAE J3016, which is endorsed by the National Highway Traffic Safety Administration. I can't say that 100% for certain since Tesla's features come and go with software updates, and the above excerpt doesn't get very specific on capabilities. But I don't believe it knew how to stop for stoplights at the time.

olliej 5y ago

The result of "autopilot" is entirely predictable. The NTSB, NHTSA, CSB, and pretty much every other safety board have repeatedly found that any system that requires someone (anyone) pay complete attention, while also not actually doing anything, will fail in the second part. It is wholly unreasonable to expect a person "in control" of a partially self driving car to have instantaneous response, it isn't even reasonable to expect 90% attention to what is going on around the car. Add to that Tesla's insistence on intentionally mislabeling things - autopilot has a clear implication, especially given their demos of it, the "full self driving" mode is clearly outright false labeling - and I think Tesla does share liability. The current model of "self driving" with the expectation the a driver will be able to immediately respond to anything the driving system misses, is inherently unsafe.

pxmpxm 5y ago
There are countless HCI studies showing the problem that you mentioned - any interface that requires full and immediate context 1/10000 times while being hands off otherwise will fail in real world use. This issue is so pronounced in mission critical systems that the problem is intentionally inverted, where the machine intervenes in the 1/10k scenarios and humans are in control the rest of the time - think flight control laws in aero or auto scram in nuclear engineering etc.

kornhole 5y ago

When I read the headline, I thought the operators of the FSD tech were being charged: Tesla or Uber. To literally slam the brakes on this, hold the companies accountable rather than individuals who believed that the products would work as advertised. If the supposed FSD still requires an individual to fully man the vehicle, what is the point of having it? It seems like bloatware surveilling you with little benefit to the user similar to what modern phones do.

varelse 5y ago
To give your car a devil tempting you to do something stupid like let down your guard and trust it to do the right thing which it will usually do right up to the point it doesn't?
azinman2 5y ago
Exactly, what is the point? It’s also clearly misnamed. Level 5 or bust.
dml2135 5y ago
Unfortunately, I think that holding the driver responsible is more legally feasible route to take. If you are successful with that, consumer confidence in these features will drop to 0 and the companies will be forced to put up or shut up with their advertising claims.
- deleted 5y ago
  [deleted]
- aiisjustanif 5y ago
  That’s ridiculous. Ruin multiple lives for a product that failed. But now that I say that. Was it Auto-pilot which legal is cruise control? Or was it Full-Self Driving and the car made a decision? Legally it’s a matter of man vs machine of it is FSD. Is man wrong if there was even the slightest attempt to correct the machine’s decision after the fact? If it is in fact legal to allow said machine to act while supervised. It could also be a matter of arguing current training and regulation for FSD, is regulation of drivers sufficient for what now is a new mode of private transportation? Should autonomous vehicles be allowed to be owned yet, do they really fit under the same umbrella as normal license? Or should they be special like a CDL or MJ licenses?

datavirtue 5y ago

"I hate this car. It is so complicated and it doesn't even take gas!"

bryan_w 5y ago

Was there a title edit or something? Why are people talking about FSD? This seems to be about autopilot

lupire 5y ago
On Tesla, commenters don't read before commenting.

boublepop 5y ago

> The misuse of Autopilot, which can control steering, speed and braking, has occurred on numerous occasions It feels wrong to call it misuse when you’re literally using it the way Tesla has previously, is currently, and will continue to market it.

gorpy7 5y ago

I have autopilot, fsd and now fsd beta. i’ve read all the comments up till now. most of the opinions seem to come from a good place but not experience. before i say “ ask me anything” let me start out by saying, within 20 seconds of activating any of these features, you need to shake the wheel to prove you’re there. there are plenty of clues before that that indicate this is not a hands off affair, but this alone is where you know it’s not going to be a ‘set it and forget it’ . ok, ask me anything.

deleted 5y ago

[deleted]