33 ms·

Tesla sued in wrongful death lawsuit that alleges Autopilot caused crash

• cmurf 7y ago
  I think it's unacceptable for automation to produce a worse result than a human in the same situation, with the same information. i.e. it's not acceptable for automation to fail danger, it must fail safe, including even if all it can do is give up (disconnects, warning tone, hands control over to the human driver). I think it's reasonable, in the narrow case where primary control is asserted and competency is claimed to be as good or better than a human, to hold automation accountable the same as a human. And in this case, if this driver acted the way autopilot did based on the same available information, we would say the driver committed suicide or was somehow incompetent. I see this as possibly a case of automation committing involuntary manslaughter (unintentional homicide from criminally negligent or reckless conduct).
  • Skunkleton 7y ago
    > ... it's not acceptable for automation to fail danger, it must fail safe... The scary thing is, by definition of what is being automated there can be no fail-safe. The only "safe" is to stop the car. If the car detected it was going to crash, then it wouldn't have had the failure in the first place. By the time the failure might be detected, it is already too late for "safe".
    • emiliobumachar 7y ago
      It's worse than that. Sudden breaking is also dangerous.
  • 1123581321 7y ago
    We don’t expect all kinds of drivers and vehicles to fit the same safety bell curve. You’ve made an assertion, but what legal framework are you are using to treat this particular human-machine interaction differently without introducing a whole new class of liability for humans and traditional manufacturers?
    • kadendogthing 7y ago
      >but what legal framework are you are using to treat this particular human-machine interaction differently without introducing a whole new class of liability for humans and traditional manufacturers? Humans are not robots?
  • stevenjohns 7y ago
    > or was somehow incompetent If they survived, they would have lost their license and been charged with some form of reckless driving. The underlying issue is that manufacturers are beta testing on the road and almost literally playing with peoples lives in the process. Legislation, although would hurt innovation, in this case is appropriate to ensure that the race to the finish doesn’t have a blood-stained legacy.
    • threeseed 7y ago
      Tesla is the only one claiming to have a working solution and is selling it to the general public. Everyone else is far more cautious.
  • modzu 7y ago
    you cant expect the human driver to be ready to take over autopilot in any meaningful way. if the driver is alert enough to the conditions and physically ready at the appropriate controls to do that, then they are already driving. still, despite it not being based on science, but legal reasons, companies pretend that humans can "take over" when alerted to. maybe a lawsuit will change that idiocy.
    • jimmaswell 7y ago
      A few people dying of their own irresponsibility does not warrant ruining it for the rest of us.
      • patejam 7y ago
        There are other people on the road, it's not just the drivers that are impacted by this. Calling it "Autopilot" is what's irresponsible.
        • jimmaswell 7y ago
          Pilots still need to take over sometimes despite the autopilot. The name is fine.
      • protomyth 7y ago
        A lot of people said the exact same thing about lawn darts and a whole host of products and activities, but sadly, society doesn't go that way.
        • modzu 7y ago
          im all for the technology and accepting it will be fallible, just not for putting the blame for its errors onto the human non-driver
  • corodra 7y ago
    Well, this brings to light the concept from the movie I, Robot. Is it a crime when an ai kills someone, unintentional or not, or is it an industrial accident? If it's suppose to replace a human task due to a "level of intelligence", is it still taxed as equipment or as an employee? Is the company (due to equipment failures) or the ai (choosing to do an action on its own) at fault? To be fair, these questions need to be hard lined pretty soon.
    • Bendingo 7y ago
      Don't know why this is downvoted - it's a good statement of the essential issue.
    • mr_toad 7y ago
      > these questions need to be hard lined pretty soon I’d argue that they’re a long way from being an issue. Firstly, current AI are far less ‘intelligent’ than almost any animal. Even a mosquito has more brain power, and we don’t think twice about killing mosquitoes. Secondly, even if an AI had similar intelligence to a human, there is no reason to believe it would be a moral creature, capable of making moral judgments, and being judged as such. Our morality evolved over thousands, probably millions, of years (or if you prefer, it was granted by some divine power). Either way, intelligence and morality aren’t synonymous.
  • lucb1e 7y ago
    > it's unacceptable for automation to produce a worse result than a human in the same situation, with the same information I don't think anyone disagrees on that. The question is: is the software worse than a human driver? Do we have enough data for a statistically significant judgement on that? Is it even autonomous enough to say anything either way, like, if the driver is required to be at attention anyway, can the software be blamed for anything in the first place? Those are the questions, I don't think there is a point saying "software must be good!"
    • jessriedel 7y ago
      >> it's unacceptable for automation to produce a worse result than a human in the same situation, with the same information > I don't think anyone disagrees on that. I disagree on that. If there's an autonomous vehicle that is better than a human in most situations, and worse in a few situations, such that the overall accident/death rate is lower, and there is no reasonable away to identify the rare dangerous situations in time to disable the autopilot, I would want to drive that car and would advise others to do so. In fact, if there was an autonomous vehicle that was almost exactly as safe as a human but slightly more dangerous (say, a 10% higher death/accident rate), I would frequently use it because the large benefits outweigh the minor statistical costs. (Indeed, I use a car at all because of its benefits over walking, busing, or staying at home, despite the higher rate of death.) If other people understood the risks, I would also suggest that they to do likewise.
  • btilly 7y ago
    This sounds good, but is impractical. First, a human who is paying attention makes better decision than a human who is surprised by being given control during a crisis. Therefore if automation sees a crisis, it's standard has to be that it is worse than a surprised human, and not how it compares to an alert driver. Second, the analysis that concludes that we're now in a specific situation where a human would do better is often beyond the powers of automation. For example it can recognize that we are now in a class of cases where, on average, automation does better. But whether it will in any particular case may be beyond its reasoning power.
    • salawat 7y ago
      This is called the Halting Problem. We don't waste time trying to make machines that do that because it's theoretically undecidable. Furthermore, you're then throwing more neural network (that has to be trained improvably across the dataset which we have no proof is the full dataset required to fully enumerate the problem space), to successfully detect the problem areas in the data going into the driving NN for edge cases. Repeat ad absurdum or until most of your self-driving car's energy is consumed trying to solve the Silicon equivalent of Plato's Cave. Or... You just teach people to drive the blessed car, which is what the absolute best a theoretical NN implementation would converge to anyway. The only benefit to a theoretically perfect NN which simulates the average driver being the car doesn't run on a processing system that is adversely effected through the recreational and voluntary (by the processing hardware) application of ethanol to the processing matrix. Well... Thermally speaking, it could end up causing serious problems if someone were dumb enough to build the capability to do that into the car for some daft reason, but we're talking existential failure of computing hardware rather than gradual degradation of functionality like you'd see with a brain, but still I think everyone gets the point and I can stop torturing this poor analogy.
      • btilly 7y ago
        What you say has just enough technical detail to seem entirely plausible to someone who doesn't understand technology. It is also entirely wrong. The problem that I am describing is not the Halting Problem. A theoretically perfect NN is also able to do a lot better than any person could. The ability to have binocular vision in multiple directions, and mix that with radar awareness of the local environment and electronic communication with other cars to be aware of obstacles that are out of sight. This technology can drive more quickly and safely than humans ever could. This sort of thing is admittedly some time off. But it is doable, and it is almost certain to happpen within our lifetimes.
• Jonanin 7y ago
  That news about this case is being spread everywhere irks me a bit, because people aren't considering the actual circumstances of the accident. Excerpt from the article: "According to the family, Mr. Huang was well aware that Autopilot was not perfect and, specifically, he told them it was not reliable in that exact location, yet he nonetheless engaged Autopilot at that location. The crash happened on a clear day with several hundred feet of visibility ahead, which means that the only way for this accident to have occurred is if Mr. Huang was not paying attention to the road, despite the car providing multiple warnings to do so." This deserves a WTF. He understood autopilot makes errors, complained to his wife several times [0] that the car usually makes errors in that exact spot, and yet wasn't paying enough attention on a clear day with ideal driving conditions to commute safely. [0] "Family members say he complained about his Tesla veering into the same barrier at the exact location of the crash and that he brought his car into the dealership several times to report a problem with the autopilot function." from https://sanfrancisco.cbslocal.com/2019/05/01/family-driver-died-tesla-autopilot-crash-files-lawsuit/ https://sanfrancisco.cbslocal.com/2019/05/01/family-driver-d...
  • eloff 7y ago
    Sounds like his smartphone is about as liable as Tesla here. If he hadn't been distracted this shouldn't have happened. He knew his autopilot doesn't handle this section of the road.
    • threeseed 7y ago
      I'm confused here. Has Tesla publicly stated which sections of the road autopilot doesn't work on ?
      • carlivar 7y ago
        The docs say to keep your hands on the wheel and stay alert at all times. Never mind Tesla CEO on national television taking hands off the wheel and not watching the road.
        • 7y ago
• saagarjha 7y ago
  The problem with Tesla's "Autopilot" is that it's marketed heavily as just that: autopilot. It's good enough to work that way 99% of the time, too, which is a recipe for disaster: things that work 99% of the time but have the potential to fail catastrophically often get glossed over by humans because it's hard to keep paying attention to something that rarely fails.
  • carlivar 7y ago
    This is exactly why Waymo and Cruise and such believe autonomy Level 3 or so is utterly dangerous and have intentionally skipped it entirely.
    • HeavenBanned 7y ago
      Do you think Tesla's insistence that LIDAR is useless has a part in this case? To me it's telling that everyone in the ASD game _except_ Tesla is using LIDAR.
      • ovao 7y ago
        Has Tesla actually asserted that LIDAR is “useless”? I know Musk has argued that LIDAR is not necessary, but “not necessary” and “useless” are very different things.
        • studentrob 7y ago
          Let's ask Elon - https://youtu.be/Ucp0TTmvqOE?t=6081 https://youtu.be/Ucp0TTmvqOE?t=6081 "Anyone who relies on LiDAR is doomed. Doomed. It's like having a whole bunch of expensive appendices. One is bad, and now you want to put in a whole bunch of them. That's ridiculous."
          • rcxdude 7y ago
            It's probably a bad idea if your plan is to sell self-driving cars to consumers. Waymo can afford a substantially higher cost premium on their self-driving cars because they are looking at providing a taxi service, not a car that people can buy.
• barrkel 7y ago
  I think it's deeply irresponsible to create a device which can drive in 90 or 95% of scenarios. It's just a way to fool humans into killing themselves.
  • deleted 7y ago
    [deleted]
  • CoolGuySteve 7y ago
    If the system is even a few percent better than humans then it would save thousands of lives when widely deployed. Musk has made this argument many times. Basically don't let perfect be the enemy of better. But if the vendor is legally liable in each case then this software is untenable.
    • deleted 7y ago
      [deleted]
    • tntn 7y ago
      Humans can be 30+% better than "humans" by 1. not using a phone and 2. not being drunk or stoned. A few percent better than "humans" is quite bad relative to a semi-responsible human driver.
      • natch 7y ago
        >A few percent better than “humans” is quite bad Yes it should be (imho) at least an order of magnitude better than humans before we consider it OK to not pay attention. Not a few percentage points better as you say. However, to overcome media drama, it probably will need to be not just one, but more like three or four orders of magnitude safer than humans, because of the “oh look, a Tesla had an accident” effect.
        • londons_explore 7y ago
          Getting more than 50 percent better than humans gets hard, since frequently the accidents are the fault of the other party.
• baoha 7y ago
  While I'm not quite sure about Tesla's responsibility, I do think CA DOT has its part in this tragic accident. Had the attenuator been replaced right after the previous accident, it could have saved the driver's life. Usually I don't complain much about the gov, but just look at the construction mess they've created on 101, it's been like that for more than 4 years!
  • threeseed 7y ago
    So don't blame the company whose claiming that their car can "autopilot" under all conditions. Instead blame the road. Interesting take.
    • goshx 7y ago
      Except the company never claimed that.
      • deleted 7y ago
        [deleted]
    • natch 7y ago
      This is a bizarrely inaccurate mischaracterization of what Tesla has ever said about autopilot.
      • unityByFreedom 7y ago
        Yes, nevermind the video that emphasizes "the driver is doing nothing" on their tesla.com/autopilot page since 2016, or that all vehicles come with hardware for full autonomy. There's nothing bizarre about threeseed's characterization. https://www.tesla.com/autopilot https://www.tesla.com/autopilot
        • ccorda 7y ago
          That is a demo of full self driving, which is a perpetually coming soon feature. You even need to pay for it separately. The actual section on what autopilot can do and what it requires of the driver is pretty clear: > Autopilot advanced safety and convenience features are designed to assist you with the most burdensome parts of driving. Autopilot introduces new features and improves existing functionality to make your Tesla safer and more capable over time. > Your Tesla will match speed to traffic conditions, keep within a lane, automatically change lanes without requiring driver input, transition from one freeway to another, exit the freeway when your destination is near, self-park when near a parking spot and be summoned to and from your garage. > Current Autopilot features require active driver supervision and do not make the vehicle autonomous. As a Model 3 owner, it does do those things quite accurately. Certainly more so than the MobileEye based system on our Volvo, which can't lane keep and stops abruptly. I find it convenient and more relaxing when driving, but I heed their warning (which I also agreed to when enabling autopilot in car) and pay attention, being prepared to take over at all times.
• patejam 7y ago
  Tesla is playing with lives and should stop offering Autopilot. Anything less than L4 autonomous driving is completely reckless. Calling it "Autopilot" when it's an L2 system should be criminal. Yes, it's an extreme stance, but we're going to have a really hard time getting out true autonomous driving when companies are playing around with people's lives. You can say "well they know the risks" but it's not a closed situation. There are others on the road who will also die because of Autopilot mistakes.
  • anonoholic 7y ago
    > Anything less than L4 autonomous driving is completely reckless. "L4 autonomous driving is completely wreck-less"?
  • goshx 7y ago
    We are going to have a really hard time getting out true autonomous driving if they don’t do this. That’s how the system will learn how to deal with real life scenarios.
    • patejam 7y ago
      You're okay training an incomplete self driving machine learning model on the general population?
      • goshx 7y ago
        I actually help it every day during my commute. They are not stupid and this isn’t so unsafe as people who have no actual experience with it claim to be.
        • threeseed 7y ago
          People have died. So it seems pretty unsafe to me.
          • tasubotadas 7y ago
• gooseus 7y ago
  It seems like a (relatively) trivial addition to the Autopilot system would be to allow the drive to tell the car when it has made a mistake and if Autopilot(s) consistently make mistakes in the same area then Autopilot should give a specific warning, or force a disengagement, when it detects that it is approaching that area. I can't imagine that this isn't already a thing or that someone at Tesla hasn't come up with this... am I missing something here?
  • goshx 7y ago
    You are not missing anything. They have that and gave details in their last Autonomy Day event.
  • deckar01 7y ago
    Tesla engineers talked about this in great depth during the recent streaming event. Manual disengagement of Autopilot is the mechanism used to trigger an upload of the sensor data. It is automatically uploaded to Tesla servers and used to train the driving model.
    • rjdagost 7y ago
      A Twitter user who rooted his Tesla claims that autopilot disengagement reports are very small (< 1 KB) and do not contain any actual sensor data, just things like GPS coordinates, speed, heading, etc.: https://twitter.com/greentheonly/status/1096322810694287361 https://twitter.com/greentheonly/status/1096322810694287361 Not the kind of information that you use to train computer vision algorithms. I found his claims to be an interesting read.
      • Maxious 7y ago
        However the rest of the thread explains that Teslas are "trigger"ed to capture imagery data (in rapid succession, almost like a video) in a variety of situations like being close to pedestrians/cyclists not just disengagements.
      • natch 7y ago
• lunulata 7y ago
  Living around silicon valley with all the tesla drivers here, you learn real fast it makes for some of the worst drivers out on the road. Regularly see them running red lights and just generally not paying attention. They crutch too hard on the auto-pilot and dick around on their smart phones, which is probably exactly what this guy was doing. It's bad for tesla drivers and it's bad for drivers around them. Hope Tesla loses this lawsuit just because of that.
  • megaremote 7y ago
    Generally, the more someone pays for their car, the more entitled they are and the more laws they break.
  • beering 7y ago
    Is it that Tesla drivers are actually worse, or that Teslas stick out to you more than other mundane cars and are pretty common around Silicon Valley?
    • lunulata 7y ago
      Good point, it could be they stick out more to me. They are super common in SV. I've personally had teslas run red lights in front of me on two different occasions within last couple months and those particularly stand out in my mind since they put my life in danger mid-intersection. I ride a motorcycle and suspect the Tesla auto pilot is particularly bad at spotting motorcyclists.
• protomyth 7y ago
  So, in a traditional embedded system, discovery can reveal the source code to help determine what went wrong (e.g. divide by zero error in x-ray machine). What are the lawyers going to get when they start looking into the Tesla's autopilot software?
  • nn3 7y ago
    Likely they will get a gigantic sea of trained deep learning weights that noone understands.
    • protomyth 7y ago
      That really isn't going to play well with a judge or jury. I can see the poor programmer up on the stand not being able to say what it meant or how the car made its decisions.
      • salawat 7y ago
        That is, unfortunately, the problem with neural nets in general. They are amazing logical constructs, but there is a fundamental opaqueness to them that absent sufficient neural network mass to convincingly simulate a human, we can't apply the same methods of formal verification and behavioral inference we could for other more specific machine implementations. No one can explain just why certain weights work at certain situations and not others. They just do. Whether someone in the justice system is comfortable effectively legislating from the bench by creating precedent holding companies liable for NN based behavior that there are no hard and fast ways for them to proof test against in the first place is another question however.
        • wbl 7y ago
          Liability is about incentives. Strict liability moves the externalities of unsafe cars onto the manufacturer.
        • protomyth 7y ago
          No one can explain just why certain weights work at certain situations and not others. They just do. I think the problem will come when some lawyer latches on to that as a way of saying the auto maker is putting a product it cannot prove or explain on the road, but still claiming it’s safe.
• CoolGuySteve 7y ago
  If the Tesla had a proximity sensor that slammed on the breaks in the last 2-3 meters, would he have survived? I ask because it's a common luxury car feature and would be easy for Tesla to implement. IIRC this crash and the semitruck crash had no evidence of the autopilot hitting the breaks according to the NTSB. Even if his velocity was only reduced by a fraction, the amount of power involved in the collision would have been reduced by whatever that fraction was. Edit: So it turns out the Model X does have automatic emergency breaking, but the preliminary report says that the Tesla actually increased its speed in the 3 seconds leading up to the crash. Sounds like a major software bug to me. Here's a review of the Model S AES compared to other cars, in particular the Tesla AES trigger can't handle when a lead car moves out of way, which is what the NTSB report says happened: https://www.caranddriver.com/features/a24511826/safety-features-automatic-braking-system-tested-explained/ https://www.caranddriver.com/features/a24511826/safety-featu... Here's the NTSB preliminary report: https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18FH011-preliminary.pdf https://www.ntsb.gov/investigations/AccidentReports/Reports/...
  • LeoPanthera 7y ago
    > If the Tesla had a proximity sensor that slammed on the breaks in the last 2-3 meters, would he have survived? I ask because it's a common luxury car feature and would be easy for Tesla to implement. My car (a Nissan) has this feature, and it comes with a warning that it works best with obstacles that are moving. (ie. other cars) Although it does work with stationary objects, it warns that if you're moving over ~35mph, it may not detect stationary obstacles, as it has trouble differentiating between them and objects on the side of the road.
  • merkaloid 7y ago
    they do have that, but as you might imagine, the emergency brakes are only there to mitigate damage in case of a crash, not to prevent them
    • CoolGuySteve 7y ago
      Yes but in this case it didn't trigger.
• jbritton 7y ago
  I think self driving cars should have both high dynamic range cameras and LIDAR and maybe time of flight cameras. Input from a LIDAR system would be much more likely to detect that barrier, and computer vision via a camera much more likely to be fooled. I think an investigation into why the computer vision system failed to detect a barrier under clear daylight conditions will show the negligence on the part of Tesla. Lane lines are frequently not well marked, and sunlight glare is a difficult problem for cameras. However, you have to be able to detect a concrete barrier, in the worst of conditions. Does Tesla have in place some kind of determination of its lane detection accuracy and then alert the driver that it is turning off auto-pilot when accuracy is low?
  • tekno45 7y ago
    Guidelines definitely say keep your hands on the wheel and be ready to take over. plus it beeps at you to take over.
  • natch 7y ago
    Yes it shuts off with a beep if it can’t handle current conditions. In case you think this kind of abrupt shutoff sounds dangerous, keep in mind this is in the current generation of the system which relies on a human driver being attentive and ready to take over at all times.
    • unityByFreedom 7y ago
      > In case you think this kind of abrupt shutoff sounds dangerous, keep in mind this is in the current generation of the system which relies on a human driver being attentive and ready to take over at all times. What is a driver to do when the system randomly decides to brake? [1] Phantom braking has been an issue for awhile, and has yet to be acknowledged by Tesla. [1] https://www.reddit.com/r/teslamotors/comments/b5yx1o/welp_this_happened_today_while_in_autopilot/ https://www.reddit.com/r/teslamotors/comments/b5yx1o/welp_th...
      • natch 7y ago
        They do acknowledge it. You asked what to do. Apply gentle force to the accelerator pedal. And that video is from a super old software version.
• carlivar 7y ago
  I know the argument is always that autopilot is not intended by Tesla to be abused (never mind the CEO on national television abusing it). However, there's what you tell humans, and then there's pragmatism regarding human nature. We should consider the latter. It's possible to be both academically correct on the warnings/instructions given and also practically wrong on human psychology.
  • deleted 7y ago
    [deleted]
  • Simple_Guy 7y ago
    Your reasoning is that people are stupid and therefore you should manipulate them 'for their own good' because they have no self-agency and self-responsibility. Needless to say, people who think like this make terrible dictators when given the chance.
    • semi-extrinsic 7y ago
      You could can easily apply this logic to say "we should revoke laws that make it illegal to drive without a seat belt", or dozens of other examples. Humans are notorious for thinking "I'm in a real hurry, I'll leave the safety system off because I likely won't get hurt". We know that laws, rules and protocols with enforcement and negative consequences are required in order to make humans act safely at scale.
    • carlivar 7y ago
      It's a good thing I did not propose that I become Supreme Dictator Strawman.
• Proven 7y ago
  It's just one of the many lawsuits against Tesla. It doesn't matter what happens with it. The thing is: Tesla will lose one or more of the many, and the crooked company is already halfway into bankruptcy. So whether they win or lose this particular lawsuit, it's only a matter of time before they fold.
• rayiner 7y ago
  I hear people say “self driving cars don’t have to be perfect, they just need to be safer than humans.” Here is a good example of why that might be harder to achieve than expected. Apparently the car had gotten confused at that exact location repeatedly before. That’s what self driving cars are going to do—if there is something “weird” it’s likely that every self driving car (at least, every one from the same manufacturer) that encounters the weird scenario will run into a problem. That could result in catastrophic failure modes at scale. What would have happened if every car on that road had been an identical Tesla? How many crashes would have happened? How long would it have taken Tesla to issue a fix? How many miles of perfect driving would be required to make up for the cluster of crash events due to that one anomaly?
  • LeoPanthera 7y ago
    To play devil's advocate: Assuming self-driving cars are connected to each other, either directly or via the internet, if one car crashes, or even just makes a minor mistake, the knowledge of how to avoid that can be transmitted to every other car on the planet.
    • mc32 7y ago
      Just as good information can be propagated, so can bad. So this had better be a robust algorithm and not one which can either be induced into something or simply learn “wrong”.
    • valine 7y ago
      Seems unlikely that the knowledge of how to avoid making a mistake could be compiled without human analysis. The technology to do that autonomously doesn’t exist. It would almost require artificial general intelligence, and if we had that self driving would be trivial.
      • bdamm 7y ago
        Automatic feature selection is the area of research you’re looking for.
      • 7y ago
• areoform 7y ago
  I would like to add over here that Tesla does indeed do sensor fusion in their cars. Their Autopilot combines radar and ultrasound with vision to decide where to drive. Commentators bringing up LIDAR are jumping the gun by assuming that this scenario isn't something that these sensors wouldn't have detected in either combination or individually to spot the anomaly. The problem over here is likely to be software due to a bug in their code than a simple lack of additional sensors (LIDAR). And this touches on deeper issues that could have profound ramifications for the autonomous driving industry and the broader industry in general. At least, in my eyes, the big problem with the autonomous car industry isn't the sensor suites they are deploying (or not), but the over-reliance on neural networks. They are black boxes with failure modes that can't be adequately mapped. See: http://www.evolvingai.org/fooling http://www.evolvingai.org/fooling What if the neural net or the system used to detect obstacles didn't see it because the precise configuration of the data fooled it? And if that's the case then what's next? How do we decide when it is okay for safety-critical systems to be opaque? How do we deal with autonomous driving if the conclusion comes out to be a "no" for this case? How should broader society deal with a yes? And who decides all of this in the first place? Possibilities like this scare me far more than the lack of LIDAR because replicating a bug like this would be next to impossible. We don't know what we don't know, and we can't explore and understand the system to suss out what we don't know. Edit: Fleshed out the idea with more questions.
  • antpls 7y ago
    > Possibilities like this scare me far more than the lack of LIDAR because replicating a bug like this would be next to impossible. We don't know what we don't know, and we can't explore and understand the system to suss out what we don't know. I don't think this is true. According to the Tesla live stream a few days ago, the cameras are always gathering data. Tesla cars could have a black box (like planes) , and they could just replay everything that happened up to M minutes before the accident, and retrain neural networks to avoid it later. According to the live, they automatised this process with the "Data Engine". But anyway, this is definitely not an excuse to beta test the system with actual customer's lives, IMHO.
  • ummonk 7y ago
    The sensor technology is a big factor here. You can use LIDAR for perception without needing a neural net. On the other hand with radar + ultrasound + vision you'll need a neural net to try to detect things like obstacles, since the raw data from those sensors doesn't directly tell you whether there is an obstacle in the way.
• Chris_Chambers 7y ago
  Imagine still being stupid enough to buy one of these things. It’s like wanting a car version of the 737 Max, except where the Boeing CEO posts memes and uses an anime avatar on twitter.
• robdachshund 7y ago
  Its almost as if we shouldn't let 4000 pound metal boxes drive themselves
• 11thEarlOfMar 7y ago
  The crux of the issue will be that autonomous capabilities will/have made cars overall safer. Tesla will be able to cite data and circumstances where the safety features saved lives, and likely more times than lives were lost. The problem is that the lives that are lost when the safety features fail are different lives than would have been lost without it at all. The families of those killed in this manner will have their day in court.
  • cududa 7y ago
    I don’t think a juries are known for valuing statistics over gut feeling, and something tells me technophobia will win here.
  • studentrob 7y ago
    > Tesla will be able to cite data and circumstances where the safety features saved lives, and likely more times than lives were lost. It would be great if Tesla would share such data, however their safety report is only a couple sentences [1] and Musk just recently refused to share more safety data because it would allow people to "turn a positive into a negative" [2] [1] https://www.tesla.com/VehicleSafetyReport https://www.tesla.com/VehicleSafetyReport [2] https://www.youtube.com/watch?v=HqvatzjHGyk&t=47m17s https://www.youtube.com/watch?v=HqvatzjHGyk&t=47m17s
• hackerpacker 7y ago
  can we just make a law that the driver is responsible/liable for the car they drive, regardless how they drive it? I mean breaks failing while you are actively driving is one thing, but to completely surrender control, that is a choice.
  • mindfulplay 7y ago
    But that's what Elon promises. He is careful never to publicly say it outright but his PR machinery wants you to believe this car can do things it literally cannot.
    • v0x 7y ago
      I really like a lot of what Tesla and Musk do, but their PR campaigns are absolutely repugnant. Like in this case where they try to make the NTSB out to be the bad guy when in reality they go completely against accepted NTSB standards in an attempt to control the narrative. It's awful.
  • Simple_Guy 7y ago
    If you surrender control to a doctor to perform a surgery, and the doctor decides to stab you in the heart, why aren't you responsible for being stabbed, since you chose to 'completely surrender control'?
• sjg007 7y ago
  Tesla is certainly on the knifes edge... they must be extremely confident
  • pauljurczak 7y ago
    or delusional ...
• rjdagost 7y ago
  Tesla made some very big claims in their recent autonomy day event- basically, they claimed that they are years ahead of competitors, while operating on "hard" mode (no lidar). And yet, a number of participants in the short autonomy day demo rides claimed that the support driver had to disengage the autopilot. Has Tesla provided any evidence which shows evidence that they are in fact far beyond competitors?
  • studentrob 7y ago
    > Has Tesla provided any evidence which shows evidence that they are in fact far beyond competitors? Evidence showing they are ahead in terms of customers? For now, maybe. In terms of safety? No. According to some fans, they sell a product you control. You're allowed to use it anywhere, therefore, they're ahead. According to Tesla, they designed a chip that's best-in-class and they share a small amount of safety data quarterly [1]. Musk will not share any more data [2] because that would allow people to "turn a positive into a negative". [1] https://www.tesla.com/VehicleSafetyReport https://www.tesla.com/VehicleSafetyReport [2] https://www.youtube.com/watch?v=HqvatzjHGyk&t=47m17s https://www.youtube.com/watch?v=HqvatzjHGyk&t=47m17s
    • rjdagost 7y ago
      Thanks for the links. Their "Vehicle Safety Report" is literally 3 sentences! I expected such a report to have much more detailed information in it.
  • tim333 7y ago
    I'm not even sure what evidence could exist to show Tesla is years ahead. Who knows how much Waymo, Cruise et al will advance next year. You can't fully see the future here.
• newnewpdro 7y ago
  Tesla should get taken to the cleaners for this, I hate that these cars w/"autopilot" are operated on the same streets I use. They should focus on making quality, well-performing electric cars. Stop using us all as beta testers for an autonomous future most of us never asked for.
• kvhdude 7y ago
  English is not my first language. I am confused by the blatant use of the term "Auto Pilot". Does it not suggest more automation than is currently feasible? Why not intelligent assist? Why is tesla/musk getting a pass here?
  • Svip 7y ago
    I think a lot of people want to be pedantic, and point out that an aeroplane's autopilot can't do all of it itself, and you therefore still need pilots to manage an airliner. However, that's because a lot of people on HN understands how a plane's autopilot works, but most people do not. They assume it is what they think a plane's autopilot is supposed to be. Since a plane could technically take off, fly and land itself by wire, but only under the best of conditions. At least, that's my theory.
  • xeromal 7y ago
    Autopilot in an airplane is similar. It can do a bit and make the trip easier, but it's not going to dodge a rock that happens to fall from the sky. It's a pretty simple system that really just relies on instrumentation. It's not using any fancy algorithms to make bold maneuvers. I might wager than Tesla's is doing more.
    • mrtksn 7y ago
      Then maybe using the Autopilot or a Tesla vehicle should be subject to rigorous training and repetitive simulator sessions every 6 months to ensure that the drivers are using it correctly and are aware of its current capacity. If it’s like an airplane, should be regulated like an airplane.
    • hyperbovine 7y ago
      Yeah but most people are not aware of that. To them it’s a button you push that flies the plane. This is why the marketing around this term is so deceptive.
  • hyperbovine 7y ago
    No, you are exactly right. “On autopilot” is English vernacular for “without thinking or paying attention”. It completely sends with wrong message, and the lawyers will have a field day with this in court.
• DorothyPhillips 7y ago
  What a terrible situation to be in, and what a loss for the family. My only contention is that wouldn't a person be a little apprehensive to let go of the wheel while on the highway? Maybe autopilot should be limited to city streets, but then I suppose that would defeat half of the purpose of auto-pilot.
• huhtenberg 7y ago
  This person died because of Tesla's cocky marketing, which leads people to believe that "Auto Pilot" is just that and that does very little to discourage this interpretation even though it is life-threatening. In this context them blaming it on the accident victim is a 100% asshole move. I am life-long Musk's fan, but Tesla trying their hardest to weasel out of any responsibility here is incredibly damaging to their reputation. The future will come, trying to accelerate its arrival at all costs is reckless.
  • mikejb 7y ago
    I generally agree with you on the Tesla marketing issues, but it's not entirely as black and white: The driver reported issues at that very location multiple times, and complained about AP not working reliably. The driver definitely was aware of flaws and issues with AP, worse yet, was aware of issues with AP at the location of the crash. Tesla's marketing may have contributed and may generally give a false sense of functionality/security, but Huang already knew it wasn't true. Yet still, at the location he knew AP had problems, he failed to pay attention to the road.
  • v_lisivka 7y ago
    Autopilot is just autopilot. There was lot of plane crashes with autopilot involved. https://www.youtube.com/watch?v=FLrIXptqZxw https://www.youtube.com/watch?v=FLrIXptqZxw
  • moduspol 7y ago
    From the article: > According to the family, Mr. Huang was well aware that Autopilot was not perfect and, specifically, he told them it was not reliable in that exact location, yet he nonetheless engaged Autopilot at that location. How can one possibly claim cocky marketing is the problem in this case?
• sytelus 7y ago
  Tesla's blog post: https://www.tesla.com/en_GB/blog/update-last-week’s-accident https://www.tesla.com/en_GB/blog/update-last-week’s-accident the driver’s hands were not detected on the wheel for six seconds prior to the collision. The driver had about five seconds and 150 meters of unobstructed view of the concrete divider Looks like sensors failed to see concrete divider in nice sunny weather and car slammed in to it at 70mph. Driver was obviously over confident on system's ability to self-drive, probably busy looking at phone and ignored warnings to put his hands on steering. In the US, there is one automotive fatality every 86 million miles across all vehicles from all manufacturers. For Tesla, there is one fatality, including known pedestrian fatalities, every 320 million miles in vehicles equipped with Autopilot hardware. If you are driving a Tesla equipped with Autopilot hardware, you are 3.7 times less likely to be involved in a fatal accident. These stats don't help when you read the guy had two kids who will now grow up fatherless for rest of their lives. Humans killing humans is very different thing than machines killing humans even if the fatality rates are 10X lower. Companies need to aggressively enforce, both hands on steering until self-driving is really really really good.
  • ehnto 7y ago
    I agree with your conclusion, but the compassion angle isn't helpful in discerning whether or not it is truly safer, and every fatality is a human death and can be just as impactul to others. This guy is no different to the 3.7x more people per mile who have died in other vehicles, so where is the concern toward those manufacturers? What are they doing wrong that their fatality rate is so much higher? If turning off automated features in a Tesla would lead to more deaths would that still be the right thing to do? At some point in automations progress this will become the Trolley Problem thought experiment, with one side being "Five people die but it was their own fault" and the other side is "One person dies but a machine did it" and we will have to decide which is morally correct.
    • roenxi 7y ago
      The moral issues will get swamped by practical ones pretty quickly. Driving while checking a phone is illegal in most jurisdictions I'm aware of. Humans driving cars could eventually be seen in a similar light - there is no defense of choosing to drive yourself if a machine that is statistically better than you was available to do the job. It would literally be endangering other road users. And it isn't clear why a rational driver would choose to drive themselves anyway, other options being available. The only blockers are (1) how good are computers really? and (2) how much extra does it cost?
• neotek 7y ago
  That's what happens when you mislead your customers into believing your self-driving technology is far more advanced than it actually is. I hope Tesla loses this case and is forced to change their bullshit marketing before more people lose their lives.
  • aneutron 7y ago
    But they actually didn't. They specifically tell you it is NOT a fully autonomous autopilot and that's it's more of an advanced cruise control. They also specifically tell you to KEEP YOUR EYES ON THE ROAD. If you fail to do so, you're asking for trouble. It is unfortunate the guy haf kids that will now grow up orphans, but judging from the information I've read in the Tesla blog post, it is clear that he was in the wrong here, at least partially, for not paying attention while driving.
    • aduric 7y ago
      So why call it 'autopilot'? Seems like bullshit marketing to me. The first association we make when we hear that term is hands-off, not 'advanced cruise control'. They're specifically using that terminology to sell their products and now they're paying the price.
      • zip1234 7y ago
        In an airplane, autopilot still requires the pilot to pay attention.
        • dsabanin 7y ago
          Most people are not pilots to know that.
        • slavik81 7y ago
          In an emergency, an airplane pilot often has time to pull out the operating handbook, flip through to the emergency checklist for their particular problem, and follow the instructions. Even for time-critical emergencies, it's recommended to read through the checklist afterwards to ensure you didn't forget anything [1]. By contrast, it would be very unsafe to be reading the Tesla owner's manual while driving. The level of attention required is much higher. [1]: http://www.tc.gc.ca/eng/civilaviation/publications/tp11575-ex12-73.htm http://www.tc.gc.ca/eng/civilaviation/publications/tp11575-e...
• CriticalCathed 7y ago
  So many people here have hardons for tesla hate. The guy fucked up, bad. Tesla is not at fault here. >"According to the family, Mr. Huang was well aware that Autopilot was not perfect and, specifically, he told them it was not reliable in that exact location, yet he nonetheless engaged Autopilot at that location. The crash happened on a clear day with several hundred feet of visibility ahead, which means that the only way for this accident to have occurred is if Mr. Huang was not paying attention to the road, despite the car providing multiple warnings to do so."
• _pmf_ 7y ago
  How do you feel as a Tesla owner knowing that after your death, Tesla will publicly post that it was your fault, based on data from your vehicle?
• robinduckett 7y ago
  Do they not have Rumble strips in the US? https://en.wikipedia.org/wiki/Rumble_strip https://en.wikipedia.org/wiki/Rumble_strip I have been in situations in my youth where I was commuting or travelling tired, and nothing alerts you like the loud rumbling sound, and I'm sure the autopilot could detect it and bring the car to a stop if the driver hasn't been alerted by the in car warning system or the rumble paint...
  • will_pseudonym 7y ago
    We do have them, though they're not required by law and are not always present.
• helloindia 7y ago
  A more nuanced dissection of whose fault it was or could be[0]. "Huang was apparently fooled many times by Autopilot. In fact, he reportedly experienced the exact same system failure that led to his fatal crash at the same location on at least seven occasions." ........... "Huang knew that Autopilot could not be relied on in the circumstances where he was commuting along the 101 freeway in Mountainview, California. Yet he persisted in both using the system and ignoring the alerts that the system apparently gave him to put his hands on the wheel and take full control." .......... "Elon Musk and Tesla should be held to account for the way they have rolled out and promoted Autopilot. But users like Walter Huang are probably not the poster children for this accountability." [0]https://www.forbes.com/sites/samabuelsamid/2019/05/01/the-problem-with-blaming-tesla-for-walter-huangs-death/#59a9c59a5c88 https://www.forbes.com/sites/samabuelsamid/2019/05/01/the-pr...
  • radcon 7y ago
    > Yet he persisted in both using the system and ignoring the alerts that the system apparently gave him to put his hands on the wheel and take full control. "Hands not detected" does NOT mean "hands not on wheel". Tesla relies on driver input (torque) to "detect" hands, which means you could have both hands on the wheel but sill get the warning to place your hands on the wheel. It seems like most people are assuming it's some kind of magical capacitive system that can detect the slightest hint of contact, which couldn't be further from the truth.
    • jsight 7y ago
      I agree, but I'm also not sure that it is a particularly important distinction. He was repeatedly warned that attention is required with this system and yet clearly was not paying attention at the time of the crash.
      • radcon 7y ago
        > clearly was not paying attention at the time of the crash. This is the fundamental point that I'm disagreeing with though. As far as I know, Tesla has a very crude system for determining driver attentiveness. They rely ONLY on steering wheel torque, which is an extremely unreliable indicator. They don't do any sort of head or eye tracking. So when Tesla claims a driver wasn't paying attention, all it really means is "we're not sure if their hands were on the wheel". My car uses the same method, and I constantly get warnings to place my hands on the wheel even when they're both already on the wheel. It's possible to have your hands on the wheel without applying any torque, and that's where this system fails.
• jordache 7y ago
  Why does this TC article need to mention he was an Apple engineer? Just trying to fill space?
• intrasight 7y ago
  "Move fast and break things" So it has to be for rapid technological progress to be made.