11 ms·
One more thing to note - anecdotal evidence indicates that Tesla cars did not attempt to center within a lane prior to an OTA update, after which multiple cars
by falcolas 8y ago
One more thing to note - anecdotal evidence indicates that Tesla cars did not attempt to center within a lane prior to an OTA update, after which multiple cars exhibited this "centering" action into gore sections (and thus required manual input to avoid an incident) on video.
To me, that this behavior was added via an update makes it even harder to predict - your car can pass a particular section of road without incident one thousand times, but an OTA update makes that one thousand and first time deadly.
Humans are generally quite poor at responding to unexpected behavior changes such as this.
- txcwpalpha 8y agoAnd this is exactly why all of these articles recently about how "great" it is that Tesla sends out frequent OTA updates are ridiculous. Frequent, unpredictable updates with changelogs that just read "Improvements and bug fixes" is fine when we're talking about a social media app, but is entirely unacceptable when we're talking about the software that controls a 2 ton hunk of metal flying at 70mph with humans inside of it. The saying has been beat to death, but it bears repeating: Tesla is a prime case where the SV mindset of "move fast and break things" has resulted in "move fast and kill people". There's a reason that other vehicle manufacturers don't send out vehicle software updates willy-nilly, and it's not because they're technologically inferior.
- slg 8y agoThis isn't an issue specific to Tesla as all automakers are now making cars that are more and more dependent on software. So what is the right way to handle these updates? You mentioned a clear flaw with OTA updates, but there are also numerous advantages. For example, the recent Tesla brake software issue was fixed with an OTA update. That immediately made cars safer. Toyota had a similar problem a few years ago and did a voluntary recall. That means many of those cars with buggy brake systems were on the road for years after a potential fix was available and were driven for billions of potentially unsafe miles.
- deleted 8y ago[deleted]
- txcwpalpha 8y ago>This isn't an issue specific to Tesla as all automakers are now making cars that are more and more dependent on software. Cars have been dependent on software for a long time (literally decades). This isn't something new. Even combustion engine cars have had software inside of them that controls the operation of the engine, and this software is vigorously tested for safety issues (because most car manufacturers understand a fault with such software could result in someone's death). Tesla seems to be the only major car manufacturer that has a problem with this. >So what is the right way to handle these updates? The way that other vehicle manufacturers (car, airplane, etc) have been doing it for decades is a pretty good way. >You mentioned a clear flaw with OTA updates, but there are also numerous advantages. For example, the recent Tesla brake software issue was fixed with an OTA update. That immediately made cars safer. There is no evidence that said OTA update made Tesla cars any safer. There is evidence that similar OTA updates have made Tesla cars more unsafe. The brake OTA that you mentioned has actually potentially done more harm than good. Tesla owners have been reporting that the same update made unexpected changes to the way their cars handle/accelerate in addition to the change in braking distance. These were forced, unpredictable changes that were introduced without warning. When you're driving a 2 ton vehicle at 70mph, being able to know exactly how your car will react in all situations, including how fast it accelerates, how well it handles, how fast it brakes, and how the autopilot will act is crucial to maintaining safety. Tesla messing with those parameters without warning is a detriment to safety, not an advantage.
- slg 8y agoI didn't say no car has ever relied on software. I said cars are becoming more reliant on software. I don't think that is a controversial statement. I also don't think it is controversial to say that other automakers also occasionally ship buggy code. The Toyota brake issue I mentioned in the previous post is one example. Additionally, the argument that we should continue to handle updates this way simply because we have done it this way for decades is the laziest possible reasoning. It is frankly surprising to see that argument on HN of all places. As for the evidence that OTA updates can make things safer, this is from Consumer Reports: >Consumer Reports now recommends the Tesla Model 3, after our testers found that a recent over-the-air (OTA) update improved the car’s braking distance by almost 20 feet. [1] That update going out immediately OTA is going to save lives compared to if Tesla waited for the cars to be serviced like other manufacturers. I don't think you can legitimately argue against that fact. [1] - https://www.consumerreports.org/car-safety/tesla-model-3-gets-cr-recommendation-after-braking-update/ https://www.consumerreports.org/car-safety/tesla-model-3-get...
- Pfhreak 8y agoThis seems testable. IANAAE (I Am Not An Automotive Engineer), but why can't you run both the new and old code side by side and if the actions they take are materially different investigate further? Like, if in one case the new code would want you to move left, and the old code goes straight, one of those behaviors is probably wrong. If the driver corrects, then the new code is probably correct, but if the driver does not, then the new code is doing something probably incorrect. At the very least, you should be able to get some sort of magnitude/fuzzy understanding of how frequently the new code is disagreeing, and you can figure out where and go check out those conditions.
- jakecopp 8y agoA/B testing self-driving car software to find bugs? Would probably work great, but that is also terrifying! But you're right, if you're really going to do a full roll out, may as well test it on a subsegment first - I'd hate for it to be used as a debugging tool though.
- akira2501 8y ago> but why can't you run both the new and old code side by side and if the actions they take are materially different investigate further? Because there are no closed facilities that you can use to actually perform any meaningful test. You could test "in-situ", but you would need an absolutely _huge_ testing area in order to accurately test and check all the different roadway configurations the vehicle is likely to encounter. You'll probably want more than one pass, some with pedestrians, some without, some in high light and some in low, etc.. It's worth noting that American's drive more than 260 billion miles each _month_. It's just an enormous problem.
- dchichkov 8y agoThis particular case might have been testable "in-situ": [ABS News Reporter] Dan Noyes also spoke and texted with Walter Huang's brother, Will, today. He confirmed Walter was on the way to work at Apple when he died. He also makes a startling claim — that before the crash, Walter complained "seven-to-10 times the car would swivel toward that same exact barrier during auto-pilot. Walter took it into dealership addressing the issue, but they couldn't duplicate it there." It is very believable that the car would swivel toward the same exact barrier on auto-pilot. BTW - I'm running a nonprofit/public dataset project aimed at increasing safety of autonomous vehicles. If anyone here wants to contribute (with suggestions / pull requests / following it on twitter / etc) - you'd be most welcome. Its: https://www.safe-av.org/ https://www.safe-av.org/
- erikpukinskis 8y agoMore accurately: move fast and save some lives and kill people.
- InternetOfStuff 8y ago> And this is exactly why all of these articles recently about how "great" it is that Tesla sends out frequent OTA updates are ridiculous. Frequent, unpredictable updates with changelogs that just read "Improvements and bug fixes" is fine when we're talking about a social media app, but is entirely unacceptable when we're talking about the software that controls a 2 ton hunk of metal I recently got downvoted for that exact line of reasoning. Looks like some people don't like to hear that :-)
- bertil 8y agoI think you got down-voted because you misrepresented what Telsa is actually doing, which is a difficult arbitrage between: - known preventable deaths from, say, not staying in the lane aggressively enough; - possible surprises and subsequent deaths. There is an ethical conundrum (that is quite different from the trolley problem) between a known fix and an unknown possibility. If both are clearly establish and you dismiss the former to have a simplistic take, yes, you would be down-voted because you are making the debate less informed. Without falling to solutionism, in that case, the remaining issue seems rather isolated to a handful of areas that look like lanes and could be either painted properly, or that Telsa cars could train to avoid. The later fix would have to be sent rapidly and could have surprising consequences -- although it seems decreasingly likely. That learning pattern (resolving unintended surprises as they happen decreasingly often) is common in software. This explains why this community prefers (to an extend) Telsa to other manufacturers. Others have preferred the surprise-free and PR-friendly option of not saving the dozens of thousand of lives dying on the road at the moment. There are ethical backgrounds to non-interventionism. As the victim of car violence, I happen to think that their position is unethical. I’m actually at what is considered an extreme position, of being in favour of Telsa (and Waymo) taking more risk than necessary and temporarily increasing the number of accidents on the road because they have a far better track record of learning from those accidents (and the subsequent press coverage) and that would lower the overall number of deaths faster. As it happens, they don’t need to: even with less than half of accidents from their counterparts, they still get spectacular learning rate.
- InternetOfStuff 8y ago
- mmusson 8y agoI tried to convince management that the “reveries” were leading to unpredictable behavior.
- abalone 8y agoTwo years ago there was an Autopark accident that made the news. Tesla blamed the driver -- very effectively.[1] But if you look closely, it likely was due to an unexpected OTA change in behavior combined with poor UI design. In the accident, the driver double-tapped the Park button which activated the Autopark feature, exited the car and walked away. The car proceeded to move forward into an object. The driver claimed he merely put it in park and never activated the auto-park feature. Tesla responded with logs proving he double-tapped and there were audible and visual warnings. Well, I looked more closely. Turns out Tesla pushed an OTA update that added this "double-tap Park to Autopark" shortcut. And it's one bad design decision after another. First, let's note the most obvious design flaw here: The difference between instructing your car to immobilize itself (park) and instructing it to move forward even if the driver was absent (Autopark) was the difference between a single and double tap on the same button. A button that you might be in the habit of tapping a couple times to make sure you've parked. So it's terrible accident-prone design from the start. Second issue is user awareness. Normally Tesla makes you confirm terms before activating new behavior, and technically they did here, but they did it in a horribly confusing way. They buried the mere mention of the Autopark shortcut under the dialog for "Require Continuous Press".[2] So if you went to turn that off -- that's the setting for requiring you to hold down a FOB button during Summon -- and you didn't read that dialog closely, you would not know that you'd also get this handy-dandy double-tap Autopark shortcut. Third is those warnings. Useless. They did not require any kind of confirmation. So if you "hit park" and then quickly exited your vehicle, you might never hear the warning or see it on the screen that, by the way, that shortcut that you didn't know existed just got triggered and is about to make the car start moving forward while you walk away. So I think it's quite plausible that the driver was not at fault here -- or at least that it was a reasonable mistake facilitated by poor design. It's unfortunate that Tesla was able to convince so many with a "data dump" that the driver was clearly at fault. I still recall that poor NYT journalist that Tesla "caught driving in circles"[3] -- while looking for a hard-to-find charger at night. Now I hope we are developing a healthier skepticism to this (now standard) response from Tesla and look more deeply at potential root causes. [1] https://www.theverge.com/2016/5/11/11658226/tesla-model-s-summon-autopilot-crash-letter https://www.theverge.com/2016/5/11/11658226/tesla-model-s-su... [2] https://youtu.be/Cg7V0gnW1Us?t=2m10s https://youtu.be/Cg7V0gnW1Us?t=2m10s [3] https://www.forbes.com/sites/joannmuller/2013/02/14/teslas-gotcha-blog-catches-new-york-times-reporter-driving-in-circles/ https://www.forbes.com/sites/joannmuller/2013/02/14/teslas-g...
- tinus_hn 8y agoThe behavior should change every day. Never changing is impossible, especially with a new product like this.
- Ntrails 8y ago> Humans are generally quite poor at responding to unexpected behavior changes such as this. I know what you mean, once you truly start to trust someone else to do a job you simply stop giving it any attention. It's being handled. So you maybe hover a bit at first, keep an eye, generally interfere. Once that stage is over, you just do other things safe in the knowledge that someone else is competently handling the task. Until it blows up in your face. This level of autopilot legitimately terrifies me. Not because it's bad, but because of the way it will make the humans who are supposed to still be responsible stop paying attention
- candiodari 8y agoYou know, I must say as a European driving in America I've done this on occasion. Combined with drivers not letting you in, it's tempting to follow the gore sections into the (barely visible) barrier. I mean, I slow down when that happens, but otherwise I seem to react pretty much the same as this Tesla car. Why can't the "gore" sections be marked like this ? 1) https://goo.gl/maps/iWEcY7hU4DJ2 https://goo.gl/maps/iWEcY7hU4DJ2 2) https://www.google.com/maps/@52.3314532,4.799684,3a,60y,221.41h,90.09t/data=!3m6!1e1!3m4!1syApvGuYRWHFGyZVmmxRBRQ!2e0!7i13312!8i6656 https://www.google.com/maps/@52.3314532,4.799684,3a,60y,221.... Differences: a) Clearly different road markings between the gore section and normal lanes b) BIG brightly colored and lit sign above the barrier c) the barrier itself has a shock absorber/deflection thing like 10 meters in front of the concrete
- stolsvik 8y ago«Lethal Regression»