24 ms·
Another Tesla on autopilot steers towards a barrier
- tjwds 9y agoThis comment from OP (beastpilot) is pretty frightening: > Yep, works for 6 months with zero issues. Then one Friday night you get an update. Everything works that weekend, and on your way to work on Monday. Then, 18 minutes into your commute home, it drives straight at a barrier at 60 MPH. > It's important to remember that it's not like you got the update 5 minutes before this happened. Even worse, you may not know you got an update if you are in a multi-driver household and the other driver installed the update. Very glad the driver had 100% of their attention on the road at that moment.
- draw_down 9y agoThe future!
- ams6110 9y ago> Very glad the driver had 100% of their attention on the road at that moment. Yes, and 100% is critical. That required pretty quick and smooth reactions. The car started to follow the right-hand lane, then suddenly re-centered on the barrier. The driver had about a second to see that and correct. That's just the sort of situation where even a slightly inattentive driver might panic and overcorrect, which could cause a spinout if the roads were wet or icy or cause a crash with parallel traffic.
- edf13 9y agoSo your basically paying (A lot) of cash to beta test their software for them... at your own (Very personal) risk!
- bjl 9y agoNot to mention the fact that, if they're software screws up and kills you, they'll rush to to Twitter in an attempt to smear you using personal data and deceptive statements.
- jacquesm 9y agoAnd endanger others who have decided to opt-out of the beta test. Keep in mind that in any accident all bets are off with respect to the traffic around you, even if you ram into a stationary barrier you could easily rebound into other traffic.
- deegles 9y agoAlso, if something goes wrong it's your fault anyways because you're supposed to keep 100% of your attention on the road.
- jacquesm 9y agoThat's the best kind of small print. The kind that disclaims all liability no matter what. I really don't get why we accept this stuff. It's akin to driving without a license. After all, as long as a Tesla 'autopilot' can't take a driving test it has no business controlling cars.
- lallysingh 9y agoI don't know why people use this thing. The cars are nice, and justify their purchase on their own, without this autopilot feature. But I don't think you can do good autopilot without lidar. People shouldn't use it.
- edf13 9y agoI really don't think the cars are that nice... this is a side from the main issue here but the build quality is poor and the design is bland - both internally and externally (That massive big iPad like screen - yuck!).
- raisedbyninjas 9y agoThe big Youtuber that rebuilds salvage Teslas made the point that almost every Tesla owning Youtuber has a video showing their car getting hauled away on a tow-truck. One of the possible reasons that Tesla service dept. doesn't provide maintenance history on salvage cars is because the amount of service would reflect poorly. It doesn't seem uncommon for a model S with 80,000 miles to have several door handle replacements, at least one whole drive unit replacement, battery contactor replacement, an infotainment system replacement, in addition to all the recall/redesigned components that are replaced before failure. I still think Tesla's are quite nice, but a bullet-proof Honda they are not.
- macawfish 9y agoNot just your own risk... There are other people on the road who share that risk. It makes me angry at Elon Musk himself. First he didn't sterilize that stupid thing he sent into interplanetary space. Now he's putting peoples lives at risk.
- ams6110 9y agoWell there's plenty of unsterilized stuff in space. The early space missions just dumped the bodily waste of the astronauts overboard.
- macawfish 9y agoYeah, but it's coming back to Earth, where it came from. This is an object that could land on Mars or another planet. Great measures have been taken in the past to ensure that other planets aren't contaminated before we've had a chance to understand their existing biology. Elon Musk is the kid who comes and knocks over some other kids' block tower for his own amusement. https://en.wikipedia.org/wiki/Planetary_protection https://en.wikipedia.org/wiki/Planetary_protection
- Someone1234 9y ago> First he didn't sterilize that stupid thing Huh? Isn't it better that way, we might spread [simple] life to other planets. I'm completely serious, I don't understand what the concern is?
- macawfish 9y agoThere may already be life on other planets. Whether or not it's simple is a short-sighted human judgement. Look at the big picture. We risk denying these planets the ability to evolve in isolation. That is a decision that cannot be reversed. Do we really want to do that? Maybe so, but it ought to be a conversation. Great measures have been taken to reduce the risk of contaminating other planets with Earth based lifeforms. Then, this belligerent guy comes along and disregards all that.
- microtherion 9y ago> The driver had about a second to see that and correct. And if there had been a crash, Tesla probably would have said that the driver had an unobstructed view of the barrier for several seconds before impact (conveniently omitting at what point AP started to swerve).
- deleted 9y ago[deleted]
- MertsA 9y agoAnd don't forget pointing out that the driver forgot to use their blinker a week before the crash so clearly they just aren't a responsible driver.
- swearwolves 9y agoFrom the OP's comments on Reddit - he said that it happens about 90% of the time on that stretch and that he has more footage of the same exact incident. He was likely prepared for it, which kinda makes it even scarier in a way. An inattentive driver would have totally botched this.
- makomk 9y agoElsewhere from the comments: "Yes the lane width detection and centering was introduced in 10.4. Our X does this now as well and while it's a welcome intoroduction in most cases every one in a while in this instance or when HOV lanes split it is unwelcome." So basically, if I'm understanding this right they introduced a new feature that positions your car a little more consistently in the lane at the small cost of sometimes steering you at full speed head-on into a barrier. Remember Tesla's first press release on the crash and how it mentioned "Tesla owners have driven this same stretch of highway with Autopilot engaged roughly 85,000 times"? I imagine the number that have driven it successfully in that lane since that update was rolled out sometime in mid-March is rather smaller...
- jacquesm 9y agoAnd that's exactly what makes Tesla PR so disingenuous, they know better than anybody what updates they did and how often their updated software passed that point and yet they will happily cite numbers they know to be wrong. So, now regarding that previous crash: did that driver (or should I say occupant) get lulled into a false sense of security because he'd been through there many times in the past and it worked until that update happened and then it suddenly didn't?
- slivym 9y agoI've got to take issue with you there. Tesla Engineering knows better than anybody what updates they did and how often they update etc. Tesla PR knows nothing about what updates the engineering team did. At least some people in Tesla PR probably don't even know the cars update their software regularly. It's bad practice for them to speak out of turn, but I can absolutely see the PR team not having a good grasp of what really indicates safety and their job is to just put out the best numbers possible.
- jacquesm 9y agoI'm sorry but Tesla PR == Tesla. If they don't have a good grasp on this they should STFU until they do. That would make Tesla even worse in my book. Their job is not to put out the best numbers possible, their job is to inform. Most likely they were more worried about the effect of their statement on their stock price than they were worried about public safety. If they do put out numbers (such as 85K trips past that point) then it is clear that they have access to engineering data with a very high resolution, it would be very bad if they only used that access to search for numbers that bolster their image.
- swiley 9y agoAnd this is why if I ever get a self driving car I want: 1) root 2) control over updates 3) everything to be completely open
- comboy 9y agoYou do not want root on your self driving car. Or to put it differently, I don't want to be driving on the same road as you with your rooted self driving car. You can be great sysadmin/coder, tesla guys may be too, but both of you changing random stuff without any communication with each other.. I've seen enough servers.
- walrus01 9y agoI am trying to imagine the typical arch Linux enthusiasts modifying the lane detection software on their cars and shudder at the thought.
- Asooka 9y agoThis is a problem that will very quickly fix itself, as morbid as it sounds...
- thrillgore 9y agoIt's one thing to remove the warning stickers from a gas-fire oven, its another when you're commanding several thousand pounds of unintentional killing machine. Keep your rooting to your phones.
- therealidiot 9y agoAll I want in a modern car is to have nothing smart in it that I'd _need_ (or, want) to root...
- oldgradstudent 9y agoEverything to be completely open is really important, but I don't want you to have root or control over update if you are using your car on a public road.
- ataturk 9y agoI just don't understand the willingness people have to put their lives in the hands of unproven technology like this. I mean, I don't even trust regular old cruise control in my cars and I keep my finger on the "cancel" button whenever I use it during long highway drives.
- prewett 9y agoAfter working in large codebases I discovered that there are all sorts of assumptions in the code, and that a small change in one function can break an assumption used a long ways elsewhere. I came to the conclusion that any change basically invalidates all your tests. An update could introduce any possible problem that wasn't there before, so now all your experience with the system as a driver is actually reset to zero.
- Barrin92 9y agoThis is just about what I expected it to look like when software engineering safety standards are let loose on the physical world.
- macawfish 9y agoPeople aren't going to tolerate this kind of stuff. These aren't cosmetic or aesthetic flaws. This is an area that deserves transparency. I'm out on the road too, and I don't get to make "consumer choices" for other people I share the road with. People are putting their market power into companies that lack the basic integrity to build these technologies with safety and transparency as the first priority, who evidently see peoples' lives as necessary sacrifices. Please remember that we are talking about vehicles speeding down the road at 70+ MPH.
- Cthulhu_ 9y agoNobody's forcing autopilot on anyone. I think instead, tesla will / should opt to disable autopilot earlier in non-ideal conditions like this - that way they won't be held liable either. ATM it really feels like they're being put on trial here.
- maxerickson 9y agoIt's not any different than the situation with bad human drivers, but anyone on the same road is exposed to these systems.
- jerrycruncher 9y agoIt is different, though. You can't push an update to stop humans from endangering themselves and/or others. Tesla, however, is able to do exactly that with their cars. That they haven't, when their technology has been proven to be fatally dangerous, is unconscionable.
- Piskvorrr 9y agoAnd conversely, you can't push an update that causes humans to start endangering themselves. (Unless Tesla has some magic way of preventing regressions, which doesn't seem to be the case)
- bjl 9y ago
- ProblemFactory 9y agoIf Tesla autopilot relies on good quality road markings, then it's not yet usable in 80% of the world.
- GoToRO 9y agoIt's not usable anywhere. What if I draw a fake line on the road? will it follow it? what if you have wet paint and another vehicle smudges it diagonally to the lane?
- aerovistae 9y agoExactly. This is what I’ve been saying. It is so dangerous. I love Tesla but I think autopilot is currently a death trap.
- donkeyd 9y ago> I think autopilot is currently a death trap. I think this is a bit of an exaggeration. As long as the driver keeps paying attention and uses it as a driving aid, not a driver replacement, everything is fine. It's the moment that people start relying on it doing something that it wasn't built for, that the problems arise.
- MadcapJake 9y ago> I think this is a bit of an exaggeration. As long as the driver keeps paying attention and uses it as a driving aid, not a driver replacement, everything is fine. It's the moment that people start relying on it doing something that it wasn't built for, that the problems arise. I think this is a bit too optimistic. People will start relying on it to be an autopilot. I think most people see this as the desired result: a car that can drive itself. Do you really think anyone wants "a car that will drive itself while you are paying full attention to every detail"? Otherwise, what is a driver really gaining from this? They're still expected to pay just as much attention (if not more) and I'd bet it's even more boring than regular driving (no interaction from the driver means it's like the world's most boring VR movie) Humans are not machines, we love to find the lazy/easy way and we love to do things rather than stare at the road, eventually people will grow complacent (hopefully not before the tech is up-to-snuff).
- walrus01 9y agoScary as hell for me since I go through this exact location on a regular basis. The location is the Seattle i-5 northbound express lanes, where the express lanes end and merge into the main freeway at Northgate.
- Y_Y 9y agoTo be fair it steers _toward_ the barrier. No barriers were harmed. Does Tesla run some sort of regression suite on this stuff? Can they get a copy of the sensor data from when this happened so they can reproduce those conditions as part of their test suite?
- Applejinx 9y agoYou don't need the sensor data, it's obvious. The lane markers widened, and 'straight' is into the (not seen yet) barrier. If the barrier end had a mockup of the ass-end of a car stuck on it (about three times as wide) the Tesla would freak out, panic brake, and still think it was in the center of a single widening lane. It's not mysterious what the car was thinking. If the overall shape of the road wasn't veering slightly to the right, the car would probably not have chosen to swerve to the left into a barrier, but the car sees the lane widening equally in both directions. Simple as that.
- Y_Y 9y agoObvious? Not to the car! My point is that gathering failing cases like this could be a a simple way to gain confidence in a self-driving system, either from the perspective of internal development at Tesla, or even for regulation. Imagine if before you were allowed to push an update to road-going cars you had to show that it doesn't crash in any of the simulations of difficult conditions that the Road Safety Authority has prepared. I'd be in favour of such a thing at least. It's not proof, but it doesn't give any false positives.
- taneq 9y agoWhat does this say about the quality of the automatic emergency braking, if it can't detect a substantial metal object directly in front of the car?
- jdmichal 9y agoDepends on how close it is and what it is. Have to weigh the energy of the impact against the pitch introduced by braking. You need the bumper to take the hit, so that it and the crumple zones can do their job. However, for a wall or highway barrier, it's probably almost always worthwhile to shed as much energy as possible. You're going to pitch yourself under the barrier where the bumper won't take the hit, like you might when colliding with another vehicle.
- taneq 9y ago> Have to weigh the energy of the impact against the pitch introduced by braking. You need the bumper to take the hit, so that it and the crumple zones can do their job. Eh? The vast majority of real frontal impacts will be under heavy braking. If a few degrees of dive under brakes is enough to compromise the crumple zones' effectiveness then your car is bad (and I don't believe any Teslas are bad in this way.)
- TheSpiceIsLife 9y agoI believe the parent is talking about small overlap crash? Small Overlap tests: this is where only a small portion of the car's structure strikes an object such as a pole or a tree, or if a car were to clip another car. This is the most demanding test because it loads the most force onto the structure of the car at any given speed. These are usually conducted at 15-20% of the front vehicle structure.[1] Modern cars are surprisingly good at small over crash tests, see this video someone linked on HN a few days ago https://www.youtube.com/watch?v=DHlj8-JcWa8 https://www.youtube.com/watch?v=DHlj8-JcWa8 1. https://en.wikipedia.org/wiki/Crash_test https://en.wikipedia.org/wiki/Crash_test
- jdmichal 9y agoI mean, this advice is pretty firmly held for hitting things like animals. Albeit for other reasons -- it helps prevent them from going over the hood and towards the windshield. Generally the pitch won't matter that much in vehicle-to-vehicle collisions. Only when the pitch would angle the front bumper completely under the other vehicle, which I admit isn't much of a concern. And not really a concern at all in vehicle-to-barrier collisions. However, if we're talking about idealized automated driving systems, I hold by my original assertion. I would expect such a system to be able to correctly analyze when releasing the brake right before collision to reduce pitch would be beneficial.
- darkerside 9y agoI wonder if Tesla has a way to visualize how disruptive a change to Autopilot. They've supposedly got all this driving data lying around. Would they be able to apply the updated algorithms to that data and quantify how many decisions would be affected, and by how much? i.e. 30% of past decisions would have been changed with the new algorithm, 5% changed by an angle of over 2 degrees. On a similar note, shouldn't customers be getting release notes for these updates? I let my customers know when we're updating a web page. Tesla should surely be informing them when cars are going to start lane centering.
- donkeyd 9y agoHow do you define a 'decision' in something like autopilot though? Also, if I remember correctly, only a couple of images per second are sent to Tesla. If there's a fraction of a second where it may decide to run into something, it may not actually appear in test results.
- maxerickson 9y agoOne way is to extract the drive path for some given sensor input. Then you can classify the drive path as safe, marginal, collision, severe collision, etc. Changes from safe to safe aren't interesting. Changes in the wrong direction are obvious issues.
- walrus01 9y agoI really don't think they could do this effectively unless each car also uploads 20GB per day of 60fps high quality video, which would collapse the cellular networks they use.
- makomk 9y agoI don't think they have that much data that's useful for this purpose. They may have billions of miles of some kinds of data, but they only upload occasional samples of the camera video that would be required to test a feature of this sort. I can't find the exact link now, but someone reverse-engineered it and it was something on the order of a few seconds a few times an hour.
- 9y ago
- fairpx 9y agoAutopilot tech in general should be open and shared among ccompanies and universities. The more brains we can put on this the better imho. If these things happen too often, Im afraid regulators will start making laws that prevent the use of this tech, which I believe has the potential to really make life better for all... one day.
- maxerickson 9y agoI think (at least in the US), the relationship the automakers have with regulators is such that if the automakers think they can make money by selling a feature, they will be able to get the regulators to come up with a way to allow it. Not necessarily one as simple as OTA updates whenever the manufacturer feels like it. As far as regulations preventing use of the tech, if level 2/3 systems end up causing more crashes than human drivers, we shouldn't allow use of them.
- quest88 9y agoI would guess people would just stop using it
- fma 9y agoI have always wondered why each company is developing their own software and algorithms... How will NTSB certify the software? Or will they? I mean, they certify car safety features now through crash tests, roll over tests, etc... Why is Tesla, or anyone manufacturer, allowed to modify vehicle safety features through OTA updates without recertification? I realize it adds more regulation...and I'm sure there can be a middle ground. But I foresee a scenario where an OTA update is pushed and the next morning there are car crashes everywhere.
- Cthulhu_ 9y agoPatents and the ability to earn money by licensing out technology. That, or hoping they are the best and get more customers through it - I think that's currently the case for Tesla, selling the only commercially available semi-autonomously driving car ATM.
- mv4 9y agoMusk wanted to regulate AI to combat existential threat; in reality, it's not super intelligent self-aware AI, but buggy/shitty AI that's trying to kill people.
- walrus01 9y agoWe will be killed by AI developers and the dunning kreuger effect, not armies of Skynet T-800s.
- lallysingh 9y agoThat's a Greek Tragedy-styled scifi story waiting to be written.
- douglaswlance 9y agoAI will bleed us slowly without alerting us to its presence.
- dwighttk 9y agoI think we are better than that
- danepowell 9y agoEveryone's afraid that AI will become too smart and take over the world. They should be afraid that it's too dumb and already has.
- pipio21 9y agoOne of the great things about self driving cars is that we are going to start "debugging Roads". From my point of view all accidents until this time could be attributed to the civil engineer that build the road as much as the driver of the car. You make a pedestrian pass between roads made only for aesthetic purposes so you put a sign forbidding the pass and call it done. But people of course use it and get killed. You put concrete barriers between roads but make it so people could crash frontally against it, something you won't find in any European high speed road without deflectors that will make vehicles not crash against concrete. Self driving will give us scientific evidence of what creates accidents like black boxes did with airplanes. Thanks to that we know that what looks like insignificant details like the color and placement of buttons, turns out to be essential.
- jacquesm 9y ago> One of the great things about self driving cars is that we are going to start "debugging Roads". We've been debugging roads since Roman times. This is mostly about debugging software, and even more about crappy software development processes. After all, if your update is not monotonically improving things you have a real problem if your product is mission critical and lives are at stake.
- kazen44 9y agoalso, roads are far harder to debug and fix then software
- jacquesm 9y agoI would not be so sure of that. Self driving in all conditions is hard. The thing that bugs me is that these '90%' solutions are released on the unsuspecting public without some serious plain language about the software capabilities and what could be expected. Marketing should not trump safety, especially not the safety of people not buying the product.
- YouKnowBetter 9y agoSuprising observation about the debugging of roads. Who knows we might even learn that certain States / Countries do better in this respect and the desing aswell as the signaling might benevit from this. Thank you.
- return0 9y agoAV driving seems to be ripe for disruption
- influx 9y agoThis is such a gross techie statement. No it’s not. Human life is on the line, this is actual engineering. This isn’t writing some stupid social app on MongoDB written in JavaScript over the weekend. Google has been at this for years, and with far better sensors, and even they aren’t there yet.
- bennettfeely 9y agoAny 16 year old can already "disrupt" the industry just by getting a license and driving. Humans are and will continue to be far more capable, rational, and economical drivers for the foreseeable future. Sounds similar to tech bros who want to radically "disrupt" carbon sequestration. Here's an actually proven idea: plant a tree.
- return0 9y agomaybe it was not obvious , AV = autonomous vehicles
- gormz 9y agoYou can pry the steering wheel out of my cold dead hands. Might be true that on paper these things cause less accidents and less deaths, but I want to be in control of my life. If I die in a car I don't want it to be because of a software bug.
- jacquesm 9y agoI hate to break it to you but it could still happen to you because both you and those Teslas are part of the same traffic stream. So you could easily get killed because of a software bug in someone else's vehicle.
- gormz 9y agotouche. seems like they're more interested in concrete barriers than other cars at the moment.
- oldgradstudent 9y agoWait for the next software update. And remember not to have an emergency requiring you to stop on the side of the road: [1] https://www.youtube.com/watch?v=fc0yYJ8-Dyo https://www.youtube.com/watch?v=fc0yYJ8-Dyo [2] https://www.theverge.com/2018/1/23/16923800/tesla-firetruck-crash-autopilot-investigation https://www.theverge.com/2018/1/23/16923800/tesla-firetruck-...
- gormz 9y agoI honestly think the fault is in the naming conventions. when you call it "Auto Pilot" it doesn't seem like you have to drive at all. Someone like Cadillac calls it "Super Cruise". I think this makes it sound much more like what it is. It's driving assistance, not self driving.
- jacquesm 9y agoThat one is ugly. If AP can't even steer around huge trucks parked on the road then that's a very serious problem. Interesting how everybody claims it is 'the drivers fault' when in fact at that moment the driver is the autopilot software. It - and not the person sitting in the left hand front seat of the car - is driving the car.
- lvoudour 9y agoI wonder what "AI updates" are going to mean from an insurance point of view. Don't know about the US, but some countries in Europe insure the driver and not the car (eg. Ireland). How far fetched is for insurers to argue that an autonomous car software update constitutes a change of driver and demand that the updates must first be validated from them before installing?
- edf13 9y agoIt's a good point. Here in the UK I am insured to drive vehicle A as specified on my insurance documents. If I modify that vehicle my insurance policy is void and I need to inform the insurance co. and probably pay a premium. In your scenario the same could be true on a software update.
- lvoudour 9y agoAnd it wouldn't be unreasonable from the insurers point of view, since any update carries a risk. What's going to happen I think is that insurance companies will probably need to vet the updates before allowing you to install them, much like android phone vendors. Running un-authorized updates will void your policy.
- iainmerrick 9y agoThis kind of video makes me wonder: what is the "autopilot" feature actually for? Do people generally like it? (I don't own a Tesla and I've never driven one.) If you're supposed to keep your hands on the wheel, and given videos like this that show you really need to keep your hands on the wheel and pay attention, is automatic steering really that big of a deal? Cruise control, now, that really is useful because it automates a trivial chore (maintaining a steady speed) and will do it well enough to improve your gas mileage. The main failure condition is "car keeps driving at full speed towards an obstacle" but an automatic emergency brake feature (again, reasonably straightforward, and standard in many new cars) can mitigate that pretty well. It seems to me that autopilot falls into an uncanny valley, where it's not simple enough to be a reliable automation or a useful safety improvement, but it also isn't smart enough to reduce the need for an alert human driver. So what's the point? If you're excited about self-driving cars because they'll reduce accidents, as many people here claim, what you should really be excited about is the more mundane incremental improvements like pedestrian airbags. Those will actually save lives right now.
- vorpalhex 9y agoSo human drivers are terrible at driving for any long period of time, especially on relatively consistent "boring" highways. Guess what computers excel at? Driving consistently on consistent highways. The Tesla Autopilot is supposed to be the always aware and paying attention portion on these cases where a human driver would be very likely to start texting or dozing off. Now, it's not fully autonomous and may well decide it can't handle a situation (or apparently try to drive you into a barrier to see if you're still awake...). In this case the human driver who is somewhat zoned out needs to take control instantly and correct the situation, until they can safely re-engage autopilot (or pull over and make sure they're still alive, etc).
- davewritescode 9y agoThe problem is autopilot actively engages you even less than highway driving. I argue this exacerbates some of the problems that cause humans to checkout. I don't know what the answer is but it feels like GM's super cruise does a more adequate job of acknowledging the realistic limits of the technology and explicitly white lists roads where the technology is available for use. I personally think that without some sort of sensors or beacons in the road, autonomous driving via camera and LIDAR sesnors is ever going to be good enough to achieve level 5 autonomous operation.
- huffmsa 9y agoGo to Washington DC, head West on Congressional, cross the Roosevelt bridge and take the exit for the GW. If there isn't a car smashed into the middle of the north/south Y junction, you'll get the chance to count the skid marks at that Y junction from humans making this exact same error. Tl dr; humans make the same mistake, at a higher frequency, which is why the accordian barrier exists.
- lafar6502 9y agohumans at least are aware that they made a mistake and can avoid collision. Tesla makes no mistake, tesla has no idea whats going on so you cannot call it a mistake. Its not a bug, its a feature.
- huffmsa 9y agoExcept it gets reported back to HQ and like with airline accidents, the systems are updated across-the-board so that the risk of future occurrences is greatly reduced. So instead of a single human learning "gee, shouldn't do that again", 100k+ vehicles learn about it.
- falcolas 9y ago"Don't worry, an update will fix it" is a cold comfort when an OTA update broke it in the first place.
- kalleboo 9y agoDoes Tesla actually do this? They should detect hotspots of people deactivating Autopilot (like in this video) and make Autopilot deactivate itself a few miles before First we had the barrier crash. Then there was the video of someone reproducing the barrier crash and manually emergency braking. Now we have this video. One thing I've learned in my software career is that if one of our customers reports a bug, 10,000 others have encountered it but not bothered reporting it. So why hasn't Tesla rolled back the software yet?!
- Gracana 9y ago
- lafar6502 9y agoha ha i dont see how this is supposed to improve safety. Maybe it's safer than having no driver at all but if we're talking about being better than typical human then the safest thing to do is to switch this thing off.
- bufferoverflow 9y agoIt still could be safer than most human drivers. We only have a few examples of their AI misbehaving. If it consistently crashed into the barriers, we would have thousands of deaths per day.
- nemetroid 9y agoFrom the linked thread: > This is my commute home- I know it does it about 90% of the time. It does it almost every evening and has for the last few weeks since 10.4 rolled out.
- e1ven 9y ago> Previous autopilot versions didn't do this. 10.4 and 12 do this 90% of the time. I didn't post it until I got .12 and it did it a few times in case it was a bug in 10.4. This is why the idea of self-updating cars terrifies me. I'd never allow autopatching on a production server - Why would I allow it on hardware I'm betting my life on?
- TheSpiceIsLife 9y agoIt adds a whole new level of frightening to the term breaking change.
- rsynnott 9y agoI'm amazed that this is even being permitted. And in almost all territories; practically no-one has banned this practice. It seems absurdly risky, especially once someone finds a security flaw in the Tesla software updater.
- dlisboa 9y ago"Move fast and break things". Taken literally.
- banku_brougham 9y ago“Move fast and break humans.”
- amq 9y agoMove fast and break barriers.
- paul7986 9y ago2018 the wild wild west of robot cars. Pay an extremely high cost including your life to ensure technological advancement. Is that not part of their marketing? If not SNL needs to do a skit! I can see the skit now, "You looked S3XY in that Telsa," "Did you get the update last night?" "Oh your right and it made me feel so S3XY until it drove me straight into a barrier at 60 MPH."
- rectang 9y agoThe ability to discern when the car needs to stop for a stationary object right in front of it needs to be an ironclad requirement for self-driving. It seems that sophisticated LIDAR is currently the best way to achieve this, but LIDAR is expensive. So companies like Uber and Tesla skimp on the LIDAR, and build self-driving cars that can more or less follow lanes but plow right into obstacles? Whoa. The feasibility of widespread "self-driving" tech is going to advance only as the cost of LIDAR falls.
- 7952 9y agoI wonder why they can't just use a less sophisticated lidar. Just have a few fixed beams that just look ahead. Do you really need a scanning system to detect an obstruction?
- freerobby 9y agoLidar isn't reliable in conditions that are less than ideal (rain, fog, snow, etc). If you build a car that is dependent on real-time lidar observations, it is worthless much of the time in some climates and some of the time in all climates. Most companies are pursuing a Lidar approach because dev_speed(Lidar approach) > dev_speed(camera approach). Tesla is pursuing a camera approach because max(camera approach) > max(lidar approach).
- Piskvorrr 9y agoIn such conditions, cameras are not unreliable, they're downright useless. Heck, even people have trouble driving during snowfall. I have a feeling that this is a massive set of scenarios that aren't even edge cases, yet SDVs are completely unprepared for them. ("No snow in coastal California - it doesn't exist at all!")
- freerobby 9y agoI think you may be imagining what a human sees in a foggy camera image, rather than what a neural network can see in it. In fact, atmospheric obstructions degrade Lidar's signal much faster than a camera's. If you check out NVidia's DriveNet demo from a couple years back, you'll see that they already have a vision-based NN that outperforms humans in fog and snow[1]. We can debate whether people should drive in those conditions at all, but today's expectations will be the baseline that SDVs are up against, and cameras are much better suited than Lidar to achieve that baseline. [1] Start at 7:21 https://www.youtube.com/watch?v=HJ58dbd5g8g https://www.youtube.com/watch?v=HJ58dbd5g8g
- damon_c 9y agoI assume they have a test suite that allows them to run 1000s of situations on the latest version of software to see if it crashes/almost crashes on any of them. If so, it seems like it would be fairly easy to add this situation to the corpus... especially if they are recording data from their cars live. They should have 100 people going out there and recording this lane split situation into their test data ASAP.
- S_A_P 9y agoThis seems to be one of those problems that you can solve 90% of the way very quickly with a seemingly simple set of rules. 1) stay in your lane 2) dont hit other objects 3) obey traffic laws Then that devil that is the details shows up...
- cmpolis 9y agoI can't understand the blindness of the Tesla/Elon fanboys in that thread; the comments on that thread are so defensive of Tesla and still show little concern for safety. Despite fairly damning video evidence, a fatality, many reports of accidents and near accidents...
- Analemma_ 9y agoPeople have a lot invested in Tesla-- sometimes emotionally, sometimes literally. It is difficult to be objective in that case.
- lisper 9y agoHere is my armchair diagnosis: right before the car veers towards the barrier it drives through a stretch of road where the only visible lane marker is on the left. Then the right lane marker comes into view at about the point where the lane starts to widen out for the lane split. The lines that will become the right and left lane markers of the split left and right lanes respectively are right next to the van in front of the Tesla, and at this point resemble the diamond lane markers in the middle of the split lanes. My guess is that the autopilot mistook these lines for the diamond lane marker and steered towards them thinking it was centering itself in the lane. If this theory turns out to be correct then Tesla is in deep trouble because this would be a very elementary mistake. The system should have known that the lanes split at this point, noticed that the distance between what it thought was the diamond lane marker and the right lane line (which was clearly visible) was wrong, and at least sounded an alarm, if not actively braked until it had a better solution. This is actually the most serious aspect of all of these crashes: the system does not seem to be aware when it is getting things wrong. I dubbed this property "cognizant failure" in my 1991 Ph.D. thesis on autonomous driving, but no one seems to have adopted it. It's not possible to engineer an autonomous system that never fails, but it is possible to engineer one in such a way that it never fails to detect that it has failed. Tesla seems to have done neither.
- jacquesm 9y agoI totally agree with your analysis. What is more worrying is that even following the rest of the traffic would have solved the problem. If you are the only car doing something then there is a good chance you are doing something wrong. Also: the lane markings are confusing but GPS and inertial readings should have clearly indicated that that is not a lane. If two systems give conflicting data some kind of alarm should be given to the driver.
- rayiner 9y agoWhen I think of self-driving car technology, I think of all of these inputs (maps, cameras, GPS, etc.) being fed into a world model from which inferences are drawn.[1] Here, the car could have known from cameras and maps that there was a lane split, and should have been able to detect that the line following was giving a conflicting inference compared to those other sources. The question is: does Tesla actually do that? Because if not, it's not really self-driving car technology. It's not a glitch in a product that's 90% of the way there; it's a fundamentally more basic technology. [1] See https://www.ri.cmu.edu/pub_files/2009/6/aimag2009_urmson.pdf https://www.ri.cmu.edu/pub_files/2009/6/aimag2009_urmson.pdf at 21-23.
- VikingCoder 9y agoTesla makes amazing cars, but their attitude to self-driving is terrible. Waymo's approach is way mo [sic] safer.
- PaulHoule 9y agoWhen are they going to release the software patch?
- rsynnott 9y agoThe scariest bit for me; in the comment's it's claimed that it has only done this for the last two software updates. Currently the worst I have to worry about when getting an OTA update is that it may make my phone less stable. Apparently in the future it may also influence precisely how my car tries to kill me.
- kozak 9y agoThe soundtrack fits the video perfectly.
- prewett 9y agoTesla says that the driver needs to be constantly watching the road in autopilot mode. But what kind of autopilot is that? If I have to constantly pay attention, I might as well just drive. I think it's safer, too, since I am bad at paying attention for something that rarely happens.
- josefresco 9y agoMy hands started to sweat watching that video - this is not good.
- jijojv 9y agohttps://www.cnbc.com/2018/01/31/apples-steve-wozniak-doesnt-believe-anything-elon-musk-or-tesla-say.html https://www.cnbc.com/2018/01/31/apples-steve-wozniak-doesnt-... "Man you have got to be ready — it makes mistakes, it loses track of the lane lines. You have to be on your toes all the time," says Wozniak. "All Tesla did is say, 'It is beta so we are not responsible. It doesn't necessarily work, so you have to be in control.' "Well you that is kinda a cheap way out of it."
- jijojv 9y agoThey should have settled https://arstechnica.com/cars/2017/04/tesla-owners-sue-enhanced-autopilot-featuressimply-too-dangerous-to-be-used/ https://arstechnica.com/cars/2017/04/tesla-owners-sue-enhanc... last year instead of dragging it on. These round of videos are more damning than ever. https://www.pacermonitor.com/public/case/21195146/Dean_Sheikh_et_al_v_Tesla,_Inc https://www.pacermonitor.com/public/case/21195146/Dean_Sheik...
- banku_brougham 9y agoThe reddit thread (of largely tesla drivers) seems to have concluded that the resolution is “stay alert especially when updates get pushed.” To me it doesn’t capture the gravity of the situation.
- dhimes 9y agoThey need to move those damn barriers. This is happening alot. /s
- planetjones 9y agoDoes anyone know how software updates are tested for such autonomous driving software ? Is there a possibility for this driver's video to be added to a regression suite and tested, or can they take the data from his car and build a regression test based on it ?
- igammarays 9y agoI sure hope they do have a regression suite with thousands of real world sensor-data samples in some normalized format made amenable to unit testing. Assuming such a normalized format is possible to generate, I expect these samples to be open-sourced and standardized. That would give the NTSB a systematized certification procedure: just run the new version of the software on all the known test cases before allowing it to be released. Still, I suspect that may not be enough - a new weird road condition or construction site can be created at any time. This is why I believe self-driving companies would be better off spending their budget upgrading and certifying specific routes as safe for automation. Certified routes could be subject to special construction protocols and regular road quality audits that ensure automated cars won't run into a non-automatable condition on those routes. It's also easy to verify and confirm that a car works on all certified routes, rather than trying to test the entire North American road network.
- syntaxing 9y agoDoes anyone know if all these recent accidents are with the old MobilEye technology or the new Nvidia system? Or was it a rumour that Tesla was moving towards Nvidia rather than MobilEye (I think MobileEye was the one that didn't want to renew their contract?)?
- ocdtrekkie 9y agoI would be careful calling it just the MobilEye system or the Nvidia system and assigning blame accordingly. They both create autonomous driving components, but at the end of the day, Tesla, Uber, etc. write their own software, and are responsible for the whole of the systems that are actually in their cars.
- kwhitefoot 9y agoIt's hard to be sure in every case because most 'news' stories and most HN comments don't specify even the model of car involved. But the incident in which the driver was killed running into the divider was a model X so that means hardware version 2 and Tesla's own software rather than Mobileye that is in, for instance my 2015 S 70D. See https://en.wikipedia.org/wiki/Tesla_Autopilot#Hardware_2 https://en.wikipedia.org/wiki/Tesla_Autopilot#Hardware_2
- antongribok 9y agoHere's the exact location where the video ends, in case anyone wants to go "troubleshooting" this weekend: https://goo.gl/maps/Y2kag6yYc3N2 https://goo.gl/maps/Y2kag6yYc3N2
- e12e 9y agoChangelog (latest changes on top): * fix issue where "Marvin" autopilot keeps complaining (rip out voicebox) * Boost intelligence of "Marvin" autopilot. This car now has the brain the size of a planet. * test new smarter "Marvin" autopilot
- mattbeckman 9y agoOne way to help might be to disable AP following an update for X days or weeks. Let AP run in a disabled simulation mode while you drive your normal routes. If it determines your course of action in any situation breaks the threshold of what it was thinking it should do next, auto-report video feed to Tesla, and prevent AP from being used in a X-mile radius around that area.
- Piskvorrr 9y agoIt seems that with current developments, that's a complicated way of saying "turn it off."
- m4x 9y agoI don't understand why people use Autopilot, or why Tesla has released it. To use it safely and according to Tesla's instructions, you have to remain 100% vigilant at all times - so you might as well just drive yourself. And if you fail to remain vigilant, which is likely since you are sitting passively in the drivers seat, you might kill somebody. Where's the upside? Why on Earth would I want to use such a product?
- kwhitefoot 9y ago> Where's the upside? Why on Earth would I want to use such a product? I use it because it makes my driving less stressful. The car and I work together which means I don't have to work so hard to keep to the speed limit, to keep a sensible distance from the car in front and stay in lane.
- dingo_bat 9y agoSo no different than a regular adaptive cruise control.
- readhn 9y agoWhy is autopilot feature still allowed to exist? It should be disabled until further improvements. This blows my mind.
- Too 9y agoThis is the biggest wtf. We've now seen at least three videos of similar trivial mistakes by autopilot only this week with one fatal crash and it is still not disabled. When aircrafts crash all planes of that model are grounded until the root cause is found. Self driving cars need similar processes.
- w_t_payne 9y agoI would argue that these incidents indicate a need for aerospace levels of quality assurance. With the right tooling this does not need to be expensive. I hope that incidents like this encourage people to develop open source tooling to support safer software and higher levels of quality. We can see how lives might depend on it.
- Piskvorrr 9y agoAerospace QA is expensive by definition, because reliability nines generally are, whatever the industry: an additional nine costs 10x. You can go cheaper, but with a corresponding decrease in reliability.
- w_t_payne 9y agoA lot (but not all) of the costs are due to additional process steps that can be automated. I think we can dramatically increase safety and quality by selectively adopting some of the concepts from aerospace and aggressively automating them. Traceability; NLP for requirements etc..
- Piskvorrr 9y agoThat is probably true, and might help - but that only gets you so far, can't program your way out of everything. For high reliability, you'll need redundancy (and thus fallback and voting and whatnot), which will a) drive up HW costs, while also b) increasing complexity. I do agree there are some LHF opportunities to learn from aviation, but doing that is insufficient.
- w_t_payne 9y agoI agree with you on the topic of redundancy. I am probably not communicating it very well; but my main motivation is a reaction to just how much better our tools need to be.
- chillingeffect 9y agoPerfect example of a system doing what it's supposed to, but that's not the right thing. I don't believe it was headed toward the center barrier - I believe it was neatly avoiding it. Here's what I think was happening: The car did an admirable job trying to stay in that left lane. Each time it saw a red and white diagonal lined barrier, it gracefully edged a comfortable distance away. The problem is, most humans catch on, "Gee, three scary-looking barriers in a row! Maybe they're trying to tell me something. "Maybe this road is closed." But I assume the car's system perceived each one as in individual, isolated obstacle to be handled without making inferences. And it did that much well, and when it realized the third or 4th barrier was impassable, it had to cross the other side of the yellow barrier and it did that well, too.
- l8again 9y agoAn honest question. Are any of these autonomous driving systems open source or peer reviewed in any way? If not, isn't it really weird that we are talking about regulations even though the underlying technology is not even peer reviewed? How do we know (mathematical proofs, etc.) that a self-driving car manufacturer has done a good enough job if all of that is proprietary?
- craftyguy 9y agoNo, no, yes, we don't.
- duiker101 9y agoIn one of the earlier discussions about this(like, last year) I remember an engeneer's comment about how one of the major manufacturer's (Toyota maybe?) code base for the monitoring system of the car was something to behold. On the lines of hundreds of global variables and basically unexisting standards. Now, I don't say that their or Tesla's autopilot are the same, but it wouldn't really surprise me either.
- deleted 9y ago[deleted]
- aabajian 9y agoDoes Tesla record when people override autopilot? Seems like this would be a good source of information about sections of road with obvious problems.
- shawkinaw 9y agoWhy isn’t the traffic ahead of the car taken into account? When I’m driving, following the car in front is a pretty good way to decide where it’s safe. Conversely, going somewhere no car ahead is going seems like a bad idea.
- paulcole 9y agoHonestly who is to say that a human driver wouldn't have done the same thing?
- sul4bh 9y agoPrevious autopilot versions didn't do this. 10.4 and 12 do this 90% of the time. I didn't post it until I got .12 and it did it a few times in case it was a bug in 10.4. It looks like software updates will soon won't just break your workflow. They might break your spine or your head or take away your life entirely.
- senatorobama 9y agoWhoever approved this change in the code needs to be out. If I was Karpathy, I'd be very worried right now that he isn't liable in a civil or criminal suit.
- decebalus1 9y agoOne thing baffles me. Where is the insurance industry in this? I would expect them to drastically increase premiums in light of these developments.
- kwhitefoot 9y agoHere in Norway insuring a Tesla is no more expensive, in fact probably cheaper, than insuring a petrol car in a similar class. I'm pretty sure that the insurance companies wouldn't do it if it meant losing money.
- chrstphrknwtn 9y agoWasn't there (or perhaps still is) a government subsidy for electric vehicle purchases in Norway? There may also be a government subsidy for insurance, to further encourage people to shift away from ICE vehicles. Cheaper insurance may have nothing to do with the quality or safety of autopilot in Tesla vehicles.
- thrillgore 9y agoCalling this well intentioned series of mistakes "Autopilot" has got to be the dumbest fucking thing Tesla has done. Actually, I stand corrected. Their stock gamble is about as dumb too.
- drderidder 9y agoI’ve always thought self driving cars are a monumentally dumb idea. There, I said it. I feel better now.
- m3kw9 9y agoMan I cannot imagine how they will debug the issue in the algorithm and how they will re train it to get that barrier mistake out
- animex 9y agoCould someone fill me in on collision detection? My 2016 BMW has Front Collision warning which can be set to min/med/max range. I've got it set to max range. It can detect a solid object that is coming up to collide with my car and warn me via sounds and HUD. Do Teslas not have some method of solid object collision that should supersede any driving directive? As per my previous comment, could a prankster paint a set of white lane markers that verve into a concrete wall or even off a cliff? That's freaking scary to me that these systems are so reliant on lane markings. At highway speed, a distracted driver, would have little time to be alerted, assess the situation and take the correct corrective action before a collision occurred.
- Tomminn 9y agoI commented on the first one that humans could easily make the mistake the Tesla made. This, however, is a scarily bad fuck up.
- pooya13 9y agoI don't mean to call the OP lier by any means. But anyone knows if there is any evidence that this video was taken on an autopilot engaged Tesla? Just want to be sure about it before I form judgment against Tesla.
- jacksmith21006 9y agoProblem is the other SDC companies are doing dangerous things trying to keep up with Google.
- dingo_bat 9y agoI think tesla needs to build a high performance cell network and upload continuous 1080p stereoscopic video at 60fps and factor that into their training. No sensor data can be a substitute for high quality video.
- tonyquart 9y agoI just read at https://www.lemberglaw.com/self-driving-autonomous-car-accident-injury-lawyers-attorneys/ https://www.lemberglaw.com/self-driving-autonomous-car-accid... about this matter, and I think that these autonomous cars haven't been ready to be publicly released to citizens now. Automakers and law makers should think about the regulation and law.