31 ms·
This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a
by brianwawok 11y ago
This is a very good Post-Mortem.
As I assumed it was kind of a corner case bug meet corner case bug met corner case bug.
This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them?
Making software is hard....
- reustle 11y agoI wouldn't worry so much. I'm sure self driving cars are going to save a lot more lives than they are going to end. Humans are terrible drivers, and the software will only get better.
- VonGuard 11y agoYeah, remember, auto-pilot in a plane needs to be 100% reliable, or everyone dies. A car needs to be, I dunno, 80%? Compared to a bad human driver, who still drives every damn day, a computer need only be about 60% reliable to be better. People suck at driving. Even a shitty self-driving car will save a ton of lives simply by obeying traffic laws.
- dreamcompiler 11y agoAn auto-pilot for an airplane is a considerably easier problem to solve. No lanes; no pedestrians; very little other traffic; three spatial degrees of freedom. That's why auto-pilots for airplanes have existed for almost a century but we're just now beginning to get self-driving cars. Humans are still better at dealing with the full panoply of crap that road driving throws at us.
- infinotize 11y agoAircraft autopilots also rely on experienced and licensed pilots to operate them and be responsible for the aircraft at all times. Self driving cars have assume the operator is not particularly capable nor paying attention to anything happening on the road.
- duaneb 11y agoDo they rely on the pilot? I was under the impression it was entirely hands off.
- durandal1 11y agoThere are many conditions unders which an aircraft autopilot will simply disconnect without previous warning and hand over controls to the pilot.
- Eugr 11y agoCurrent generation of autopilots doesn't handle traffic avoidance or make any routing decisions - they just follow pre-programmed routes at pre-programmed speed and altitude (or climb/descent profile). But even this relatively simple level of automation causes problems - pilots start to rely on automation too much, and when things go south they are not capable to deal with it. Airlines recognize it, and put more emphasis on hand-flying during training and routine operations, so pilots don't lose their basic piloting skills. It's not a new problem - there is an excellent training video from 1997 - "Children of the Magenta": https://www.youtube.com/watch?v=pN41LvuSz10 https://www.youtube.com/watch?v=pN41LvuSz10
- schwarrrtz 11y agoThis video also inspired an excellent podcast from 99 Percent Invisible about the challenges and dangers of automation. I would highly recommend listening. http://99percentinvisible.org/episode/children-of-the-magenta-automation-paradox-pt-1/ http://99percentinvisible.org/episode/children-of-the-magent...
- ocdtrekkie 11y agoGoogle Self-Driving Cars currently fail approximately every 1,500 miles, according to their own report. We are a LONG way away from being able to separate the operator's attention from driving.
- boydc 11y agoSelf-driving car could be better than human in average. But as long as there are human drivers who drive better than self-driving software, it would be disaster for these drivers. We definitely do not want some technique than do good for majority but do horrible things for minority, right?
- scarecrowbob 11y agoI;m not following your argument here. A single driver's ability isn't the only risk factor... if I'm a great driver but every one else sucks (that's how it for everyone already, right :D ), then an overall increase in the population's driving ability helps me, right?
- boydc 11y agothe overall increase helps you indeed. But do you want use self-driving software if you are a great driver(or you think you are)? I do not because I want to be more safer by driving myself. If great drivers like to drive themselves. Others wants too because they do not trust these great drivers. In everyone driver's eyes, there are only two kinds of drivers 1 ) bad driver slower than me. 2 ) mad driver faster than me.
- jchrisa 11y agoWhy would it be a disaster?
- llamataboot 11y agoThis would be true only if your driving ability only affected your chance to die, but your driving ability has an effect on everyone else's safety on the road as well!
- boydc 11y agoConsider you are a damn good driver, better than self-driving software. If self-driving software can reduce your risk by giving you a safer environment(by replacing lots of bad drivers), but it will increase your risk when handling risks(because it's not as good as you). Would you like to choose self-driving software? The point here is, no matter how good the driving environment goes, I do not want to lost any chance to survive(If I'm a good driver).
- zdkl 11y agoNot sure low-probability/high-damage events are comparable to high-prob/"low"-dmg in the first place and that's not a trivial question to handle in real-time
- ceejayoz 11y ago> Yeah, remember, auto-pilot in a plane needs to be 100% reliable, or everyone dies. First, they're not anywhere near 100% reliable. They can fail on their own, and they'll also intentionally shut themselves off if the instruments they rely on fail. https://en.wikipedia.org/wiki/Air_France_Flight_447 https://en.wikipedia.org/wiki/Air_France_Flight_447 Second, an autopilot failure shouldn't lead to death if the pilots are competent and paying attention.
- duaneb 11y agoWhat does a failed autopilot look like? Would a pilot do any better with an "aerodynamic stall"? I know little about planes and it seems like that'd be a big problem with or without a pilot driving.
- ceejayoz 11y agoAll pilots are trained to recover from a stall. A failed autopilot could look like all sorts of things, from just automatically disconnecting itself (usually with a loud warning alert) to issuing incorrect instructions (which is why the pilots are supposed to be awake and alert while it's engaged, watching the instruments).
- neurotech1 11y agoActually a key finding in AF447 was that pilots were not trained on how to recognize and recover from a high altitude stall. It is not like flying a Cessna 150. The junior first officer didn't realize the aircraft had stalled. Pilots were trained on the procedure for recovering from a low altitude stall; 100% or TOGA thrust and power out of it while minimizing altitude loss. Training has now changed for both low altitude and high altitude stall recovery.
- ceejayoz 11y agoIIRC, part of the issue was that the two pilots issued contradictory joystick commands, which the plane averaged to zero. Which is a bit terrifying.
- duaneb 11y agoBut autopilot for planes is actually much easier than negotiating traffic with irrational humans with road rage. You can coordinate with air traffic for takeoffs and landings, and there is very little to run into at tens of thousands of feet.
- raverbashing 11y ago> auto-pilot in a plane needs to be 100% reliable, or everyone dies Actually it is much more simpler than a self-driving car. And if there is a problem it disengages.
- vacri 11y agoAutopilot in a plane usually doesn't involve autonavigation, whereas autodrive in a car generally requires navigation. Autodrive in a car without navigation is basically 'cruise control'.
- dreamcompiler 11y ago'Cruise control' does not (at least it didn't until very recently) even attempt to avoid collisions with neighboring cars or keep the car in its lane. Even without navigation, autodrive in a car is a considerably more difficult problem than either cruise control or autopilot in a plane.
- vacri 11y agoMy point is that 'autopilot' is more like 'cruise control', and except in advanced cases, is not analagous to 'autodrive'.
- brianwawok 11y agoI know this logically. But emotionally I know how many bugs I have written in my life. I know software devs are human.. aka I know how the sausage is made.
- kohanz 11y agoYes, but have you been part of the development & testing effort for mission-critical software (e.g. a class 1 or 2 medical device?). It's not true in all cases, but for the most part the level of QA that goes into the devices before release is significantly higher than that of your average product. This is why regulation is required.
- deleted 11y ago[deleted]
- praxulus 11y agoGCE downtime just means people lose money, it's not life-or-death. Skimping on QA in order to reduce costs and get to market faster is a perfectly reasonable decision when the consequences are so mundane.
- flurdy 11y agoI understand what you mean but that is generalising too much what people use GCE, public clouds, self hosted servers for, and especially going forward. It is not all convenience applications, game backends etc. What people these days use AWS/GCE for is so varied, even public sector use AWS Gov Region for example. Downtime consequences is not just money lost but can be life-and-death and for some application they need solid QA even if hosted in a public cloud. It may (emphasise 'may') be how they share medical data via GCE/AWS that gets delayed just before a surgery (ok, edge case) or how they update bugs in a critical GPS model that happen to be used by an ambulance, or even a taxi used by pregnant lady that is about to drop, etc. Or simple general medical self diagnosis information site that by chance could have saved someone in that time slot. Or any other random non medical usage which involves a server and data of some kind that happen to be in GCE. Yes critical real time systems often are on-premise or in self hosted data centres, but more and more are not especially if viewed as not critical but in some cases indirectly are critical.
- ambago 11y agoEspecially since "driver-error" is the cause of 94% of motor vehicle crashes in the U.S.[1], with 32,675 people killed and 2.3 million injured in 2014.[2] Worldwide, motor-vehicle crashes cause over 1.2 million deaths each year and are the leading cause of death for people between the ages of 15-29 years old.[3] It's estimated that self-driving cars could reduce vehicle crashes by approximately 90%! [4] [1] http://www-nrd.nhtsa.dot.gov/pubs/812115.pdf http://www-nrd.nhtsa.dot.gov/pubs/812115.pdf [2] http://www-nrd.nhtsa.dot.gov/Pubs/812219.pdf http://www-nrd.nhtsa.dot.gov/Pubs/812219.pdf [3] http://www.who.int/violence_injury_prevention/road_safety_status/2015/en/ http://www.who.int/violence_injury_prevention/road_safety_st... [4] http://www.mckinsey.com/industries/automotive-and-assembly/our-insights/ten-ways-autonomous-driving-could-redefine-the-automotive-world http://www.mckinsey.com/industries/automotive-and-assembly/o...
- takeda 11y agoThis is true for a single car, but self driving cars are introducing something that did not happen before. Imagine majority of cars are self driving cars and they all malfunctions due to bug in software update.
- executesorder66 11y ago> and they all malfunctions due to bug in software update. That's assuming everyone with a self driving car is driving the exact same model and they all updated at the exact same time. Chances are there will be many different models and manufactures so an OTA update with a bug will only affect a much smaller percentage of the self driving cars.
- nzoschke 11y agoSeconded. Fast recovery of the problem, fast to publish a postmortem, and a very thurough postmortem. Outages suck, but are inevitable even for Google. With a response like this Google has gained even more trust from me.
- kyrra 11y agoGoogle's take on postmortems is really nice. As the SRE book points out, they are seen as a learning tool for others. Most internal postmortems are available for anyone within the company to see and learn from. As well, they are always blameless. No fingers are pointed at the person who caused the issue in the postmortem. They explain the issue, what happened, and how it can be prevented in the future. Pair this with the outage tracking tools and you can find all the outages that have happened across Google and what caused them. Then there is DiRT[0] testing to try and catch problems in a controlled manner. Having things break randomly through Google's infrastructure and you have to see if your service's setup and oncall people handle it properly is a really awesome exercise. [0] http://queue.acm.org/detail.cfm?id=2371516 http://queue.acm.org/detail.cfm?id=2371516 The opinions stated here are my own, not necessarily those of Google. Edit: Changed from saying "all" to "most" postmortems being available to Googlers to see.
- Thaxll 11y agoself driving cars will be safer than BGP that' s for sure.
- clebio 11y agoOh geez. This comment needs to go into the pantheon of all-time nerdy humor.
- ddispaltro 11y agoAgreed, I think a good postmortem distinguishes great companies from just good companies. The depth on philosophy, reasoning and then action is very digestible.
- fweespee_ch 11y ago> This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Yes. However, the current failure rate of human drivers being improved on is the standard I care about. http://www.cnbc.com/2015/10/29/crash-data-for-self-driving-cars-may-not-tell-whole-story.html http://www.cnbc.com/2015/10/29/crash-data-for-self-driving-c... > After crunching the data, Schoettle and Sivak concluded there's an average of 9.1 crashes involving self-driving vehicles per million miles traveled. That's more than double the rate of 4.1 crashes per one million miles involving conventional vehicles. That is the only number that matters to me. Google gets that to 4.0 per million miles and I'd say they are good to go.
- brianwawok 11y agoSo crashes like that include drunk drivers, drugged drivers, and texting drivers. What is the crashes per miles for a paying attention driver? If it is 1 per million miles, the self driving car would need to be a lot lower. Now if it was 4am and I am falling asleep at the wheel, I bet any self driving car would beat me. So cool to turn on, but maybe not for a daytime cruise...
- bcook 11y agoWhy are you comparing self-driving cars to exclusively a "paying attention driver"? For self-driving cars to be safer than human drivers, there is no requirement that the self-driving cars should be better/safer than the best human driver... the self-driving car simply needs to be safer than the majority of humans.
- ams6110 11y agoIt points out that a lot depends on the driver. I drive attentively, and moreover I like driving. I would never want a self-driving car for myself. However some seem to think that should be the only choice because they can beat accident averages that include drivers who drink, text, do makeup, masturbate, whatever while driving.
- 11y ago
- jfoster 11y agoAs long as the edge case bugs in self driving cars come up less frequently than human error, it's an overall improvement.
- ocdtrekkie 11y agoCurrently, they don't. Google cars fail every 1,500 miles on average.
- 8note 11y agodo you have a comparison for the number of miles between human errors?
- ocdtrekkie 11y agoIt's hard to get an exact figure, particularly because of unreported accidents, and various sources. But I believe insurance companies have previously stated it's about one in every 250,000 miles. For the sake of giving a wide berth for unreported accidents, and to not give humans the benefit of the doubt, I've been using the rough figure of 150,000 miles between accidents. I don't have a great source for it though, and if anyone finds a good source, it'd be fantastic.
- jfoster 11y agoWhere is that figure coming from? I'm curious what kinds of failures they have.
- ocdtrekkie 11y agohttps://news.ycombinator.com/item?id=11492569 https://news.ycombinator.com/item?id=11492569 <- I detailed it a bit more in this comment, including linking the source report from Google.
- jfoster 11y agoI think when you say they "fail", I think you are referring to the disengagements, right? The report you link to there says: “Immediate manual control” disengage thresholds are set conservatively. Our objective is not to minimize disengages; rather, it is to gather as much data as possible to enable us to improve our self-driving system. Also, table 4 reports the number of disengagements (for any reason) each month, as well as the miles driven each month. In the most recent month in that table, it's actually 16 disengagements over 43275.9 miles. That's approximately one disengagement every 2705 miles; about the distance from Sacramento, CA to Washington, DC. At the start of 2015 it was only 343 miles per disengagement; 53 disengagements over 18192.1 miles. The pace of improvement is incredible, especially considering disengagements are set conservatively. Can a human drive from Sacramento CA to Washington DC without a single close call or mistake along the way? I really doubt it. This technology will be saving lives soon.
- nxzero 11y agoIdea that edge cases in autonomous vehicles would result in 30,000+ deaths a year to me is a stretch. If you dispute this, please explain. If your position is that one death is too many, that is illogical relative to the option of letting people drive cars.
- ocdtrekkie 11y agoCurrently, the self-driving software fails out on a Google Self-Driving Car every 1,500 miles. If the car suddenly stops trying to drive in the road, and the driver isn't attentive (or worse, if Google gets their way and convinces the laws to change so they don't have to have steering wheels) that's a lot of deaths. I'm not saying it won't get better, but pretending self-driving cars is a cure-all right now is hilarious and insane.
- gizmo385 11y agoSource? What kind of failure are you talking about? Minor hiccups or full failures which stop the car entirely?
- ocdtrekkie 11y agoGoogle's report from December 2015: http://static.googleusercontent.com/media/www.google.com/en//selfdrivingcar/files/reports/report-annual-15.pdf http://static.googleusercontent.com/media/www.google.com/en/... Over 424,000 miles driven: 272 times the car had a 'system failure' and immediately returned control to the driver with only a couple seconds of warning. (Approx. every 1,558 miles.) A car mid-traffic spontaneously dropping control of the vehicle would likely create a large number of accidents. 13 car accidents prevented via human intervention (Approx. every 32,615 miles), 10 of which would've been the self-driving car's at-fault (Approx. every 42,400 miles). These virtual accidents were tested with the telemetry recorded during the incident, and it was determined had the human test driver not intervened, an accident would've occurred. Total of these events is 285, which is approximately every 1,487 miles driven. For useful comparison, a rough human average (when you add a large margin to account for unreported accidents) is somewhere around one accident every 150,000 miles driven. (Insurance companies see them every 250,000 miles approximately, I believe.)
- ksou32 11y agoHumans have bugs all the time, you can just faint for no reason while driving . ..
- llamataboot 11y agoMy consolation in that fact is that weird edge cases happen with human driven cars as well. Someone has a seizure and crashes, or more commonly reaches for a cigarette, the radio, their phone. People hit ice or water and overcorrect their spin. People drive too fast. Etc etc etc. Not even all edge cases, many common modes of failure. I except self-driving cars that kill people will be a huge emotional issue for a lot of people in accepting them, but for me, i just want them to be safer than human drivers, which isn't THAT high of a bar to cross.
- fizzbatter 11y agoYup. People seem to be overly critical with automated car failures. Personally, i think automated cars are going to easily be better than humans in the working cases (both human and ai are concious). Next, i expect to see fully operational backup systems. Eg, if a monitoring system decides that the primary system is failing for whatever reason, be it bug or unhandled road condition (tree/etc), the backup system takes over and solely attempts to get the driver off the road, and into a safe location. Humans often fail, but often can attempt to recover. And, as bad as we may be at even recovering, we know to try and avoid oncoming traffic. Computers (currently) are very bad at recovering when they fail. I feel like having a computer driving, in the event of failure, is akin to a narcoleptic driver - when it goes wrong, it goes really wrong. Hence why i hope to see a backup system, completely isolated, and fully intent on safely changing lanes or finding a suitable area to pull over.
- nostalgiac 11y agoSounds good in theory. Until the bug that causes failure is also present in the monitoring system, and as such doesn't fail over to the backup system. AKA exactly what happened here to Google.
- msellout 11y agoHumans also make mistakes in corner cases. I'm no more afraid of an auto-steering car than a human-steered car.
- ben_jones 11y agoSelf driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that might actually happen to me! [1]: https://en.wikipedia.org/wiki/List_of_motor_vehicle_deaths_in_U.S._by_year https://en.wikipedia.org/wiki/List_of_motor_vehicle_deaths_i...
- llamataboot 11y agoNo because negligent driving doesn't just put the driver at risk - it puts everyone else on the road plus pedestrians at risk
- wefarrell 11y agoExactly, negligent drivers don't just kill themselves and there is very little you can do to prevent one from killing you.
- 794CD01 11y agoThey don't have to be safer than driving is today. They can be significantly less safe while still being an improvement for society because drivers will be able to focus on other activities while travelling instead of wasting that time focusing on driving the car.
- Grambo 11y agoActually they have to be significantly safer than driving today. People would rather be unsafe and in control than not in control and a tiny bit safer. I know personally if a self driving car could only drive as well as I could then I'd still want to be the one driving.
- oldmanjay 11y ago
- stcredzero 11y agoThis is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Formal systems?
- tremon 11y agoFormal systems are still built on a model of the outside world, not on the world itself FAFAIK. Even if your formal coverage is 100%, you can not anticipate all weird edge cases the real world can come up with.
- eloff 11y agoYes but how many people drive stoned, drunk, or distracted? How many people drive aggressively, speeding, or erratically? How many people do dumb things on the road? As a software engineer I know that there will be bugs and some will likely kill people. But as a driver who has driven many years in less civilized countries, I know that human beings are terrible drivers. Who would you rather share the road with, computer drivers that drive like your grandma, or a bunch of humans? It's a no-brainer right?
- pveierland 11y agoIt also showcases the great thing about self-driving cars. Even though accidents will happen, when it does there will be plenty of sensor data and logs which can be examined to find the exact cause in a post-mortem. An improvement to the software can then be made, and millions of cars deployed can all effectively learn from a single accident. With humans, the amount of knowledge gained and the collective improvement of driving behavior from a single accident is low, and each accident mostly provides some data points to tracked statistics. With machines, great systematic improvements are made possible over time such that the remaining edge cases will become increasingly improbable.
- marcosdumay 11y agoJust like planes. I'll have to point that this is necessary, but not sufficient for enabling an ever improving, extremely safe activity. Aviation also have a just right amount of blame running in the system that is hard to replicate on any other area.
- xiphias 11y agoWhat does post-mortem mean in this context? The software one (after an accident) or the human one (after death)? I think It's crazy that the word gets back the original meaning..
- yeukhon 11y agoAirplanes are equipped with software and many pilots would turn on auto-pilots after a long take off. Bugs are everywhere, and it's just a matter of time before one is so critical and kill people. So our best bet is better quality assurance through proof and overtesting (do this incrementally!)
- Dylan16807 11y ago> As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. I wouldn't really agree with that. There were two pieces of code designed to perform checks on new configs and cancel them. They both failed. Neither of those checks is a corner case. If you had a spec sheet for the system that manages IP blocks, that functionality would be listed as a feature right up front.
- godgod 11y agoImagine the day when the software powering that self driving car chooses to avoid the pedestrian by driving your car off a cliff. Software can kill.
- Donzo 11y agoA 1-in-one-million occurrence will happen a thousand times each day when you perform a billion operations.
- erichocean 11y agoAre BGP updates for Google's own router configurations really so frequent that they can't pay an engineer to at least monitor the propagation of configuration changes? In this case, a human would have instantly seen that the update was a) rejected (as explained in the postmortem), and b) holy shit, WHY DID THE ROUTER CHANGE ITS OWN CONFIGURATION TO BLOW AWAY ALL OF THE GCE ROUTES!?! I'm all for automation, but WTF? Insert even a semi-competent engineer in the loop to monitor the configuration change as it propagates around and the entire problem could have been addressed almost trivially, as the human engineers eventually decided to do.
- jvolkman 11y agoThe human part of any process will also eventually fail, and it's much more difficult to fix human bugs. Better to shoot for full automation.
- heisnotanalien 11y agoBut a computer could do the same thing. It would be possible to alert on all the routes being blown away?
- DanielDent 11y agoAny sufficiently large system quickly reaches a point where a human has difficulty tracking what the system should look like. Google has at least tens of data center locations, each of which will have multiple physical failure domains. There are also many discontiguous routes being announced at all of their network PoPs. They have substantially more PoPs than data centers. It very quickly gets too much to reasonably expect people to be able to keep track of what the system should look like, let alone grasping what it does look like.
- dsl 11y agoFirst of all, BGP is core to Google's load balancing architecture. So within a single datacenter you probably have at least a few dozen devices down stream from each edge router. Secondly, I'm seeing just shy of 500 individual prefixes, 282 directly connected peers (other networks), and a presence at over 100 physical internet exchanges, just for one of Google's four ASes. Would you be able to read over that configuration and tell me if it has errors?
- mikx007 11y agoAssuming bugs are never intentional and mostly random... Maybe instead of one autopilot software, self driving cars of the future will have several, developed by completely different teams. Then a self driving car can take some sort of average or most common output instruction (thus minimizing the risk of random bugs/edge cases...etc.)
- profeta 11y agowhy do you think that will wait for cars? https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25
- draw_down 11y agoWell, humans driving cars is already a disaster, so.
- lugg 11y agoThis wasn't an edge case. It was two bugs in two sections of code both designed to recover from a serious problem. It sounds like both sections of code were not tested properly at the very least. Sounds to me like someone just didn't bother to test the failsafe part of the code.
- packetslave 11y ago...and you're basing this on what, exactly? It's easy to pontificate about what people "didn't bother to test" based on zero information.
- lugg 11y agoFirst bug: In a failure case, it should remove the failing config, not all of them. Pretty hard thing to miss if you test for it with any level of basic unit test or similar. Second bug: canary failure should prevent further propogation of the bad config. A little more difficult to test with automated tests due to requiring a connection. It sounds like this was in fact tested, but the usage between the two bits of software was not tested. A good integration test would have caught this. But I wouldn't call that required. I would at least however think it was required that the use case of that particular code to be at least manually checked because, you know it's a feature for disaster prevention / recovery. There was enough information to deduce this pretty easily. Although they did tend to glaze over it in the write-up, almost purposefully. For all those spouting that this was a good postmortem, not really, it's a good covering of ones ass, a good spin, sidestepping the real root cause. What has slas and "here take credits" got to do with a postmortem? I'm not really sure why I got downvoted for this. The post mortem was good but it wasn't something I'd aim to strive for. I like gcloud and I'll keep using it but I find the response to this thing a little bit hard to swallow.
- magicalist 11y ago> I'm not really sure why I got downvoted for this. Because you have an apparently incredibly simple mental model for the system and so of course tests for it seem simple?
- robmcm 11y agoThe key thing to remember is a bug in autonomous driving doesn't mean the car swerves off the road at 100mph. If the software crashes or fails the car can come to a stop quite quickly without harm and allowing for human intervention. Having said that I am still scared, I'm not sure how well Tesla auto pilot will handle a tire blowout at 70mph. Perhaps better than I would, but I would much rather I was in control.