12 ms·
Lessons from the PG&E outage
- ChrisArchitect 10mo agoRelated context: Waymo halts service during S.F. blackout after causing traffic jams https://news.ycombinator.com/item?id=46342412 https://news.ycombinator.com/item?id=46342412
- jtchang 10mo agoHow is this mode not a standard part of their disaster recovery plan? Especially in sf and the bay area they need to assume an earthquake is going to take out a lot of infrastructure. Did they not take into account this would happen?
- vlovich123 10mo ago> While we successfully traversed more than 7,000 dark signals on Saturday, the outage created a concentrated spike in these requests. This created a backlog that, in some cases, led to response delays contributing to congestion on already-overwhelmed streets. We established these confirmation protocols out of an abundance of caution during our early deployment, and we are now refining them to match our current scale. While this strategy was effective during smaller outages, we are now implementing fleet-wide updates that provide the Driver with specific power outage context, allowing it to navigate more decisively. Sounds like it was and you’re not correctly understanding the complexity of running this at scale.
- jeroenhd 10mo agoSounds like their disaster recovery plan was insufficient, intensified traffic jams in already congested areas because of "backlog", and is now being fixed to support the current scale. The fact this backlog created issues indicates that it's perhaps Waymo that doesn't understand the complexity of running at that scale, because their systems got overwhelmed.
- shafyy 10mo agoThey probably do, they just don't give a shit. It's still the "move fast and break things" mindset. Internalize profits but externalize failures to be carried by the public. Will there be legal consequences for Waymo (i.e. fines?) for this? Probably not...
- atherton94027 10mo agoWaymo cars have been proven safer than human drivers in California. At the same time, 40k people die each year in the US in car accidents caused by human drivers. I'm very happy they're moving fast so hopefully fewer people die in the future
- browningstreet 10mo agoWhat Waymo profits? They're one-of-one still. Having ridden in a Waymo many times, there's very little "move fast and break things" leaking in the experience. They can simulate power outages as much as they want (testing) but the production break had some surprises. This is a technical forum.. most of us have been there.. bad things happened, plans weren't sufficient, we can measure their response on the next iteration in terms of how they respond to production insufficiencies in the next event. Also, culturally speaking, "they suck" isn't really a working response to an RCA.
- Animats 10mo agoIf the onboard software has detected an unusual situation it doesn't understand, moving may be a bad idea. Possible problems requiring a management decision include flooding, fires, earthquakes, riots, street parties, power outages, building collapses... Handling all that onboard is tough. For different situations, a nearby "safe place" to stop varies. The control center doesn't do remote driving, says Waymo. They provide hints, probably along the lines of "back out, turn around, and get out of this area", or "clear the intersection, then stop and unload your passenger". Waymo didn't give much info. For example, is loss of contact with the control center a stop condition? After some number of seconds, probably. A car contacting the control center for assistance and not getting an answer is probably a stop condition. Apparently here they overloaded the control center. That's an indication that this really is automated. There's not one person per car back at HQ; probably far fewer than that. That's good for scaling.
- gorfian_robot 10mo agorelying on essentially remote dispatch to resolve these errors states is a disaster
- michaelt 10mo ago> For example, is loss of contact with the control center a stop condition? Almost certainly no - you don’t want the vehicle to enter a tunnel, then stop half way through due to a lack of cell signal. Rather, areas where signal dropouts are common would be made into no-go areas for route planning purposes.
- deleted 10mo ago[deleted]
- bgwalter 10mo ago[flagged]
- dash2 10mo agoPeople downvoting you may think that this is an uninteresting quibble: we may not find it very surprising that sometimes Waymo asks for human guidance, and we don't necessarily think "autonomous" is an all or nothing designator.
- bgwalter 10mo agoDefinition in the Oxford dictionary: "Of, pertaining to, or characterized by autonomy; self-governing, independent; free of external influence or control." Self-driving car advertisers like Musk or Waymo just want to co-opt this term because it sounds cool. The term also deliberately hides the fact that these vehicles surveil and track you. EDIT: It is the full definition in the printed Shorter Oxford English Dictionary (which is a large two volume publication). It is understandable that morons downvote it.
- ssl-3 10mo agoIn this context, I think I prefer to use this definition from the Oxford Advanced Learner's Dictionary: "(of a vehicle) that has the technology to drive itself without a person in control" I think it fits the state of affairs well-enough. https://www.oxfordlearnersdictionaries.com/us/definition/english/autonomous https://www.oxfordlearnersdictionaries.com/us/definition/eng...
- bgwalter 10mo agoThat is a sad state of affairs, I hope it does not make it into the printed edition. The same applies to "autonomous drones", which are the most remote assisted machines imaginable. But of course the advertising departments want to evoke an image of the Marlboro man saddling his horse rather than a GPS tracked, surveillance riddled, face scanning, remote assisted contraption.
- rdiddly 10mo agoNo one seems sufficiently outraged that a private company's equipment blocked the public roads during an emergency.
- doctorpangloss 10mo agoOn the contrary, I would prefer HN detach all threads expressing "concern." That way we don't have to make a subjective call if a comment is "concern" or "concern trolling" at all - they are equally uninteresting and do not advance curiosity.
- ACCount37 10mo agoBased. Anyone complaining about HN being "insufficiently outraged" should go to Twitter and never return.
- rdiddly 10mo agoI was actually wondering more about the people whose streets they are. Didn't mean to indicate that I or anyone cares what HN thinks.
- rcxdude 10mo agoThe internet lens tends to distort what's happening on the ground quite a lot. I would expect the people living there have different things to direct their ire to.
- tengbretson 10mo ago> No one seems sufficiently outraged Harvesting outrage is about the only reliable function the internet seems to have at this point. You're not seeing enough of it?
- rdiddly 10mo agoI've seen plenty but about the wrong things.
- Papazsazsa 10mo agoThe symbolic irony of this situation is almost too rich to bear.
- xnx 10mo agoInteresting that some legacy safety/precaution code caused more timid and disruptive driving behavior than the current software route planner would've chosen on its own.
- prpl 10mo agoI suspected this. They were moving, but randomly to an observer. I’d seen about 2 out of maybe 20 stopped Waymos navigating around Arguello and Geary area in SF Saturday at 6PM. What was worse was that there was little to no connectivity service across all 3 main providers deeper in the power outage area as well - Spruce and Geary or west of Park Presidio (I have 2 phones, with Google Fi/T-Mobile, AT&T, and Verizon).
- black_13 10mo ago[dead]
- ec109685 10mo agoDo Waymo’s have Starlink or another satellite based provider backup? Otherwise, what do they if cell service goes down and they need to phone home for confirmation?
- apexalpha 10mo agoCell services is usually around for a while when power goes down. I doubt they have more than that.
- trollbridge 10mo agoThat seems like a major oversight. Adding Starlink wouldn’t add that much marginal cost.
- voidUpdate 10mo ago> "the resulting congestion required law enforcement to manually manage intersections" Does anyone know if a Waymo vehicle will actually respond to a LEO giving directions at a dark intersection, or if it will just disregard them in favour of treating it as a 4 way stop?
- fc417fc802 10mo agoI suddenly find that I really want an answer to this as well because I'm now imagining what might ensue if one of these attempted to board a car ferry. Typically there's a sign "turn headlights off", you're expected to maintain something like 5 mph (the flow of traffic should never stop), and you get directed by a human to cross multiple lane markings often deviating from the path that the vehicle immediately in front of you took.
- voidUpdate 10mo agoI think that Waymo isn't concerned about those types of scenario because they only operate in a limited area, and can tune their systems to operate best in that area (EG not worrying about car ferries, human-operated parking lots etc)
- jvanderbot 10mo agoRight. People still imagine that Level 5 is going to happen, and it is at best a long way off. You're talking full AI at some point.
- nostrademons 10mo agoCar ferries don't really make much sense in a Waymo-ubiquitous world. It's not your vehicle; there isn't really a reason why you would need to have the same vehicle on the other side of ferry ride. You're better off having one Waymo network on one side of the waterway, a separate Waymo network on the other side, and then a passenger-only ferry with a much higher passenger capacity (and oftentimes, they go much faster, since you can have hull forms like wave-piercing catamarans, hydrofoils, and hovercraft when you aren't carrying cars).
- moomoo11 10mo agoTesla FSD would never have this issue according to Elon Musk.
- bagels 10mo ago- written from my flying roadster
- joshribakoff 10mo agoThis reads to me, an angry resident, as an AI generated article that attempts to leverage the chaos that they caused, for marketing purposes — not as any sort of genuine remorse — underscoring why we shouldn’t be banning AI regulation in the USA.
- nzoschke 10mo ago> we are now implementing fleet-wide updates That ~1000 drivers on the road are all better trained on what to do in the next power outage is incredible. There will always be unexpected events and mistakes made on the roads. Continual improvement that is locked in algorithmically across the entire fleet is way better than any individual driver's learning / training / behaviorior changes.
- imoverclocked 10mo agoHumans seemed to navigate this just fine, even with all the Waymo road blocks and without extra training. If every unknown requires a software update, this system is doomed to repeat this behavior over and over in the long term.
- charcircuit 10mo ago>seemed to navigate this just fine From my understanding the reason the Waymos didn't handle this was because humans were breaking traffic rules and going when they shouldn't have been. If most humans navigated it correctly, then waynos would have handled this better.
- potato3732842 10mo agoThis attitude is exactly how the Waymos came to handle the problem so poorly in the first place. The principal Skinner "everyone else else is wrong" bit is just icing on the cake. Can't just program it to be all "when confused copy others" because it will invariably violate the letter of the law and people will screech. So they pick the legally safe but obviously not effective option, have it behave like a teenager on day 1 of drivers ed and basically freeze up. Of course that most certainly does not scale whatsoever, but it covers their asses so it's what they gotta do.
- bluGill 10mo agotraffic safety engineers often have influence on the letter of the law. We would all be better off if people followed it (humans are bad judges of the exceptions)
- yonran 10mo agoThe blog post makes no mention of the cellular network congestion/dropped packets that affected people during the power outage. I had bars but was unable to load websites for most of the day. Were Waymos unaffected by the network problems, or were request timeouts encompassed in the word “backlog” used by the blog post?
- AlotOfReading 10mo agoThe networking on AVs is usually redundant across multiple cellular networks to deal with coverage and outage issues. They also use business sims, which usually have a slightly higher network priority than consumers. If waymo's also negotiated to use one of the infrastructure QCIs instead, it would take some seriously disastrous network conditions for them to experience meaningful congestion.
- fowl2 10mo agoI wonder if the cars also have some sort of mesh (LoRaWAN?) network to help each other out in temporary dead zones, emergencies, etc.
- AlotOfReading 10mo agoBandwidth is problematic for mesh. A remote assistance situation would involve at least 4 high definition camera streams, and ideally with minimal latency. It's challenging to put that much data onto public spectrum even if you wanted to make a custom radio.
- fowl2 10mo agoI was thinking more for communicating the “mass disruption in progress” message - or even just distributing the manually verified “traffic lights down” data between vehicles.
- schwede 10mo agoSending power outage context to the vehicles does not seem like enough of a response. I hope at least they have internal plans for more. For large, complex systems, you want multiple layers of protections. The response feels way too reactive when they could use this incident to guide improvements across the board.
- hnburnsy 10mo ago>The situation was severe enough that the San Francisco Department of Emergency Management advised residents to stay home, underscoring the extraordinary nature of the weekend’s disruptions. Waymo cannot point to this as an extenuating circumatance when they where a major contributing factor.
- WorldPeas 10mo agoPardon my being under-informed, but does anyone know why Civic Center, the Presidio, the Park, and the Golden Gate were all dark the longest? Was there some separated municipal circuit they were on that was restored last as it was more complicated? Entered the thread thinking there would be more discussion on the actual architectural mishaps of the grid here rather than those of Waymo alone.
- srhack 10mo agoThe way all Waymos are updated to learn from this incident reminds me of Pluribus.
- Schiendelman 10mo agoPluribus is certainly in part a commentary on software machine learning models!
- cantalopes 10mo agoI find always find it kind of funny when an article starts with "At [company], our mission is to be the world’s [the product outranking competitors in its domain]". I mean, come on, unless you really are a nonprofit trying to save the planet or something (no, building a better X is not saving the planet), your mission is to get rich and raise into a monopoly in your field