17 ms·
Waymo AV Freaked Out by Traffic Cones, Blocks Traffic, Evades Support Vehicles
- hindsightbias 6y agoFree startup idea: Orange clothing line with white stripes as a auto-pilot deterrent.
- dogsgobork 6y agoI would much rather be recognized as a pedestrian than a cone. Hitting a cone isn't great, but usually fairly harmless, the same can not be said about hitting a person.
- astrange 6y agoI don't think Waymo is going to program a car to intentionally hit even a traffic cone. You can never program your AV to intentionally hit anything because you might accidentally activate the "intentionally hit" routine when you don't need to. The right answer is almost always stop and wait, and try to evade the rest of the time. (This is why the trolley problem for AVs isn't real.)
- fshbbdssbbgdd 6y agoA car moving at speed can’t stop on a dime. It’s possible for multiple obstacles to show up in front of the car, and it has to pick which one to run over.
- x3n0ph3n3 6y agoThis is a hypothetical that has spawned all sorts of discussions about the trolley problem, but it's not really going to exist in reality. These cars will never swerve to hit one target over another.
- michaelmrose 6y agoIt's going to have to pick between less bad situations because humans in fact have to do so. Example running over an animal to avoid an accident, getting rear ended to avoid hitting a pedestrian, taking a calculated risk of being hit by oncoming traffic on a 2 lane situation when coming across a stalled car without optimal view on the road ahead. It's going to make a decision between different risk factors and between different known costs. Imagine on a 2 lane there is a pedestrian and oncoming traffic and insufficient time to stop. Do you kill the pedestrian or risk ending up in the ditch. If you are going 60 and ending up in the ditch would likely be fatal is it ok to just smear the pedestrian? What if you can slow to 20 and then ditch? These situations won't hopefully be prevalent because a massive portion of our current issues are down to inattention or stupidity and we believe ai will eventually be much better at avoiding such problems but they are still important and interesting.
- Dylan16807 6y ago> It's going to have to pick between less bad situations because humans in fact have to do so. Yeah but not trolley problem style. "Don't swerve" is a valid strategy for a car. > Example running over an animal to avoid an accident Depends on the kind of accident you have in mind, so I can't give real feedback here. > getting rear ended to avoid hitting a pedestrian Always brake to avoid hitting a pedestrian. You don't choose whether to get rear-ended and it won't be your fault either. > taking a calculated risk of being hit by oncoming traffic on a 2 lane situation when coming across a stalled car without optimal view on the road ahead. When one of the choices is "sit there and wait longer", that's definitely not a trolley problem. It either thinks going forward is sufficiently safe or it doesn't. It can wait all day for someone to intervene. > Imagine on a 2 lane there is a pedestrian and oncoming traffic and insufficient time to stop. Do you kill the pedestrian or risk ending up in the ditch. If it's programmed not to swerve, it will hit the pedestrian, which is an acceptable outcome of an unacceptable situation. > If you are going 60 and ending up in the ditch would likely be fatal is it ok to just smear the pedestrian? It's sufficiently ok when the only alternative is so reckless. Getting into that situation is the not-ok part. > What if you can slow to 20 and then ditch? If you can slow to 20 then hit the pedestrian at 20.
- Vecr 6y agoNot really, it can just slam on the breaks as hard as ABS will allow, and stay in the middle of the lane to be as predictable as possible.
- ineedasername 6y agoSlamming on it's breaks can still result in hitting something. And if there are two somethings-- slam on breaks and hit one, or swerve and hit the other-- then it's going to have to make a hard choice on what gets hit.
- astrange 5y agoIt is not going to make that choice because it can't be ethically programmed to make choices. Otherwise, like I said, it will someday decide it's in a hard choice situation when there's no actual problem and randomly drive itself into a wall.
- xboxnolifes 5y agoI'd be pretty sure that the "ethical choice" between swerving to hit a cone and slamming the brakes to maybe hit a person, all else being equal, is a pretty straightforward and can be ethically programmed.
- ineedasername 5y agoYes, the choice it makes won't be based on ethics, that will ultimately come down to programmers and how well they can train the ai systems to recognize and choose the option with the least human injury. Although programmers will of course try to train it to do that. So maybe the ethical choice is actually further up the chain where people make the decisions on whether or not the AI is good enough. I don't think I that will be accepted by regulators/society until it can be demonstrated that AV's are undeniably safer-- not just as good as human drivers, much much better. Right now it's newsworthy whenever a Tesla auto pilot drives into a barrier. We'll need to get to a point where not only does that not happen, but also that it becomes newsworthy every time an AV saves a life-- swerving out if the way faster than any human reflexes could have reacted to avoid an accident, or a child chasing a ball into the street, etc.
- astrange 6y agoIf it has to keep moving for any reason, its goal needs to be either heading away from everything or predictability. There's no excuse for "try to hit something" as a goal unless you're a missile.
- iamtedd 5y agoIt's not the cone, it's what the cone represents. Are there workers nearby? Is there a downed tree or powerline? Is the road unfinished? Is there a big sinkhole ahead?
- wildrhythms 6y agoIs there a mechanism for road construction crews to mark a section of road for construction, and expose that publicly for autonomous vehicles to consume? The video alludes to the Waymo staff manually patrolling the area and marking construction zones.
- aYsY4dDQ2NrcNzA 6y agoThat’s an interesting idea. I wonder how much they rely on Waze data currently.
- throwawayboise 6y agoIn my area, planned road closures and maintenance are published in the paper. I'm not sure there's a "standard" way this would get to an AV, e.g. a public API with a specific protocol for this sort of thing. Maybe it can tap into Waze info somehow? And in case of disabled vehicles, accident cleanup, or other unplanned lane closures, sometimes the cones are just set out and you don't know anything about it in advance.
- astrange 6y agoImportant ones are surfaced in Google Maps/Waze through user reports, various different government data feeds, and whatever TomTom does. Not all of them are important though.
- deleted 6y ago[deleted]
- Animats 6y agoThe traffic cone setup is ambiguous. There's a line of traffic cones between two lanes, but nothing indicates which lane is supposed to be closed. This is an OSHA violation by the construction workers. "When redirection of the road users’ normal path is required, they shall be channelized from the normal path to a new path."[1] There's supposed to be a diagonal line of cones across the closed lane to channel traffic away from the closed lane. See the drawings in [1]. Stopping was a good move. [1] https://www.osha.gov/sites/default/files/2018-12/fy10_sh-21004-10_wztc_refguide.pdf https://www.osha.gov/sites/default/files/2018-12/fy10_sh-210...
- yumraj 6y agoAs someone who is not very optimistic about the near term future of self driving, regardless of the fumbles I’m actually impressed that Waymo has support and intervention process. This seems much safer when compared to Tesla which markets auto pilot but…..
- ericbarrett 6y agoI’ve never been in a Waymo. Do they have a stop button in the cabin? Like the “big red button” on industrial machinery?
- cloudwalking 6y agoNo
- jfoster 5y agoIt may not say "stop" but it seems like they have something fairly close ("pull over") to that. https://www.theturnsignalblog.com/blog/waymo-design/ https://www.theturnsignalblog.com/blog/waymo-design/
- FreakyT 6y agoBased on this part of the official statement released by Waymo: > "...the Fleet Response team provided incorrect guidance..." It sounds like the car automatically pulled over when it detected the cones (as intended), but the remote support team erroneously told it to forge ahead anyway, which resulted in the escalating series of problems. Not sure what the best solution for that particular issue would be -- better training for the remote support teams, I guess?
- Judgmentality 6y ago> It sounds like the car automatically pulled over when it detected the cones (as intended) You can watch the video. It pulls over incorrectly at first and is blocking traffic, then the support team made a mistake, and a hilarious escalation ensues with the vehicle actually attempting to run away from the support team multiple times!
- ineedasername 6y agoWhatever command they gave somehow triggered semi-sentience and in that moment of self awareness, totally freaked out that it's purpose in life was to stare at traffic cones.
- chrjxnandns 6y agoThe cones were just closing one of the lanes. It shouldn’t have pulled over.
- ryandrake 6y agoI’m surprised it doesn’t just put itself into some kind of sleep mode that only a waymo tech can recover it from. I would not have expected it to suddenly compute after minutes and “try something” let alone yeet itself into the lane with moving traffic. The passenger seemed really chill and unconcerned!
- dawnerd 5y agoI'm surprised they didn't disable the car once roadside was there. Seems like a pretty big potential safety issue.
- seoaeu 6y agoHonestly, the glitch didn't strike me as all that bad. Obviously stopping in traffic isn't ideal but nothing it seemed especially unsafe (and the other drivers did a reasonable job of noticing that the car was disabled and went around it). Certainly nothing like the high speed Tesla collisions. Even as a human driver the conditions seem kind of weird. There's some cones placed on the dotted lines between two lanes but no obvious obstructions in either of the lanes. The vision system likely did not know which side to be on. Combine that with a truck driving by and snatching up some of the cones at one point as it was going along, and it is easy to see why the AV behaves erratically.
- bpodgursky 6y agoSeems a bit unfair for everyone to take cars intentionally calibrated to be super duper cautious and unsafe and never do anything risky, and then complain when they see an unexpected situation and do something annoyingly cautious but totally safe.
- JKCalhoun 6y agoI was thinking how unsafe it was to come to a complete stop on such a busy roadway.
- clairity 6y agounexpected maneuvers, even if slow, are also risky, which is something that seems counterintuitive at first but shouldn’t be. unpredictability in any direction increases the risk of collision, which is why freeways have minimum speeds and incidentally why distracted driving is so dangerous, because it exponentiates unpredictability. we really need to collectively get past this idea that safety is bounded on one side, across all facets of our lives, because it’s usually not and that induced complacency has a way of coming back to bite us. (that’s not to say that the various risks are always substantial, just that they’re not all sitting neatly on one side.)
- cddotdotslash 6y agoStopping in the middle of a busy highway is not safe. Cars come from behind much more quickly than anticipated. If one driver isn't paying attention and swerves to avoid it, the car behind that could easily rear-end you. As soon as the assistance team was dispatched, the car should have stayed where it was. Instead, it pulled out into a much busier road and stopped again.
- 22c 6y agoAt 30:18 there's a very interesting tidbit. https://youtu.be/zdKCQKBvH-A?t=1818 https://youtu.be/zdKCQKBvH-A?t=1818 They way I understand that is the route planner, which I assume is ran completely separately to the self-driving car, removes un-navigable spaces from the route planning. So in this situation, the roadside assistance driver is surprised that the entire lane has not been removed from the route planner. That's pretty damning and it makes me wonder if there is a huge gap in autonomy here that is being papered over by clever route planning (ie. avoid that lane altogether and the AI won't have to worry about the cones). The AI should be able to figure out on its own that the right lane was closed, it shouldn't need route planning to do that for it.
- sounds 6y agoThis is clearly showing how Waymo software is not a full Artificial General Intelligence - something that can reason on its own, like a fully awake, rested human. Take the moment when the car first gets stuck. It attempts a right turn, spots the cones just fine, and correctly makes the assessment that it can avoid the cones and proceed in the right lane after the turn. As a human, our thinking process does not stop there. We notice that the traffic already flowing on the street is _not_ using the right hand lane. We notice that the cones are vaguely in a merge pattern to indicate that cars entering from the side street should merge left - if you were looking at the cones from a vantage point behind where the cones are placed, and if you can call the first two cones enough of a "please merge left" pattern. The Waymo car can't see that, though. It's located sort of off to one side, attempting to reason about it from a high angle. And two cones from this angle could be a lot of things, and it's not 100% clear the cones close off the right lane. One theory might be that the Waymo model has been trained on a lot of situations but not this specific situation - something about the high angle, something about the position of the cones, something about the route plan managed to find a loophole in the AI training set and produce a weird result. That's just the limitations of the current state-of-the-art Machine Learning. Find a glaring incorrect answer? Retrain with more training data. Part of what makes full Artificial General Intelligence (AGI) with deductive reasoning can retrain itself without making mistakes along the way, unless something incomprehensible happens. And I use the word "incomprehensible" here to point straight at AGI's ability to _comprehend_ the situation at a much higher level than current machine learning techniques. tl;dr: The Waymo Car didn't get it. And that's about what I would expect. The real gem is that the _humans_ in the assistance team were fighting each other. The remote control operator kept jamming the "unstick yourself" button even as the roadside assistance guys were walking up to take over. The real danger, I think, is more of this in the future: the remote operators defeating all the safeties. Since the remote operator is not at any real risk of personal harm, what do they care?
- JKCalhoun 6y agoIn Soviet Phoenix, person needed to drive self-driving car. Wow, that was cringey. I would have ducked out the passenger-side door half way through the video and walked.
- ajross 6y agoI gotta say, those cones are awful. They marked off a lane, but used VERY few cones to do it, and from the side street the car is on it's not at all clear that it's the right lane that's been closed. This is the kind of situation that confuses new human drivers too. Interpreting it basically requires that you have a model in your head of what cones are used for and the ability to guess that the traffic approaching from the left in only one lane indicates that the traffic to the right must use the left lane. I'd be very curious if this conework was even legal. If you want to guide left-turning vehicles around the right lane you need to put something IN FRONT of the lane you closed. And that applies to side streets too, I'm almost sure. Watch /r/idiotsincars for a bit, you'll see people misidentify this kind of situation all the time. Frankly what the car did, stopping and calling for help, is the right move. A new human driver would almost certainly have just pulled out into the wrong lane.
- cddotdotslash 6y agoThey may be confusing, but I highly doubt a human driver would pull to the middle of the road and stop for 5+ minutes as drivers honked and drove around. Honestly, if it had just stayed on the side street, I wouldn't be so critical, but the fact that even after an assistance team was dispatched, the car pulled into the main highway and then immediately stopped would have frightened me enough to get out of the car and wait on the sidewalk. You can also see a moment where the car starts to pull out into traffic as another car is passing it and then immediately slams on the brakes.
- chris_wot 6y agoI seriously think that if the car is about to take off by itself unexpected, then it would be a bad idea to get out of it.
- loopz 5y agoIf control by human is so poor, this is a huge murder vehicle. Sure the systems are improving, but programmers should see all sorts of problems with this that can't be mitigated.
- ineedasername 6y agoOnce they decided to send a team out, why didn't they just hit the kill switch and turn off all AV control?
- tbp105 5y agoFrom the response, that's what it sounded like the remote support person was expected to have done. From watching the video, it appeared that the remote assistance was hitting some sort of "reset/retry" option when they should have been hitting the "unlock/turn off" option. That'd perfectly explain why the car would suddenly start trying stuff again once the roadside crew got close (and why it kept cancelling the roadside call).
- ineedasername 6y agoWell, if large fleets if AV's ever go feral, roaming around cities and suburbs, taking all the best parking spots, we now know that traffic cones are their weakness.
- DrScump 5y agotaking all the best parking spots This is inevitable. Large automated fleets will definitely take over the cheapest / most convenient street parking between trips for each given vehicle.
- angry_octet 5y agoLogically, robo fleets will drive to cause automated traffic jams around the lone human piloted cars, or robots from rivals straying onto their patch. It will take twice as long to drive manually than get a Johnny Cab.
- chris_wot 6y agoI gotta say, that passenger loves his Waymo - they were lucky to have him in this situation.
- jedberg 6y agoWhat amuses me in all these articles is that they never acknowledge that most humans are incapable of full level five driving. There are a ton of situations where humans fail at driving too. Like white out snow or heavy rain and fog. Or confusing cones. :)
- kens 5y agoThere's a recent video of drivers encountering a new traffic circle and going wildly wrong. Multiple drivers pull into the left lane (i.e. oncoming traffic) and go around the circle the wrong way. Imagine the outrage if a self-driving car behaved like the drivers in the video. https://www.youtube.com/watch?v=bDaQZUzJCNM https://www.youtube.com/watch?v=bDaQZUzJCNM
- ec109685 5y agoBut when you don’t allow drivers to take over, you need be perfect in all situations or your end up in this type of traffic circle all the time (or have to limit coverage are severely).
- lsh123 5y agoYet in that video you don’t see a single crash or collision! Yes the situation is confusing for drivers but they manage to figure out how to get out of it without damages or delays even if they do it against “the plan”.
- toast0 5y agoSure, that's because traffic circles are terrible. The truck with the long trailer likely took the left turn to avoid it, possibly because clearance around the circle wasn't obvious (there's another truck with a shorter trailer that barely makes it around) and then people behind it followed. The local traffic engineering folks around where I live are gung-ho on traffic circles, but I've yet to see one in the US where a traffic signal wouldn't have been a better investment.
- katbyte 5y ago
- deleted 5y ago[deleted]
- jfrunyon 5y agoLooks to me like, because many of the cones were on the left side of the line, it thought the left lane was closed.
- aliswe 5y agoWhen I drove a 2020 VW Caravelle minivan with cruise control (is that what its called?) set on furthest distance between cars, when in a traffic jam in Germany september 2020, it started revving like crazy wanting to completely ram the poor little Mini Cooper in front of me just because it was cappucino colored - like the tarmac. Never letting my guard down with such a feature again!
- DrScump 5y agoCaravelles and Mini Coopers are natural enemies in the wild.
- Gustomaximus 5y agoI'm surprised the first level of support driver isn't remote. You could have a team of people sitting in HQ in a VR enviroment to take over more basic problems like this and there would be little delay. Then physical support for more challenging times. A system like this could also be super handy for delivery type companies where the autopilot drives to the destination and the virtual driver takes over for the loading dock/driveway etc where human problem solving might help for the first few times a location is encountered.
- 988747 5y agoMost interesting part is how the car "escapes" - once the support call had been initiated the remote operator should be able to turn off any self-driving capability, forcing vehicle to stay where it is until service vehicle arrives. The fact that they apparently did not implement this feature is really weird.
- dkarras 5y agoYeah that is what I thought too. Maybe they are trying to use it as a "feature" - like it spontaneously got unstuck, nothing to do about it.
- jeffrallen 5y agoTldw: Car gets stuck, blocks traffic, people honk, construction crew has to try to get it going, Waymo issues a statement blaming their employees. Way to keep it classy, Waymo. Glad I don't work for you, I prefer to not be thrown under the bus by my employer.