10 ms·
California halts Pony.ai's driverless testing permit after accident
- jzer0cool 5y agoWhat were requirements before getting the permit in first place? What is allowed/disallowed after obtaining permit?
- Jensson 5y ago> On Oct. 28, a Pony.ai vehicle operating in autonomous mode hit a road center divider and a traffic sign in Fremont after turning right This is really bad, if their software can't even avoid hitting a stationary object then it isn't ready to test on public roads.
- viraptor 5y agoIf it does that more than people then it's not ready. People hit signs and dividers all the time. It's hard to come to realistic conclusions based on one case.
- Jensson 5y agoA bad driver risks his own life so he has good reasons to try to improve. A bad self driving car program doesn't risk the founders life so he has no reason to improve it if regulations doesn't force him to.
- pishpash 5y agoIt risks the founder's net worth so there are some reasons to improve, maybe not enough.
- barbazoo 5y agoDepends on the details of course but I'd expect an autonomous vehicle to be much much better at not hitting things. Especially stationary objects. People are bad drivers because they have limited vision, limited attention, are emotional and are easily distracted by all kinds of things. Just because an autonomous car has a similarly bad track record as the average human, doesn't mean it's ready for the road.
- viraptor 5y agoIt may be much better. This is a single case and we get every single case reported. The problem is that computer vision is not the same as human vision and at some point we may end up with a situation close to "this car drives itself perfectly apart from running over a completely obvious, well visible person once a year". And it's going to be a really hard one to process for many people.
- therockspush 5y agoIs that same lever being applied to everyone thats testing this right now, or is this making an example? I know nothing about nothing, so it's a real question.
- KennyBlanken 5y agoWell, Tesla's autopilot has been slamming into emergency vehicles on the sides of highways for years and the feds have finally gotten around to investigating things. Their full self driving beta requires constant vigilance because it likes to randomly veer at pedestrians and light poles. As a sidenote, it's irritating that Reuters keeps calling it an "accident." It was a crash.
- TaylorAlexander 5y agoI interviewed at a self driving car startup in 2016 that was testing their early prototypes on public roads! They were also using their investor money to rent out some ten million dollar home in the Silicon Valley hills as their office. Big red flags all around. I declined their request for a second round interview.
- rosndo 5y ago>They were also using their investor money to rent out some ten million dollar home in the Silicon Valley hills as their office Is this supposed to be a bad thing? Why? The focus on the price of the house seems pretty strange in an area with a particularly high price-to-rent ratio. But I guess $20k/mo just doesn’t sound as spicy.
- Jensson 5y agoI think he meant that they rent out their own home to their own company to extract investor money from the company.
- rosndo 5y agoThat would be a very strange detail to leave out. But sure, it’s the only way their comment makes any sense. Otherwise, spending investor money on office space seems like a perfectly normal thing to do. The $ amount described doesn’t seem too unusual for the area, and a house might offer significant advantages over traditional office space for a startup that is working on cars.
- Jensson 5y agoAnther explanation could be that many startups try to give out the vibe of "just look at how much investor cash we can burn, come join in on the fun!", and are very open about that. I can understand why you wouldn't want to join such a company, even though that is exactly what many others are looking for.
- longcommonname 5y agoEvery other self driving company out there has the same type of incident.
- jowday 5y agoI think this one got revoked because the car was actually in autonomous mode without a safety driver in the car.
- Jensson 5y agoAfter they got a permit to drive without a safety driver? I don't think so. The permit should be for when the technology is mature enough to never do these kinds of mistakes.
- threeseed 5y ago> This is really bad, if their software can't even avoid hitting a stationary object then it isn't ready to test on public roads. Just slap a Beta tag on it and she'll be right.
- deleted 5y ago[deleted]
- randmeerkat 5y ago> This is really bad, if their software can't even avoid hitting a stationary object then it isn't ready to test on public roads. Almost as bad as a Tesla crashing into a stopped fire truck. Why is Tesla “autopilot” still allowed on Californian roads? https://www.wired.com/story/tesla-autopilot-why-crash-radar/ https://www.wired.com/story/tesla-autopilot-why-crash-radar/
- jeremyjh 5y agoWell, it wasn't driverless. The driver was arrested for DUI. Its not even known in that article that autopilot was turned on.
- randmeerkat 5y agoThe article goes on to say that two other drivers who said they had autopilot on also crashed into fire trucks… How many fire trucks do Teslas need to crash into for it to be considered a problem?
- dawnerd 5y agoOne could argue that normal cruise control on normal cars would also crash into fire trucks. Ultimately drivers are responsible in both cases, it's their fault 100% end of story. PS Tesla rolled a software update to detect emergency vehicles and slow down.
- mikeyouse 5y agoThe cruise control on my Subaru probably wouldn't hit a fire truck since it has pre-collision braking.. but in any case, it's not sold as "Autopilot" or "Full Self Driving" so the limitations are pretty clear. There's a reason several countries stopped Tesla from marketing their nonsense like they do in the US - it turns out when you say things are self driving, consumers believe you and don't read the fine print. https://www.reuters.com/article/us-tesla-autopilot-germany/german-court-bans-tesla-ads-claiming-its-cars-are-capable-of-autonomous-driving-idUSKCN24F1T5 https://www.reuters.com/article/us-tesla-autopilot-germany/g...
- jakear 5y agoIt's awe-inspiring how much more quickly a human brain can learn versus a computer. These companies have expended tens of thousands of engineer-hours of some one the best minds in the industry to build models derived from tens of thousands of hours of training data, and still I'd place money on a novice human driver with just ~100 hours of experience besting any AI in the majority of "strange" driving circumstances. Of course, if that human brain is tired, stressed, and checking their phone, it's a whole different story :)
- Jensson 5y agoThe driving part is usually normal algorithm code and not machine learning though. Machine learning is for object detection, what the software does with that knowledge is regular programming.
- dundarious 5y agoVery much so. The tens of thousands of hours of training is like the learning babies do while looking at jangling keys. Then the driving part is more static -- I'm not sure that it's entirely regular programming, there may be some training of models there. But my understanding is it's certainly not learning/training _while_ driving, and _almost_ certainly it's not even learning to drive. It's just doing more learning for object detection and tracking, like a baby watching the world from a pram. So I feel like no matter how many autonomous hours these cars accumulate on the road, they have effectively 0 hours of "learning to drive".
- Jensson 5y ago> I'm not sure that it's entirely regular programming, there may be some training of models there I doubt it, not anything serious at least. Machine learning isn't at the level where it can be used to navigate environments better than human coded algorithms can. Maybe if you had a datacenter worth of computing power to do it, like AlphaStar, but you don't have that in a car. And even if you had that I still doubt it, Alphastar had a perfect representation of the game and could simulate the effects of moves perfectly.
- pavel_lishin 5y agoAren't there entire abandoned cities, and yet-to-be-filled suburb developments in the world? Are autonomous driving companies not using those, stuffed full of "stunt" drivers who are employed to do nothing but simulate regular city traffic?
- xvedejas 5y agoWhat would that accomplish? The goal is for these cars to self-drive in locations where people live, and that does include mapping those locations over many drives. Autonomous vehicles still have a good safety record, and California's willingness to suspend a permit when nobody was injured does signal to me that they'll take it very seriously when a company's safety record falters.
- Syonyk 5y agoThe current state of self driving cars can't handle basic, well behaved, standard traffic. There's no way it's ready for the sort of adversarial stuff you'd get hiring a bunch of people to drive around simulating traffic for a car. I guarantee it's a great way to find out how your car is at dealing with being cut off, at people running stop signs, etc - and there's certainly value in doing that. Just, not while you still can't avoid hitting stationary objects along the edge of the road.
- KennyBlanken 5y ago> The current state of self driving cars can't handle basic, well behaved, standard traffic. Some self-driving cars can't. At one end, you have the Uber self-driving vehicles which were caught committing numerous traffic offenses by dashcams and cyclist helmet cams within the first days of their illegally testing their fleet on public roads, and then one of their cars ran over a pedestrian that the vehicle's sensors detected but the computers misclassified, somehow - and Uber had disabled the SUV's own auto-braking systems that probably would have saved the woman's life. At the other end you have Google's self-driving cars have had, I think one at-fault crash across the entire fleet in many, many years and millions of miles of testing? They've been involved in a lot of crashes, but every single one until their one slow-speed scrape with a bus was the fault of another driver. That was 5 years ago or so. Somewhere in-between is Tesla's FSD "Beta" which, as has been shown by numerous youtube videos (at least, the ones that Tesla doesn't abuse DMCA to take down), behaves extremely erratically repeatedly in the space of less than half an hour in an urban environment. This includes veering off a turn at pedestrians on a curb corner, and in another incident veering toward a light pole. In general, it seems to have a lot of problems making turns.
- sydthrowaway 5y agoStill believe you need AGI for viable SDCs
- Jensson 5y agoDepends on how you define "viable self driving car". If that means it needs to drive at least as well as humans in every scenario you tend to use a car in, then yeah probably, there are so many places with overgrown roads, avoid getting stuck in sand etc. If however you just want it to navigate highways and city streets, then probably not. The self driving cars wont be perfect around people and human drivers, no, but pedestrians and human drivers can update their own behaviour to compensate. It took a while for pedestrians to understand how to behave around human drivers, the same thing will happen for autonomous cars.
- oblio 5y agoHumans are generally predictable, even crazy ones, since we get to interact with them from day 1, after birth. Plus in many places there are decent tests in place that block some of the crazies. Computers, not so much.
- Animats 5y agoHere's the crash report: "On October 28, 2021, after turning right onto Fremont Blvd from Cushing Pkwy, the Pony.ai Autonomous Vehicle ("Pony.ai AV") performed a left lane change maneuver in autonomous mode. While performing the lane change, the Pony.ai AV came into contact with a center divider on Fremont Blvd. and the traffic sign that was posted on the divider. The Pony.ai AV suffered moderate damage to the front of the vehicle and the undercarriage. There were no injuries and no other vehicles involved. Fremont Police Department were called to report the incident and the damaged street sign. Pony.ai has subsequently worked with local authorities to resolve all issues related to the damaged sign." It's not a serious incident, but it is a fully autonomous vehicle under good conditions going somewhere it was absolutely not supposed to go. So DMV revoked their license for full autonomous operation without a safety driver. They have to go back to testing with a safety driver. [1] https://www.dmv.ca.gov/portal/file/pony-ai_102821-pdf/ https://www.dmv.ca.gov/portal/file/pony-ai_102821-pdf/
- tyingq 5y agoStreetview positioned at a right turn from Cushing onto southbound Fremont Blvd: https://www.google.com/maps/@37.4905025,-121.9488087,3a,24.2y,134.28h,86.87t/data=!3m6!1e1!3m4!1s1QKyKisG_B_UQaEfmT9h1Q!2e0!7i16384!8i8192 https://www.google.com/maps/@37.4905025,-121.9488087,3a,24.2... You can see the curb rashed up and the pretty improvised fix to the sign from a previous accident I suppose, since the map says March, 2021. https://imgur.com/a/U8cokYd https://imgur.com/a/U8cokYd
- tlrobinson 5y agoI doubt that was where the impact was if the car was turning right from Cushing onto Freemont, since it’s 2 lanes over from the turning lane.
- ehnto 5y agoThe report said it performed a left lane change after the initial turn, so it could still be.
- 5y ago
- ck2 5y agoAgain, who is going to be put in jail when the first person is murdered by a driverless car? Because that's the only motivation for keeping things safe, if it's just a fine corporations will just cut a check and write it off and shrug. Cyclist on side of road, pedestrian wearing the wrong thing crossing the road, etc. Sure a human could have killed them too. But that human will be put on trial/jail.
- lazide 5y agoWell, technically murder requires a human assailant and intent, which the car won’t have. Unless the programmer intends it anyway, then they are the murderer. Same as a lion can’t murder a human. It’s just being a lion, doing lion things.
- DarkUranium 5y agoOn the other hand, crows commonly partake in murders. ((I had to.))
- Jensson 5y agoAlready happened: https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg The end result is that Uber ended their self driving car program and paid the family money, while the grunt worker who was put in the car without proper training got charged with negligent homicide and that trial is still going. The person responsible for that program should be charged, not the safety driver, but that is how things goes. You can't just put random people in a car and call them safety drivers, and charging such a person with homicide doesn't really get to the root of the problem.
- mavhc 5y agoThe problem is the justice system assumes 1 person kills 1 other person. In reality there were 100s of people in the chain of events, from the safety driver being bored and watching their phone, to the person who hired them and didn't train them properly, to the person who invented the safety driver system and didn't think people would get bored. Then you have the people who made the decision to turn off some of the subsystems of the car and then test it at night with no extra precautions, and every programmer who let that happen. The the management of the company who provided the wrong incentives to their employees. And that's before you get to the people who designed the roads, and the lack of social care to look after people who for some reason are homeless
- captainmuon 5y agoI believe the first generations of driverless cars have to be overly cautious. If it can't be 100% sure that it's model corresponds to reality, and that nothing is going to jump out of the bush, then it has to stop. A safe car is going to stop or slow down at least once per trip, for the forseeable future. I think the car should also talk about what it thinks, to get people to understand the limitations of the technology. "There could be a child between those parking cars, slowing down" "I cannot see the road markings, emergency stop" "Road signs don't match my data, slowing down" "Train mode activated, following virtual track" Then, and only then, you can do the second step. Automated cars have the potential to be much safer than cars with human drivers. You can accept some accidents, in order to prevent many more. But you can only do that if the public and lawmakers have confidence in the technology. Right now it is way too much of a black box with wierd failure modes.
- mayormcmatt 5y agoThis reminds me of a situation I saw with another autonomous car company's vehicle in SF a couple weeks back. I truly didn't know what the correct move by the vehicle should have been. The situation: The vehicle was on Vallejo crossing west to east over Sansome Street, a one-way, two-lane thoroughfare. For Sansome, there is no stop sign here, but there is for Vallejo. The vehicle stopped at its sign, then began to proceed across the intersection, until two pedestrians entered the crosswalk from south to north along the far side of Sansome. The vehicle stopped in the middle of the street to wait for the pedestrians to cross. There was no cross traffic coming, but what if there were? Seems like a situation ripe for a side collision, especially because Sansome is brisk at this point. I really don't know what the capabilities of the sensors are, or how situations are assessed, so I'm assuming it did what it did based on its situational awareness. Wondered if the vehicle could have backed up to the stop sign it came from if there was cross traffic, or if it could have broken the rules and gone around the pedestrians if there was danger of a side collision. The intersection: https://goo.gl/maps/TisMyYc1RhKgJWFf8 https://goo.gl/maps/TisMyYc1RhKgJWFf8 PS: there is a garage at that corner that almost exclusively contains Waymo vehicles, and this was not a Waymo car.
- darepublic 5y agowith all the billions being spent on fully self driving cars why doesn't a company build a fake indoor town where it can simulate various driving conditions, weather, lighting etc and put fake pedestrian everywhere and then have the AI be able to learn in an environment simultaneously more challenging (since you can throw all kinds of shit at it) and forgiving (since mistakes while training aren't hurting people).