8 ms·

Waymo recalls software after two self-driving cars hit the same truck

212 points by reteltech 3y ago

Fricken 3y ago
A pick-up truck being towed crooked and backwards. Both vehicles failed to read the situation in the same manner. Autonomous vehicles have various redundant systems built-in that can take priority and override false positives. I was previously under the assumption that one of the really important reasons for Lidar is that it can get you closer to an absolute truth about whether something is a solid object, and where that hypothetically solid object is relative to the position of the vehicle, regardless of what the classifier thinks it is seeing. So did the lidar fail to read the solid object, or was the lidar, was it de-prioritized? or was it simply not available as a fallback? Presumably Radar and proximity sensors were also involved. What were they doing? This is a fascinating edge case, and I hope to hear about the real reason for the 2 incidents.
- Mageek 3y ago
  The article says it: “We determined that due to the persistent orientation mismatch of the towed pickup truck and tow truck combination, the Waymo AV incorrectly predicted the future motion of the towed vehicle.” It was detected, but it predicted the truck would move in a way that it didn’t end up moving.
  - pests 3y ago
    How do you drive into a solid war if you have lidar though? To say nothing of predictions, the object is where it's at.... where it's at at that moment. You don't need to predict where it's at now... because you know where it's at.
    - avalys 3y ago
      It wasn’t a solid wall, it was another vehicle. The brakes don’t respond immediately - you need to be able to detect that a collision is imminent several seconds before it actually occurs. This means you have to also successfully exclude all the scenarios where you are very close to another car, but a collision is not imminent because the car will be out of the way by the time you get there. Yes, at some point before impact the Waymo probably figured out that it was about to collide. But not soon enough to do anything about it.
      - antisthenes 3y ago
        This is why autonomous vehicles need a hard cap on the speed limit they are driving at. If the max speed is 35mph, that allows a good braking system to respond by safely stopping from LIDAR info 99% of the time.
        lotsoweiners 3y ago
        Yeah but then people aren’t going to want to take them because they are slow.
        antisthenes 3y ago
        The average speed of a commuting car is around ~23mph, when you account for stops and red lights. 35mph is only a hindrance if you commute by freeway every day, which many people don't. People will take them if they're priced right and they can do things in the car without having to pay attention to the road.
        roxgib 3y ago
        But autonomous vehicles tend to perform best on freeways, that's where they are most likely to be allowed
      - Out_of_Characte 3y ago
        My brakes respond immediatly,for all intents and purposes as a human with a reaction time. I'm at fault if the person in front of me stops and I dont have the stopping distance to avoid a collision. I get that self driving software is difficult. But theres no excuse for this type of accident.
        awhitby 3y ago
        That might be true for the simple case of following within a lane, although you only have to drive around to realize most drivers do not leave adequate following distance at all times to make this a pure physics problem. And neither is a good driver watching only the car in front, but also the brake lights of the cars in front of that, to help anticipate the car in front's likely actions. But take an even slightly more complex example: you're on a two lane roadway and the car in the other lane changes into your lane, leaving inadequate stopping distance for you. You brake as hard as you safely can (maybe you have a too-close follower, too), but still there will be a few seconds when you could not, in fact, avert a collision if for some reason the car in front braked. I have no idea what the legal situation would be: is it their fault if the crash happens within 3 seconds but yours if it happens after you've had time but failed to restablish your needed stopping distance? Honestly even in the simple one lane case, I doubt you can slam your brakes on the interstate for no reason then expect to avoid any liability for the crash, blaming your follower for following too close. Driving has a bunch of rules, then an awful lot of common sense and social interaction on top of them to make things actually work.
    - bobsomers 3y ago
      You can't drive if you only use the current "frame" of data as the basis for your decision. Imagine driving on the highway, a comfortable distance behind a lead vehicle. The planning software would want to slam on the brakes without predicting that the blob of sensor data in front of you is going to continue moving forward at highway speeds. That motion prediction enables the planning software to know that the space in front of your vehicle will be unoccupied by the time you reach it. A similar prediction error was the reason Cruise rear ended the bendy bus in SF a while back. It segmented the front and rear halves of the bus as two separate entities rather than a connected one, and mispredicted the motion of the rear half of the bus.
      - hmcq6 3y ago
        > That motion prediction enables the planning software to know that the space in front of your vehicle will be unoccupied by the time you reach it. I think we're all on the same page about this part but what's confusing and hilarious is why would the correct answer ever be to drive into an unmoving object? If they tried to avoid the truck and swerved and hit a different vehicle there would be no confusion here. But the self driving algorithm is effectively committing suicide (Kamikaze). That's novel. My guess is that the self-driving car was not able to recognize the truck until it was very close and the sudden appearance of the truck is interpreted by the algorithm as if the truck is moving very fast. And the best answer in that case would be to let the truck pass (basically do what the waymo did). But that means the lidar information about the shape not moving is being deprioritized in favor of the recognized object being calculated to move fast. A situation which could only really occur if a speeding vehicle plowed through a stationary object. Fantastic solution for avoiding a situation like this -> https://www.youtube.com/watch?v=BbjjlvOxDYk https://www.youtube.com/watch?v=BbjjlvOxDYk But a bad solution for avoiding a stationary object.
        clhodapp 3y ago
        Who said it was an unmoving object? Maybe I missed something in the story, but I got the sense that this happened in motion. The towed truck would have essentially been moving forward at an angle, hanging across the lane boundary.
    - lazide 3y ago
      You can’t drive more than a few MPH unless you’re reacting based on the expected future, rather than the current one. It’s why it’s so difficult to do (actually) and the ability to do it well is just as much about the risk appetite of the one responsible as anything else - because knowing if a car is likely to pull out at the light into traffic, or how likely someone is to be hiding in a bush or not is really hard. But that is what humans deal with all the time while driving. Because no one can actually know the future, and predicting the future is fundamentally risky. And knowing when to hold ‘em, and when to fold ‘em is really more of an AGI type thing.
    - sokoloff 3y ago
      In self-driving, you are making predictions about where the object is right now based on the synthesis of data from your sensors (and often filtering information from past estimates of the object position). These might be high-precision, high-accuracy predictions, but they're predictions nonetheless. (It's been quite some years since I worked on vision-based self-driving, so my experience is non-zero but also quite dated.)
  - PH95VuimJjqBqy 3y ago
    I don't find that acceptable in any way, no human driver is going to do that and by that I mean no human driver is going to drive into something just because it moved in a way they didn't expect. they're going to slam on the brakes and the only way that's going to happen is if momentum is too high. I understand we have to have explanations or we can't fix them, but it's just as important to understand this should never have happened even WITH the described failure. If I had to guess, there's code to avoid stopping at every little thing and that code took precedence (otherwise rides would not be enjoyable). And I get the competing interests here but there must be a comparison to humans when these incidents happen.
    - duped 3y ago
      > no human driver is going to drive into something just because it moved in a way they didn't expect I would actually put money that this is the cause of most crashes involving multiple moving cars. Hell, a friend of mine got into an accident two weeks ago where they t-boned somebody that turned onto a median when they didn't expect it.
      - PH95VuimJjqBqy 3y ago
        which is dealt with the explanation about momentum. If your friend saw them and never attempted to stop then your friend needs to lose their license.
    - piombisallow 3y ago
      "no human driver"? Really? Ever? Are you willing to bet on that assertion? Even if the human driver downs a bottle of vodka before driving?
      - PH95VuimJjqBqy 3y ago
        well if we're going to play that game, what if the human suffers a medical emergency! or what if they're driving a bus and they have to keep above 60 mph! --- I guess it's my fault, I didn't list every single contingency possible...
ralph84 3y ago
Link should go to the blog post not CNN. https://waymo.com/blog/2024/02/voluntary-recall-of-our-previous-software/ https://waymo.com/blog/2024/02/voluntary-recall-of-our-previ...
im3w1l 3y ago
This is my biggest fear with self-driving cars. Correlated failures. As a society we are extremely good at dealing with independent accidents. We can calculate very precisely how many people will die of traffic in a given year and we can account for it, we can have insurances, and we can decide exactly how much we are willing to spend to save a life on the margin. But if everything is fine, everything is fine, everything is fine, and then all hell breaks lose? We are not as good at dealing with that.
- add-sub-mul-div 3y ago
  Yeah, there's nothing akin to a software update that would cause the entire fleet of human drivers to start driving badly or unexpectedly all at once. We also know how to hold individuals accountable for independent accidents. We know we won't get justice when people will inevitably start to get killed by standard corporate greed, incompetence, enshittification.
  - gdestus 3y ago
    I imagine this can be remedied by slow rolling non critical updates out so that the entire fleet doesn't get upended by Bobby Tables at once. You could trivially observe the daily change in accidents/collisions/whatever and adjust fire
    - deleted 3y ago
      [deleted]
    - shermantanktop 3y ago
      Works great until the problem condition is not evenly distributed in place and time. Imagine that the release goes out in June but can’t handle icy roads; or the release goes out and can’t handle leap years; or it goes to cars in Iowa but has a problem interpreting ocean mist.
    - kevinventullo 3y ago
      That remedy already applies to massive profitable services like Google and Facebook, yet they still have outages caused by sloppy configuration pushes.
Klaus23 3y ago
Sounds like they were relying solely on their neural network path prediction, which failed when the truck was dragged at an odd angle. A simple lidar moving object segmentation, which doesn't even know what it's looking at but can always spit out reasonable path predictions, would probably have saved them. I think Mobileye is doing something like this, but they release so little data, which is always full of marketing bullshit, that it is hard to know what exactly they are working on.
- digitallis42 3y ago
  It's unlikely to be neutral network based. This sounds like a model prediction failure. You take a mathematical model of car motion: the rear wheels generally don't steer. The front steered wheels can cause the car to drive along an arc. If you want to predict the arc that will be driven, you take the initial starting heading of the vehicle and project forward in time based on your understanding of the vehicles steering angle. For most "driving in lane at velocity" cases, you generally would assume that the vehicle has very little steering angle input. We're now getting to see where autonomy needs to develop "spider sense": the scene in front of me feels wrong because some element isn't following the expected behavior maybe in ways that can't really be rationalized about, so we'll become much more conservative/defensive when dealing with it.
  - Klaus23 3y ago
    It is definitely based on neural networks: https://www.youtube.com/watch?v=RpiN3LyMLB8 https://www.youtube.com/watch?v=RpiN3LyMLB8
  - rasz 3y ago
    Its NN black box alright, for a hand crafted model you would have something like: if prediction != reality {mismatch+= whatever the time tick is} if mismatch > second (or two) {abort driver assist}
  - im3w1l 3y ago
    I'm thinking it might make sense to have a sort of hierarchy of models. The stupidest model predicts that everything will be stationary. The second model predicts that everything will travel in a straight line. The third model tries to predict a circular arc based on same fusion of path history and observed steering input. Fourth model uses a notion of action to predict whats going to happen, like "the car is changing lanes". Fifth model uses body language and common sense to predict intention "the pedestrian wants to cross". Each model can potentially predict longer into the future but also has more complexity and things that can go wrong. So you keep track of how well each model is doing (on an object basis) and if one level is failing then you fall back on a stupider one. You might also want to increase caution if your models are not doing well (lower speed and increased safety distance).
pinkmuffinere 3y ago
I wish there was a picture of the strange towing configuration. I wonder if I would be confused as well, although my guess is that I’d read the situation correctly
- midasuni 3y ago
  I’ve been confused occasionally by road situations. Somehow I’ve never managed to drive into a multi-ton object.
  - ghodith 3y ago
    Can't say the same for everyone.
  - Cacti 3y ago
    Well that car has been going around saying at least it hasn’t been posting uninformative, useless anecdotes on HN.
carbonatedmilk 3y ago
One of the best things I've learnt recently is how to apply the zero blame, process improvement approach that (many) air safety regulators take to my own teams. I'd sat through 'five whys' style postmortems before, but it was reading air safety investigation reports that finally got me to understand it and make it a useful part of how we get better at our jobs. By comparison, the way we're investigating and responding to self-driving safety incidents still seems very primitive. Why is that?
- esafak 3y ago
  Where did you learn about that approach, and how is it different?
  - sokoloff 3y ago
    Not GP, but: There is an explicit culture and mechanism of blamelessness around safety concerns and minor violations/deviations, which is incredibly helpful. Read about the ASRS* program (admin'd by NASA, with anonymity for non-intentional issues, prohibition on use of submissions for enforcement purposes, and explicit "get out punishment" card from the FAA): https://asrs.arc.nasa.gov/overview/immunity.html https://asrs.arc.nasa.gov/overview/immunity.html The FAA is also explicitly including "evidence of voluntary compliance" and "just culture" in its approach to aviation safety, and explicitly changed its goal from enforcement and proof to ensuring compliance [with enforcement being only one available tool]: https://www.faa.gov/about/initiatives/cp https://www.faa.gov/about/initiatives/cp (PDF pres: https://www.faa.gov/sites/faa.gov/files/2022-09/The%20Compliance%20Program.pdf https://www.faa.gov/sites/faa.gov/files/2022-09/The%20Compli... ) I'd also read a bunch of aviation reports: https://www.ntsb.gov/Pages/monthly.aspx https://www.ntsb.gov/Pages/monthly.aspx (more detailed reports are available approximately 2 years after the occurrence date and more details are available for fatal or air carrier occurrences, so if you don't care which ones to read, filter for those to start). If you're more video oriented, watch @blancolirio, @NTSBgov, @AirSafetyInstitute, or @pilot-debrief. (I'd skip @ProbableCause-DanGryder.) For a short summary, there is an intense focus on determining the facts (who, what, when, where, maybe some guesses as to why) and drawing conclusions about primary and contributing causes from there. * Aviation Safety Reporting System
andreareina 3y ago
Do we have a picture of the truck? I'm having difficulty imagining it given that surely the tow truck would want the towed vehicle in-line to make driving go smoothly?
- rainbowzootsuit 3y ago
  The towed vehicle has its rear driven wheels up on a tow hook type of tow truck and It sounds like a locked steering wheel that has been turned. This would lead to the angled tracking of the front wheels. This would be common for a debt recovery or when a city impounded the vehicle where it's taken without cooperation of the owner. https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Ftse1.mm.bing.net%2Fth%3Fid%3DOIP.cV8_J2zhbyHk3k-9XaUB8AHaEw%26pid%3DApi&f=1&ipt=2588b4114e062d55f9e2b9816431a247925e13f491dc1afa57c4162710787941&ipo=images https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2F...
Terr_ 3y ago
[Recycled from a older submission] Well, I feel kinda vindicated by this news, after previously noting: > People worry that ways and times [self-driving cars] are unsafe (separate from overall rates) will be unusual, less-predictable, or involve a novel risk-profile. In this example, having a secretly cursed vehicle configuration is something that we don't normally think of as a risk-factor from human drivers. _______ As an exaggerated thought experiment, imagine that autonomous driving achieves a miraculous reduction in overall accident/injury rate down to just 10% of when humans were in charge... However of the accidents that still happen, half are spooky events where every car on the road targets the same victim for no discernible reason. From the perspective of short-term utilitarianism, an unqualified success, but it's easy to see why it would be a cause of concern that could block adoption.
- camel_gopher 3y ago
  Human drivers just have secretly cursed driver configurations.
  - Terr_ 3y ago
    But not systematically, unless someone invents a completely novel tier of outrage-inducing bumper-sticker.
- deleted 3y ago
  [deleted]
adrians1 3y ago
This is what people don't appreciate when quoting those statistics about how self-driving cars are safer than humans: when a human driver causes an accident, it was because that particular person did something wrong. When a self-driving car handles a situation wrongly that's a big issue, because all the self-driving cars run the same software.
- skybrian 3y ago
  Yes, but when the bug was fixed, it was fixed everywhere.
  - skygazer 3y ago
    That phrasing makes it easy to underestimate the astronomical dimensions of edgecase-space, and what necessarily remains unexplored after such a fix.
  - bsaul 3y ago
    how do we know for sure that it was fixed ? What are the conditions that are fixed ?
    - what_ever 3y ago
      Since the comment thread is comparing humans, how do you know the human has fixed their bug?
      - bsaul 3y ago
        Humans don't have bugs. they may have disease, or mental troubles, but we're pretty good at assessing them. Thanks in part to their ability to communicate.
        RunningDroid 3y ago
        I'd argue that humans don't have bugs but their mental models of things do, mostly as a consequence of the fact a model's complexity increases with it's accuracy.
rkagerer 3y ago
That poor tow truck driver, must have felt like Skynet was really out to get him.

212 points by reteltech 3y ago

A pick-up truck being towed crooked and backwards. Both vehicles failed to read the situation in the same manner. Autonomous vehicles have various redundant systems built-in that can take priority and override false positives. I was previously under the assumption that one of the really important reasons for Lidar is that it can get you closer to an absolute truth about whether something is a solid object, and where that hypothetically solid object is relative to the position of the vehicle, regardless of what the classifier thinks it is seeing. So did the lidar fail to read the solid object, or was the lidar, was it de-prioritized? or was it simply not available as a fallback? Presumably Radar and proximity sensors were also involved. What were they doing? This is a fascinating edge case, and I hope to hear about the real reason for the 2 incidents.

Mageek 3y ago
The article says it: “We determined that due to the persistent orientation mismatch of the towed pickup truck and tow truck combination, the Waymo AV incorrectly predicted the future motion of the towed vehicle.” It was detected, but it predicted the truck would move in a way that it didn’t end up moving.
- pests 3y ago
  How do you drive into a solid war if you have lidar though? To say nothing of predictions, the object is where it's at.... where it's at at that moment. You don't need to predict where it's at now... because you know where it's at.
  - avalys 3y ago
    It wasn’t a solid wall, it was another vehicle. The brakes don’t respond immediately - you need to be able to detect that a collision is imminent several seconds before it actually occurs. This means you have to also successfully exclude all the scenarios where you are very close to another car, but a collision is not imminent because the car will be out of the way by the time you get there. Yes, at some point before impact the Waymo probably figured out that it was about to collide. But not soon enough to do anything about it.
    - antisthenes 3y ago
      This is why autonomous vehicles need a hard cap on the speed limit they are driving at. If the max speed is 35mph, that allows a good braking system to respond by safely stopping from LIDAR info 99% of the time.
      - lotsoweiners 3y ago
        Yeah but then people aren’t going to want to take them because they are slow.
        antisthenes 3y ago
        The average speed of a commuting car is around ~23mph, when you account for stops and red lights. 35mph is only a hindrance if you commute by freeway every day, which many people don't. People will take them if they're priced right and they can do things in the car without having to pay attention to the road.
        roxgib 3y ago
        But autonomous vehicles tend to perform best on freeways, that's where they are most likely to be allowed
    - Out_of_Characte 3y ago
      My brakes respond immediatly,for all intents and purposes as a human with a reaction time. I'm at fault if the person in front of me stops and I dont have the stopping distance to avoid a collision. I get that self driving software is difficult. But theres no excuse for this type of accident.
      - awhitby 3y ago
        That might be true for the simple case of following within a lane, although you only have to drive around to realize most drivers do not leave adequate following distance at all times to make this a pure physics problem. And neither is a good driver watching only the car in front, but also the brake lights of the cars in front of that, to help anticipate the car in front's likely actions. But take an even slightly more complex example: you're on a two lane roadway and the car in the other lane changes into your lane, leaving inadequate stopping distance for you. You brake as hard as you safely can (maybe you have a too-close follower, too), but still there will be a few seconds when you could not, in fact, avert a collision if for some reason the car in front braked. I have no idea what the legal situation would be: is it their fault if the crash happens within 3 seconds but yours if it happens after you've had time but failed to restablish your needed stopping distance? Honestly even in the simple one lane case, I doubt you can slam your brakes on the interstate for no reason then expect to avoid any liability for the crash, blaming your follower for following too close. Driving has a bunch of rules, then an awful lot of common sense and social interaction on top of them to make things actually work.
  - bobsomers 3y ago
    You can't drive if you only use the current "frame" of data as the basis for your decision. Imagine driving on the highway, a comfortable distance behind a lead vehicle. The planning software would want to slam on the brakes without predicting that the blob of sensor data in front of you is going to continue moving forward at highway speeds. That motion prediction enables the planning software to know that the space in front of your vehicle will be unoccupied by the time you reach it. A similar prediction error was the reason Cruise rear ended the bendy bus in SF a while back. It segmented the front and rear halves of the bus as two separate entities rather than a connected one, and mispredicted the motion of the rear half of the bus.
    - hmcq6 3y ago
      > That motion prediction enables the planning software to know that the space in front of your vehicle will be unoccupied by the time you reach it. I think we're all on the same page about this part but what's confusing and hilarious is why would the correct answer ever be to drive into an unmoving object? If they tried to avoid the truck and swerved and hit a different vehicle there would be no confusion here. But the self driving algorithm is effectively committing suicide (Kamikaze). That's novel. My guess is that the self-driving car was not able to recognize the truck until it was very close and the sudden appearance of the truck is interpreted by the algorithm as if the truck is moving very fast. And the best answer in that case would be to let the truck pass (basically do what the waymo did). But that means the lidar information about the shape not moving is being deprioritized in favor of the recognized object being calculated to move fast. A situation which could only really occur if a speeding vehicle plowed through a stationary object. Fantastic solution for avoiding a situation like this -> https://www.youtube.com/watch?v=BbjjlvOxDYk https://www.youtube.com/watch?v=BbjjlvOxDYk But a bad solution for avoiding a stationary object.
      - clhodapp 3y ago
        Who said it was an unmoving object? Maybe I missed something in the story, but I got the sense that this happened in motion. The towed truck would have essentially been moving forward at an angle, hanging across the lane boundary.
  - lazide 3y ago
    You can’t drive more than a few MPH unless you’re reacting based on the expected future, rather than the current one. It’s why it’s so difficult to do (actually) and the ability to do it well is just as much about the risk appetite of the one responsible as anything else - because knowing if a car is likely to pull out at the light into traffic, or how likely someone is to be hiding in a bush or not is really hard. But that is what humans deal with all the time while driving. Because no one can actually know the future, and predicting the future is fundamentally risky. And knowing when to hold ‘em, and when to fold ‘em is really more of an AGI type thing.
  - sokoloff 3y ago
    In self-driving, you are making predictions about where the object is right now based on the synthesis of data from your sensors (and often filtering information from past estimates of the object position). These might be high-precision, high-accuracy predictions, but they're predictions nonetheless. (It's been quite some years since I worked on vision-based self-driving, so my experience is non-zero but also quite dated.)
- PH95VuimJjqBqy 3y ago
  I don't find that acceptable in any way, no human driver is going to do that and by that I mean no human driver is going to drive into something just because it moved in a way they didn't expect. they're going to slam on the brakes and the only way that's going to happen is if momentum is too high. I understand we have to have explanations or we can't fix them, but it's just as important to understand this should never have happened even WITH the described failure. If I had to guess, there's code to avoid stopping at every little thing and that code took precedence (otherwise rides would not be enjoyable). And I get the competing interests here but there must be a comparison to humans when these incidents happen.
  - duped 3y ago
    > no human driver is going to drive into something just because it moved in a way they didn't expect I would actually put money that this is the cause of most crashes involving multiple moving cars. Hell, a friend of mine got into an accident two weeks ago where they t-boned somebody that turned onto a median when they didn't expect it.
    - PH95VuimJjqBqy 3y ago
      which is dealt with the explanation about momentum. If your friend saw them and never attempted to stop then your friend needs to lose their license.
  - piombisallow 3y ago
    "no human driver"? Really? Ever? Are you willing to bet on that assertion? Even if the human driver downs a bottle of vodka before driving?
    - PH95VuimJjqBqy 3y ago
      well if we're going to play that game, what if the human suffers a medical emergency! or what if they're driving a bus and they have to keep above 60 mph! --- I guess it's my fault, I didn't list every single contingency possible...

ralph84 3y ago

Link should go to the blog post not CNN. https://waymo.com/blog/2024/02/voluntary-recall-of-our-previous-software/ https://waymo.com/blog/2024/02/voluntary-recall-of-our-previ...

im3w1l 3y ago

This is my biggest fear with self-driving cars. Correlated failures. As a society we are extremely good at dealing with independent accidents. We can calculate very precisely how many people will die of traffic in a given year and we can account for it, we can have insurances, and we can decide exactly how much we are willing to spend to save a life on the margin. But if everything is fine, everything is fine, everything is fine, and then all hell breaks lose? We are not as good at dealing with that.

add-sub-mul-div 3y ago
Yeah, there's nothing akin to a software update that would cause the entire fleet of human drivers to start driving badly or unexpectedly all at once. We also know how to hold individuals accountable for independent accidents. We know we won't get justice when people will inevitably start to get killed by standard corporate greed, incompetence, enshittification.
- gdestus 3y ago
  I imagine this can be remedied by slow rolling non critical updates out so that the entire fleet doesn't get upended by Bobby Tables at once. You could trivially observe the daily change in accidents/collisions/whatever and adjust fire
  - deleted 3y ago
    [deleted]
  - shermantanktop 3y ago
    Works great until the problem condition is not evenly distributed in place and time. Imagine that the release goes out in June but can’t handle icy roads; or the release goes out and can’t handle leap years; or it goes to cars in Iowa but has a problem interpreting ocean mist.
  - kevinventullo 3y ago
    That remedy already applies to massive profitable services like Google and Facebook, yet they still have outages caused by sloppy configuration pushes.

Klaus23 3y ago

Sounds like they were relying solely on their neural network path prediction, which failed when the truck was dragged at an odd angle. A simple lidar moving object segmentation, which doesn't even know what it's looking at but can always spit out reasonable path predictions, would probably have saved them. I think Mobileye is doing something like this, but they release so little data, which is always full of marketing bullshit, that it is hard to know what exactly they are working on.

digitallis42 3y ago
It's unlikely to be neutral network based. This sounds like a model prediction failure. You take a mathematical model of car motion: the rear wheels generally don't steer. The front steered wheels can cause the car to drive along an arc. If you want to predict the arc that will be driven, you take the initial starting heading of the vehicle and project forward in time based on your understanding of the vehicles steering angle. For most "driving in lane at velocity" cases, you generally would assume that the vehicle has very little steering angle input. We're now getting to see where autonomy needs to develop "spider sense": the scene in front of me feels wrong because some element isn't following the expected behavior maybe in ways that can't really be rationalized about, so we'll become much more conservative/defensive when dealing with it.
- Klaus23 3y ago
  It is definitely based on neural networks: https://www.youtube.com/watch?v=RpiN3LyMLB8 https://www.youtube.com/watch?v=RpiN3LyMLB8
- rasz 3y ago
  Its NN black box alright, for a hand crafted model you would have something like: if prediction != reality {mismatch+= whatever the time tick is} if mismatch > second (or two) {abort driver assist}
- im3w1l 3y ago
  I'm thinking it might make sense to have a sort of hierarchy of models. The stupidest model predicts that everything will be stationary. The second model predicts that everything will travel in a straight line. The third model tries to predict a circular arc based on same fusion of path history and observed steering input. Fourth model uses a notion of action to predict whats going to happen, like "the car is changing lanes". Fifth model uses body language and common sense to predict intention "the pedestrian wants to cross". Each model can potentially predict longer into the future but also has more complexity and things that can go wrong. So you keep track of how well each model is doing (on an object basis) and if one level is failing then you fall back on a stupider one. You might also want to increase caution if your models are not doing well (lower speed and increased safety distance).

pinkmuffinere 3y ago

I wish there was a picture of the strange towing configuration. I wonder if I would be confused as well, although my guess is that I’d read the situation correctly

midasuni 3y ago
I’ve been confused occasionally by road situations. Somehow I’ve never managed to drive into a multi-ton object.
- ghodith 3y ago
  Can't say the same for everyone.
- Cacti 3y ago
  Well that car has been going around saying at least it hasn’t been posting uninformative, useless anecdotes on HN.

carbonatedmilk 3y ago

One of the best things I've learnt recently is how to apply the zero blame, process improvement approach that (many) air safety regulators take to my own teams. I'd sat through 'five whys' style postmortems before, but it was reading air safety investigation reports that finally got me to understand it and make it a useful part of how we get better at our jobs. By comparison, the way we're investigating and responding to self-driving safety incidents still seems very primitive. Why is that?

esafak 3y ago
Where did you learn about that approach, and how is it different?
- sokoloff 3y ago
  Not GP, but: There is an explicit culture and mechanism of blamelessness around safety concerns and minor violations/deviations, which is incredibly helpful. Read about the ASRS* program (admin'd by NASA, with anonymity for non-intentional issues, prohibition on use of submissions for enforcement purposes, and explicit "get out punishment" card from the FAA): https://asrs.arc.nasa.gov/overview/immunity.html https://asrs.arc.nasa.gov/overview/immunity.html The FAA is also explicitly including "evidence of voluntary compliance" and "just culture" in its approach to aviation safety, and explicitly changed its goal from enforcement and proof to ensuring compliance [with enforcement being only one available tool]: https://www.faa.gov/about/initiatives/cp https://www.faa.gov/about/initiatives/cp (PDF pres: https://www.faa.gov/sites/faa.gov/files/2022-09/The%20Compliance%20Program.pdf https://www.faa.gov/sites/faa.gov/files/2022-09/The%20Compli... ) I'd also read a bunch of aviation reports: https://www.ntsb.gov/Pages/monthly.aspx https://www.ntsb.gov/Pages/monthly.aspx (more detailed reports are available approximately 2 years after the occurrence date and more details are available for fatal or air carrier occurrences, so if you don't care which ones to read, filter for those to start). If you're more video oriented, watch @blancolirio, @NTSBgov, @AirSafetyInstitute, or @pilot-debrief. (I'd skip @ProbableCause-DanGryder.) For a short summary, there is an intense focus on determining the facts (who, what, when, where, maybe some guesses as to why) and drawing conclusions about primary and contributing causes from there. * Aviation Safety Reporting System

andreareina 3y ago

Do we have a picture of the truck? I'm having difficulty imagining it given that surely the tow truck would want the towed vehicle in-line to make driving go smoothly?

rainbowzootsuit 3y ago
The towed vehicle has its rear driven wheels up on a tow hook type of tow truck and It sounds like a locked steering wheel that has been turned. This would lead to the angled tracking of the front wheels. This would be common for a debt recovery or when a city impounded the vehicle where it's taken without cooperation of the owner. https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Ftse1.mm.bing.net%2Fth%3Fid%3DOIP.cV8_J2zhbyHk3k-9XaUB8AHaEw%26pid%3DApi&f=1&ipt=2588b4114e062d55f9e2b9816431a247925e13f491dc1afa57c4162710787941&ipo=images https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2F...

Terr_ 3y ago

[Recycled from a older submission] Well, I feel kinda vindicated by this news, after previously noting: > People worry that ways and times [self-driving cars] are unsafe (separate from overall rates) will be unusual, less-predictable, or involve a novel risk-profile. In this example, having a secretly cursed vehicle configuration is something that we don't normally think of as a risk-factor from human drivers. _______ As an exaggerated thought experiment, imagine that autonomous driving achieves a miraculous reduction in overall accident/injury rate down to just 10% of when humans were in charge... However of the accidents that still happen, half are spooky events where every car on the road targets the same victim for no discernible reason. From the perspective of short-term utilitarianism, an unqualified success, but it's easy to see why it would be a cause of concern that could block adoption.

camel_gopher 3y ago
Human drivers just have secretly cursed driver configurations.
- Terr_ 3y ago
  But not systematically, unless someone invents a completely novel tier of outrage-inducing bumper-sticker.
deleted 3y ago
[deleted]

adrians1 3y ago

This is what people don't appreciate when quoting those statistics about how self-driving cars are safer than humans: when a human driver causes an accident, it was because that particular person did something wrong. When a self-driving car handles a situation wrongly that's a big issue, because all the self-driving cars run the same software.

skybrian 3y ago
Yes, but when the bug was fixed, it was fixed everywhere.
- skygazer 3y ago
  That phrasing makes it easy to underestimate the astronomical dimensions of edgecase-space, and what necessarily remains unexplored after such a fix.
- bsaul 3y ago
  how do we know for sure that it was fixed ? What are the conditions that are fixed ?
  - what_ever 3y ago
    Since the comment thread is comparing humans, how do you know the human has fixed their bug?
    - bsaul 3y ago
      Humans don't have bugs. they may have disease, or mental troubles, but we're pretty good at assessing them. Thanks in part to their ability to communicate.
      - RunningDroid 3y ago
        I'd argue that humans don't have bugs but their mental models of things do, mostly as a consequence of the fact a model's complexity increases with it's accuracy.

rkagerer 3y ago

That poor tow truck driver, must have felt like Skynet was really out to get him.