5 ms·

It wasn't that the sensors were confused or something - the AI had no idea how to cope with pedestrians in the road for whatever reason, so threw them in the "O

It wasn't that the sensors were confused or something - the AI had no idea how to cope with pedestrians in the road for whatever reason, so threw them in the "Other" category. At least 98% of objects are categorised, the rest don't matter. /s

rvense 7y ago
TBH this does sound exactly like something you'd expect to happen if you took some Javascript ninja used to moving fast and breaking things and asked them to build a car. But surely Uber has more sense than that...
- svrtknst 7y ago
  Yes... surely.
- dsfyu404ed 7y ago
  Coming from the defense (go ahead, clutch your pearls, I won't be offended) industry I just can't understand why people ship things that fail into in potentially dangerous states on some known non-trivial fraction of inputs. If the purpose of the software were entertainment or some other low stakes thing I'd understand but software controlling things that can kill people or lose lots of money need to be able to handle unexpected inputs gracefully.
  - Eric_WVGG 7y ago
    Because the penalties and rewards don’t align.
  - Majromax 7y ago
    > I just can't understand why people ship things that fail into in potentially dangerous states on some known non-trivial fraction of inputs Because that kind of safety-conscious culture needs to be deliberately enforced, from the top all the way down, by a system that takes a long view. If we're making an analogy to the defense industry, then building self-driving cars as a tech startup is closer to the Anarchist's Cookbook than Northrop Grumman.
  - targonca 7y ago
    Because you don't want to be the one whose team is lacking in agility on the next company all-hands meeting ;)
- peteradio 7y ago
  Or managers barking at sheepish recent grads/h1bs who don't know how to or can't push back. Does not seem like a very strong engineering culture.
- fortran77 7y ago
  What are you saying, that a Haskell or Rust elite programmer wouldn't have made this mistake!?
  - nosianu 7y ago
    The strong types in those languages would have prevented a reference to a "Person" object to be treated like a reference to an "Other" object... /s
Gupie 7y ago
Isn't the bigger problem that it did not classify the object as being on a collision path. It doesn't matter what the object is, brake if you are going to hit it.
- fabian2k 7y ago
  From what I read in some of the earlier reports on this, the car didn't have the ability to emergency brake in autonomous mode. It was disabled at that time, so it could only brake in regular traffic, not for obstacles that appear suddenly.
  - mannykannot 7y ago
    That is right - so now we have two major errors. The justification made for disabling the emergency braking - that it would interfere with data gathering - might appear reasonable at first sight, but it does not stand up to scrutiny, for if the emergency braking is triggered, the driving system has already made a mistake, and you already have the data on that malfunction.
    - kjaftaedi 7y ago
      Yes, but accidental emergency breaking can cause just as bad of an accident as this, it all just depends on the scenario. They had hired a driver to sit behind the wheel to monitor the road and the car for exactly this reason. The driver they hired decided to watch a movie on their phone instead of paying attention to the road.
      - mannykannot 7y ago
        > Yes, but accidental emergency breaking can cause just as bad of an accident as this, it all just depends on the scenario. If Volvo's system is that dangerous, then it should not be on the road at all - but there is no evidence that it is, you are just making a speculative argument. > They had hired a driver to sit behind the wheel to monitor the road and the car for exactly this reason. That is no reason to disable a safety feature that would add safety in depth. > The driver they hired decided to watch a movie on their phone instead of paying attention to the road. That was a major error - a crime, in fact - but, unfortunately, also an entirely predictable scenario that cannot be dismissed on the grounds that dealing with it would make testing more difficult or expensive. So now we have three errors.
wongarsu 7y ago
Consistently hitting the "other" category would be a massive improvement over what happened. According to the timeline for the pedestrian was initially classified as a vehicle, later as a bicycle, with short periods in between where she's classified as "other" or "unknown". And of course when an object changes classification it loses all history and is initially predicted to be static. Despite detecting her for 5.6 seconds the system never had more than 1.5 seconds of tracking history because it couldn't make up it's mind and kept throwing everything away.

rvense 7y ago

TBH this does sound exactly like something you'd expect to happen if you took some Javascript ninja used to moving fast and breaking things and asked them to build a car. But surely Uber has more sense than that...

svrtknst 7y ago
Yes... surely.
dsfyu404ed 7y ago
Coming from the defense (go ahead, clutch your pearls, I won't be offended) industry I just can't understand why people ship things that fail into in potentially dangerous states on some known non-trivial fraction of inputs. If the purpose of the software were entertainment or some other low stakes thing I'd understand but software controlling things that can kill people or lose lots of money need to be able to handle unexpected inputs gracefully.
- Eric_WVGG 7y ago
  Because the penalties and rewards don’t align.
- Majromax 7y ago
  > I just can't understand why people ship things that fail into in potentially dangerous states on some known non-trivial fraction of inputs Because that kind of safety-conscious culture needs to be deliberately enforced, from the top all the way down, by a system that takes a long view. If we're making an analogy to the defense industry, then building self-driving cars as a tech startup is closer to the Anarchist's Cookbook than Northrop Grumman.
- targonca 7y ago
  Because you don't want to be the one whose team is lacking in agility on the next company all-hands meeting ;)
peteradio 7y ago
Or managers barking at sheepish recent grads/h1bs who don't know how to or can't push back. Does not seem like a very strong engineering culture.
fortran77 7y ago
What are you saying, that a Haskell or Rust elite programmer wouldn't have made this mistake!?
- nosianu 7y ago
  The strong types in those languages would have prevented a reference to a "Person" object to be treated like a reference to an "Other" object... /s

Gupie 7y ago

Isn't the bigger problem that it did not classify the object as being on a collision path. It doesn't matter what the object is, brake if you are going to hit it.

fabian2k 7y ago
From what I read in some of the earlier reports on this, the car didn't have the ability to emergency brake in autonomous mode. It was disabled at that time, so it could only brake in regular traffic, not for obstacles that appear suddenly.
- mannykannot 7y ago
  That is right - so now we have two major errors. The justification made for disabling the emergency braking - that it would interfere with data gathering - might appear reasonable at first sight, but it does not stand up to scrutiny, for if the emergency braking is triggered, the driving system has already made a mistake, and you already have the data on that malfunction.
  - kjaftaedi 7y ago
    Yes, but accidental emergency breaking can cause just as bad of an accident as this, it all just depends on the scenario. They had hired a driver to sit behind the wheel to monitor the road and the car for exactly this reason. The driver they hired decided to watch a movie on their phone instead of paying attention to the road.
    - mannykannot 7y ago
      > Yes, but accidental emergency breaking can cause just as bad of an accident as this, it all just depends on the scenario. If Volvo's system is that dangerous, then it should not be on the road at all - but there is no evidence that it is, you are just making a speculative argument. > They had hired a driver to sit behind the wheel to monitor the road and the car for exactly this reason. That is no reason to disable a safety feature that would add safety in depth. > The driver they hired decided to watch a movie on their phone instead of paying attention to the road. That was a major error - a crime, in fact - but, unfortunately, also an entirely predictable scenario that cannot be dismissed on the grounds that dealing with it would make testing more difficult or expensive. So now we have three errors.

wongarsu 7y ago

Consistently hitting the "other" category would be a massive improvement over what happened. According to the timeline for the pedestrian was initially classified as a vehicle, later as a bicycle, with short periods in between where she's classified as "other" or "unknown". And of course when an object changes classification it loses all history and is initially predicted to be static. Despite detecting her for 5.6 seconds the system never had more than 1.5 seconds of tracking history because it couldn't make up it's mind and kept throwing everything away.