9 ms·

I worked on the autonomous pod system at Heathrow airport[1]. We used a very conservative control methodology; essentially the vehicle would remain stopped unle

by nkoren 8y ago

I worked on the autonomous pod system at Heathrow airport[1]. We used a very conservative control methodology; essentially the vehicle would remain stopped unless it received a positive "GO" signal from multiple independent sensor and control systems. The loss of any "GO" signal would result in an emergency stop. It was very challenging to get those all of those "GO" indicators reliable enough to prevent false positives and constant emergency-braking. The reason we were ultimately able to do this is because we were operating in a fully-segregated environment of our own design. We could be certain that every other vehicle in the system was something that should be fully under our control, so anything even slightly anomalous should be treated as a hazard situation. There are a lot of limitations to this approach, but I'm confident that it could carry literally billions of passengers without a fatality. It is overwhelmingly safe. Operating in a mixed environment is profoundly different. The control system logic is fully reversed: you must presume that it is safe to proceed unless a "STOP" signal received. And because the interpretation of image & LIDAR data is a rather... fuzzy... process, that "STOP" signal needs to have fairly liberal thresholds, otherwise your vehicle will not move. Uber made a critical mistake in counting on a human-in-the-loop to suddenly take control of the vehicle (note: this is why Type 3 automation is something I'm very dubious about), but it's important to understand that if you want autonomous vehicles to move through mixed-mode environments at the speeds which humans drive, then it is absolutely necessary for them to take a fuzzy, probabilistic approach to safety. This will inevitably result in fatalities -- almost certainly fewer than when humans drive, but plenty of fatalities nonetheless. The design of the overall system is system is inherently unsafe. Do you find this unacceptable? If so, then then ultimately the only way to address this is through changing the design of the streets and/or our rules about how they are be used. These are fundamentally infrastructural issues. Merely swapping out vehicle control systems -- robot vs. human -- will be less revolutionary than many expect. 1: http://www.ultraglobalprt.com/ http://www.ultraglobalprt.com/

mcguire 8y ago
I'm with you up until the "almost certainly" part. Is there any actual hard data on that?
aetherson 8y ago
It is the case that even very good driverless cars of the future will cause fatalities now and then. Even if they're safer than human drivers. Don't conflate that with Uber's screw-up here. This wasn't a situation where a fatality was unavoidable or where a very safe system had a once-in-a-blue-moon problem. It's one where they just drove around not-very-safe cars.
- dubbel 8y ago
  > It is the case that even very good driverless cars of the future will cause fatalities now and then. Even if they're safer than human drivers. And that is why I am so mad at Uber. They are compromising the public trust in autonomous cars with their reckless release policy. And thereby potentially endangering even more lives, as we have to convince the public of the advantages of this technology.
  - sanderjd 8y ago
    I agree with all of this except the tense: haven't they shut the whole thing down at this point with no immediate plans to start it up again? Or am I mis-remembering that.
    - bagels 8y ago
      They were testing in three places: San Francisco, Arizona, and Pittsburgh. They didn't want to get a license from California (probably because they couldn't follow the safety regulations), so they threw a tantrum and moved to AZ. Then after this fatality, they shut the AZ program down and are just testing in Pittsburgh.
      - aetherson 8y ago
        That's not true. They shut down everywhere after this fatality. They just said that they'll shut down AZ permanently (not that AZ would probably let them do it anyway), and resume testing in Pittsburgh sometime soon, in a more limited way (which apparently the Pittsburgh mayor isn't wild about).
jacquesm 8y ago
> The loss of any "GO" signal would result in an emergency stop. That's an E-stop chain and that's exactly how it should work. But the software as described in the NTSB report was apparently bad enough that they essentially hardwired an override on their emergency stop. The software equivalent of putting a steel bar into a fuse receptacle. The words that come to mind are 'criminal negligence'. The vehicle would not have been able to do an E-stop even if it was 100% sure it had to do just that, nor did it warn the human luggage. The problem here is not that the world is so unsafe that you will have to make compromises to get anywhere at all, the problem here is that the software is still so buggy that there is no way to safely navigate common scenarios. Pedestrian on the road at night is one that I've encountered twice on my trips and they did not lead to any fatalities because when I can't see I slow down. If 6 seconds isn't enough to make a decision you have no business being on the road in the first place.
- blattimwind 8y ago
  > Pedestrian on the road at night is one that I've encountered twice on my trips and they did not lead to any fatalities because when I can't see I slow down. If 6 seconds isn't enough to make a decision you have no business being on the road in the first place. I've seen a few people comment on the footage that they too would have run the pedestrian over, to which my only response is: I sure hope you don't have a driver's license [anymore]!
  - hcknwscommenter 8y ago
    The vast majority of those people are being (purposely?) deluded by a very misleading video showing an apparently pitch black section of road. In reality, it was a lighted road and the dashcam footage had a very compressed dynamic range.
    - forapurpose 8y ago
      > it was a lighted road The report is a bit ambiguous about that: The videos show that the pedestrian crossed in a section of roadway not directly illuminated by the roadway lighting.
maigret 8y ago
The problem is that drivers rarely maintain the safety distance they should have to not endanger themselves. BUT in that case, the car should have also noticed if there were near traffic behind. Doing nothing in that case doesn’t seem the right decision at all. Very good write anyways... indeed many things will have to change - probably the infrastructure, the vehicles, the software, the way pedestrians move, and driver behavior as well.
nmca 8y ago
This highlights an interesting general point - in many situations, there is no simple safe fallback policy. On a highway, an emergency stop is not safe. This is a general problem in AI safety and is covered nicely in this youtube video, as well as the paper referenced there - https://www.youtube.com/watch?v=lqJUIqZNzP8 https://www.youtube.com/watch?v=lqJUIqZNzP8
- jacquesm 8y ago
  > On a highway, an emergency stop is not safe. That depends, there could simply be no traffic behind you, which an experienced driver and hopefully and automated one would be monitoring. Besides, there are many situations on the highway where an E-stop is far safer than any of the alternatives even if there is traffic behind you. Driving as though nothing has changed in the presence of an E-stop worthy situation is definitely not the right decision.
  - beenBoutIT 8y ago
    How intelligent is the ML driving the car? If the car slowed down and hit the 49 year old at a reduced speed the insurance payout to a now severely disabled individual would be far more expensive than the alternative insurance pay out with a pedestrian fatality. A choice between paying out for 40 years worth of around-the-clock medical care vs. a one-time lump-sum payout to the victim's family would be pretty obvious from a corporate point of view.
    - jacquesm 8y ago
      Are you seriously suggesting that the better software strategy is to aim for the kill because it is cheaper than possibly causing 'only' injury? That should be criminal. I'm all for chalking this one up to criminal negligence and incompetence, outright malice is - for now - off the table, unless someone leaks meeting notes from Uber where they discussed that exact scenario.
      - beenBoutIT 8y ago
boxcardavin 8y ago
This is absolutely the right analysis of how these systems work and why you can't expect autonomous cars to halt traffic deaths. What the Uber crash has shown us is that the tolerance for AVs killing people is probably exactly zero, not some (very meaningful) reduction like 10x or 100x less. My company didn't start with this zero tolerance thing in our minds, but it turns out our self-delivering electric bicycles have a huge advantage for real world safety because they weigh ~60lbs when in autonomous mode and are limited to 12mph. This equals the kinetic energy of myself walking at a brisk pace, or basically something that won't kill purely from blunt force impact. I think the future for autonomy will be unlocked by low mass and low speed vehicles, not cars converted to drive themselves.
- jacquesm 8y ago
  > What the Uber crash has shown us is that the tolerance for AVs killing people is probably exactly zero, not some (very meaningful) reduction like 10x or 100x less. It hasn't shown that at all. It has documented beyond reasonable doubt that Uber should not be allowed to participate in real world tests of autonomous vehicles. There are plenty of situations where people would fully accept a self driving vehicle killing someone but this isn't one of those.
- econochoice 8y ago
  > This equals the kinetic energy of myself walking at a brisk pace, or basically something that won't kill purely from blunt force impact. This is an analogy that cannot completely map to cycling. A fall at any speed from a bike is literally a potentially crippling or deadly scenario.
  - boxcardavin 8y ago
    When a rider is onboard there’s no autonomy, the bike is only self-delivering.
  - grkvlt 8y ago
    Potentially deadly? Maybe, sure, but at low speeds, up to 10 mph say, it is incredibly unlikely that falling off a bicycle (even with no helmet) will do more than cause bruises and damaged ego.
robk 8y ago
I love your pods at T5! They're super fun.
hcknwscommenter 8y ago
Sure, but "[t]he system is not designed to alert the operator." At least they could have alerted the operator. This seems like reckless endangerment or negligent homicide. Luckily for Uber they hit a poor person and no one will hold them responsible. 1.3 seconds is a long time for the operator to act.
YeGoblynQueenne 8y ago
>> This will inevitably result in fatalities -- almost certainly fewer than when humans drive, but plenty of fatalities nonetheless. Where is the almost-certainty coming from that the fatalities would be fewer compared to humans driving? And what does "almost" mean in this case?
- majewsky 8y ago
  Computers have vastly lower reaction time than humans. Computers have sensory input that humans lack (LIDAR). Computers don't get drowsy or agitated. And "almost" is always a good idea when talking about a future that looks certain. Takes into account the unknown unknowns. And the known unknowns (cough hacking cough).
  - YeGoblynQueenne 8y ago
    Fast reaction times, good sensors and unyielding focus are not enough to drive safely. An agent also needs situational awareness and an understanding of the entities in its environment and their relations. Without the ability to understand its environment and react appropriately to it, all the good the fast reaction times will do to an AI agent is to let it take the wrong decisions faster than a human being. Just saying "computers" and waving our hands about won't magically solve the hard problems involved in full autonomy. Allegedly, the industry has some sort of plan to go from where we are now (sorta kinda level-2 autonomy) to full, level-5 autonomy where "computers" will drive more safely than humans. It would be very kind of the industry if they could share that plan with the rest of us, because for the time being it sounds just like what I describe above, saying "computers" and hand-waving everything else.
- nkoren 8y ago
  That's a sociopolitical question more than a technical one. I posit that: 1.) Road safety -- as far as the current operating concept of cars is concerned (eg., high speeds in mixed environments) -- is not a problem that can be "solved". At best it can only ever be approximated. The quality of approximation will correspond to the number of fatalities. Algorithm improvements will yield diminishing returns: the operating domain is fundamentally unsafe, and will always result in numerous fatalities even when driven "perfectly". 2.) With regards to factors that contribute to driving safety, there are some things that computers are indisputably better at than humans (raw reaction time). There are other things that humans are still better at than computers (synthesising sensory data into a cohesive model of the world, and then reasoning about that world). Computers are continually improving their performance, however. While we don't have all the theories worked out for how machines will eventually surpass human performance in these domains, we don't have a strong reason to believe that machines won't surpass human performance in these domains. The only question is when. (I don't have an answer to this question). 3.) So the question is not "when will autonomous driving be safe" (it won't be), but rather: "what is the minimum level of safety we will accept from autonomous driving?" I'm quite certain that the bar will be set much higher for autonomous driving than for human driving. This is because risk perception -- especially as magnified by a media that thrives on sensationalism -- is based on how "extraordinary" an event seems, much more than how dangerous it actually is. Look at the disparities in sociopolitical responses to, say, plane crashes and Zika virus, versus car crashes and influenza. Autonomous vehicles will be treated more as the former than the latter, and therefore the scrutiny they receive will be vastly higher. 4.) So basically, driverless cars will only find a routine place on the road if and when they have sufficiently fewer fatalities than human driving. My assertion was a bit tautological in this respect, but basically, if they're anywhere near as dangerous as human drivers, then they won't be a thing at all. 5.) Personally, I think that the algorithms won't be able to pass this public-acceptability threshold on their own, because even the best-imaginable algorithm, if adopted on a global basis, would still kill hundreds of thousands of people every year. That's still probably too many. I expect that full automation eventually will become the norm, but only as enabled by new types of infrastructure / urban design which enable it to be safer than automation alone.

by nkoren 8y ago

mcguire 8y ago

I'm with you up until the "almost certainly" part. Is there any actual hard data on that?

aetherson 8y ago

It is the case that even very good driverless cars of the future will cause fatalities now and then. Even if they're safer than human drivers. Don't conflate that with Uber's screw-up here. This wasn't a situation where a fatality was unavoidable or where a very safe system had a once-in-a-blue-moon problem. It's one where they just drove around not-very-safe cars.

dubbel 8y ago
> It is the case that even very good driverless cars of the future will cause fatalities now and then. Even if they're safer than human drivers. And that is why I am so mad at Uber. They are compromising the public trust in autonomous cars with their reckless release policy. And thereby potentially endangering even more lives, as we have to convince the public of the advantages of this technology.
- sanderjd 8y ago
  I agree with all of this except the tense: haven't they shut the whole thing down at this point with no immediate plans to start it up again? Or am I mis-remembering that.
  - bagels 8y ago
    They were testing in three places: San Francisco, Arizona, and Pittsburgh. They didn't want to get a license from California (probably because they couldn't follow the safety regulations), so they threw a tantrum and moved to AZ. Then after this fatality, they shut the AZ program down and are just testing in Pittsburgh.
    - aetherson 8y ago
      That's not true. They shut down everywhere after this fatality. They just said that they'll shut down AZ permanently (not that AZ would probably let them do it anyway), and resume testing in Pittsburgh sometime soon, in a more limited way (which apparently the Pittsburgh mayor isn't wild about).

jacquesm 8y ago

> The loss of any "GO" signal would result in an emergency stop. That's an E-stop chain and that's exactly how it should work. But the software as described in the NTSB report was apparently bad enough that they essentially hardwired an override on their emergency stop. The software equivalent of putting a steel bar into a fuse receptacle. The words that come to mind are 'criminal negligence'. The vehicle would not have been able to do an E-stop even if it was 100% sure it had to do just that, nor did it warn the human luggage. The problem here is not that the world is so unsafe that you will have to make compromises to get anywhere at all, the problem here is that the software is still so buggy that there is no way to safely navigate common scenarios. Pedestrian on the road at night is one that I've encountered twice on my trips and they did not lead to any fatalities because when I can't see I slow down. If 6 seconds isn't enough to make a decision you have no business being on the road in the first place.

blattimwind 8y ago
> Pedestrian on the road at night is one that I've encountered twice on my trips and they did not lead to any fatalities because when I can't see I slow down. If 6 seconds isn't enough to make a decision you have no business being on the road in the first place. I've seen a few people comment on the footage that they too would have run the pedestrian over, to which my only response is: I sure hope you don't have a driver's license [anymore]!
- hcknwscommenter 8y ago
  The vast majority of those people are being (purposely?) deluded by a very misleading video showing an apparently pitch black section of road. In reality, it was a lighted road and the dashcam footage had a very compressed dynamic range.
  - forapurpose 8y ago
    > it was a lighted road The report is a bit ambiguous about that: The videos show that the pedestrian crossed in a section of roadway not directly illuminated by the roadway lighting.

maigret 8y ago

The problem is that drivers rarely maintain the safety distance they should have to not endanger themselves. BUT in that case, the car should have also noticed if there were near traffic behind. Doing nothing in that case doesn’t seem the right decision at all. Very good write anyways... indeed many things will have to change - probably the infrastructure, the vehicles, the software, the way pedestrians move, and driver behavior as well.

nmca 8y ago

This highlights an interesting general point - in many situations, there is no simple safe fallback policy. On a highway, an emergency stop is not safe. This is a general problem in AI safety and is covered nicely in this youtube video, as well as the paper referenced there - https://www.youtube.com/watch?v=lqJUIqZNzP8 https://www.youtube.com/watch?v=lqJUIqZNzP8

jacquesm 8y ago
> On a highway, an emergency stop is not safe. That depends, there could simply be no traffic behind you, which an experienced driver and hopefully and automated one would be monitoring. Besides, there are many situations on the highway where an E-stop is far safer than any of the alternatives even if there is traffic behind you. Driving as though nothing has changed in the presence of an E-stop worthy situation is definitely not the right decision.
- beenBoutIT 8y ago
  How intelligent is the ML driving the car? If the car slowed down and hit the 49 year old at a reduced speed the insurance payout to a now severely disabled individual would be far more expensive than the alternative insurance pay out with a pedestrian fatality. A choice between paying out for 40 years worth of around-the-clock medical care vs. a one-time lump-sum payout to the victim's family would be pretty obvious from a corporate point of view.
  - jacquesm 8y ago
    Are you seriously suggesting that the better software strategy is to aim for the kill because it is cheaper than possibly causing 'only' injury? That should be criminal. I'm all for chalking this one up to criminal negligence and incompetence, outright malice is - for now - off the table, unless someone leaks meeting notes from Uber where they discussed that exact scenario.
    - beenBoutIT 8y ago

boxcardavin 8y ago

This is absolutely the right analysis of how these systems work and why you can't expect autonomous cars to halt traffic deaths. What the Uber crash has shown us is that the tolerance for AVs killing people is probably exactly zero, not some (very meaningful) reduction like 10x or 100x less. My company didn't start with this zero tolerance thing in our minds, but it turns out our self-delivering electric bicycles have a huge advantage for real world safety because they weigh ~60lbs when in autonomous mode and are limited to 12mph. This equals the kinetic energy of myself walking at a brisk pace, or basically something that won't kill purely from blunt force impact. I think the future for autonomy will be unlocked by low mass and low speed vehicles, not cars converted to drive themselves.

jacquesm 8y ago
> What the Uber crash has shown us is that the tolerance for AVs killing people is probably exactly zero, not some (very meaningful) reduction like 10x or 100x less. It hasn't shown that at all. It has documented beyond reasonable doubt that Uber should not be allowed to participate in real world tests of autonomous vehicles. There are plenty of situations where people would fully accept a self driving vehicle killing someone but this isn't one of those.
econochoice 8y ago
> This equals the kinetic energy of myself walking at a brisk pace, or basically something that won't kill purely from blunt force impact. This is an analogy that cannot completely map to cycling. A fall at any speed from a bike is literally a potentially crippling or deadly scenario.
- boxcardavin 8y ago
  When a rider is onboard there’s no autonomy, the bike is only self-delivering.
- grkvlt 8y ago
  Potentially deadly? Maybe, sure, but at low speeds, up to 10 mph say, it is incredibly unlikely that falling off a bicycle (even with no helmet) will do more than cause bruises and damaged ego.

robk 8y ago

I love your pods at T5! They're super fun.

hcknwscommenter 8y ago

Sure, but "[t]he system is not designed to alert the operator." At least they could have alerted the operator. This seems like reckless endangerment or negligent homicide. Luckily for Uber they hit a poor person and no one will hold them responsible. 1.3 seconds is a long time for the operator to act.

YeGoblynQueenne 8y ago

>> This will inevitably result in fatalities -- almost certainly fewer than when humans drive, but plenty of fatalities nonetheless. Where is the almost-certainty coming from that the fatalities would be fewer compared to humans driving? And what does "almost" mean in this case?

majewsky 8y ago
Computers have vastly lower reaction time than humans. Computers have sensory input that humans lack (LIDAR). Computers don't get drowsy or agitated. And "almost" is always a good idea when talking about a future that looks certain. Takes into account the unknown unknowns. And the known unknowns (cough hacking cough).
- YeGoblynQueenne 8y ago
  Fast reaction times, good sensors and unyielding focus are not enough to drive safely. An agent also needs situational awareness and an understanding of the entities in its environment and their relations. Without the ability to understand its environment and react appropriately to it, all the good the fast reaction times will do to an AI agent is to let it take the wrong decisions faster than a human being. Just saying "computers" and waving our hands about won't magically solve the hard problems involved in full autonomy. Allegedly, the industry has some sort of plan to go from where we are now (sorta kinda level-2 autonomy) to full, level-5 autonomy where "computers" will drive more safely than humans. It would be very kind of the industry if they could share that plan with the rest of us, because for the time being it sounds just like what I describe above, saying "computers" and hand-waving everything else.
nkoren 8y ago
That's a sociopolitical question more than a technical one. I posit that: 1.) Road safety -- as far as the current operating concept of cars is concerned (eg., high speeds in mixed environments) -- is not a problem that can be "solved". At best it can only ever be approximated. The quality of approximation will correspond to the number of fatalities. Algorithm improvements will yield diminishing returns: the operating domain is fundamentally unsafe, and will always result in numerous fatalities even when driven "perfectly". 2.) With regards to factors that contribute to driving safety, there are some things that computers are indisputably better at than humans (raw reaction time). There are other things that humans are still better at than computers (synthesising sensory data into a cohesive model of the world, and then reasoning about that world). Computers are continually improving their performance, however. While we don't have all the theories worked out for how machines will eventually surpass human performance in these domains, we don't have a strong reason to believe that machines won't surpass human performance in these domains. The only question is when. (I don't have an answer to this question). 3.) So the question is not "when will autonomous driving be safe" (it won't be), but rather: "what is the minimum level of safety we will accept from autonomous driving?" I'm quite certain that the bar will be set much higher for autonomous driving than for human driving. This is because risk perception -- especially as magnified by a media that thrives on sensationalism -- is based on how "extraordinary" an event seems, much more than how dangerous it actually is. Look at the disparities in sociopolitical responses to, say, plane crashes and Zika virus, versus car crashes and influenza. Autonomous vehicles will be treated more as the former than the latter, and therefore the scrutiny they receive will be vastly higher. 4.) So basically, driverless cars will only find a routine place on the road if and when they have sufficiently fewer fatalities than human driving. My assertion was a bit tautological in this respect, but basically, if they're anywhere near as dangerous as human drivers, then they won't be a thing at all. 5.) Personally, I think that the algorithms won't be able to pass this public-acceptability threshold on their own, because even the best-imaginable algorithm, if adopted on a global basis, would still kill hundreds of thousands of people every year. That's still probably too many. I expect that full automation eventually will become the norm, but only as enabled by new types of infrastructure / urban design which enable it to be safer than automation alone.