12 ms·

This is what's going to happen. If you've ever seen a machine learning algorithm in action, this isn't surprising at all. Basically, they'll behave as expected

This is what's going to happen. If you've ever seen a machine learning algorithm in action, this isn't surprising at all. Basically, they'll behave as expected some well known percentage of the time. But when they don't, the result will not be just a slight deviation from the normal algorithm, but a very unexpected one. So we will have overall a much smaller number of deaths caused by self driving cars, but ones that do happen will be completely unexpected and scary and shitty. You can't really get away from this without putting these cars on rails. Moreover, the human brain won't like processing these freak accidents. People die in car crashes every damn day. But we have become really accustomed to rationalizing that: "they were struck by a drunk driver", "they were texting", "they didn't see the red light", etc. These are "normal" reasons for bad accidents and we can not only rationalize them, but also rationalize how it wouldn't happen to us: "I don't drive near colleges where young kids are likely to drive drunk", "I don't text (much) while I drive", "I pay attention". But these algorithms will not fail like that. Each accident will be unique and weird and scary. I won't be surprised if someone at some point wears a stripy outfit, and the car thinks they are a part of the road, and tries to explicitly chase them down until they are under the wheels. Or if the car suddenly decides that the road continues at a 90 degree angle off a bridge. Or that the splashes from a puddle in front is actually an oncoming car and it must swerve into the school kids crossing the perpendicular road. It'll always be tragic, unpredictable and one-off.

orbitur 9y ago
> I won't be surprised if someone at some point wears a stripy outfit, and the car thinks they are a part of the road, and tries to explicitly chase them down until they are under the wheels. Or if the car suddenly decides that the road continues at a 90 degree angle off a bridge. Or that the splashes from a puddle in front is actually an oncoming car and it must swerve into the school kids crossing the perpendicular road. Are you working on the next season of Black Mirror? In all seriousness, my fear (and maybe not fear, maybe it's happy expectation in light of the nightmare scenarios) is that if a couple of the "weird and terrifying" accidents happen, the gov't would shut down self-driving car usage immediately.
- jessaustin 9y ago
  If that's bad, what happens when the robocars get hacked?
  - stefano 9y ago
    Like we have botnets made out of thousands (millions?) of compromised computers, we could have entire fleets of compromised cars, rentable on the black market using cryptocurrency, that could be used to commit crimes (e.g. homicide) while keeping the killer anonymous. Scary stuff. I hope these self-driving cars will be able and designed to work while completely offline, with no built-in way to ever connect to a network. But given the biggest player in the field seems to be Google, they'll probably be always connected in order to send data to the mothership and receive ads to show you.
    - ctchocula 9y ago
      Using these cars to commit homicide was actually one of the plot points in the second book of Three Body Problem trilogy by Liu Cixin. Very much recommended if you are into sci-fi.
      - jessaustin 9y ago
        Those books are interesting in that Liu seems to get away with seriously portraying narratives that would be out of bounds in the "approved" popular culture of the West. Murder by robocar is a minor example, but others include portrayal of the inherent weakness of societies in which men are effeminate and the superiority of leaving strategic decisions to military authorities. (I don't particularly agree with those propositions, but they are certainly present in the books.)
        njarboe 9y ago
        I'm half-way done with the last book in the trilogy and I am finding the assumptions and viewpoints of the world from a Chinese view quite interesting. The one that struck me most was how he presents humanities greatest strength over the technologically superior aliens is the human ability to conceal their true thoughts and the possibility of deception. Quite different from Christianity's high value placed on honesty.
ttoinou 9y ago
I won't be surprised if someone at some point wears a stripy outfit, and the car thinks they are a part of the road Shouldn't be an issue with 3D cameras
- iliis 9y ago
  Well, in that case a non-textured or reflective clothing could have a similar effect.
  - bduerst 9y ago
    Wouldn't LIDAR pick it up still?
    - IgorPartola 9y ago
      Imagine someone carrying a couple of two by fours they are holding vertically. They then stop on the sidewalk to check their phone at just the right angle. I am not really giving specific examples, as much as trying to illustrate that the way ML systems fail isn't by being slightly off from the intended programming, but by being really off.
- IncRnd 9y ago
  You can't prove a negative and should be careful about promising what may turn out to be false. There is potentially quite a bit of money to be made by people with the auto version of slipping on a pickle jar. When there is money to be made, talented but otherwise misguided people apply their efforts.
bertomartin 9y ago
Seems like you're jumping to conclusions here. Let's wait to see what exactly happened. I highly doubt that any of these companies just use "straight" ML. For complex applications, there's generally a combination of rules-based algorithm and statistical ML based ones applied to solve problems. So to simply: highly suspect predictions aren't just blinded followed.
obastani 9y ago
Very little of what goes into a current-generation self-driving car is based on machine learning [1]. The reason is exactly your point -- algorithmic approaches to self-driving are much safer and more predictable than a machine learning algorithms. Instead, LIDAR should exactly identify potential obstacles to the self-driving car on the road. The extent to which machine learning is used is to classify whether each obstacle is a pedestrian, bicyclist, another car, or something else. By doing so, the self-driving car can improve its ability to plan, e.g., if it predicts that an obstacle is a pedestrian, it can plan for the event that the pedestrian is considering crossing the road, and can reduce speed accordingly. However, the only purpose of this reliance on the machine learning classification should be to improve the comfort of the drive (e.g., avoid abrupt braking). I believe we can reasonably expect that within reason, the self-driving car nevertheless maintains an absolute safety guarantee (i.e., it doesn't run into an obstacle). I say "within reason", because of course if a person jumps in front of a fast moving car, there is no way the car can react. I think it is highly unlikely that this is what happened in the accident -- pedestrians typically exercise reasonable precautions when causing the road. [1] https://www.cs.cmu.edu/~zkolter/pubs/levinson-iv2011.pdf https://www.cs.cmu.edu/~zkolter/pubs/levinson-iv2011.pdf
- IgorPartola 9y ago
  But that's the issue: identifying a pedestrian vs a snowman or a mailbox or a cardboard cutout is important when deciding whether to swerve left or right. It's an asymptotic problem: you'll never get 100% identification, and based on that, even the rigid algorithms will make mistakes. LIDAR is also not perfect when the road is covered in 5 inches of snow and you can't tell where the lanes are. Or at predicting a driver that's going to swerve into your lane because they spilled coffee on their lap or had a stroke. With erratic input, you will get erratic output. Even the best ML vision algorithm will sometimes produce shit output, which will become input to the actual driving algorithm.
  - PyroLagus 9y ago
    > Or at predicting a driver that's going to swerve into your lane because they spilled coffee on their lap or had a stroke. Neither are humans, and a self-driving car can react much faster than any human ever could.
gfodor 9y ago
Do you have any experience developing autonomous driving algorithms? Because you are making a lot of broad claims about their characteristics that only someone with a fairly deep level of expertise could speculate about.
Gudin 9y ago
Or take a slow approach like Google Waymo which was almost perfect so far on the roads. Uber is rushing it, and this could cost whole industry.
- utopcell 9y ago
  exactly. I would personally never trust an Uber self-driving car, specifically because I've lost trust in the company itself.
legohead 9y ago
Your comment reminded me of the One Pixel Attack [1] and my joke about wearing a giant yellow square costume... [1] https://github.com/Hyperparticle/one-pixel-attack-keras https://github.com/Hyperparticle/one-pixel-attack-keras
dalbasal 9y ago
Another quirk of people, particularly acting via "People in Positions of Authority" is that they will need to do something to prevent next time. Why did this happen? What steps have we make sure it will never happen again? These are both methods of analysing & fixing problems and methods of preserving a decision making authority. Sometimes this degrades into a cynical "something must be done" for the sake of doing, but... it's not all (or even mostly) cynical. It just feels wrong going forward without correction, and we won't tolerate this from our decision makers. EVen if we will, they will assume (out of habit) that we won't We can't know how this happened. There is nothing to do. ..and.. this will happen again, but at a rate lower than human driver's more less opaque accidents.... I'm not sure how that works as an alternative finding out what went wrong and doing something. Your comment is easily translated into "you knew there was a glitch in the software, but you let this happen anyway." Something will need to be done.
- jakobegger 9y ago
  It‘s amazing how quickly things can happen after an accident. In a nearby town, people have petitioned for a speed limit for a long time. Nothing happened until a 6 year old boy was killed. Within a few weeks a speed limit was in place.
  - supertrope 9y ago
    Safety rules are written in blood. Often someone has to die before action is taken. But eventually people forget, get sloppy, or consider the rules completely useless. See: nightclub fires.
- grumdan 9y ago
  Even if we assume that we wanted to address this for real, I fear that it will be next to impossible to actually assess whether whatever mistake caused this has actually been addressed when all the technology behind it is proprietary. I can easily see people being swayed by a well-written PR speech about how "human safety" is their "top priority" without anything substantial actually being done behind the scenes. I think any attempts to address such issues have to come with far-ranging transparency regulations on companies, possibly including open-sourcing (most of) their code. I don't think regulatory agencies alone would have the right incentives to actually check up on this properly.
anonu 9y ago
Totally agree on your premise that we can rationalize humans killing humans - but we cannot do so with machines killing humans. If self-driving cars really are safer in the long-run for drivers and pedestrians - maybe what people need is a better grasp on probability and statistics? And self-driving car companies need to show and publicize the data that backs this claim up to win the trust of the population.
- falcolas 9y ago
  If a human kills a human, we have someone in the direct chain of command that we can punish. If an algorithm kills a person... who do we punish? How do we punish them in a severe enough way to encourage making things better? Perhaps, similar to airline crashes, we should expect Uber to pay out to the family, plus a penalty fine. 1m per death? 2? What price to we put on a life?
  - IgorPartola 9y ago
    It's not just about punishment though. It's about our brains being (mostly) good at saying "well they were unlucky, but I'd never find myself in that situation, so I'm OK." And with some brains wired for anxiety doing the opposite and instead only thinking about how they are the ones that would be in all those situations.
  - drewmol 9y ago
    >How do we punish them in a severe enough way to encourage making things better? This is a tough problem, but if we stop or limit the allocation of resources to protect the secretive intellectual property that is autonomoilusly running people over, that is the most effective incentive I can see. Plus it's pretty easy to do. We don't even have to force them to disclose anything, by affording less legal protection, their employees will open-source it for us.
- rebuilder 9y ago
  I'd be surprised if you could educate this problem away just by publishing statistics. Generally, people don't seem to integrate statistics well on an emotional level, but do make decisions based on emotional considerations. I mean, people play the lottery. That's a guaranteed loss, statistically speaking. In fact, it's my understanding that, where I live, you're more likely to get hit by a (human-operated) car on your way to get your lottery ticket than you are to win any significant amount of money. But still people brave death for a barely-existent chance at winning money!
thearn4 9y ago
> Moreover, the human brain won't like processing these freak accidents. I think this is really key. The ability to put the blame on something tangible, like the mistakes of another person, somehow allows for more closure than if it was a random technical failure.
derefr 9y ago
> These are "normal" reasons for bad accidents and we can not only rationalize them, but also rationalize how it wouldn't happen to us: "I don't drive near colleges where young kids are likely to drive drunk", "I don't text (much) while I drive", "I pay attention". Tangent: is there a land vehicle designed for redundant control, the way planes are? I've always wondered how many accidents would have been prevented if there were classes of vehicles (e.g. large trucks) that required two drivers, where control could be transferred (either by push or pull) between the "pilot" and "copilot" of the vehicle. Like a driving-school car, but where both drivers are assumed equally fallible.
- greedo 9y ago
  Pilots don't share control of an aircraft; the copilot may help with some tasks, but unless the captain relinquishes control of the yoke (etc.) he's flying it. So you'd still have issues where a cars "pilot" gets distracted, or makes a poor decision.
  - Piskvorrr 9y ago
    It's even more complex - there's the temporary designation "pilot flying" and "pilot not flying", with a handover protocol and whatnot: https://aviation.stackexchange.com/questions/5078/how-is-aircraft-control-positively-transferred-between-pilot-flying-and-pilot-no https://aviation.stackexchange.com/questions/5078/how-is-air...
pvelagal 9y ago
May be for now, autonomous driving should be limited to freeways and roads that dont have pedestrian crossings/pavements.
- jusonchan 9y ago
  This is a great point. Solve it one step at a time. But the problem is Uber's business plan is to replace drivers with autonomous vehicles ferrying passengers. i.e. take the driver cost out of the equation. Same goes for Waymo and others trying to enter/play in this game. It's always about monetization which kills/slows innovation. Just highway-mode is not going to make a lot of money except in the trucking business and I bet they will succeed soon enough and reduce transportation costs. But passenger vehicles, not so much. May help in reducing fatigue related accidents but not a money making business for a multi-billion dollar company. That being said, really sad for the victim in this incident.
  - nugi 9y ago
    From what I could tell, it never was a money making business. Uber now and forever, operates at a loss.
coldtea 9y ago
>So we will have overall a much smaller number of deaths caused by self driving cars Why? This is what the self-driving cars industry insists on, but has nowhere near been proven (only BS stats, under ideal conditions, no rain, no snow, selected roads, etc -- and those as reported by the companies itself). I can very well imagine a greater than average human driving AI. But I can also imagine being able to write it anytime soon not being a law of nature. It might take decades or centuries to get out of some local maxima. General AI research had also promised the moon once again in the 60s and 70s, and it all died with little to show of in the 80s. It was always "a few years down the line". I'm not so certain that we're gonna get this good car AI anytime soon.
- naasking 9y ago
  > This is what the self-driving cars industry insists on, but has nowhere near been proven Because machines have orders of magnitude fewer failure modes than humans, but with greater efficiency. It's why so much human labour has been automated. There's little reason to think driving will be any different. You can insist all you like that the existing evidence is under "ideal conditions", but a) that's how humans pass their driving tests too, and b) we've gone from self-driving vehicles being a gleam in someone's eye to actual self-driving vehicles on public roads in less than 10 years. Inclement weather won't take another 10 years. It's like you're completely ignoring the clear evidence of rapid advancement just because you think it's a hard problem, while the experts actually building these systems expect fully automated transportation fleets within 15 years.
  - frgtpsswrdlame 9y ago
    Your comment is just random hopeful assertions though... >It's why so much human labour has been automated. But how much human labour that is as complicated as driving has been automated? As far as I can tell automation is very, very bad when it needs to interact with humans who may behave unexpectedly. >b) we've gone from self-driving vehicles being a gleam in someone's eye to actual self-driving vehicles on public roads in less than 10 years. Inclement weather won't take another 10 years. >It's like you're completely ignoring the clear evidence of rapid advancement just because you think it's a hard problem, while the experts actually building these systems expect fully automated transportation fleets within 15 years. Actually plenty of experts within the field disagree with you. “I tell adult audiences not to expect it in their lifetimes. And I say the same thing to students,” he says. “Merely dealing with lighting conditions, weather conditions, and traffic conditions is immensely complicated. The software requirements are extremely daunting. Nobody even has the ability to verify and validate the software. I estimate that the challenge of fully automated cars is 10 orders of magnitude more complicated than [fully automated] commercial aviation.” Steve Shladover, transportation researcher at the University of California, Berkeley http://www.automobilemag.com/news/the-hurdles-facing-autonomous-vehicles/ http://www.automobilemag.com/news/the-hurdles-facing-autonom... With autonomous cars, you see these videos from Google and Uber showing a car driving around, but people have not taken it past 80 percent. It's one of those problems where it's easy to get to the first 80 percent, but it's incredibly difficult to solve the last 20 percent. If you have a good GPS, nicely marked roads like in California, and nice weather without snow or rain, it's actually not that hard. But guess what? To solve the real problem, for you or me to buy a car that can drive autonomously from point A to point B—it's not even close. There are fundamental problems that need to be solved. Herman Herman, Director of the National Robotics Engineering Center @ CMU https://motherboard.vice.com/en_us/article/d7y49y/robotics-lab-uber-gutted-says-driving-cars-are-not-even-close-carnegie-mellon-nrec https://motherboard.vice.com/en_us/article/d7y49y/robotics-l...
Splines 9y ago
Interesting insight. While self-driving cars should reduce the number of accidents, there are going to be some subset of people who are excellent drivers for which self-driving cars will increase their accident rates (for example, the kind of person who stays home when its icy, but decides that their new self-driving car can cope with the conditions).
emodendroket 9y ago
> So we will have overall a much smaller number of deaths caused by self driving cars, but ones that do happen will be completely unexpected and scary and shitty. You can't really get away from this without putting these cars on rails. One of the big questions I have about autonomous driving is if it's really a better solution to the problems it's meant to solve than more public transportation.
IanCal 9y ago
> This is what's going to happen. If you've ever seen a machine learning algorithm in action, this isn't surprising at all. Basically, they'll behave as expected some well known percentage of the time. But when they don't, the result will not be just a slight deviation from the normal algorithm, but a very unexpected one. Do we even know yet what's happened? It seems rather in bad taste to take someones death, not know the circumstances then wax lyrical about how it matches what you'd expect.
yelloweyes 9y ago
Very well put. It boggles my mind that a forum full of computer programmers can look at autonomous cars and think "this is a good idea". They are either delusional and think their code is a gift to humanity or they haven't put much thought into it.
- web007 9y ago
  I don't believe I'm delusional, my code is certainly medium-to-okay. I've put a lot of thought into this. I think autonomous cars are a very good idea, I want to work on building them and I want to own one as soon as safely possible. Autonomous cars, as they exist right now, are not up to the task at hand. That's why they should still have safety drivers and other safeguards in place. I don't know enough to understand their reasoning, but I was very surprised when Waymo removed safety drivers in some cases. This accident is doubly surprising, since there WAS a safety driver in the car in this case. I'll be interested to see the analysis of what happened and what failures occurred to let this happen. Saying that future accidents will be "unexpected" and therefore scary is FUD in its purest form, fear based on uncertainty and doubt. It will be very clear exactly what happened and what the failure case was. Even as the parent stated, "it saw a person with stripes and thought they were road" - that's incredibly stupid, but very simple and explainable. It will also be explainable (and expect-able) the other failures that had to occur for that failure to cause a death. What set of systems (multiple cameras, LIDAR, RADAR, accelerometers, maps, GPS, etc.) had to fail in what combined way for such a failure? Which one of N different individual failures could have prevented the entire failure cascade? What change needs to take place to prevent future failures of this sort - even down to equally stupid reactions to failure as "ban striped clothing"? Obviously any changes should take place in the car itself, either via software or hardware modifications, or operational changes i.e. maximum speed, minimum tolerances / safe zones, even physical modifications to configuration of redundant systems. After that should any laws or norms be changed, should roads be designed with better marking or wider lanes? Should humans have to press a button to continue driving when stopped at a crosswalk, even if they don't have to otherwise operate the car? Lots of people have put a lot of thought into these scenarios. There is even an entire discipline around these questions and answers, functional safety. There's no one answer, but autonomy engineers are not unthinking and delusional.
- 9y ago

orbitur 9y ago

> I won't be surprised if someone at some point wears a stripy outfit, and the car thinks they are a part of the road, and tries to explicitly chase them down until they are under the wheels. Or if the car suddenly decides that the road continues at a 90 degree angle off a bridge. Or that the splashes from a puddle in front is actually an oncoming car and it must swerve into the school kids crossing the perpendicular road. Are you working on the next season of Black Mirror? In all seriousness, my fear (and maybe not fear, maybe it's happy expectation in light of the nightmare scenarios) is that if a couple of the "weird and terrifying" accidents happen, the gov't would shut down self-driving car usage immediately.

jessaustin 9y ago
If that's bad, what happens when the robocars get hacked?
- stefano 9y ago
  Like we have botnets made out of thousands (millions?) of compromised computers, we could have entire fleets of compromised cars, rentable on the black market using cryptocurrency, that could be used to commit crimes (e.g. homicide) while keeping the killer anonymous. Scary stuff. I hope these self-driving cars will be able and designed to work while completely offline, with no built-in way to ever connect to a network. But given the biggest player in the field seems to be Google, they'll probably be always connected in order to send data to the mothership and receive ads to show you.
  - ctchocula 9y ago
    Using these cars to commit homicide was actually one of the plot points in the second book of Three Body Problem trilogy by Liu Cixin. Very much recommended if you are into sci-fi.
    - jessaustin 9y ago
      Those books are interesting in that Liu seems to get away with seriously portraying narratives that would be out of bounds in the "approved" popular culture of the West. Murder by robocar is a minor example, but others include portrayal of the inherent weakness of societies in which men are effeminate and the superiority of leaving strategic decisions to military authorities. (I don't particularly agree with those propositions, but they are certainly present in the books.)
      - njarboe 9y ago
        I'm half-way done with the last book in the trilogy and I am finding the assumptions and viewpoints of the world from a Chinese view quite interesting. The one that struck me most was how he presents humanities greatest strength over the technologically superior aliens is the human ability to conceal their true thoughts and the possibility of deception. Quite different from Christianity's high value placed on honesty.

ttoinou 9y ago

I won't be surprised if someone at some point wears a stripy outfit, and the car thinks they are a part of the road Shouldn't be an issue with 3D cameras

iliis 9y ago
Well, in that case a non-textured or reflective clothing could have a similar effect.
- bduerst 9y ago
  Wouldn't LIDAR pick it up still?
  - IgorPartola 9y ago
    Imagine someone carrying a couple of two by fours they are holding vertically. They then stop on the sidewalk to check their phone at just the right angle. I am not really giving specific examples, as much as trying to illustrate that the way ML systems fail isn't by being slightly off from the intended programming, but by being really off.
IncRnd 9y ago
You can't prove a negative and should be careful about promising what may turn out to be false. There is potentially quite a bit of money to be made by people with the auto version of slipping on a pickle jar. When there is money to be made, talented but otherwise misguided people apply their efforts.

bertomartin 9y ago

Seems like you're jumping to conclusions here. Let's wait to see what exactly happened. I highly doubt that any of these companies just use "straight" ML. For complex applications, there's generally a combination of rules-based algorithm and statistical ML based ones applied to solve problems. So to simply: highly suspect predictions aren't just blinded followed.

obastani 9y ago

Very little of what goes into a current-generation self-driving car is based on machine learning [1]. The reason is exactly your point -- algorithmic approaches to self-driving are much safer and more predictable than a machine learning algorithms. Instead, LIDAR should exactly identify potential obstacles to the self-driving car on the road. The extent to which machine learning is used is to classify whether each obstacle is a pedestrian, bicyclist, another car, or something else. By doing so, the self-driving car can improve its ability to plan, e.g., if it predicts that an obstacle is a pedestrian, it can plan for the event that the pedestrian is considering crossing the road, and can reduce speed accordingly. However, the only purpose of this reliance on the machine learning classification should be to improve the comfort of the drive (e.g., avoid abrupt braking). I believe we can reasonably expect that within reason, the self-driving car nevertheless maintains an absolute safety guarantee (i.e., it doesn't run into an obstacle). I say "within reason", because of course if a person jumps in front of a fast moving car, there is no way the car can react. I think it is highly unlikely that this is what happened in the accident -- pedestrians typically exercise reasonable precautions when causing the road. [1] https://www.cs.cmu.edu/~zkolter/pubs/levinson-iv2011.pdf https://www.cs.cmu.edu/~zkolter/pubs/levinson-iv2011.pdf

IgorPartola 9y ago
But that's the issue: identifying a pedestrian vs a snowman or a mailbox or a cardboard cutout is important when deciding whether to swerve left or right. It's an asymptotic problem: you'll never get 100% identification, and based on that, even the rigid algorithms will make mistakes. LIDAR is also not perfect when the road is covered in 5 inches of snow and you can't tell where the lanes are. Or at predicting a driver that's going to swerve into your lane because they spilled coffee on their lap or had a stroke. With erratic input, you will get erratic output. Even the best ML vision algorithm will sometimes produce shit output, which will become input to the actual driving algorithm.
- PyroLagus 9y ago
  > Or at predicting a driver that's going to swerve into your lane because they spilled coffee on their lap or had a stroke. Neither are humans, and a self-driving car can react much faster than any human ever could.

gfodor 9y ago

Do you have any experience developing autonomous driving algorithms? Because you are making a lot of broad claims about their characteristics that only someone with a fairly deep level of expertise could speculate about.

Gudin 9y ago

Or take a slow approach like Google Waymo which was almost perfect so far on the roads. Uber is rushing it, and this could cost whole industry.

utopcell 9y ago
exactly. I would personally never trust an Uber self-driving car, specifically because I've lost trust in the company itself.

legohead 9y ago

Your comment reminded me of the One Pixel Attack [1] and my joke about wearing a giant yellow square costume... [1] https://github.com/Hyperparticle/one-pixel-attack-keras https://github.com/Hyperparticle/one-pixel-attack-keras

dalbasal 9y ago

Another quirk of people, particularly acting via "People in Positions of Authority" is that they will need to do something to prevent next time. Why did this happen? What steps have we make sure it will never happen again? These are both methods of analysing & fixing problems and methods of preserving a decision making authority. Sometimes this degrades into a cynical "something must be done" for the sake of doing, but... it's not all (or even mostly) cynical. It just feels wrong going forward without correction, and we won't tolerate this from our decision makers. EVen if we will, they will assume (out of habit) that we won't We can't know how this happened. There is nothing to do. ..and.. this will happen again, but at a rate lower than human driver's more less opaque accidents.... I'm not sure how that works as an alternative finding out what went wrong and doing something. Your comment is easily translated into "you knew there was a glitch in the software, but you let this happen anyway." Something will need to be done.

jakobegger 9y ago
It‘s amazing how quickly things can happen after an accident. In a nearby town, people have petitioned for a speed limit for a long time. Nothing happened until a 6 year old boy was killed. Within a few weeks a speed limit was in place.
- supertrope 9y ago
  Safety rules are written in blood. Often someone has to die before action is taken. But eventually people forget, get sloppy, or consider the rules completely useless. See: nightclub fires.
grumdan 9y ago
Even if we assume that we wanted to address this for real, I fear that it will be next to impossible to actually assess whether whatever mistake caused this has actually been addressed when all the technology behind it is proprietary. I can easily see people being swayed by a well-written PR speech about how "human safety" is their "top priority" without anything substantial actually being done behind the scenes. I think any attempts to address such issues have to come with far-ranging transparency regulations on companies, possibly including open-sourcing (most of) their code. I don't think regulatory agencies alone would have the right incentives to actually check up on this properly.

anonu 9y ago

Totally agree on your premise that we can rationalize humans killing humans - but we cannot do so with machines killing humans. If self-driving cars really are safer in the long-run for drivers and pedestrians - maybe what people need is a better grasp on probability and statistics? And self-driving car companies need to show and publicize the data that backs this claim up to win the trust of the population.

falcolas 9y ago
If a human kills a human, we have someone in the direct chain of command that we can punish. If an algorithm kills a person... who do we punish? How do we punish them in a severe enough way to encourage making things better? Perhaps, similar to airline crashes, we should expect Uber to pay out to the family, plus a penalty fine. 1m per death? 2? What price to we put on a life?
- IgorPartola 9y ago
  It's not just about punishment though. It's about our brains being (mostly) good at saying "well they were unlucky, but I'd never find myself in that situation, so I'm OK." And with some brains wired for anxiety doing the opposite and instead only thinking about how they are the ones that would be in all those situations.
- drewmol 9y ago
  >How do we punish them in a severe enough way to encourage making things better? This is a tough problem, but if we stop or limit the allocation of resources to protect the secretive intellectual property that is autonomoilusly running people over, that is the most effective incentive I can see. Plus it's pretty easy to do. We don't even have to force them to disclose anything, by affording less legal protection, their employees will open-source it for us.
rebuilder 9y ago
I'd be surprised if you could educate this problem away just by publishing statistics. Generally, people don't seem to integrate statistics well on an emotional level, but do make decisions based on emotional considerations. I mean, people play the lottery. That's a guaranteed loss, statistically speaking. In fact, it's my understanding that, where I live, you're more likely to get hit by a (human-operated) car on your way to get your lottery ticket than you are to win any significant amount of money. But still people brave death for a barely-existent chance at winning money!

thearn4 9y ago

> Moreover, the human brain won't like processing these freak accidents. I think this is really key. The ability to put the blame on something tangible, like the mistakes of another person, somehow allows for more closure than if it was a random technical failure.

derefr 9y ago

> These are "normal" reasons for bad accidents and we can not only rationalize them, but also rationalize how it wouldn't happen to us: "I don't drive near colleges where young kids are likely to drive drunk", "I don't text (much) while I drive", "I pay attention". Tangent: is there a land vehicle designed for redundant control, the way planes are? I've always wondered how many accidents would have been prevented if there were classes of vehicles (e.g. large trucks) that required two drivers, where control could be transferred (either by push or pull) between the "pilot" and "copilot" of the vehicle. Like a driving-school car, but where both drivers are assumed equally fallible.

greedo 9y ago
Pilots don't share control of an aircraft; the copilot may help with some tasks, but unless the captain relinquishes control of the yoke (etc.) he's flying it. So you'd still have issues where a cars "pilot" gets distracted, or makes a poor decision.
- Piskvorrr 9y ago
  It's even more complex - there's the temporary designation "pilot flying" and "pilot not flying", with a handover protocol and whatnot: https://aviation.stackexchange.com/questions/5078/how-is-aircraft-control-positively-transferred-between-pilot-flying-and-pilot-no https://aviation.stackexchange.com/questions/5078/how-is-air...

pvelagal 9y ago

May be for now, autonomous driving should be limited to freeways and roads that dont have pedestrian crossings/pavements.

jusonchan 9y ago
This is a great point. Solve it one step at a time. But the problem is Uber's business plan is to replace drivers with autonomous vehicles ferrying passengers. i.e. take the driver cost out of the equation. Same goes for Waymo and others trying to enter/play in this game. It's always about monetization which kills/slows innovation. Just highway-mode is not going to make a lot of money except in the trucking business and I bet they will succeed soon enough and reduce transportation costs. But passenger vehicles, not so much. May help in reducing fatigue related accidents but not a money making business for a multi-billion dollar company. That being said, really sad for the victim in this incident.
- nugi 9y ago
  From what I could tell, it never was a money making business. Uber now and forever, operates at a loss.

coldtea 9y ago

>So we will have overall a much smaller number of deaths caused by self driving cars Why? This is what the self-driving cars industry insists on, but has nowhere near been proven (only BS stats, under ideal conditions, no rain, no snow, selected roads, etc -- and those as reported by the companies itself). I can very well imagine a greater than average human driving AI. But I can also imagine being able to write it anytime soon not being a law of nature. It might take decades or centuries to get out of some local maxima. General AI research had also promised the moon once again in the 60s and 70s, and it all died with little to show of in the 80s. It was always "a few years down the line". I'm not so certain that we're gonna get this good car AI anytime soon.

naasking 9y ago
> This is what the self-driving cars industry insists on, but has nowhere near been proven Because machines have orders of magnitude fewer failure modes than humans, but with greater efficiency. It's why so much human labour has been automated. There's little reason to think driving will be any different. You can insist all you like that the existing evidence is under "ideal conditions", but a) that's how humans pass their driving tests too, and b) we've gone from self-driving vehicles being a gleam in someone's eye to actual self-driving vehicles on public roads in less than 10 years. Inclement weather won't take another 10 years. It's like you're completely ignoring the clear evidence of rapid advancement just because you think it's a hard problem, while the experts actually building these systems expect fully automated transportation fleets within 15 years.
- frgtpsswrdlame 9y ago
  Your comment is just random hopeful assertions though... >It's why so much human labour has been automated. But how much human labour that is as complicated as driving has been automated? As far as I can tell automation is very, very bad when it needs to interact with humans who may behave unexpectedly. >b) we've gone from self-driving vehicles being a gleam in someone's eye to actual self-driving vehicles on public roads in less than 10 years. Inclement weather won't take another 10 years. >It's like you're completely ignoring the clear evidence of rapid advancement just because you think it's a hard problem, while the experts actually building these systems expect fully automated transportation fleets within 15 years. Actually plenty of experts within the field disagree with you. “I tell adult audiences not to expect it in their lifetimes. And I say the same thing to students,” he says. “Merely dealing with lighting conditions, weather conditions, and traffic conditions is immensely complicated. The software requirements are extremely daunting. Nobody even has the ability to verify and validate the software. I estimate that the challenge of fully automated cars is 10 orders of magnitude more complicated than [fully automated] commercial aviation.” Steve Shladover, transportation researcher at the University of California, Berkeley http://www.automobilemag.com/news/the-hurdles-facing-autonomous-vehicles/ http://www.automobilemag.com/news/the-hurdles-facing-autonom... With autonomous cars, you see these videos from Google and Uber showing a car driving around, but people have not taken it past 80 percent. It's one of those problems where it's easy to get to the first 80 percent, but it's incredibly difficult to solve the last 20 percent. If you have a good GPS, nicely marked roads like in California, and nice weather without snow or rain, it's actually not that hard. But guess what? To solve the real problem, for you or me to buy a car that can drive autonomously from point A to point B—it's not even close. There are fundamental problems that need to be solved. Herman Herman, Director of the National Robotics Engineering Center @ CMU https://motherboard.vice.com/en_us/article/d7y49y/robotics-lab-uber-gutted-says-driving-cars-are-not-even-close-carnegie-mellon-nrec https://motherboard.vice.com/en_us/article/d7y49y/robotics-l...

Splines 9y ago

Interesting insight. While self-driving cars should reduce the number of accidents, there are going to be some subset of people who are excellent drivers for which self-driving cars will increase their accident rates (for example, the kind of person who stays home when its icy, but decides that their new self-driving car can cope with the conditions).

emodendroket 9y ago

> So we will have overall a much smaller number of deaths caused by self driving cars, but ones that do happen will be completely unexpected and scary and shitty. You can't really get away from this without putting these cars on rails. One of the big questions I have about autonomous driving is if it's really a better solution to the problems it's meant to solve than more public transportation.

IanCal 9y ago

> This is what's going to happen. If you've ever seen a machine learning algorithm in action, this isn't surprising at all. Basically, they'll behave as expected some well known percentage of the time. But when they don't, the result will not be just a slight deviation from the normal algorithm, but a very unexpected one. Do we even know yet what's happened? It seems rather in bad taste to take someones death, not know the circumstances then wax lyrical about how it matches what you'd expect.

yelloweyes 9y ago

Very well put. It boggles my mind that a forum full of computer programmers can look at autonomous cars and think "this is a good idea". They are either delusional and think their code is a gift to humanity or they haven't put much thought into it.

web007 9y ago
I don't believe I'm delusional, my code is certainly medium-to-okay. I've put a lot of thought into this. I think autonomous cars are a very good idea, I want to work on building them and I want to own one as soon as safely possible. Autonomous cars, as they exist right now, are not up to the task at hand. That's why they should still have safety drivers and other safeguards in place. I don't know enough to understand their reasoning, but I was very surprised when Waymo removed safety drivers in some cases. This accident is doubly surprising, since there WAS a safety driver in the car in this case. I'll be interested to see the analysis of what happened and what failures occurred to let this happen. Saying that future accidents will be "unexpected" and therefore scary is FUD in its purest form, fear based on uncertainty and doubt. It will be very clear exactly what happened and what the failure case was. Even as the parent stated, "it saw a person with stripes and thought they were road" - that's incredibly stupid, but very simple and explainable. It will also be explainable (and expect-able) the other failures that had to occur for that failure to cause a death. What set of systems (multiple cameras, LIDAR, RADAR, accelerometers, maps, GPS, etc.) had to fail in what combined way for such a failure? Which one of N different individual failures could have prevented the entire failure cascade? What change needs to take place to prevent future failures of this sort - even down to equally stupid reactions to failure as "ban striped clothing"? Obviously any changes should take place in the car itself, either via software or hardware modifications, or operational changes i.e. maximum speed, minimum tolerances / safe zones, even physical modifications to configuration of redundant systems. After that should any laws or norms be changed, should roads be designed with better marking or wider lanes? Should humans have to press a button to continue driving when stopped at a crosswalk, even if they don't have to otherwise operate the car? Lots of people have put a lot of thought into these scenarios. There is even an entire discipline around these questions and answers, functional safety. There's no one answer, but autonomy engineers are not unthinking and delusional.
9y ago