14 ms·

>With self-driving cars, I think people kind of accept that now. That even if you don’t quite know how a self-driving car does it all, if it has a lot fewer acc

>With self-driving cars, I think people kind of accept that now. That even if you don’t quite know how a self-driving car does it all, if it has a lot fewer accidents than a person-driven car then it’s a good thing. Unless there is a condition in the environment that causes a systemic failure involving large number of self-driving cars at the same time. This happens all the time with other types of software. People shouldn't assume that machine-learned models are magically different and ignore all common-sense rules we apply to safety-critical applications. In your daily job, would you be willing to bet your entire car on a black box piece of code that seems reliable, because it worked most of the time in the past? Also, recall that flash crash of the stock market. Imagine something like that, but with lots of cars on a highway. If you don't understand "how a self-driving car does it", you are unlikely to install appropriate safeguards. PS: Please don't insult your intelligence by replying that we don't understand how people drive either, so it makes any opaque self-driving software okay.

donaldknuth123 8y ago
Just look at the airline industry, planes are flown by software 90% of the time. Do you also worry about a global blackout of autopilot software? Seems like a silly thing to get stuck on. Our banking and electrical grid systems are far more prone to failure than anything built in SV.
AndrewKemendo 8y ago
This is a common conflation of intelligence evaluation criteria which is discussed in detail in Jose Hernandez-Orallo's 2017 book Measure of all minds. It can be simultaneously true that we cannot describe in a causally deterministic way how a system works while also determining with some confidence that the system is performing as desired. This is true of all human systems. So if you don't know how people drive, but have enough confidence to let them drive based on performance in evaluation scenarios, then similarly we should be able to let machines drive assuming they perform equally as well as humans over the same contexts without knowing how they drive either.
- ddingus 8y ago
  Not until they can demonstrate agency.
  - AndrewKemendo 8y ago
    Do you want machines to have agency or do you want to control them? You can't have both.
    - gmfawcett 8y ago
      Surely there is a continuum between total agency and total control.
      - ddingus 8y ago
        Exactly.
    - ddingus 8y ago
      Sure you can, just as we do with people.
b_tterc_p 8y ago
1) good self driving cars aren’t machine learning models. They leverage machine learning for certain tasks like object classification, but it’s not some black box neural net that learned how to get the car from A to B. 2) you’re describing how a system can crash. Driverless cars don’t share state, besides, I guess, vast scale astronomical and weather conditions. It’s not likely something would cause all cars to crash because of a collective hiccup. There may be a set of circumstances that consistently cause individual cars to crash, but that’s not really the same thing. And we can understand how these events are processed, so unlike people, we can adapt for future repeats of this scenario.
- leereeves 8y ago
  Driverless cars are part of a system that includes GPS, mapping, OTA firmware updates and updates about traffic conditions, regulations, weather, etc. Errors or malicious attacks upstream could lead to disruptive or even dangerous behavior by a lot of cars at the same time.
  - b_tterc_p 8y ago
    Malicious actions are a concern, but they’re at about the same level of concern as alternative acts of terrorism. Unintentional errors in the things you list don’t feel like they’re huge threats. I would Imagine incidents with at most tens of cars and a relatively low risk at that. Not good, but not a scalable threat.
  - rjplatte 8y ago
    Some car companies are more conscious about this than others. Tesla has a refreshingly anti-fragile approach, where Waymo is heavily relying on data captured through massive amounts of infrastructure.
    - DennisP 8y ago
      That sounds interesting. How is Tesla's approach anti-fragile?
chongli 8y ago
systemic failure involving large number of self-driving cars at the same time This happens to human drivers every year. Snow, freezing rain, fog; these adverse weather conditions can lead to large numbers of collisions at the same time. Expecting self driving cars to never have a problem during rare environmental corner cases is to expect them to be vastly superior to human drivers all the time. If you're waiting for that day before you consent to self-driving cars sharing the road with you, you'll be waiting a long time.
- deleted 8y ago
  [deleted]
- m0zg 8y ago
  >> Snow, freezing rain, fog; Well, self-driving cars can currently handle none of the aforementioned conditions. That's why they are being tested where those conditions don't exist, and that's why we're about 20 years from widespread adoption. It rains and snows in much of the rest of the world.
  - chongli 8y ago
    It rains and snows in much of the rest of the world. Yes, I live in the rest of the world and my family has been affected by fatal collisions brought on by adverse weather. I remain skeptical that self driving cars will be able to handle all of these things. I do think they'll be okay for mundane driving where the biggest risk is driver distractions, however. I would like to see cars and maybe even roads made smarter in order to make driving safer. If all the cars on the road were part of a wireless network and they could signal to one another when they're braking or turning, then other vehicles could respond more quickly. Heck, these systems could even be included in phones and pedestrian crossings so that cars could be signalled instantly when a pedestrian steps into the road in order to cross.
    - Swenrekcah 8y ago
      I used to think that this kind of system must be the future, but now I can only think of the greatly increased attack surface if my self-controlled car is not only taking directions from visual input and other sensors, but also all sorts of other unknown devices.
chrisfosterelli 8y ago
> would you be willing to bet your entire car on a black box piece of code that seems reliable, because it worked most of the time in the past For the average non-technical person, this is pretty much their relationship with all technology. I don't think it's exclusive to tech either. I know lots of people that have absolutely no idea how the mechanical side of a car works, but they trust their lives to one at 100km/hr because it has worked generally pretty well in the past.
- apta 8y ago
  > I know lots of people that have absolutely no idea how the mechanical side of a car works, but they trust their lives to one at 100km/hr because it has worked generally pretty well in the past. Car safety has made great progress in the past 10 or so years. People don't have to know how they work under the hood as long as designers and engineers already built safety into them and these designs have been heavily tested by third parties (e.g. Euro NCAP, IIHS, etc) Self driving cars will have to prove their safety, especially in extreme weather and non paved roads.
  - chrisfosterelli 8y ago
    Definitely! Everyone here agrees that they need to prove themselves effective, and there is more work to do there. I think Geoff's point that the parent poster disagrees with is that it isn't necessary to completely understand every single aspect of the model's weights in order to be able to empirically validate it as safe.
  - criddell 8y ago
    Every year something like 30,000 people are killed in car crashes in the US. Say the introduction of self-driving cars cuts that number in half and also say that of those 15,000 deaths, 2/3 of them are due to software or system errors, that's still progress, isn't it?
munificent 8y ago
I think your concern here is less about flaws in the AI training and more about scale issues when you have a million cars with the same brain and thus the same flaw. I share your concern. Human brains have lots of flaws too, but at least every human driver has a different brain, so some fraction of mistakes cancel out. But another way to look at this is the fact that "cloning" the same AI brain across multiple cars is equivalent to a single AI brain controlling many cars. And that does have human analogies where you have a single person who has a dramatic amount of power. Any flaws in that human's reasoning can be disastrous. Trump having access to the nuclear launch codes comes to mind. In practice, the way we mitigate this is by surrounding that human with advisors and other brains to try to help shore up deficiencies. We could do something similar with AI. Have a couple of differently-trained models and compare their outputs. If they disagree wildly, then try to gracefully shut down the system.
bwanab 8y ago
The vast majority of failures in automobile usage today are caused by human conditions that will never affect a machine. Alcohol, drugs, fatigue, social distractions, aggressive driving. Google something like "top causes of automobile accidents". No list I found had exogenous causes (rain, snow, etc) in the top 5. For machine driving to top that isn't going to be that hard given that we are our own worst enemy on the road.
- vertexFarm 8y ago
  This is still a biased argument. Once it scales up, you'll be able to google "top causes of self-driving car accidents." They will have their own flaws and quirks that we don't anticipate yet, and we don't have any good reason to believe the death toll will be inherently less with computers than with people. Plus, there will be exogenous causes of accidents that are far down on the list for humans, but for AI it might represent a much higher risk and this changes the nature of that list entirely. There just aren't reliable statistics on what the safety of this tech will be at this point in time. All we have is extrapolations with lots of bias and little independent review. Maybe it will eventually be worth it, but that experiment will come with a great many human deaths and we can't simply brush aside the ethics of that. I personally think it won't prove to be safer than humans for many decades. This is far more of a complex problem than a lot of the developers think, and I believe there's a case of engineering arrogance going on here as well. On the surface it seems like we have something very close to a viable product, but the remaining little details and issues are important and represent an enormous amount of work, where we are going to see diminishing returns and enormous difficulty making a system that is flexible and robust enough to handle all the required situations more reliably than the average driver on its own for thousands of miles, scaled up to millions of units around the world. They will be good enough for niche applications, but the dream of a general self-driving infrastructure is pretty far off.
  - b_tterc_p 8y ago
    This is a straw man too. We don’t flip a switch and suddenly all cars are driverless, and then we discover that they can’t handle all situations and everyone dies. We slowly scale it out to environments where it is provably safe. We will not see “top causes of self driving car accidents” because known conditions will merit additional development or restriction on usage. What kind of thing could conceivably make that list? Something like snow doesn’t fit, because they won’t run in the snow unless they’re safe to run in the snow. You don’t need to be better than humans for all situations. You need to be safe. Humans are generally well above that threshold, except for when they’re drunk, high, tired, distracted, or otherwise stupid. It doesn’t really make sense to compare deaths per mile for cars vs attentive humans in stable conditions. Both can honestly be reduced to the point at which they’re practically 0 unless it’s the victim’s fault. We do need to be attentive to edge cases that cause cars not to be safe, but we can also observe the overall pattern and draw conclusions on how likely unknown edge cases are to be encountered.
ivl 8y ago
So, this reminds me of a nuclear meltdown vs. the constant pollution form a coal power plant nearby a city. Even with one of those single, ugly, massive failures they could still very well cause fewer deaths.
tlb 8y ago
You could divide this into two kinds of failures: one local to a physical area (such as a traffic light that somehow always looks green to the self-driving cameras) and global ones (such as virus, or a bug like a GPS time rollover causing all cars to do something crazy simultaneously.) Local ones seem unlikely to cause casualties larger than events that fool large numbers of human drivers, like sinkholes or malfunctioning traffic lights or black ice on a freeway. Some of these will probably happen, and the systems will be improved, and they'll happen less and less often. Global ones could be very large, and there are several instances of global, cascading failures in other systems like telephone or data networks. There are known ways to prevent many classes of failures, but getting to 100.000000% is hard.
vertexFarm 8y ago
Also there's a lack of compelling evidence that self-driving cars have a lot fewer accidents than people do, and this argument is always worded in a deceptive way to kind of quickly shoehorn that assumption in and hope nobody notices. Even measuring this metric reliably is hard, and it's not helped at all by the fact that most of these cars in the real world rely on a human operator as a backup. How do we separate their responsibility? And where's the robust, independent review? Any system that requires a human operator to passively observe a boring process for hours at a time, yet snap to attention immediately and without warning to take control and prevent human deaths is simply insane. That's the diametric opposite of how a safe system should be designed. If the human is required for safe operation, they should just be driving at all times to maintain vigilance. I don't see people talk about that much. The assumption seems to be that AI will quickly get better and the human back-up will not be needed--that the human is kind of a temporary workaround. Well I've seen plenty of engineering projects where the temporary fix becomes a long-term pain in the ass because the difficulty of the real solution was massively underestimated and under-budgeted. We might get stuck with these terribly designed AI-human hybrid systems for many decades as AI keeps getting pushed into the horizon and we slowly realize that this is one of those hard problems that doesn't progress the same as most other tech does. There are pure driverless systems out there, but they don't cope well with many of the incongruities and fringe situations which are an unfortunate reality on the open road in most of our country. I think they will be confined to niche uses for a long, long time.
nikofeyn 8y ago
why do people (not you, the quoted part) keep talking as if self-driving cars have already reached the point where they are safer than humans?
- YeGoblynQueenne 8y ago
  Good point. The answer is probably that nobody really cares about safety, they just want to have their cool toys right now. If people really cared about safety, they 'd be advocating for the only certain way to reduce car accidents: fewer cars (or in other words, more public transport, more trains, etc). Except that wouldn't make sense because the whole point is to sell more cars. Self driving "safety" is just the latest marketing gimmick.
mkolodny 8y ago
Despite your warning, I think it's important to say that we don't understand exactly how the brain works. Yet, we still allow people to drive. To know whether we should trust someone to drive, we created a test (driver's license test). I don't see why this would be any different for deciding whether computers should be allowed to drive - we can just make sure self-driving cars pass a road test. We could even do better than driver's license tests and have a human driver at the wheel during the test just in case.
- Barrin92 8y ago
  we don't just let people drive because they can pass a road test. There's probably 12 year old's that can pass a road test with enough training. We let people drive because we know that they are fully capable agents able to make decisions even in case of failure. In other words, they are responsible, reliable, fully cognisant individuals, far beyond the technical ability of operating a machine. A self-driving car is like a parrot trained to speak. Not only is it not an agent aware of its larger surrounding and potential implications of getting into unknown situations. It also has no real understanding of what driving is at all, it's even stupider than a pet. And because its software is usually deployed to thousands if not millions of devices at the same time, there is little to no variance to protect from extreme failure.
  - seanmcdirmid 8y ago
    > We let people drive because we know that they are fully capable agents able to make decisions even in case of failure. In other words, they are responsible, reliable, fully cognisant individuals, far beyond the technical ability of operating a machine. I wish I could share your optimism in humanity, but what you just claimed sounds far from being true universally. Take your typical driver and throw in a failure situation they have never trained before (because, who has time to train them for every possible, or evens few likely failures), and it is not likely to end well.
    - Barrin92 8y ago
      >Take your typical driver and throw in a failure situation they have never trained before (because, who has time to train them for every possible, or evens few likely failures Humans have a base line of common sense reasoning and the ability to create coherent models of the world. A human driver isn't (rare exceptions aside) going to drive up an impossible street or try to dodge a street sign mistaken for a kangoroo in West Germany. There's nice examples of how an ML algorithm can be tricked into all sorts of pathological behaviour by choosing adversarial images, there's no baseline of reason to fall back on. Imagine a model that has one of these fatal edge cases baked in and a thousand cars suddenly mistake a flash of light for a u-turn. Sure, one or two human drivers out of a ten thousand might drive that irrationally on occasion, but compared to a machine we can usually cope with this. And I don't even want to imagine what happens if this turns malicious. Imagine a bad actor on a bridge above a highway with some artifact that messes automated cars recognition up and the developers forgot to put some hard fail-safe in for that piece of road.
gowld 8y ago
> PS: Please don't insult your intelligence by replying that we don't understand how people drive either, so it makes any opaque self-driving software okay. I won't insult my intelligence; I'll ask why you trust people even though you don't know how they work and you know that at a high rate they intentionally impair themselves before driving? Recall that time when a human trader lost a billion dollars? https://traderhq.com/traders-cost-company-billions/ https://traderhq.com/traders-cost-company-billions/ Remember when the economy tanked because everyone in finance used Black-Scholes model that seemed reliable, because it worked most of the time in the past?
seanmcdirmid 8y ago
> Unless there is a condition in the environment that causes a systemic failure involving large number of self-driving cars at the same time. This happens all the time already: there is a condition in the environment (eg when it snows in Seattle) there is a systematic failure involving large numbers of people driven cars at the same time. People just don’t have experience (erm, “training”) with the weird unusual environmental condition and fail to deal with it. Not surprising that computers would have similar problems. > PS: Please don't insult your intelligence by replying that we don't understand how people drive either, so it makes any opaque self-driving software okay. But we actually do know how well people drive. We have data on failure rates for sure. We don’t know how they drive, that is completely opaque.

donaldknuth123 8y ago

Just look at the airline industry, planes are flown by software 90% of the time. Do you also worry about a global blackout of autopilot software? Seems like a silly thing to get stuck on. Our banking and electrical grid systems are far more prone to failure than anything built in SV.

AndrewKemendo 8y ago

This is a common conflation of intelligence evaluation criteria which is discussed in detail in Jose Hernandez-Orallo's 2017 book Measure of all minds. It can be simultaneously true that we cannot describe in a causally deterministic way how a system works while also determining with some confidence that the system is performing as desired. This is true of all human systems. So if you don't know how people drive, but have enough confidence to let them drive based on performance in evaluation scenarios, then similarly we should be able to let machines drive assuming they perform equally as well as humans over the same contexts without knowing how they drive either.

ddingus 8y ago
Not until they can demonstrate agency.
- AndrewKemendo 8y ago
  Do you want machines to have agency or do you want to control them? You can't have both.
  - gmfawcett 8y ago
    Surely there is a continuum between total agency and total control.
    - ddingus 8y ago
      Exactly.
  - ddingus 8y ago
    Sure you can, just as we do with people.

b_tterc_p 8y ago

1) good self driving cars aren’t machine learning models. They leverage machine learning for certain tasks like object classification, but it’s not some black box neural net that learned how to get the car from A to B. 2) you’re describing how a system can crash. Driverless cars don’t share state, besides, I guess, vast scale astronomical and weather conditions. It’s not likely something would cause all cars to crash because of a collective hiccup. There may be a set of circumstances that consistently cause individual cars to crash, but that’s not really the same thing. And we can understand how these events are processed, so unlike people, we can adapt for future repeats of this scenario.

leereeves 8y ago
Driverless cars are part of a system that includes GPS, mapping, OTA firmware updates and updates about traffic conditions, regulations, weather, etc. Errors or malicious attacks upstream could lead to disruptive or even dangerous behavior by a lot of cars at the same time.
- b_tterc_p 8y ago
  Malicious actions are a concern, but they’re at about the same level of concern as alternative acts of terrorism. Unintentional errors in the things you list don’t feel like they’re huge threats. I would Imagine incidents with at most tens of cars and a relatively low risk at that. Not good, but not a scalable threat.
- rjplatte 8y ago
  Some car companies are more conscious about this than others. Tesla has a refreshingly anti-fragile approach, where Waymo is heavily relying on data captured through massive amounts of infrastructure.
  - DennisP 8y ago
    That sounds interesting. How is Tesla's approach anti-fragile?

chongli 8y ago

systemic failure involving large number of self-driving cars at the same time This happens to human drivers every year. Snow, freezing rain, fog; these adverse weather conditions can lead to large numbers of collisions at the same time. Expecting self driving cars to never have a problem during rare environmental corner cases is to expect them to be vastly superior to human drivers all the time. If you're waiting for that day before you consent to self-driving cars sharing the road with you, you'll be waiting a long time.

deleted 8y ago
[deleted]
m0zg 8y ago
>> Snow, freezing rain, fog; Well, self-driving cars can currently handle none of the aforementioned conditions. That's why they are being tested where those conditions don't exist, and that's why we're about 20 years from widespread adoption. It rains and snows in much of the rest of the world.
- chongli 8y ago
  It rains and snows in much of the rest of the world. Yes, I live in the rest of the world and my family has been affected by fatal collisions brought on by adverse weather. I remain skeptical that self driving cars will be able to handle all of these things. I do think they'll be okay for mundane driving where the biggest risk is driver distractions, however. I would like to see cars and maybe even roads made smarter in order to make driving safer. If all the cars on the road were part of a wireless network and they could signal to one another when they're braking or turning, then other vehicles could respond more quickly. Heck, these systems could even be included in phones and pedestrian crossings so that cars could be signalled instantly when a pedestrian steps into the road in order to cross.
  - Swenrekcah 8y ago
    I used to think that this kind of system must be the future, but now I can only think of the greatly increased attack surface if my self-controlled car is not only taking directions from visual input and other sensors, but also all sorts of other unknown devices.

chrisfosterelli 8y ago

> would you be willing to bet your entire car on a black box piece of code that seems reliable, because it worked most of the time in the past For the average non-technical person, this is pretty much their relationship with all technology. I don't think it's exclusive to tech either. I know lots of people that have absolutely no idea how the mechanical side of a car works, but they trust their lives to one at 100km/hr because it has worked generally pretty well in the past.

apta 8y ago
> I know lots of people that have absolutely no idea how the mechanical side of a car works, but they trust their lives to one at 100km/hr because it has worked generally pretty well in the past. Car safety has made great progress in the past 10 or so years. People don't have to know how they work under the hood as long as designers and engineers already built safety into them and these designs have been heavily tested by third parties (e.g. Euro NCAP, IIHS, etc) Self driving cars will have to prove their safety, especially in extreme weather and non paved roads.
- chrisfosterelli 8y ago
  Definitely! Everyone here agrees that they need to prove themselves effective, and there is more work to do there. I think Geoff's point that the parent poster disagrees with is that it isn't necessary to completely understand every single aspect of the model's weights in order to be able to empirically validate it as safe.
- criddell 8y ago
  Every year something like 30,000 people are killed in car crashes in the US. Say the introduction of self-driving cars cuts that number in half and also say that of those 15,000 deaths, 2/3 of them are due to software or system errors, that's still progress, isn't it?

munificent 8y ago

I think your concern here is less about flaws in the AI training and more about scale issues when you have a million cars with the same brain and thus the same flaw. I share your concern. Human brains have lots of flaws too, but at least every human driver has a different brain, so some fraction of mistakes cancel out. But another way to look at this is the fact that "cloning" the same AI brain across multiple cars is equivalent to a single AI brain controlling many cars. And that does have human analogies where you have a single person who has a dramatic amount of power. Any flaws in that human's reasoning can be disastrous. Trump having access to the nuclear launch codes comes to mind. In practice, the way we mitigate this is by surrounding that human with advisors and other brains to try to help shore up deficiencies. We could do something similar with AI. Have a couple of differently-trained models and compare their outputs. If they disagree wildly, then try to gracefully shut down the system.

bwanab 8y ago

The vast majority of failures in automobile usage today are caused by human conditions that will never affect a machine. Alcohol, drugs, fatigue, social distractions, aggressive driving. Google something like "top causes of automobile accidents". No list I found had exogenous causes (rain, snow, etc) in the top 5. For machine driving to top that isn't going to be that hard given that we are our own worst enemy on the road.

vertexFarm 8y ago
This is still a biased argument. Once it scales up, you'll be able to google "top causes of self-driving car accidents." They will have their own flaws and quirks that we don't anticipate yet, and we don't have any good reason to believe the death toll will be inherently less with computers than with people. Plus, there will be exogenous causes of accidents that are far down on the list for humans, but for AI it might represent a much higher risk and this changes the nature of that list entirely. There just aren't reliable statistics on what the safety of this tech will be at this point in time. All we have is extrapolations with lots of bias and little independent review. Maybe it will eventually be worth it, but that experiment will come with a great many human deaths and we can't simply brush aside the ethics of that. I personally think it won't prove to be safer than humans for many decades. This is far more of a complex problem than a lot of the developers think, and I believe there's a case of engineering arrogance going on here as well. On the surface it seems like we have something very close to a viable product, but the remaining little details and issues are important and represent an enormous amount of work, where we are going to see diminishing returns and enormous difficulty making a system that is flexible and robust enough to handle all the required situations more reliably than the average driver on its own for thousands of miles, scaled up to millions of units around the world. They will be good enough for niche applications, but the dream of a general self-driving infrastructure is pretty far off.
- b_tterc_p 8y ago
  This is a straw man too. We don’t flip a switch and suddenly all cars are driverless, and then we discover that they can’t handle all situations and everyone dies. We slowly scale it out to environments where it is provably safe. We will not see “top causes of self driving car accidents” because known conditions will merit additional development or restriction on usage. What kind of thing could conceivably make that list? Something like snow doesn’t fit, because they won’t run in the snow unless they’re safe to run in the snow. You don’t need to be better than humans for all situations. You need to be safe. Humans are generally well above that threshold, except for when they’re drunk, high, tired, distracted, or otherwise stupid. It doesn’t really make sense to compare deaths per mile for cars vs attentive humans in stable conditions. Both can honestly be reduced to the point at which they’re practically 0 unless it’s the victim’s fault. We do need to be attentive to edge cases that cause cars not to be safe, but we can also observe the overall pattern and draw conclusions on how likely unknown edge cases are to be encountered.

ivl 8y ago

So, this reminds me of a nuclear meltdown vs. the constant pollution form a coal power plant nearby a city. Even with one of those single, ugly, massive failures they could still very well cause fewer deaths.

tlb 8y ago

You could divide this into two kinds of failures: one local to a physical area (such as a traffic light that somehow always looks green to the self-driving cameras) and global ones (such as virus, or a bug like a GPS time rollover causing all cars to do something crazy simultaneously.) Local ones seem unlikely to cause casualties larger than events that fool large numbers of human drivers, like sinkholes or malfunctioning traffic lights or black ice on a freeway. Some of these will probably happen, and the systems will be improved, and they'll happen less and less often. Global ones could be very large, and there are several instances of global, cascading failures in other systems like telephone or data networks. There are known ways to prevent many classes of failures, but getting to 100.000000% is hard.

vertexFarm 8y ago

Also there's a lack of compelling evidence that self-driving cars have a lot fewer accidents than people do, and this argument is always worded in a deceptive way to kind of quickly shoehorn that assumption in and hope nobody notices. Even measuring this metric reliably is hard, and it's not helped at all by the fact that most of these cars in the real world rely on a human operator as a backup. How do we separate their responsibility? And where's the robust, independent review? Any system that requires a human operator to passively observe a boring process for hours at a time, yet snap to attention immediately and without warning to take control and prevent human deaths is simply insane. That's the diametric opposite of how a safe system should be designed. If the human is required for safe operation, they should just be driving at all times to maintain vigilance. I don't see people talk about that much. The assumption seems to be that AI will quickly get better and the human back-up will not be needed--that the human is kind of a temporary workaround. Well I've seen plenty of engineering projects where the temporary fix becomes a long-term pain in the ass because the difficulty of the real solution was massively underestimated and under-budgeted. We might get stuck with these terribly designed AI-human hybrid systems for many decades as AI keeps getting pushed into the horizon and we slowly realize that this is one of those hard problems that doesn't progress the same as most other tech does. There are pure driverless systems out there, but they don't cope well with many of the incongruities and fringe situations which are an unfortunate reality on the open road in most of our country. I think they will be confined to niche uses for a long, long time.

nikofeyn 8y ago

why do people (not you, the quoted part) keep talking as if self-driving cars have already reached the point where they are safer than humans?

YeGoblynQueenne 8y ago
Good point. The answer is probably that nobody really cares about safety, they just want to have their cool toys right now. If people really cared about safety, they 'd be advocating for the only certain way to reduce car accidents: fewer cars (or in other words, more public transport, more trains, etc). Except that wouldn't make sense because the whole point is to sell more cars. Self driving "safety" is just the latest marketing gimmick.

mkolodny 8y ago

Despite your warning, I think it's important to say that we don't understand exactly how the brain works. Yet, we still allow people to drive. To know whether we should trust someone to drive, we created a test (driver's license test). I don't see why this would be any different for deciding whether computers should be allowed to drive - we can just make sure self-driving cars pass a road test. We could even do better than driver's license tests and have a human driver at the wheel during the test just in case.

Barrin92 8y ago
we don't just let people drive because they can pass a road test. There's probably 12 year old's that can pass a road test with enough training. We let people drive because we know that they are fully capable agents able to make decisions even in case of failure. In other words, they are responsible, reliable, fully cognisant individuals, far beyond the technical ability of operating a machine. A self-driving car is like a parrot trained to speak. Not only is it not an agent aware of its larger surrounding and potential implications of getting into unknown situations. It also has no real understanding of what driving is at all, it's even stupider than a pet. And because its software is usually deployed to thousands if not millions of devices at the same time, there is little to no variance to protect from extreme failure.
- seanmcdirmid 8y ago
  > We let people drive because we know that they are fully capable agents able to make decisions even in case of failure. In other words, they are responsible, reliable, fully cognisant individuals, far beyond the technical ability of operating a machine. I wish I could share your optimism in humanity, but what you just claimed sounds far from being true universally. Take your typical driver and throw in a failure situation they have never trained before (because, who has time to train them for every possible, or evens few likely failures), and it is not likely to end well.
  - Barrin92 8y ago
    >Take your typical driver and throw in a failure situation they have never trained before (because, who has time to train them for every possible, or evens few likely failures Humans have a base line of common sense reasoning and the ability to create coherent models of the world. A human driver isn't (rare exceptions aside) going to drive up an impossible street or try to dodge a street sign mistaken for a kangoroo in West Germany. There's nice examples of how an ML algorithm can be tricked into all sorts of pathological behaviour by choosing adversarial images, there's no baseline of reason to fall back on. Imagine a model that has one of these fatal edge cases baked in and a thousand cars suddenly mistake a flash of light for a u-turn. Sure, one or two human drivers out of a ten thousand might drive that irrationally on occasion, but compared to a machine we can usually cope with this. And I don't even want to imagine what happens if this turns malicious. Imagine a bad actor on a bridge above a highway with some artifact that messes automated cars recognition up and the developers forgot to put some hard fail-safe in for that piece of road.

gowld 8y ago

> PS: Please don't insult your intelligence by replying that we don't understand how people drive either, so it makes any opaque self-driving software okay. I won't insult my intelligence; I'll ask why you trust people even though you don't know how they work and you know that at a high rate they intentionally impair themselves before driving? Recall that time when a human trader lost a billion dollars? https://traderhq.com/traders-cost-company-billions/ https://traderhq.com/traders-cost-company-billions/ Remember when the economy tanked because everyone in finance used Black-Scholes model that seemed reliable, because it worked most of the time in the past?

seanmcdirmid 8y ago

> Unless there is a condition in the environment that causes a systemic failure involving large number of self-driving cars at the same time. This happens all the time already: there is a condition in the environment (eg when it snows in Seattle) there is a systematic failure involving large numbers of people driven cars at the same time. People just don’t have experience (erm, “training”) with the weird unusual environmental condition and fail to deal with it. Not surprising that computers would have similar problems. > PS: Please don't insult your intelligence by replying that we don't understand how people drive either, so it makes any opaque self-driving software okay. But we actually do know how well people drive. We have data on failure rates for sure. We don’t know how they drive, that is completely opaque.