15 ms·

I find it odd the way people talk about moving through the levels sequentially without really acknowledging there's a massive phase change that happens somewher

by CPLX 10y ago

I find it odd the way people talk about moving through the levels sequentially without really acknowledging there's a massive phase change that happens somewhere between level two and four, where a self-driving car could easily be way more dangerous than the average human driver. That comes as soon as you have a situation where the automation has to be able to handle all foreseeable scenarios without failing because the human is no longer paying attention. It's a fixture of complex adaptive systems analysis that one encounters scenarios where having anything short of perfection is just as good as useless. The implicit assumption in this post is that there are linear returns to incremental progress. That's not necessarily true at all. It's quite possible -- I would argue likely -- that we'll be very tantalizingly close-but-not-good-enough to full automation for a decade or three before we actually change the fundamental nature of transportation.

bluGill 10y ago
Just remember that self driving cars do not have to be perfect, they just need to be better than humans. The use is at just over 1 death per million miles. If self driving cars can reach .5 they are better already. If the cars can detect the times they are bad and pull over (screw your appointment) that might be enough.
- aetherson 10y ago
  Would you get into a self-driving car that you knew had fewer fatalities per mile than the overall human average, but more fatalities per mile than the average of non-drunk, non-sleepy human drivers below the age of 70 and above the age of 20 with more than 5 years of driving experience? Serious question.
  - rsync 10y ago
    No, I would not. We, as a society, are about to learn some hard lessons about software. As in, deuteronomy-level lessons - things that will be (figuratively) carved into stone. We will gain a new appreciation for the value of human decision making.
    - maxerickson 10y ago
      We will gain a new appreciation for the value of human decision making. A pretty big chunk of automobile collisions are directly attributable to poor human decision making. For example, nearly 1/3 of fatal collisions in the US involve a driver with a BAC of 0.08 or higher.
      - Retric 10y ago
        That statistic includes drunk drivers who where not at fault. ed: And if using the NTSB numbers sober drivers with drunk passengers. PS: Drunk driving is stupid and dangerous, but poor statistics bother me. Drunk drivers are more often night time drivers adjusting for that closer to ~1/4.
        maxerickson 10y ago
        So only 1/4 of traffic fatalities are attributable to a single category of poor decisions. Oh boy. Do you have a good source that tracks fault in alcohol involved collisions? (I'm not trying to argue, I'm curious to look at it)
        Retric 10y ago
        I don't want to defend driving while impaired in any way. Sadly, it's still really common to drive while impaired. 121 million self-reported episodes of alcohol-impaired driving among U.S. adults each year Just remember, even if you are sober, driving home from a party at 3AM is still significantly more dangerous than average.
        maxerickson 10y ago
        So 1/4 is just a seat of the pants estimate? I sloppily implied causation above, but I didn't actually say 1/3 of crashes are attributable to alcohol, I said that a "big chunk" are attributable to poor decisions and separately pointed out the large percentage of fatal crashes involving alcohol.
  - Retric 10y ago
    Yes, that would be better than I was when sleepy making it a net safety gain when driving home late. As an added bonus I can turn it on in a traffic jam and relax as that's stressful but not fast enough to be dangerous.
  - deleted 10y ago
    [deleted]
  - Eridrus 10y ago
    What's the price and safety delta? Btw, what is the death/accident rates for non-drunk people? Including accudents from drunk drivers crashing into them.
  - JamesBarney 10y ago
    Yes. Lets assume a. 1 fatality per 100 million miles driven as an average driver[0] b. .5 fatalities per 100 million miles driven as robo driver c. .25 fatalities per 100 million miles driven as 20-70 sober well rested driver I'm 31yr old male so I have an average life expectancy of 47 more years. That means I lose about .5/100 million * 47 yrs * 365days * 24hrs * 60mins * 60s = 7.4 seconds life lost per mile in robo-car. Sober/well-rested that drops to 3.7 seconds of life lost per mile. So I'd be giving up 3.7 seconds of life to be able to read/work/sleep instead of drive for every mile. For me this is a good deal @ high speeds(trade 3.7 seconds of life for a minute of time), and an absolute steal at lower speeds(usual commute averages 4 minutes/mile.) [0]https://www-fars.nhtsa.dot.gov/Main/index.aspx https://www-fars.nhtsa.dot.gov/Main/index.aspx
    - aetherson 10y ago
      This is a well-reasoned answer, but probably not the same trade-off that most people would make. For example, if a car was twice as bad as the average driver (2 fatalities per 100 million miles), then you'd say 29.6 seconds per mile -- maybe something you wouldn't do at freeway speeds, but you would do at city street speeds. But I don't think that most people would get into a car that was twice as likely to crash as an average human driver (including drunks and so forth). Which suggests that most people have a different approach to this question.
      - antisthenes 10y ago
        The average movement speed of a car over a trip is ~22 mph anyway. The gains from coordinated robotic driving could very well be enough to actually reduce speed limits by a significant value, which not only reduces overall fatalities (due to laws of physics), but also makes the equation better for the robotic cars, which might convince more people to switch, leading to a positive feedback loop. That's an optimistic scenario, though.
      - hx87 10y ago
        I would if the drive were limited to city streets, since a crash there has a much lower probability of injury and death compared to a highway.
  - random_upvoter 10y ago
    When a piece of software crashes out of its model space, it becomes infinitely stupid. A drunk driver, while impaired, retains a level of understanding of the situation but a program has nothing to fall back on. I suspect that accidents with automated driving, while rare, will tend to be more severe. Think train wrecks.
    - CPLX 10y ago
      > When a piece of software crashes out of its model space, it becomes infinitely stupid. Well said.
    - ghaff 10y ago
      The liability situation at scale will also be "interesting." With the exception of pharmaceutical side-effects (which, in the case of vaccines are even explicitly dealt with through the government), it's hard to think of another consumer product, properly maintained and operated, where faults randomly kill people because "stuff happens" without consequences. GM didn't get to play the engineering is complicated and there are always tradeoffs card with its ignition switches. (And, yes, I know this was a case of a known problem but product defects still usually trigger liability in any case.)
    - stale2002 10y ago
      Not really. If something goes wrong, 99.9% of the time, the solution is to slam on the brakes, and maybe swerve a little to avoid an immediate obstacle. If a self driving car only does this, it will be a way better driver than a human. Talking about weird edge cases misses the point. Edge case = slam on the brakes.
      - random_upvoter 10y ago
        You assume that the program will recognize the edge cases. But will it recognize the edge cases of recognizing the edge cases?
      - galdosdi 10y ago
        Really? So easy to think of scenarios where there _is_ no safe answer. For example, what if it's dense fast traffic (say, one car space between each car, 50mph) in random black ice conditions and sensors suddenly begin failing so the software no longer trusts is inputs? Based on recent good inputs, accelerating, swerving, or breaking are all dangerous, and based on the loss of "vision" continuing at current speed is dangerous. The safest thing to do might be to begin honking, flashing hazards, and slowing down very slowly, but could still be way riskier than just handing control to a human driver who (in this scenario) can still see. I can think of tons of similar situations. I have a feeling a lot of folks who think it'll be so easy are people who have mainly experience driving in easy areas and conditions. EDIT: I could have easily made the scenario way worse: the road is curvy, or random lanes are closing and reopening due to construction. Loss of visibility then means continuing straight has a high chance of causing a serious wreck quickly rather than maybe eventually
  - figers 10y ago
    Self driving car can avert accidents caused by drunk / sleepy drivers a human wouldn't react to in time. There are videos of Tesla's automatically avoiding many dangerous situations and those cars are the very first of the first self driving cars to come...
  - notheguyouthink 10y ago
    I think in that example, i would move the bar to exactly what you cited. I think there's an implicit understanding in the OPs statement that means "better than average competent humans". Although, another way to look at is: If you fall below the bar for the average human, including drunks/etc, aren't you by definition more safe using this AI system? If your question/argument is: What about those who are safer than average - then i think the answer is, they shouldn't use self driving, only the "poor" drivers should. Of course the next question is how do you determine that. Perhaps infractions on your driving record would inhibit you from driving. If you get caught for drunk driving, that's it, no more license, purely automated driving (even if that means no rain driving, etc). If you're above 70, or etc.
  - nsgi 10y ago
    Would you get into a car with a driver under 20, above 70, with less than five years of driving experience or who is a little sleepy?
  - stale2002 10y ago
    Do you use Taxis? How do you know that they aren't sleepy/inexperienced/a bad driver?
- ghaff 10y ago
  More like 1 per 100 million miles in the US. And remember that's all cars. I'm sure it would be significantly lower if you consider only late model cars with modern safety features. Obviously, a lot of people still die in traffic accidents. But the bar is actually still fairly high for automated systems. I also suspect (though I don't have the numbers) that even if automation were just for highway driving that would, by itself, significantly reduce the fatalities.
- naner 10y ago
  I don't think this is true, people are much less forgiving when machines make fatal mistakes. Also people will be hostile towards the perceived loss of autonomy.
- wtvanhest 10y ago
  I am being somewhat knit-picky, because your point is: "Self driving cars only have to be better than humans who are already very imperfect" But... It isn't just deaths that need to be prevented. It is a combination of total damages, delays, deaths etc. If self driving cars take forever to make a turn or have other AI failures like driving too slow, it won't be acceptable to a large number of people. In addition, you will see a situation in the future where partially automated cars Level 2 or 3 are safer than current driving modes. That means that in order to go from level 3 to level 4, you need the car to be safer than level 3. I want a fully automated self driving car so badly, but I suspect I have a long way before I can hop in a car with no steering wheel and sip an IPA while being rushed home from a tough day at the office.
- schintan 10y ago
  When I drive, at least I am in control. I don't give any thoughts to accident statistics. A self driving car may become safer statistically, but I don't see myself in a self driving car that is safer on an average but still has a non-zero chance of getting me in an accident.
  - yoz-y 10y ago
    There will never be a non-zero chance of getting into accident. But if the car is a safer driver than you then it is a net win.
    - schintan 10y ago
      Factually that may be true, but its a matter of belief. When I drive, I am 100% confident that I will not get into an accident. A self driving car with <100% safety is a net loss in my mind.
      - Al-Khwarizmi 10y ago
        If you are 100% confident that you won't have an accident, you're deluding yourself. Unexpected phenomena like an animal crossing the road, a drunk or reckless driver driving against traffic or barging in from an intersection where you have the priority, etc. can get anyone.
        tbrownaw 10y ago
        If you are 100% confident that you won't have an accident, you're deluding yourself. So? That's something humans tend to do, and anyone trying to sell self-driving cars will have to account for it.
        yoz-y 10y ago
        I think that when, at some point, it will be proven that driverless cars are safer then the regulators should come in and simply ban humans from driving vehicles that do not expressly require to do "weird stuff".
- nradov 10y ago
  In terms of improving safety, I think most people are looking at it backwards. Instead of having the computer drive and expecting the human to take control in dangerous situations we should have the human drive and allow the computer to take control temporarily when it detects a high probability of an imminent collision. This could prevent most collisions and is technically easier to accomplish. In fact some vehicles already have front crash prevention systems and stability control systems that automatically brake and cut the throttle (2 control axes). http://www.iihs.org/iihs/topics/t/automation-and-crash-avoidance/topicoverview http://www.iihs.org/iihs/topics/t/automation-and-crash-avoid... So manufacturers could build upon that baseline by adding more sensor inputs and control axes for collision avoidance. Instead of just braking when a large obstacle appears in front the vehicle could steer around obstacles, exert force on the steering wheel to prevent running off the road, limit throttle inputs when approaching curves, etc. Some aircraft autopilots already have similar functionality by automatically climbing to avoid terrain.
- rhino369 10y ago
  If I'm the government, I'm going to make sure they have to be better than humans with self driving co-pilot. You also can't compare deaths per million miles if the self driving cars don't operate in dangerous conditions. You'd have to compare apple miles to apple miles and orange miles to orange miles. A lot of people die in car wrecks in bad conditions. A car that only kills .5 people per million miles, but doesn't work in the rain, snow, dark, whatever, likely isn't safer than a person.
bpyne 10y ago
"It's a fixture of complex adaptive systems analysis that one encounters scenarios where having anything short of perfection is just as good as useless." Great point. It's one that not enough people are emphasizing. I need to be physically engaged in the driving experience or I won't be attentive in the least. It's been a trait since childhood that my mental state is tied to physical activity. A car that keeps me physically uninvolved until it decides that it needs me would be a death trap for me and anyone around me.
- ghaff 10y ago
  It's not just you. It's a well-researched phenomenon. I do think there's the potential to enable full autonomy in well-defined use cases while requiring manual control elsewhere. (Still problematic--how do you enforce?) But the idea that a driver can take control on short notice from an automated system is a non-starter. Maybe you can assume there's an adult with supervisory control in the vehicle but you certainly can't assume they're prepared to deal with urgent emergencies.
Fricken 10y ago
GM, Ford, Waymo, Uber, Toyota, Apple, Volvo and Zoox have all opted to skip SEO levels 2 and 3, due to overtrust and the handoff problem respectively. These are most of the biggest players. The Germans appear to be largely in a state of clusterfuck, Mobileye is making serious amendments to their strategy, and that pretty much leaves Tesla, who are already in too deep with the incremental approach to back out now.
sandworm101 10y ago
And that transition has been repeatedly tried, and abandoned, in less-demanding autopilot fields. Trains, planes and ships are all less computationaly difficult yet attempts to remove drivers from the equation have come to nothing. If it isnt 100% than it just isnt acceptable.
- aurelianito 10y ago
  Paris metro line 14 has no driver in the trains. Madrid'd airport has a fully automated train shuttle between terminals. I have ridden both of them and they were quite successful removing the drivers on those.
  - sandworm101 10y ago
    Such trains are on short and closed routes above or below ground, or at least protected behind fences. They arent cargo-hauling trains crossing open country. Those all have crews.
    - aurelianito 10y ago
      Sure. But you are moving goals now.
ForrestN 10y ago
There are cultural and legal dimensions here, though. Will some countries prefer partial autonomy/incrementalism in legalizing autonomous cars? Will some people feel more comfortable getting used to autonomous cars while they are still able to drive them when they feel like it?
MR4D 10y ago
Imagine some drunk guy hops into his car in bad weather. He taps the auto-cruise button and then passes out. Snow, rain, construction, - something - hinders the ride somewhere along the way home. Car cannot process the issue correctly, and slows to a stop, alerting the driver. Driver, being sound asleep, cannot hear the annoying alert tone. Alert continues, and possibly after some time, alerts authorities of it's disabled status. I'll bet that in the near future (next decade or so), there will be rising incidents of traffic being blocked due to a stopped automated car who's driver is not responsive, but the computer couldn't handle some construction or weather, and went into fail-safe mode, simply slowing down and stopping (but not pulling over due to lack of space). Heck, we're halfway there with people checking FB on their phones at stoplights, so anticipating this seems reasonable. We haven't solved that problem, and I don't think we'll solve the one you raise (not for a much longer time, at least). Valid problem, but a solution is likely far off.
- alkonaut 10y ago
  Until cars are reliably at the highest level of replacing humans along the entire way of any drive (I.e cars don't have drivers seats) then obviously it must be a criminal offense to be the "driver" while drunk, even if the driver was hoping to sleep all the way home. What has happened here is really better: the drunk guy is asleep in a stopped car, rather than asleep in a car that just hit another car.
  - _archon_ 10y ago
    Interestingly, he can still be arrested for DWI. This would be an interesting case, though, because the mens rea is clearly not present; the human intended to be conveyed by a capable transportation system, and didn't intend or attempt to control a vehicle while in motion. I look forward to the result of this case for legal precedent reasons.
    - CPLX 10y ago
      For what it's worth DWI and its variations are usually strict liability statutes.
alkonaut 10y ago
I think we will soon be in a situation where self driving cars are better than humans for 10% of driving scenarios(highway, gridlock commute, some parking, ...) but nearly useless in others. I think the acceptance for half decent AI will be much lower than it is for poor human drivers, and the hardest parts of autonomous driving is still a very long way out. That's probably fine though, if the automation can reliably discover when the environment changes. Especially since perhaps a majority of the time spent driving is in the very few "easy" scenarios. Human drivers are probably happy to drive 10% of the commute time for the next 2-4 decades. What this means however is that the dream of completely autonomous Ubers and transport trucks is probably much further out than some might hope.
- ghaff 10y ago
  >if the automation can reliably discover when the environment changes. That sounds about right. The ability to read/write/etc. while driving on the highway would actually be a huge win for a lot of people. And, equally important, it seems achievable in, perhaps, a single digit number of years. But there would need to be an unambiguous mode change from "you can do Facebook now" to "you're driving" with plenty of time to make that shift. This isn't as sexy as roboUber, shared cars, etc. but I'm inclined to agree that getting the last 10% of the way to unsupervised self-driving in essentially all situations is going to take a very long time.
  - shuntress 10y ago
    On the highway on-ramp you engage auto pilot. When you are 5,2, and 1/4 mile away from the off ramp you get a warning that auto pilot is preparing to disengage. If you do not accept and manually turn off auto-pilot, it slows down safely to a stop in the break-down lane.
    - nradov 10y ago
      That's exactly how Volvo's Drive Me project is supposed to work. http://www.volvocars.com/intl/about/our-innovation-brands/intellisafe/autonomous-driving/drive-me# http://www.volvocars.com/intl/about/our-innovation-brands/in...
Animats 10y ago
The big change is between levels 2 and 3. That changes who's responsible. At level 2, the driver is responsible for accidents; at level 3, the manufacturer is. California and Michigan have already enacted that into law, that's what NHTSA recommends, Volvo's CEO publicly endorses that, and the big US automakers accept it. Tesla finally accepts it. Uber is still whining, but that's Uber. Volvo's is starting a 100 vehicle test in one city in Sweden. Automation is only available on some major roads. The driver is not required to pay attention in automatic mode. There's a progress bar - minutes until the end of automatic driving. There are previously located points at the end of automatic driving regions where the vehicle can pull over and stop, should the driver not take over when requested. So that's what entry level full autonomy looks like. That's a good level 3. Those go on the road any day now; the vehicles have been built and tested, and the first of the 100 drivers, who are car purchasers, not Volvo employees, have already been chosen.[1] And yes, they work in snow. The test is in Gothenburg, Sweden; they have to work in snow. [1] http://www.volvocars.com/intl/about/our-innovation-brands/intellisafe/autonomous-driving/drive-me/real-life http://www.volvocars.com/intl/about/our-innovation-brands/in...
- secabeen 10y ago
  You have hit the nail on the head here. The 2/3 liability transition is a big one. My guess is that companies will actually delay deploying level 3 systems until they are nearly perfect. This will result in systems that seem really really good at driving (especially in sunny southern california), yet still insist that the driver wiggle the wheel or show some other sign of attentiveness to prevent liability transfer.
  - jrowley 10y ago
    I can't wait for the Show HN hacks to get around these requirements.
    - nradov 10y ago
      That was already done 2 years ago. Some idiot just tied a bottle to the steering wheel of his Mercedes to trick it into sensing that he was still driving. https://www.youtube.com/shared?ci=ALqiEkdD36Q https://www.youtube.com/shared?ci=ALqiEkdD36Q
oconnor663 10y ago
I think there are some basic physical advantages that might make the computers leap ahead of the humans, at least in terms of safety. Things like cameras pointing in all directions, really fast reaction times, and good steering/braking technique. Those don't necessarily help you navigate complicated intersections, so there might still be plenty of practical problems in the phase transition, but maybe fewer safety problems? It could even be that the safety improvements are so dramatic that they become the pressure for us to engineer around the other issues. (Like marking lanes electronically and giving traffic cops new equipment and whatever else.)
nradov 10y ago
One edge case I have seen is law enforcement officers using loudspeakers on their cars to give drivers specific instructions such as to pull over in a certain spot or move out of a certain area. How will level 5 autonomous vehicles handle that case? There is no useful visual signal, just the audio. Will they need speech recognition and NLP that works in noisy environments?
- carb 10y ago
  In theory you could have an API integration with law enforcement that allow them to switch the car into "pull over" mode.
  - nradov 10y ago
    In theory new technology can do all sorts of magical things. In reality the San Jose Police department is broke and can't even afford to hire enough officers to comply with federal guidelines. Where is the funding going to come from?
  - dmitrygr 10y ago
    Scary... Also: hacked in...3...2...1...

by CPLX 10y ago

bluGill 10y ago

Just remember that self driving cars do not have to be perfect, they just need to be better than humans. The use is at just over 1 death per million miles. If self driving cars can reach .5 they are better already. If the cars can detect the times they are bad and pull over (screw your appointment) that might be enough.

aetherson 10y ago
Would you get into a self-driving car that you knew had fewer fatalities per mile than the overall human average, but more fatalities per mile than the average of non-drunk, non-sleepy human drivers below the age of 70 and above the age of 20 with more than 5 years of driving experience? Serious question.
- rsync 10y ago
  No, I would not. We, as a society, are about to learn some hard lessons about software. As in, deuteronomy-level lessons - things that will be (figuratively) carved into stone. We will gain a new appreciation for the value of human decision making.
  - maxerickson 10y ago
    We will gain a new appreciation for the value of human decision making. A pretty big chunk of automobile collisions are directly attributable to poor human decision making. For example, nearly 1/3 of fatal collisions in the US involve a driver with a BAC of 0.08 or higher.
    - Retric 10y ago
      That statistic includes drunk drivers who where not at fault. ed: And if using the NTSB numbers sober drivers with drunk passengers. PS: Drunk driving is stupid and dangerous, but poor statistics bother me. Drunk drivers are more often night time drivers adjusting for that closer to ~1/4.
      - maxerickson 10y ago
        So only 1/4 of traffic fatalities are attributable to a single category of poor decisions. Oh boy. Do you have a good source that tracks fault in alcohol involved collisions? (I'm not trying to argue, I'm curious to look at it)
        Retric 10y ago
        I don't want to defend driving while impaired in any way. Sadly, it's still really common to drive while impaired. 121 million self-reported episodes of alcohol-impaired driving among U.S. adults each year Just remember, even if you are sober, driving home from a party at 3AM is still significantly more dangerous than average.
        maxerickson 10y ago
        So 1/4 is just a seat of the pants estimate? I sloppily implied causation above, but I didn't actually say 1/3 of crashes are attributable to alcohol, I said that a "big chunk" are attributable to poor decisions and separately pointed out the large percentage of fatal crashes involving alcohol.
- Retric 10y ago
  Yes, that would be better than I was when sleepy making it a net safety gain when driving home late. As an added bonus I can turn it on in a traffic jam and relax as that's stressful but not fast enough to be dangerous.
- deleted 10y ago
  [deleted]
- Eridrus 10y ago
  What's the price and safety delta? Btw, what is the death/accident rates for non-drunk people? Including accudents from drunk drivers crashing into them.
- JamesBarney 10y ago
  Yes. Lets assume a. 1 fatality per 100 million miles driven as an average driver[0] b. .5 fatalities per 100 million miles driven as robo driver c. .25 fatalities per 100 million miles driven as 20-70 sober well rested driver I'm 31yr old male so I have an average life expectancy of 47 more years. That means I lose about .5/100 million * 47 yrs * 365days * 24hrs * 60mins * 60s = 7.4 seconds life lost per mile in robo-car. Sober/well-rested that drops to 3.7 seconds of life lost per mile. So I'd be giving up 3.7 seconds of life to be able to read/work/sleep instead of drive for every mile. For me this is a good deal @ high speeds(trade 3.7 seconds of life for a minute of time), and an absolute steal at lower speeds(usual commute averages 4 minutes/mile.) [0]https://www-fars.nhtsa.dot.gov/Main/index.aspx https://www-fars.nhtsa.dot.gov/Main/index.aspx
  - aetherson 10y ago
    This is a well-reasoned answer, but probably not the same trade-off that most people would make. For example, if a car was twice as bad as the average driver (2 fatalities per 100 million miles), then you'd say 29.6 seconds per mile -- maybe something you wouldn't do at freeway speeds, but you would do at city street speeds. But I don't think that most people would get into a car that was twice as likely to crash as an average human driver (including drunks and so forth). Which suggests that most people have a different approach to this question.
    - antisthenes 10y ago
      The average movement speed of a car over a trip is ~22 mph anyway. The gains from coordinated robotic driving could very well be enough to actually reduce speed limits by a significant value, which not only reduces overall fatalities (due to laws of physics), but also makes the equation better for the robotic cars, which might convince more people to switch, leading to a positive feedback loop. That's an optimistic scenario, though.
    - hx87 10y ago
      I would if the drive were limited to city streets, since a crash there has a much lower probability of injury and death compared to a highway.
- random_upvoter 10y ago
  When a piece of software crashes out of its model space, it becomes infinitely stupid. A drunk driver, while impaired, retains a level of understanding of the situation but a program has nothing to fall back on. I suspect that accidents with automated driving, while rare, will tend to be more severe. Think train wrecks.
  - CPLX 10y ago
    > When a piece of software crashes out of its model space, it becomes infinitely stupid. Well said.
  - ghaff 10y ago
    The liability situation at scale will also be "interesting." With the exception of pharmaceutical side-effects (which, in the case of vaccines are even explicitly dealt with through the government), it's hard to think of another consumer product, properly maintained and operated, where faults randomly kill people because "stuff happens" without consequences. GM didn't get to play the engineering is complicated and there are always tradeoffs card with its ignition switches. (And, yes, I know this was a case of a known problem but product defects still usually trigger liability in any case.)
  - stale2002 10y ago
    Not really. If something goes wrong, 99.9% of the time, the solution is to slam on the brakes, and maybe swerve a little to avoid an immediate obstacle. If a self driving car only does this, it will be a way better driver than a human. Talking about weird edge cases misses the point. Edge case = slam on the brakes.
    - random_upvoter 10y ago
      You assume that the program will recognize the edge cases. But will it recognize the edge cases of recognizing the edge cases?
    - galdosdi 10y ago
      Really? So easy to think of scenarios where there _is_ no safe answer. For example, what if it's dense fast traffic (say, one car space between each car, 50mph) in random black ice conditions and sensors suddenly begin failing so the software no longer trusts is inputs? Based on recent good inputs, accelerating, swerving, or breaking are all dangerous, and based on the loss of "vision" continuing at current speed is dangerous. The safest thing to do might be to begin honking, flashing hazards, and slowing down very slowly, but could still be way riskier than just handing control to a human driver who (in this scenario) can still see. I can think of tons of similar situations. I have a feeling a lot of folks who think it'll be so easy are people who have mainly experience driving in easy areas and conditions. EDIT: I could have easily made the scenario way worse: the road is curvy, or random lanes are closing and reopening due to construction. Loss of visibility then means continuing straight has a high chance of causing a serious wreck quickly rather than maybe eventually
- figers 10y ago
  Self driving car can avert accidents caused by drunk / sleepy drivers a human wouldn't react to in time. There are videos of Tesla's automatically avoiding many dangerous situations and those cars are the very first of the first self driving cars to come...
- notheguyouthink 10y ago
  I think in that example, i would move the bar to exactly what you cited. I think there's an implicit understanding in the OPs statement that means "better than average competent humans". Although, another way to look at is: If you fall below the bar for the average human, including drunks/etc, aren't you by definition more safe using this AI system? If your question/argument is: What about those who are safer than average - then i think the answer is, they shouldn't use self driving, only the "poor" drivers should. Of course the next question is how do you determine that. Perhaps infractions on your driving record would inhibit you from driving. If you get caught for drunk driving, that's it, no more license, purely automated driving (even if that means no rain driving, etc). If you're above 70, or etc.
- nsgi 10y ago
  Would you get into a car with a driver under 20, above 70, with less than five years of driving experience or who is a little sleepy?
- stale2002 10y ago
  Do you use Taxis? How do you know that they aren't sleepy/inexperienced/a bad driver?
ghaff 10y ago
More like 1 per 100 million miles in the US. And remember that's all cars. I'm sure it would be significantly lower if you consider only late model cars with modern safety features. Obviously, a lot of people still die in traffic accidents. But the bar is actually still fairly high for automated systems. I also suspect (though I don't have the numbers) that even if automation were just for highway driving that would, by itself, significantly reduce the fatalities.
naner 10y ago
I don't think this is true, people are much less forgiving when machines make fatal mistakes. Also people will be hostile towards the perceived loss of autonomy.
wtvanhest 10y ago
I am being somewhat knit-picky, because your point is: "Self driving cars only have to be better than humans who are already very imperfect" But... It isn't just deaths that need to be prevented. It is a combination of total damages, delays, deaths etc. If self driving cars take forever to make a turn or have other AI failures like driving too slow, it won't be acceptable to a large number of people. In addition, you will see a situation in the future where partially automated cars Level 2 or 3 are safer than current driving modes. That means that in order to go from level 3 to level 4, you need the car to be safer than level 3. I want a fully automated self driving car so badly, but I suspect I have a long way before I can hop in a car with no steering wheel and sip an IPA while being rushed home from a tough day at the office.
schintan 10y ago
When I drive, at least I am in control. I don't give any thoughts to accident statistics. A self driving car may become safer statistically, but I don't see myself in a self driving car that is safer on an average but still has a non-zero chance of getting me in an accident.
- yoz-y 10y ago
  There will never be a non-zero chance of getting into accident. But if the car is a safer driver than you then it is a net win.
  - schintan 10y ago
    Factually that may be true, but its a matter of belief. When I drive, I am 100% confident that I will not get into an accident. A self driving car with <100% safety is a net loss in my mind.
    - Al-Khwarizmi 10y ago
      If you are 100% confident that you won't have an accident, you're deluding yourself. Unexpected phenomena like an animal crossing the road, a drunk or reckless driver driving against traffic or barging in from an intersection where you have the priority, etc. can get anyone.
      - tbrownaw 10y ago
        If you are 100% confident that you won't have an accident, you're deluding yourself. So? That's something humans tend to do, and anyone trying to sell self-driving cars will have to account for it.
        yoz-y 10y ago
        I think that when, at some point, it will be proven that driverless cars are safer then the regulators should come in and simply ban humans from driving vehicles that do not expressly require to do "weird stuff".
nradov 10y ago
In terms of improving safety, I think most people are looking at it backwards. Instead of having the computer drive and expecting the human to take control in dangerous situations we should have the human drive and allow the computer to take control temporarily when it detects a high probability of an imminent collision. This could prevent most collisions and is technically easier to accomplish. In fact some vehicles already have front crash prevention systems and stability control systems that automatically brake and cut the throttle (2 control axes). http://www.iihs.org/iihs/topics/t/automation-and-crash-avoidance/topicoverview http://www.iihs.org/iihs/topics/t/automation-and-crash-avoid... So manufacturers could build upon that baseline by adding more sensor inputs and control axes for collision avoidance. Instead of just braking when a large obstacle appears in front the vehicle could steer around obstacles, exert force on the steering wheel to prevent running off the road, limit throttle inputs when approaching curves, etc. Some aircraft autopilots already have similar functionality by automatically climbing to avoid terrain.
rhino369 10y ago
If I'm the government, I'm going to make sure they have to be better than humans with self driving co-pilot. You also can't compare deaths per million miles if the self driving cars don't operate in dangerous conditions. You'd have to compare apple miles to apple miles and orange miles to orange miles. A lot of people die in car wrecks in bad conditions. A car that only kills .5 people per million miles, but doesn't work in the rain, snow, dark, whatever, likely isn't safer than a person.

bpyne 10y ago

"It's a fixture of complex adaptive systems analysis that one encounters scenarios where having anything short of perfection is just as good as useless." Great point. It's one that not enough people are emphasizing. I need to be physically engaged in the driving experience or I won't be attentive in the least. It's been a trait since childhood that my mental state is tied to physical activity. A car that keeps me physically uninvolved until it decides that it needs me would be a death trap for me and anyone around me.

ghaff 10y ago
It's not just you. It's a well-researched phenomenon. I do think there's the potential to enable full autonomy in well-defined use cases while requiring manual control elsewhere. (Still problematic--how do you enforce?) But the idea that a driver can take control on short notice from an automated system is a non-starter. Maybe you can assume there's an adult with supervisory control in the vehicle but you certainly can't assume they're prepared to deal with urgent emergencies.

Fricken 10y ago

GM, Ford, Waymo, Uber, Toyota, Apple, Volvo and Zoox have all opted to skip SEO levels 2 and 3, due to overtrust and the handoff problem respectively. These are most of the biggest players. The Germans appear to be largely in a state of clusterfuck, Mobileye is making serious amendments to their strategy, and that pretty much leaves Tesla, who are already in too deep with the incremental approach to back out now.

sandworm101 10y ago

And that transition has been repeatedly tried, and abandoned, in less-demanding autopilot fields. Trains, planes and ships are all less computationaly difficult yet attempts to remove drivers from the equation have come to nothing. If it isnt 100% than it just isnt acceptable.

aurelianito 10y ago
Paris metro line 14 has no driver in the trains. Madrid'd airport has a fully automated train shuttle between terminals. I have ridden both of them and they were quite successful removing the drivers on those.
- sandworm101 10y ago
  Such trains are on short and closed routes above or below ground, or at least protected behind fences. They arent cargo-hauling trains crossing open country. Those all have crews.
  - aurelianito 10y ago
    Sure. But you are moving goals now.

ForrestN 10y ago

There are cultural and legal dimensions here, though. Will some countries prefer partial autonomy/incrementalism in legalizing autonomous cars? Will some people feel more comfortable getting used to autonomous cars while they are still able to drive them when they feel like it?

MR4D 10y ago

Imagine some drunk guy hops into his car in bad weather. He taps the auto-cruise button and then passes out. Snow, rain, construction, - something - hinders the ride somewhere along the way home. Car cannot process the issue correctly, and slows to a stop, alerting the driver. Driver, being sound asleep, cannot hear the annoying alert tone. Alert continues, and possibly after some time, alerts authorities of it's disabled status. I'll bet that in the near future (next decade or so), there will be rising incidents of traffic being blocked due to a stopped automated car who's driver is not responsive, but the computer couldn't handle some construction or weather, and went into fail-safe mode, simply slowing down and stopping (but not pulling over due to lack of space). Heck, we're halfway there with people checking FB on their phones at stoplights, so anticipating this seems reasonable. We haven't solved that problem, and I don't think we'll solve the one you raise (not for a much longer time, at least). Valid problem, but a solution is likely far off.

alkonaut 10y ago
Until cars are reliably at the highest level of replacing humans along the entire way of any drive (I.e cars don't have drivers seats) then obviously it must be a criminal offense to be the "driver" while drunk, even if the driver was hoping to sleep all the way home. What has happened here is really better: the drunk guy is asleep in a stopped car, rather than asleep in a car that just hit another car.
- _archon_ 10y ago
  Interestingly, he can still be arrested for DWI. This would be an interesting case, though, because the mens rea is clearly not present; the human intended to be conveyed by a capable transportation system, and didn't intend or attempt to control a vehicle while in motion. I look forward to the result of this case for legal precedent reasons.
  - CPLX 10y ago
    For what it's worth DWI and its variations are usually strict liability statutes.

alkonaut 10y ago

I think we will soon be in a situation where self driving cars are better than humans for 10% of driving scenarios(highway, gridlock commute, some parking, ...) but nearly useless in others. I think the acceptance for half decent AI will be much lower than it is for poor human drivers, and the hardest parts of autonomous driving is still a very long way out. That's probably fine though, if the automation can reliably discover when the environment changes. Especially since perhaps a majority of the time spent driving is in the very few "easy" scenarios. Human drivers are probably happy to drive 10% of the commute time for the next 2-4 decades. What this means however is that the dream of completely autonomous Ubers and transport trucks is probably much further out than some might hope.

ghaff 10y ago
>if the automation can reliably discover when the environment changes. That sounds about right. The ability to read/write/etc. while driving on the highway would actually be a huge win for a lot of people. And, equally important, it seems achievable in, perhaps, a single digit number of years. But there would need to be an unambiguous mode change from "you can do Facebook now" to "you're driving" with plenty of time to make that shift. This isn't as sexy as roboUber, shared cars, etc. but I'm inclined to agree that getting the last 10% of the way to unsupervised self-driving in essentially all situations is going to take a very long time.
- shuntress 10y ago
  On the highway on-ramp you engage auto pilot. When you are 5,2, and 1/4 mile away from the off ramp you get a warning that auto pilot is preparing to disengage. If you do not accept and manually turn off auto-pilot, it slows down safely to a stop in the break-down lane.
  - nradov 10y ago
    That's exactly how Volvo's Drive Me project is supposed to work. http://www.volvocars.com/intl/about/our-innovation-brands/intellisafe/autonomous-driving/drive-me# http://www.volvocars.com/intl/about/our-innovation-brands/in...

Animats 10y ago

The big change is between levels 2 and 3. That changes who's responsible. At level 2, the driver is responsible for accidents; at level 3, the manufacturer is. California and Michigan have already enacted that into law, that's what NHTSA recommends, Volvo's CEO publicly endorses that, and the big US automakers accept it. Tesla finally accepts it. Uber is still whining, but that's Uber. Volvo's is starting a 100 vehicle test in one city in Sweden. Automation is only available on some major roads. The driver is not required to pay attention in automatic mode. There's a progress bar - minutes until the end of automatic driving. There are previously located points at the end of automatic driving regions where the vehicle can pull over and stop, should the driver not take over when requested. So that's what entry level full autonomy looks like. That's a good level 3. Those go on the road any day now; the vehicles have been built and tested, and the first of the 100 drivers, who are car purchasers, not Volvo employees, have already been chosen.[1] And yes, they work in snow. The test is in Gothenburg, Sweden; they have to work in snow. [1] http://www.volvocars.com/intl/about/our-innovation-brands/intellisafe/autonomous-driving/drive-me/real-life http://www.volvocars.com/intl/about/our-innovation-brands/in...

secabeen 10y ago
You have hit the nail on the head here. The 2/3 liability transition is a big one. My guess is that companies will actually delay deploying level 3 systems until they are nearly perfect. This will result in systems that seem really really good at driving (especially in sunny southern california), yet still insist that the driver wiggle the wheel or show some other sign of attentiveness to prevent liability transfer.
- jrowley 10y ago
  I can't wait for the Show HN hacks to get around these requirements.
  - nradov 10y ago
    That was already done 2 years ago. Some idiot just tied a bottle to the steering wheel of his Mercedes to trick it into sensing that he was still driving. https://www.youtube.com/shared?ci=ALqiEkdD36Q https://www.youtube.com/shared?ci=ALqiEkdD36Q

oconnor663 10y ago

I think there are some basic physical advantages that might make the computers leap ahead of the humans, at least in terms of safety. Things like cameras pointing in all directions, really fast reaction times, and good steering/braking technique. Those don't necessarily help you navigate complicated intersections, so there might still be plenty of practical problems in the phase transition, but maybe fewer safety problems? It could even be that the safety improvements are so dramatic that they become the pressure for us to engineer around the other issues. (Like marking lanes electronically and giving traffic cops new equipment and whatever else.)

nradov 10y ago

One edge case I have seen is law enforcement officers using loudspeakers on their cars to give drivers specific instructions such as to pull over in a certain spot or move out of a certain area. How will level 5 autonomous vehicles handle that case? There is no useful visual signal, just the audio. Will they need speech recognition and NLP that works in noisy environments?

carb 10y ago
In theory you could have an API integration with law enforcement that allow them to switch the car into "pull over" mode.
- nradov 10y ago
  In theory new technology can do all sorts of magical things. In reality the San Jose Police department is broke and can't even afford to hire enough officers to comply with federal guidelines. Where is the funding going to come from?
- dmitrygr 10y ago
  Scary... Also: hacked in...3...2...1...