4 ms·

When a piece of software crashes out of its model space, it becomes infinitely stupid. A drunk driver, while impaired, retains a level of understanding of the s

When a piece of software crashes out of its model space, it becomes infinitely stupid. A drunk driver, while impaired, retains a level of understanding of the situation but a program has nothing to fall back on. I suspect that accidents with automated driving, while rare, will tend to be more severe. Think train wrecks.

CPLX 10y ago
> When a piece of software crashes out of its model space, it becomes infinitely stupid. Well said.
ghaff 10y ago
The liability situation at scale will also be "interesting." With the exception of pharmaceutical side-effects (which, in the case of vaccines are even explicitly dealt with through the government), it's hard to think of another consumer product, properly maintained and operated, where faults randomly kill people because "stuff happens" without consequences. GM didn't get to play the engineering is complicated and there are always tradeoffs card with its ignition switches. (And, yes, I know this was a case of a known problem but product defects still usually trigger liability in any case.)
stale2002 10y ago
Not really. If something goes wrong, 99.9% of the time, the solution is to slam on the brakes, and maybe swerve a little to avoid an immediate obstacle. If a self driving car only does this, it will be a way better driver than a human. Talking about weird edge cases misses the point. Edge case = slam on the brakes.
- random_upvoter 10y ago
  You assume that the program will recognize the edge cases. But will it recognize the edge cases of recognizing the edge cases?
- galdosdi 10y ago
  Really? So easy to think of scenarios where there _is_ no safe answer. For example, what if it's dense fast traffic (say, one car space between each car, 50mph) in random black ice conditions and sensors suddenly begin failing so the software no longer trusts is inputs? Based on recent good inputs, accelerating, swerving, or breaking are all dangerous, and based on the loss of "vision" continuing at current speed is dangerous. The safest thing to do might be to begin honking, flashing hazards, and slowing down very slowly, but could still be way riskier than just handing control to a human driver who (in this scenario) can still see. I can think of tons of similar situations. I have a feeling a lot of folks who think it'll be so easy are people who have mainly experience driving in easy areas and conditions. EDIT: I could have easily made the scenario way worse: the road is curvy, or random lanes are closing and reopening due to construction. Loss of visibility then means continuing straight has a high chance of causing a serious wreck quickly rather than maybe eventually
  - sjwright 10y ago
    In that situation I'd rather trust my life to that computer software than to a human driver (including myself) who simply couldn't be trusted to act safely if their vision was suddenly impaired. Still, no driverless car should ever let itself be in dense fast traffic to begin with. Only a human driver would ever be dumb enough to do something so stupid and dangerous. I would expect all high speed roads to be continuously validated by the car network, mapped in detail and kept up to date by the sensors of each car as it drives it. Your car should be streaming a model of the upcoming road from the cloud and be able to fall back onto it in the event of sensor failure. Handing over control to a human driver only works when one is actually in the car. And paying attention. And competent.
  - bsder 10y ago
    > Really? So easy to think of scenarios where there _is_ no safe answer. Then the human is no better or worse, no? One of the improvements is that an autonomous car will avoid getting into a situation it can't dig itself out of. And, at times, it will detect those situations long before a human will. My favorite counterexample for people like you is driving in Southern California in the rain. An autonomous car NOW is probably better than 25-30% of the drivers on the road in that situation.
- jellicle 10y ago
  If something goes wrong, as we've seen with Tesla, the car tends to barrel full-speed into very solid obstacles. Because, duh, something went wrong. Human errors tend to be misjudging something and then getting into a situation that, due to physics, they can't quite get out of. So you can't quite stop the car in time and you hit the car in front of you at 15mph. Car errors are going to be things like "I thought the tractor trailer was a billboard or a bird so I hit it at 60mph with no braking". Or "I lost all sensor data and got stuck in a loop so I drove off the bridge". It's a fundamentally different sort of error.
  - Retric 10y ago
    People also fall asleep at the wheel, with cruse control there now driving randomly. I watched someone driving on a nice sunny day on a low traffic interstate suddenly make a sharp right turn crossing our lane to fly off the highway and start rolling. I have no idea why this happened, but everyone in the car survived so it's not even part of the traffic fatality statistics.
- jcranmer 10y ago
  Most of my driving (as in 90% by mileage) is on highways, and most of that is 70-mph rural highways. The first edge case I'd think of, totally not a weird edge case at all, is sudden 0-visibility fog banks at 70mph. If you think it is a good idea to slam on the brakes in such conditions, someone is likely to lose a life.
  - sjwright 10y ago
    No driverless car should be wholly reliant on visible light sensors. I imagine that an accurate map of the road ahead and front-facing radar would be all that's needed to respond to the situation as safely as any human driver could.
sjwright 10y ago
One hopes the typical self-driving car isn't relying on just one model space in one software process. Surely these systems would have some mechanism for a robust, simple, safety-oriented software process that is continuously running and can take over at any moment, with the singular task of ensuring a safe stop. It should be continuously preparing and revising a stopping strategy. It should even be able to do this with any half its sensors returning spurious or misleading data. Or even if all sensors are suddenly jammed, it should be able to blindly execute the stopping strategy prepared moments prior.

CPLX 10y ago

> When a piece of software crashes out of its model space, it becomes infinitely stupid. Well said.

ghaff 10y ago

The liability situation at scale will also be "interesting." With the exception of pharmaceutical side-effects (which, in the case of vaccines are even explicitly dealt with through the government), it's hard to think of another consumer product, properly maintained and operated, where faults randomly kill people because "stuff happens" without consequences. GM didn't get to play the engineering is complicated and there are always tradeoffs card with its ignition switches. (And, yes, I know this was a case of a known problem but product defects still usually trigger liability in any case.)

stale2002 10y ago

Not really. If something goes wrong, 99.9% of the time, the solution is to slam on the brakes, and maybe swerve a little to avoid an immediate obstacle. If a self driving car only does this, it will be a way better driver than a human. Talking about weird edge cases misses the point. Edge case = slam on the brakes.

random_upvoter 10y ago
You assume that the program will recognize the edge cases. But will it recognize the edge cases of recognizing the edge cases?
galdosdi 10y ago
Really? So easy to think of scenarios where there _is_ no safe answer. For example, what if it's dense fast traffic (say, one car space between each car, 50mph) in random black ice conditions and sensors suddenly begin failing so the software no longer trusts is inputs? Based on recent good inputs, accelerating, swerving, or breaking are all dangerous, and based on the loss of "vision" continuing at current speed is dangerous. The safest thing to do might be to begin honking, flashing hazards, and slowing down very slowly, but could still be way riskier than just handing control to a human driver who (in this scenario) can still see. I can think of tons of similar situations. I have a feeling a lot of folks who think it'll be so easy are people who have mainly experience driving in easy areas and conditions. EDIT: I could have easily made the scenario way worse: the road is curvy, or random lanes are closing and reopening due to construction. Loss of visibility then means continuing straight has a high chance of causing a serious wreck quickly rather than maybe eventually
- sjwright 10y ago
  In that situation I'd rather trust my life to that computer software than to a human driver (including myself) who simply couldn't be trusted to act safely if their vision was suddenly impaired. Still, no driverless car should ever let itself be in dense fast traffic to begin with. Only a human driver would ever be dumb enough to do something so stupid and dangerous. I would expect all high speed roads to be continuously validated by the car network, mapped in detail and kept up to date by the sensors of each car as it drives it. Your car should be streaming a model of the upcoming road from the cloud and be able to fall back onto it in the event of sensor failure. Handing over control to a human driver only works when one is actually in the car. And paying attention. And competent.
- bsder 10y ago
  > Really? So easy to think of scenarios where there _is_ no safe answer. Then the human is no better or worse, no? One of the improvements is that an autonomous car will avoid getting into a situation it can't dig itself out of. And, at times, it will detect those situations long before a human will. My favorite counterexample for people like you is driving in Southern California in the rain. An autonomous car NOW is probably better than 25-30% of the drivers on the road in that situation.
jellicle 10y ago
If something goes wrong, as we've seen with Tesla, the car tends to barrel full-speed into very solid obstacles. Because, duh, something went wrong. Human errors tend to be misjudging something and then getting into a situation that, due to physics, they can't quite get out of. So you can't quite stop the car in time and you hit the car in front of you at 15mph. Car errors are going to be things like "I thought the tractor trailer was a billboard or a bird so I hit it at 60mph with no braking". Or "I lost all sensor data and got stuck in a loop so I drove off the bridge". It's a fundamentally different sort of error.
- Retric 10y ago
  People also fall asleep at the wheel, with cruse control there now driving randomly. I watched someone driving on a nice sunny day on a low traffic interstate suddenly make a sharp right turn crossing our lane to fly off the highway and start rolling. I have no idea why this happened, but everyone in the car survived so it's not even part of the traffic fatality statistics.
jcranmer 10y ago
Most of my driving (as in 90% by mileage) is on highways, and most of that is 70-mph rural highways. The first edge case I'd think of, totally not a weird edge case at all, is sudden 0-visibility fog banks at 70mph. If you think it is a good idea to slam on the brakes in such conditions, someone is likely to lose a life.
- sjwright 10y ago
  No driverless car should be wholly reliant on visible light sensors. I imagine that an accurate map of the road ahead and front-facing radar would be all that's needed to respond to the situation as safely as any human driver could.

sjwright 10y ago

One hopes the typical self-driving car isn't relying on just one model space in one software process. Surely these systems would have some mechanism for a robust, simple, safety-oriented software process that is continuously running and can take over at any moment, with the singular task of ensuring a safe stop. It should be continuously preparing and revising a stopping strategy. It should even be able to do this with any half its sensors returning spurious or misleading data. Or even if all sensors are suddenly jammed, it should be able to blindly execute the stopping strategy prepared moments prior.