8 ms·

I imagine we'd treat it like we do getting hit by a drunk driver. Vilify reckless programmers, and don't think about it.

deegles 10y ago
This would have a dramatic chilling effect on hiring for self-driving car software developers, which would ironically make them less safe.
- lallysingh 10y ago
  It might lead to some decent work on formal verification of programs.
  - iopq 10y ago
    The problem is that perfectly-working software can easily get you in an accident. "Well, the bus didn't yield, and the programming said it would" It's not quite possible to write a car that avoids ALL accidents because a car has a speed and a turning radius and breaks only work so fast.
    - ben_jones 10y ago
      Still would be nice if the software could go "shit I'm about to get into an accident" and deploy air bags etc. that much faster. Plus now that I'm thinking about it, most accidents probably occur with some option that, had it been utilized, would've prevented the incident. Imagine the insane accident-avoiding swerve maneuvers a software program could potentially pull off.
    - TeMPOraL 10y ago
      > It's not quite possible to write a car that avoids ALL accidents because a car has a speed and a turning radius and breaks only work so fast. It's not possible to have a self-driving car avoid all accidents, but one can presumably get pretty close. The realtime data from sensors give you enough information about the car itself, its surroundings and other objects around it to continuously compute a safety envelope - a subspace of the phase space of controllable parameters (like input, steering) within which the car can stop safely - and then make one of the goals to aggressively steer the car to remain in that envelope. This approach should be able to automagically handle things like safe driving distances or pedestrians suddenly running into the street. Of course there will be a lot of details to account for when implementing this software, but it's important to realize that we have enough computing power to let the car continuously have every possible backup plan for almost any contingency in its electronic brain.
    - seanp2k2 10y ago
      To other replies, since we've reached max depth: yes, and that's why it makes no sense to me to equip such badly-spec'd vehicles with the self-driving bits first (Tesla S excluded). I drive a higher-than-normal performance vehicle for the exact reason that it gives me more options when I need to get out of a bad situation. I can out-break, out-swerve (more lateral grip), and out-accelerate most other cars on the road. The stopping distance (factor of tire grip and break power) has definitely helped me avoid accidents. Lower body roll = more grip during more extreme maneuvers + more control during them. Top Gear discussed speed limits being set based on worst-case breaking distances. Here's an analysis of that: http://www.jmp.co.uk/forward-thinking/update/top-gear-and-speed-limit-debate http://www.jmp.co.uk/forward-thinking/update/top-gear-and-sp... but TL;DR performance vehicles can be safer since they're more capable. Think what an F1 or rally car could do with computers driving it and avoiding accidents.
      - sah2ed 10y ago
        Mind me asking which performance car you drive?
  - elcapitan 10y ago
    Is there actually some way of formal verification of software that is driven mostly by machine learning? I imagine there's a small core of code that runs some trained models, but how are they being formally verified? How do we know there's not a blind spot in that model that turns out to be fatal under certain conditions?
    - lallysingh 10y ago
      I'd imagine that you'd have some hybrid of ML and traditional code, and may be able to reason statistically about the ML sections, and user traditional (verified) code to cut the tail on the distribution. All pipe dreams of mine, but the research potential here could be worth flaming truckloads of grant money :-)
  - dfox 10y ago
    The problem with formal verification of systems like this is that checking whether software matches the specification or whether the specification is self-consistent is not the interesting problem. Whether specification matches the real world is the interesting problem and there is no formal way to verify that.
- pjlegato 10y ago
  For this reason, most self-driving car development will only happen at large companies like Google, Tesla, Ford, etc. because they are the only ones who will be able to afford to purchase a massive general liability insurance policy.
  - seanp2k2 10y ago
    I think that the other part with insurance is that insurers have no idea what the risk involved actually is (since it's not been around long) so they aim way high to cover themselves.
    - nothrabannosir 10y ago
      They should aim as high as they can; the price of a product is not bounded by its cost, but by what people are willing to pay.

deegles 10y ago

This would have a dramatic chilling effect on hiring for self-driving car software developers, which would ironically make them less safe.

lallysingh 10y ago
It might lead to some decent work on formal verification of programs.
- iopq 10y ago
  The problem is that perfectly-working software can easily get you in an accident. "Well, the bus didn't yield, and the programming said it would" It's not quite possible to write a car that avoids ALL accidents because a car has a speed and a turning radius and breaks only work so fast.
  - ben_jones 10y ago
    Still would be nice if the software could go "shit I'm about to get into an accident" and deploy air bags etc. that much faster. Plus now that I'm thinking about it, most accidents probably occur with some option that, had it been utilized, would've prevented the incident. Imagine the insane accident-avoiding swerve maneuvers a software program could potentially pull off.
  - TeMPOraL 10y ago
    > It's not quite possible to write a car that avoids ALL accidents because a car has a speed and a turning radius and breaks only work so fast. It's not possible to have a self-driving car avoid all accidents, but one can presumably get pretty close. The realtime data from sensors give you enough information about the car itself, its surroundings and other objects around it to continuously compute a safety envelope - a subspace of the phase space of controllable parameters (like input, steering) within which the car can stop safely - and then make one of the goals to aggressively steer the car to remain in that envelope. This approach should be able to automagically handle things like safe driving distances or pedestrians suddenly running into the street. Of course there will be a lot of details to account for when implementing this software, but it's important to realize that we have enough computing power to let the car continuously have every possible backup plan for almost any contingency in its electronic brain.
  - seanp2k2 10y ago
    To other replies, since we've reached max depth: yes, and that's why it makes no sense to me to equip such badly-spec'd vehicles with the self-driving bits first (Tesla S excluded). I drive a higher-than-normal performance vehicle for the exact reason that it gives me more options when I need to get out of a bad situation. I can out-break, out-swerve (more lateral grip), and out-accelerate most other cars on the road. The stopping distance (factor of tire grip and break power) has definitely helped me avoid accidents. Lower body roll = more grip during more extreme maneuvers + more control during them. Top Gear discussed speed limits being set based on worst-case breaking distances. Here's an analysis of that: http://www.jmp.co.uk/forward-thinking/update/top-gear-and-speed-limit-debate http://www.jmp.co.uk/forward-thinking/update/top-gear-and-sp... but TL;DR performance vehicles can be safer since they're more capable. Think what an F1 or rally car could do with computers driving it and avoiding accidents.
    - sah2ed 10y ago
      Mind me asking which performance car you drive?
- elcapitan 10y ago
  Is there actually some way of formal verification of software that is driven mostly by machine learning? I imagine there's a small core of code that runs some trained models, but how are they being formally verified? How do we know there's not a blind spot in that model that turns out to be fatal under certain conditions?
  - lallysingh 10y ago
    I'd imagine that you'd have some hybrid of ML and traditional code, and may be able to reason statistically about the ML sections, and user traditional (verified) code to cut the tail on the distribution. All pipe dreams of mine, but the research potential here could be worth flaming truckloads of grant money :-)
- dfox 10y ago
  The problem with formal verification of systems like this is that checking whether software matches the specification or whether the specification is self-consistent is not the interesting problem. Whether specification matches the real world is the interesting problem and there is no formal way to verify that.
pjlegato 10y ago
For this reason, most self-driving car development will only happen at large companies like Google, Tesla, Ford, etc. because they are the only ones who will be able to afford to purchase a massive general liability insurance policy.
- seanp2k2 10y ago
  I think that the other part with insurance is that insurers have no idea what the risk involved actually is (since it's not been around long) so they aim way high to cover themselves.
  - nothrabannosir 10y ago
    They should aim as high as they can; the price of a product is not bounded by its cost, but by what people are willing to pay.