2 ms·

> one day the manufacturer pushes a mandatory, automatic over-the-air update that contains a critical bug This is a hidden two-fault scenario, which isn't alwa

by ajross 3y ago

> one day the manufacturer pushes a mandatory, automatic over-the-air update that contains a critical bug This is a hidden two-fault scenario, which isn't always "bad engineering" but is usually a smell in that direction. You're (1) accepting that the operator/owner/whatever of the vehicle is technically responsible, (2) acknowledging that this supervision is likely to work in practice, then (3) imagining a hardware/systems failure in the car and simultaneously (4) supposing that the supervision you took for granted in (2) fails at the same time. Basically, you don't need point 3 to make your point about safety logically. You added it to change the villain from the driver to the manufacturer.

lolinder 3y ago
I'm explicitly not accepting points 1 or 2. I set up the scenario to show that no matter how careful the consumer is, their supervision will not work in practice (since the vehicle's behavior can change at any time and without warning) and therefore they cannot be held responsible for the behavior of the vehicle. Either the manufacturer should not be allowed to market it (because it cannot be operated safely) or the manufacturer should be considered the operator.
- ajross 3y ago
  > I set up the scenario to show that no matter how careful the consumer is, their supervision will not work in practice (since the vehicle's behavior can change at any time and without warning) Sorry, can you explain that better? The whole point to the metaphor of "supervision" (e.g. of party A by party B) is that party A is not trusted to behave correctly in all circumstances. You seem to be saying that supervision can only work if B is never required to intervene at all (and by extension, won't, if A's behavior changes)? In which case there's no supervision happening, right? Again, it really seems like you're just playing tricks to make the manufacturer into an unsolvable villain. I mean, you get that's ridiculous right? We trust our cars not to fail already. And have, for a century. And sometimes they do fail, and when that happens we adjust regulatory strategies to minimize risk. Why is any of that different when there's a scary robot involved?
  - djbusby 3y ago
    Did you miss the part where the automobile features and behaviors can suddenly change because of software? But my 1971 Buick is almost exactly the same as it was. There are no sudden changes. When the operator doesn't have full Agency of their hardware, can they be held accountable for it's unexpected behavior?
    - ajross 3y ago
      > When the operator doesn't have full Agency of their hardware, can they be held accountable for it's unexpected behavior? Yes? Because that's what "supervision" means? Again, I'm very much not following your logic. The whole idea behind supervised autonomy[1] is that the device may not behave correctly. Period. If you aren't capable of supervising your car after an update you clearly weren't doing any supervision before the update. [1] Or supervised anything! Would you argue against holding parents accountable for childproofing their homes just because their kids got an "update" and learned to walk? Again I really don't think you're thinking this through.

by ajross 3y ago

lolinder 3y ago

I'm explicitly not accepting points 1 or 2. I set up the scenario to show that no matter how careful the consumer is, their supervision will not work in practice (since the vehicle's behavior can change at any time and without warning) and therefore they cannot be held responsible for the behavior of the vehicle. Either the manufacturer should not be allowed to market it (because it cannot be operated safely) or the manufacturer should be considered the operator.

ajross 3y ago
> I set up the scenario to show that no matter how careful the consumer is, their supervision will not work in practice (since the vehicle's behavior can change at any time and without warning) Sorry, can you explain that better? The whole point to the metaphor of "supervision" (e.g. of party A by party B) is that party A is not trusted to behave correctly in all circumstances. You seem to be saying that supervision can only work if B is never required to intervene at all (and by extension, won't, if A's behavior changes)? In which case there's no supervision happening, right? Again, it really seems like you're just playing tricks to make the manufacturer into an unsolvable villain. I mean, you get that's ridiculous right? We trust our cars not to fail already. And have, for a century. And sometimes they do fail, and when that happens we adjust regulatory strategies to minimize risk. Why is any of that different when there's a scary robot involved?
- djbusby 3y ago
  Did you miss the part where the automobile features and behaviors can suddenly change because of software? But my 1971 Buick is almost exactly the same as it was. There are no sudden changes. When the operator doesn't have full Agency of their hardware, can they be held accountable for it's unexpected behavior?
  - ajross 3y ago
    > When the operator doesn't have full Agency of their hardware, can they be held accountable for it's unexpected behavior? Yes? Because that's what "supervision" means? Again, I'm very much not following your logic. The whole idea behind supervised autonomy[1] is that the device may not behave correctly. Period. If you aren't capable of supervising your car after an update you clearly weren't doing any supervision before the update. [1] Or supervised anything! Would you argue against holding parents accountable for childproofing their homes just because their kids got an "update" and learned to walk? Again I really don't think you're thinking this through.