3 ms·

The fact that it attempts to do more things can, in fact, make it more dangerous in the specific situations where a human interrupt is needed. Human attention w

by jprete 3y ago

The fact that it attempts to do more things can, in fact, make it more dangerous in the specific situations where a human interrupt is needed. Human attention works better the more it is needed, so if you need a human in the loop, you're best off with a system where their input is needed fairly often compared to a system where their input is needed once every long while. The other factor is that human attention may be needed in a situation where it is very hard to actively fix the situation if you're not already in the OODA loop. If you suddenly realize you need to override the car, and you're in the middle of a complex merge and the reason you need to override the car is because someone else screwed up, it may not be possible to solve the problem with the extra cost of your attention spinning up.

wilg 3y ago
I understand what you're saying, but if you're not already in the OODA loop then you aren't actively monitoring the car. You could say, well, maybe the system lulls you into not being in that loop by taking most of the driving tasks away. I would say that having driven thousands of miles with it, it certainly doesn't feel like that to me. I'm still very, instinctively aware of how the car should be acting and even minor deviations from what I would do myself are very conspicuous and cause me to actively double-check its behavior. Also, in its current state, input is needed quite frequently. And the empirical data available (which could definitely be higher fidelity, granted) appears to suggest that the actual safety record is quite good. So we're left with a trade-off: there are some scenarios where a more featureful system could elevate risk, but there are other scenarios where it lowers them. It's not obvious on paper how to balance them, and the empirical (and, for me, experiential) data indicates the trade-off is a reasonable one. Perhaps a large amount of the concern around this comes from people not being aware of what it's like to operate a car in this mode. Maybe people think you really don't have to pay attention? It knows when you've got hands on the wheel, it knows when you're looking at your watch or your phone or the car's display, and it will pretty aggressively prevent you from getting complacent. Not only will it beep, it might lock you out from using it entirely. (Lockouts never happens to me, but I have seen tweets from it happening to others.) There was an old pre-Waymo Google talk about how partial driving assistance was dangerous for this reason, and I bought that argument but I no longer find it convincing having used such a system a lot.

by jprete 3y ago

wilg 3y ago

I understand what you're saying, but if you're not already in the OODA loop then you aren't actively monitoring the car. You could say, well, maybe the system lulls you into not being in that loop by taking most of the driving tasks away. I would say that having driven thousands of miles with it, it certainly doesn't feel like that to me. I'm still very, instinctively aware of how the car should be acting and even minor deviations from what I would do myself are very conspicuous and cause me to actively double-check its behavior. Also, in its current state, input is needed quite frequently. And the empirical data available (which could definitely be higher fidelity, granted) appears to suggest that the actual safety record is quite good. So we're left with a trade-off: there are some scenarios where a more featureful system could elevate risk, but there are other scenarios where it lowers them. It's not obvious on paper how to balance them, and the empirical (and, for me, experiential) data indicates the trade-off is a reasonable one. Perhaps a large amount of the concern around this comes from people not being aware of what it's like to operate a car in this mode. Maybe people think you really don't have to pay attention? It knows when you've got hands on the wheel, it knows when you're looking at your watch or your phone or the car's display, and it will pretty aggressively prevent you from getting complacent. Not only will it beep, it might lock you out from using it entirely. (Lockouts never happens to me, but I have seen tweets from it happening to others.) There was an old pre-Waymo Google talk about how partial driving assistance was dangerous for this reason, and I bought that argument but I no longer find it convincing having used such a system a lot.