How do you make the system decide that a situation is uncertain? It might have high conviction in an outcome only because it lacks general understanding. You ca

How do you make the system decide that a situation is uncertain? It might have high conviction in an outcome only because it lacks general understanding. You cannot just handwave the problem away saying "oh just pull over when the system doesn't know". You are ignoring the most difficult class of errors, high conviction but incorrect predictions. This is not a question of reducing complexity, as even very simple statistical tools will sometimes give you high conviction (and yet incorrect) predictions because they are simplified models. The assumption that error conditions are easier to recognize and control than normal operating conditions is insidious. For example, an AI could misidentify a patch of ice with very high conviction that it is line marking. A human could too, but we are talking about an AI that has to recognize when it's unsafe and pull over (as you wrote). I'm sure you've seen the case where an NN taught to recognize wolves did so by picking up the presence of snow as the sole feature in a photo. Only because that happened to be the most distinguishing feature of images with a wolf on it! That AI would only recognize a wolf as long as the photo had some snow. It'd also of course misidentify mere snowy landscapes as a wolf. The mistake is silly, but issues like this are very difficult to diagnose once the inputs become more complex.