In near-critical situations there tend to be disjoint optima in how the group of participants need to behave in order to avoid an incident. If the automated sys

In near-critical situations there tend to be disjoint optima in how the group of participants need to behave in order to avoid an incident. If the automated system is programmed to prefer the more - hm how would you call it - calm, civilized, obedient "minimum", but the participants the more agressive one, I can feel your problem. Sometimes just squeezing through resolves a situation more directly and quickly. The designers of those systems are usually not allowed to prefer that. Even if the system would recognize it properly (which needs much more processing power, because the search space is much bigger and, as mentioned, non-convex. I don't think you should be downvoted, because that is one of the core ethical aspects we need to discuss about those systems. And how they interact with humans. SV dystopia "soon there won't be humans in the loop" doesn't help here.