I think a reasonable standard is: if someone dies, is seriously injured, or likely would have died or been seriously injured if the driver hadn't intervened in

I think a reasonable standard is: if someone dies, is seriously injured, or likely would have died or been seriously injured if the driver hadn't intervened in a situation that a normally competent driver wouldn't have had a problem then the self-driving system in question should be disabled globally until the vendor of the system can reasonably convince the NTSB or other relevant government agency that they've taken adequate precautions that it's very unlikely to happen again. It's to be expected that people will occasionally die in car accidents. Some things are hard to avoid. If someone on a bicycle comes shooting out into a busy street when they don't have the right away and no one could see them coming, they're probably going to get run over. My general expectation for self-driving vehicles is: don't create unsafe situations that weren't there before, don't behave in unpredictable ways, and if you can't avoid running someone over at least do it in a way that the consequences fall on the person doing risky things. In the above example, swerving to miss the biker is fine if it can be done safely, but swerving to miss the biker and instead running over someone on the sidewalk instead is not. (Or, in other words: if you can't avoid an accident, at least be predictable and try to obey the general principles of right-of-way.) Swerving at a bike for no reason as in the video above should result in suspension of self-driving features (assuming the video is real and wasn't staged in some way).