The key is how we respond when the software fails. I agree, but if the first step to solving a problem is understanding that it exists then the first principle

The key is how we respond when the software fails. I agree, but if the first step to solving a problem is understanding that it exists then the first principle here must be to acknowledge that software systems are fallible and therefore any surprising or reasonably contested result they produce should be treated with proper caution until further information can be gathered. So many of the problems we see when modern technology goes wrong start with assuming it didn't. At that point, it's not even about how you respond to the failure, because you're denying that the failure ever happened. Big software companies with considerable lobbying power seem to be particularly good at convincing people who aren't technical experts, including most politicians, judges, juries and reporters, that this is the case. A corollary to this is that we desperately need more technological awareness among our politicians, lawyers, journalists and other relevant professions. Tech has become too big to be a minor issue you delegate to some random advisor in a basement office. It affects almost everything we do today, sometimes profoundly, and failing to understand that will inevitably lead to some horrible outcomes as we've seen all too vividly today.