The danger is that AI can be biased, so if we don't know how it's making it's decisions it could be stupid. I mean, imagine that making an insurance claim with

The danger is that AI can be biased, so if we don't know how it's making it's decisions it could be stupid. I mean, imagine that making an insurance claim with a black-sounding name results in a 5% greater chance of being rejected. How would we even know if this is the case? And, how do we prevent that? Now, of course humans are biased too, but there's no guarantee that the biases of humans are the same as the biases of whatever AI model is chosen. And with humans we can hold them accountable to some degree. We need that accountability with AI agents.