I mean no responsible developer would take the output of a ML model and then act on that without any tests on whether the output makes sense or not. That's the

I mean no responsible developer would take the output of a ML model and then act on that without any tests on whether the output makes sense or not. That's the way to handle unexplainable AI. There's also opportunity to look at uncertainty estimation, i.e. look at the epistemic uncertainty of the model and use it as a proxy for potential error. That seems to be a main thrust for AI right now. I'm currently writing a paper looking at this for dynamical systems.