You, the person expecting an explanation, won't have access to the model to twiddle with the parameters and see what happens. In this case I'm assuming somethin

You, the person expecting an explanation, won't have access to the model to twiddle with the parameters and see what happens. In this case I'm assuming something like "the bank explaining to you why your loan was accepted/rejected", not just the model explaining to the bank employees why it was accepted/rejected, because this is where the social friction is and why anybody cares at all about explanations.