Perhaps a sequence of sentences, almost a series in the mathematical sense, where you iterate your explanation-finder, generating more sentences in the series,

Perhaps a sequence of sentences, almost a series in the mathematical sense, where you iterate your explanation-finder, generating more sentences in the series, until it's sufficiently detailed for the question you care about today. Also, remember that the explanation doesn't really have to fully contain the semantics of the model. It doesn't have to fully encode the weights, just activate sufficiently sympathetic structures in a human brain.