I don't have a rigorous argument but as I recall Rice's theorem says that computers cannot compute nontrivial properties about programs, in general. So we canno

I don't have a rigorous argument but as I recall Rice's theorem says that computers cannot compute nontrivial properties about programs, in general. So we cannot have virus checkers, etc. (Well we actually do have imperfect virus checkers, of course). So following that, a nontrivial property such as self-explanation, e.g. correctly explain what a neural network is doing, for all possible inputs. (But again this is not fully clear to me, someone might point out that for a specific program this is different and Rice's theorem is irrelevant.)