Things can be really complex and fully explainable at the same time. A program could have millions of variables and you could zoom in on a variable and explain

Things can be really complex and fully explainable at the same time. A program could have millions of variables and you could zoom in on a variable and explain its purpose exactly. In a neural net, the creators of the neural net in general don't know what it means for the weight at some node being 0.46557, or how the system would behave if you changed it to 0.5. CNNs tend to use the first few layers to detect surface features like edges, and later layers to detect more abstract features. This was an empirical discovery! The creators of the CNNs were like holy shit, these layers detect edges! Anyway I think there's a substantial difference between building really complex systems (that yes may appear as black boxes to outsiders) and systems where the designers themselves generally don't know how the thing works.