I guess I didn’t expect my comment to be so popular, and I was kind of tired when I wrote it, it was 2 in the morning. I guess my point was not anything really

I guess I didn’t expect my comment to be so popular, and I was kind of tired when I wrote it, it was 2 in the morning. I guess my point was not anything really about complexity (or the evolution of giraffes as HN seems to be very excited to correct me about). My point was that an AI giving an explanation for its output will almost certainly not explain how the model was generated or trained, because that is the secret sauce. It can only give an explanation from within its own domain of knowledge, unlike a human who can make an informed decision based on facts and then explain it from a higher-level viewpoint, or from the viewpoint of the person asking for the explanation. For a concrete example, imagine an AI for determining if a given video is of someone shoplifting. I give it a video and it says “yes, suspicious hand movements (78%), avoiding staff (65%), change of clothing (45%).” Here the AI has said why it thinks shoplifting is occurring, and a savvy user could guess what types of videos the AI was trained on. Buuuut, we are left wondering all sorts of important questions. How was the training data collected? Did it represent a diverse set of shoplifters and patrons? What movements are considered suspicious? What constitutes avoidance? What is the false positive rate of the AI? What was the motive of the people creating the AI… So to go back to my initial analogy, The AI can explain its internal process of coming to a decision, but it cannot explain the human environment in which it was created. In the same way, we can use science to construct useful narratives for “why” specific features of an animal fit with its environment, but we fail miserably to explain “why” the entire environment came to be.