Here's a concrete famous example. I'm sure I'll remember the details wrong, but the gist is the kind of thing that keeps happening. A while back, people built

Here's a concrete famous example. I'm sure I'll remember the details wrong, but the gist is the kind of thing that keeps happening. A while back, people built a model for medical imaging that learned to distinguish between images of patients with vs without some disease (can't remember that detail). It did well, but failed in the real world. It turned out that instead of learning to recognize features of the disease at hand, it learned to recognize some tiny feature of whether the image came from a specific hospital that collected part of the dataset, or something stupid like that. Saying "maybe model X does the same thing as humans" is proven wrong for X after X after X. At this point, the default assumption should be that ML techniques are different from humans unless proven otherwise.