Statistics Ph.D. here. This phenomenon is the scourge of our existence in the big-data age. It's why we have endless stories about miracle medical diagnostic

Statistics Ph.D. here. This phenomenon is the scourge of our existence in the big-data age. It's why we have endless stories about miracle medical diagnostic methods that can tell everything about you from a retina image or a recording of your voice or gait or whatever. When the method fizzles out in practice, there's not an equivalent news story. And don't get me started on those annoying personality tests for employers claiming to predict job performance. Everyone can now do a train-test split, use a canned algorithm to get a classifier with good apparent accuracy on the hold-out data set. Like, yeah, so what. That accuracy score is only a fair estimate of how accurate the method will be out in the general population under the most restrictive circumstances (keywords: the generalization problem, the representative training set problem, stationarity, data drift/model drift). And when the subject of study is humans rather than atoms - humans that can change their behaviors in response to being studied and classified (keywords: Hawthorne effect, Goodhart's law) - all bets on future predictive ability are off. The paper talks about decorrelating age/gender/ethnicity. Statistical corrections (keywords: causal methods, confounding, instrumental variables) aren't magic; they should be thought of as best-efforts attempts to cope with an ultimately intractable problem, not as something that, if done right, solves it. So, it's perfectly possible that the vision model is picking up on subtler demographic clues. Finding apparent patterns that are correlated with any trait of interest in the population has always been easy, and is now essentially automated. That's superstition / stereotyping / phrenology / pseudoscience, all the stuff humanity worked so hard to get away from. Designing a clean experiment to isolate a consistent effect that's akin to a law of nature? That's science, and it's hard, and the scientific method and statistical methodology can only tell you the pitfalls, not give a recipe for making discoveries. For most things - we find patterns. They work for prediction for a while, and then they don't. Finding genuine causal phenomena from the data - that's a rare and precious thing. (no blame on this study's authors; they're highlighting an curious phenomenon, not suggesting that that it be used for decision making or claiming to have found a law of society or biology)