It isn't "science fiction" though, we have a mountain of examples that demonstrate how probabilistic AI fails when the training dataset doesn't perfectly align

It isn't "science fiction" though, we have a mountain of examples that demonstrate how probabilistic AI fails when the training dataset doesn't perfectly align with the eventual application. It isn't restricted to AI either, there have been numerous studies showing that cross-race identification to be garbage data. <Insert other race> all look the same: https://onlinelibrary.wiley.com/doi/abs/10.1002/acp.898 https://onlinelibrary.wiley.com/doi/abs/10.1002/acp.898 This bus is an ostrich: https://arxiv.org/abs/1312.6199 https://arxiv.org/abs/1312.6199 Racist autofocus: https://sitn.hms.harvard.edu/flash/2020/racial-discrimination-in-face-recognition-technology https://sitn.hms.harvard.edu/flash/2020/racial-discriminatio... Google thinks black people look like gorillas: https://www.wired.com/story/when-it-comes-to-gorillas-google-photos-remains-blind https://www.wired.com/story/when-it-comes-to-gorillas-google... And here is an argument to look forward to: should the training dataset be build to represent the general population, or the specific subgroup that the software would be most often encountering? Because that FBI UCR...