You assume that all of the biases will eventually cancel out. This is not necessarily correct, especially in the case of societal bias -- an example of this is

You assume that all of the biases will eventually cancel out. This is not necessarily correct, especially in the case of societal bias -- an example of this is racism, where everyone went out of their way to find reasons why PoC were inferior to white people; we know this to be false (At least, the majority of the world does. America is still having some problems with this), but still have hundereds of years of biased data to the opposite effect, because of societal attitudes. This trap falls hand in hand with the fact that large amounts of scientific data is not being published (Example, clinical research: [1][2][3]) All the AI knows is the training set. This is given a necessary bias by the researchers in the process of creating the AI. The best an AI can do is exaggerate the biases that it learned from the training set (Which is, after all, the main point of using AIs -- to exaggerate favorable biases to produce data that the researches believe is valid). [1]: www.badscience.net/2009/10/and-now-nerd-news/ [2]: https://www.researchgate.net/topic/Negative-Data-in-Science https://www.researchgate.net/topic/Negative-Data-in-Science [3]: http://www.nature.com/news/half-of-us-clinical-trials-go-unpublished-1.14286 http://www.nature.com/news/half-of-us-clinical-trials-go-unp...