How about you select a representative (i.e. random and statistically significant) sample of the population and then ask them their opinions about certain (espec

How about you select a representative (i.e. random and statistically significant) sample of the population and then ask them their opinions about certain (especially controversial) parts of your data, and then weigh your data according to these opinions? That's just an idea that occurred to me (in 30 seconds of thought) which could probably make the training data significantly more unbiased. But I'm sure there are research scientists who can come up with better methods for sampling data in a more unbiased fashion. Note that this is not an all or nothing approach. Your training data could presumably be 100% biased or 0% biased, but also any value in-between. The goal is to try to make it as close to 0% biased as feasible, given whatever effort you're comfortable expending.