Every model designer is going to come to the table with his own set of lenses and biases based on this own life experiences. These biases are incorporated into

Every model designer is going to come to the table with his own set of lenses and biases based on this own life experiences. These biases are incorporated into our models based on the metrics that we optimize for and the features that we incorporate into our models. It may comforting to think that all our biases are “baked out” of the models during the training process but if the researcher had been a black man do you believe that they would have arrived at identical models? Even if the model designer found ways to compensate for his own bias the world that these models operate in are inherently biased. Training data may be subtly biased in ways that are difficult to detect. A model trained on recidivism is likely to be biased if, for example, a lower class black male or upper class white women have different risks of being arrested for the same crime. Assuming the models themselves are unbiased as trained they may create bias with subtle errors when pushed into production or due to second order effects, feedback loops or errors or manipulation of data. For example during the credit bubble entire industries found ways to raise borrowes FICO scores. There is also the inherent corporate bias that the researcher pointed out which the bias of the almighty dollar. Compensating for bias is expensive and may even impact salability if an unbiased model doesn’t “gel” with the expectation of customers who are biased. For example a judge may feel pressure of homogenize his decisions or abdicate his responsibility by rubber stamping the models decisions. Currently we compensate for this in a democratic society with checks and balances. What are the checks and balances for a computer algorithm that is a trade secret for a for-profit corporate entity?