There is a very real danger of models being biased in a way that doesn't show up when you apply these crude hacks to inputs. It seems to me we have to be much

There is a very real danger of models being biased in a way that doesn't show up when you apply these crude hacks to inputs. It seems to me we have to be much more deliberate, much more analytical, and much more thorough in testing models if we want to substantially reduce or even eliminate discrimination. Yes, you can A/B test the model if you can design reasonable experiments. You still don't have the general discrimination test because you have to define what a reasonable input distribution and what reasonable outputs are. If an employer is looking to hire an engineer with a CS degree from a top-tier university, and they use an AI model to evaluate resumes and it returns a number of successes on black people very similar to the population distribution of graduates from those programs is the model discriminatory? There are still hard problems here because any natural baseline you use for a model may in fact be wrong and designing a reasonable distribution of input data is almost impossibly hard as well.