These are not "fairness" issues; these are process feedback issues. The same problem pops up if you're using algorithmic selection of machine parts to test for

These are not "fairness" issues; these are process feedback issues. The same problem pops up if you're using algorithmic selection of machine parts to test for failure, attempting to programmatically evaluate patches for code quality, writing fraud detection algorithms, etc.