I think the true question is how you decide if an algorithm is actually biased or not. At a first shot I would suggest something like this: Divide some otherwis

I think the true question is how you decide if an algorithm is actually biased or not. At a first shot I would suggest something like this: Divide some otherwise randomly sampled testing data into two sets, depending on the characteristic in question. If the ROC curve is approximately the same for both sets, I'd guess you could say the algorithm is not biased.