Here's how I see it. It really becomes a problem when you have a situation like Google's, where we are to take their word that their algorithm is fair when the

Here's how I see it. It really becomes a problem when you have a situation like Google's, where we are to take their word that their algorithm is fair when the algorithm is not audited, or realistically auditable by a trusted, disinterested party. But at the more mundane level, these limitations simply reflect the practical limit of data collection, and also the limit of the ability of the entity that wants to act based on the data collected. To go to the example of mortgage default, it is not as though an inquiry (beyond particular cases) into the causes behind the correlation behind zip code and default is the best use of resources for the most altruistic organization. It is costly to investigate, and it's possible that the altruistic organization has no means to act on it in any significant manner in the first place. And the analysis does not mention how this data is already emancipatory, which is also important when considering alternatives or replacement. Absent aggregated statistics, it would be more difficult for everyone to get a loan. These potentially false reifications are already far less false than previous methods of "my friend trusts this guy". It would have been more convincing if an alternative method was proposed, however unfeasibly it would scale, that would do the loan applicant more justice.