> But the thing is, the formalist output of algorithms is not necessarily in agreement with what political power actually does in the real world. I think this

> But the thing is, the formalist output of algorithms is not necessarily in agreement with what political power actually does in the real world. I think this is where your argument makes a pretty big assumption. It's not just problematic that formalisms don't yield outputs in line with whatever political agendas are at play[1]. It's that formalisms don't necessarily yield outputs in line with reality, by definition. They yield outputs in line with their models. Sometimes that model is explicitly encoded in some chunks of logic. Sometimes that model is implicitly encoded by optimizations run over massive swathes of data. Neither approach guarantees anything other than a best fit model for the reality presented in the data. This is why these systems are so good at playing games. The information is perfect (usually). The rules of the universe have been agreed upon by all parties, and all permutations of state are equally observable and explicable according to those rules. In such an artificial setting, you can be pretty confident that your model is emulating the true set of rules to the best of its ability; that it cleaves pretty close to reality. In reality, however, information is far from perfect. We can't agree on what the rules are. We don't really have a lot of data, relative to how much stuff happens. The data we do have often isn't particularly granular. Often times, it's based off of abstractions and metrics that are -not- algorithmically based (test scores being a typical example), that may encode biases[1]. We don't know the degree to which different features interfere with each other, and the degree to which they can be considered independent. Thus we don't know whether a given feature selection is 'good'[2]. And most importantly, we can't decide on what the right questions to ask are, and thus we don't know if the data we -do- collect even paints a sufficiently complete picture to model in the first place. So I think the concern is predicated around something simpler than you're describing, but somewhat obscured by poor choice of terminology. Algorithms can be trusted, mathematically. It's the data that's the problem. And with sufficient bad data, or even just insufficient good data, you can get those trustworthy algorithms to paint you any kind of picture you want. Even if you don't know you want it. Even if it's not so much a thing you want, but more a thing you don't not want[3]. And if anything, algorithmic decisionmaking risks reinforcing extant power structures far more than toppling them; because they're often the ones collecting the data in the first place. --- [1] I really dislike that linked Stucchio post. The toy example about automatically normalizing for biases makes a pretty huge assumption; that some higher-level, easily-accessible metric will provide enough signal so as to drown out the bias. I'm not convinced that this is ever the case in social/political reality, and I'm almost completely convinced that it's never the case. But this is my particular bias. [2] This is somewhat analogous to Dijkstra's quip on testing being sufficiently only to prove the presence of bugs, not their absence. In a world of imperfect information, you can't be sure that you're right; you can only be sure that a given line of thinking is wrong. [3] c.f. Yudkowsky's revision of Hanlon's Razor