7 ms·

I've been thinking all the recent hostility towards algorithms - including ProPublica's deliberate lies about them which are cited by this very article (https:/

I've been thinking all the recent hostility towards algorithms - including ProPublica's deliberate lies about them which are cited by this very article (https://www.chrisstucchio.com/blog/2016/propublica_is_lying.html https://www.chrisstucchio.com/blog/2016/propublica_is_lying....) - and what might be motivating it. I have the following theory. Consider the proposition of formalism - the idea that the stated public rules must, in fact, drive the expression of political power. It's pretty clear that we don't live in a world governed by formalism. As one example, intimately related to many of the critiques of algorithmic decisionmaking, we claim to favor individual fairness (the proposition that individuals should be judged on their own individual merits) yet in practice we push for equality of outcomes (the proposition that processes should make decisions that do not vary across certain reference classes). However, algorithmic decisionmaking actually implements formalism. If you tell an algorithm to predict which student is most likely to graduate college, or which loan applicant is most likely to repay, the algorithm will do exactly that to the best of it's ability given the available data. In many cases, the algorithm will even correct bias in it's inputs: https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html https://www.chrisstucchio.com/blog/2016/alien_intelligences_... But the thing is, the formalist output of algorithms is not necessarily in agreement with what political power actually does in the real world. For example, political power in the world strives to have proportional representation of different racial groups in college, yet algorithms aiming to predict grades/graduation probabilities would actively penalize blacks (who underperform relative to what a race-neutral model predicts). As such, algorithms are a huge threat to existing power structures, because they both implement formalism and they also reveal the lies behind our existing stated principles. I'd love to hear what others think about this theory.

Terr_ 10y ago
Reminds me of this rather dark/depressing take on corporate structures: > Bureaucracies are structures designed to do certain things very efficiently and competently: those that are by default in the best interests of the Sociopaths. They are also designed to do certain things incompetently: those expensive things that the organization is expected to do, but would cut into Sociopath profits if actually done right. [0] http://www.ribbonfarm.com/2011/10/14/the-gervais-principle-v-heads-i-win-tails-you-lose/ http://www.ribbonfarm.com/2011/10/14/the-gervais-principle-v...
sp332 10y ago
There's no reason to think that the bulk of women or black people would have different merits than the bulk of white men. So when we see those outcomes diverging, we assume that some something other than individual merits caused it and we act to bring the outcomes back in line with the assumed-equal merits. In other words, forcing equality of outcomes is just supposed to be correcting for some external thing that does not treat people according to merits.
- yummyfajitas 10y ago
  I'm aware of the theory. But algorithms are another way to formally provide equality of opportunity. The algo formally encodes "admit the best students" or "give out loans to the best credit risks". That's what it does. Maybe your ad-hoc procedure will be reproduced, maybe not. My belief is that "equality of opportunity" is a lie, just a smokescreen to gain equal outcomes, and automated formalism is a threat to this.
  - sp332 10y ago
    Your algorithm would take literally any form of discrimination that had been applied to a student so far in their lives as a signal to discriminate against them even further. It wouldn't look at the reasons a student's grades were low and consider whether it was their "merits".
    - yummyfajitas 10y ago
      Yes, if your actual goal is some sort of compensation for historical wrongs, you need to openly admit that and formally encode it in your algorithm. That's formalism.
dmreedy 10y ago
> But the thing is, the formalist output of algorithms is not necessarily in agreement with what political power actually does in the real world. I think this is where your argument makes a pretty big assumption. It's not just problematic that formalisms don't yield outputs in line with whatever political agendas are at play[1]. It's that formalisms don't necessarily yield outputs in line with reality, by definition. They yield outputs in line with their models. Sometimes that model is explicitly encoded in some chunks of logic. Sometimes that model is implicitly encoded by optimizations run over massive swathes of data. Neither approach guarantees anything other than a best fit model for the reality presented in the data. This is why these systems are so good at playing games. The information is perfect (usually). The rules of the universe have been agreed upon by all parties, and all permutations of state are equally observable and explicable according to those rules. In such an artificial setting, you can be pretty confident that your model is emulating the true set of rules to the best of its ability; that it cleaves pretty close to reality. In reality, however, information is far from perfect. We can't agree on what the rules are. We don't really have a lot of data, relative to how much stuff happens. The data we do have often isn't particularly granular. Often times, it's based off of abstractions and metrics that are -not- algorithmically based (test scores being a typical example), that may encode biases[1]. We don't know the degree to which different features interfere with each other, and the degree to which they can be considered independent. Thus we don't know whether a given feature selection is 'good'[2]. And most importantly, we can't decide on what the right questions to ask are, and thus we don't know if the data we -do- collect even paints a sufficiently complete picture to model in the first place. So I think the concern is predicated around something simpler than you're describing, but somewhat obscured by poor choice of terminology. Algorithms can be trusted, mathematically. It's the data that's the problem. And with sufficient bad data, or even just insufficient good data, you can get those trustworthy algorithms to paint you any kind of picture you want. Even if you don't know you want it. Even if it's not so much a thing you want, but more a thing you don't not want[3]. And if anything, algorithmic decisionmaking risks reinforcing extant power structures far more than toppling them; because they're often the ones collecting the data in the first place. --- [1] I really dislike that linked Stucchio post. The toy example about automatically normalizing for biases makes a pretty huge assumption; that some higher-level, easily-accessible metric will provide enough signal so as to drown out the bias. I'm not convinced that this is ever the case in social/political reality, and I'm almost completely convinced that it's never the case. But this is my particular bias. [2] This is somewhat analogous to Dijkstra's quip on testing being sufficiently only to prove the presence of bugs, not their absence. In a world of imperfect information, you can't be sure that you're right; you can only be sure that a given line of thinking is wrong. [3] c.f. Yudkowsky's revision of Hanlon's Razor
nkurz 10y ago
Consider the proposition of formalism - the idea that the stated public rules must, in fact, drive the expression of political power. I'm not familiar with this usage of "formalism". I searched your site, but didn't find other mentions. Your gloss helps[1], but is there a reference you could point to? It's pretty clear that we don't live in a world governed by formalism. Are you saying that we don't live in such a world, or that we can safely assume that almost everyone realizes that we do not live in such a world? The first seems arguably true, the second strikes me as likely false. we claim to favor individual fairness (the proposition that individuals should be judged on their own individual merits) yet in practice we push for equality of outcomes (the proposition that processes should make decisions that do not vary across certain reference classes) Your use of "we" here seems forced. You[2] are the only clear party to this "we", but since you offer this example as a contradiction, presumably you do not actually put yourself into both categories. I suspect there is actually a division between these groups, with few claiming to belong to both. political power in the world strives to have proportional representation of different racial groups in college I'm not sure if I disagree with this, or if I missed the beginning of a counterfactual. I'd agree that some people claim to strive for proportional representation of races in college. Some of these think of this as a goal in and of itself, some assume it's a convenient metric for "equal opportunity" or a related goal, and others find it beneficial to claim to be for this without doing any striving at all. But to say that "political power in the world strives" either seems at odds with the evidence --- which I think shows that there are still racial disparities in college attendance and outcomes both in the US and the world --- or implies that "political power" is a relatively weak force that is prevented from having its way by something stronger. Or maybe you are defining power differently than I would. I tend to a simplified Foucauldian worldview[3], where "power" is strong and "truth" is weak. In my view, the best way to understand what "power" is striving for is to ignore everything it claims and look instead at how the world actually behaves. While not always accurate, I think the actual behavior of the world is usually a better starting point in determining what "power" is striving for than listening to its consciously crafted claims. When the world does not behave in a way that most benefits those with power, it's likely we have misidentified the true holder of power, or have mistaken its stated goals for its true goals. As such, algorithms are a huge threat to existing power structures, because they both implement formalism and they also reveal the lies behind our existing stated principles. I liked the distinction that the authors of the linked article make between "syntactic transparency" and "algorithmic accountability". They claim that the real harm comes from "opaque algorithms", and that "interpretability" is something that should receive more emphasis. I think this is compatible with your theory, since an opaque algorithm is less likely to reveal the divorce between what is espoused and what is practiced, and an easily interpretable one is more threatening. What do you think? [1] In your linked post, you said "Unlike me in this blog post, they didn't gloss over the statistical test - instead, the used the one that came built into R." I wasn't sure what you meant here. Do you maybe mean "they did gloss over" (made it superficially lustrous by omitting essential details) or maybe that "didn't gloss" (failed to define their terms as with a glossary)? [2] I don't know if it's clear to the audience that 'yummyfajitas' and the 'chrisstucchio' of the linked blog are related. I think they are the same person, but maybe I'm missing a subtle distinction of persona? [3] http://www.powercube.net/other-forms-of-power/foucault-power-is-everywhere/ http://www.powercube.net/other-forms-of-power/foucault-power...

Terr_ 10y ago

Reminds me of this rather dark/depressing take on corporate structures: > Bureaucracies are structures designed to do certain things very efficiently and competently: those that are by default in the best interests of the Sociopaths. They are also designed to do certain things incompetently: those expensive things that the organization is expected to do, but would cut into Sociopath profits if actually done right. [0] http://www.ribbonfarm.com/2011/10/14/the-gervais-principle-v-heads-i-win-tails-you-lose/ http://www.ribbonfarm.com/2011/10/14/the-gervais-principle-v...

sp332 10y ago

There's no reason to think that the bulk of women or black people would have different merits than the bulk of white men. So when we see those outcomes diverging, we assume that some something other than individual merits caused it and we act to bring the outcomes back in line with the assumed-equal merits. In other words, forcing equality of outcomes is just supposed to be correcting for some external thing that does not treat people according to merits.

yummyfajitas 10y ago
I'm aware of the theory. But algorithms are another way to formally provide equality of opportunity. The algo formally encodes "admit the best students" or "give out loans to the best credit risks". That's what it does. Maybe your ad-hoc procedure will be reproduced, maybe not. My belief is that "equality of opportunity" is a lie, just a smokescreen to gain equal outcomes, and automated formalism is a threat to this.
- sp332 10y ago
  Your algorithm would take literally any form of discrimination that had been applied to a student so far in their lives as a signal to discriminate against them even further. It wouldn't look at the reasons a student's grades were low and consider whether it was their "merits".
  - yummyfajitas 10y ago
    Yes, if your actual goal is some sort of compensation for historical wrongs, you need to openly admit that and formally encode it in your algorithm. That's formalism.

dmreedy 10y ago

> But the thing is, the formalist output of algorithms is not necessarily in agreement with what political power actually does in the real world. I think this is where your argument makes a pretty big assumption. It's not just problematic that formalisms don't yield outputs in line with whatever political agendas are at play[1]. It's that formalisms don't necessarily yield outputs in line with reality, by definition. They yield outputs in line with their models. Sometimes that model is explicitly encoded in some chunks of logic. Sometimes that model is implicitly encoded by optimizations run over massive swathes of data. Neither approach guarantees anything other than a best fit model for the reality presented in the data. This is why these systems are so good at playing games. The information is perfect (usually). The rules of the universe have been agreed upon by all parties, and all permutations of state are equally observable and explicable according to those rules. In such an artificial setting, you can be pretty confident that your model is emulating the true set of rules to the best of its ability; that it cleaves pretty close to reality. In reality, however, information is far from perfect. We can't agree on what the rules are. We don't really have a lot of data, relative to how much stuff happens. The data we do have often isn't particularly granular. Often times, it's based off of abstractions and metrics that are -not- algorithmically based (test scores being a typical example), that may encode biases[1]. We don't know the degree to which different features interfere with each other, and the degree to which they can be considered independent. Thus we don't know whether a given feature selection is 'good'[2]. And most importantly, we can't decide on what the right questions to ask are, and thus we don't know if the data we -do- collect even paints a sufficiently complete picture to model in the first place. So I think the concern is predicated around something simpler than you're describing, but somewhat obscured by poor choice of terminology. Algorithms can be trusted, mathematically. It's the data that's the problem. And with sufficient bad data, or even just insufficient good data, you can get those trustworthy algorithms to paint you any kind of picture you want. Even if you don't know you want it. Even if it's not so much a thing you want, but more a thing you don't not want[3]. And if anything, algorithmic decisionmaking risks reinforcing extant power structures far more than toppling them; because they're often the ones collecting the data in the first place. --- [1] I really dislike that linked Stucchio post. The toy example about automatically normalizing for biases makes a pretty huge assumption; that some higher-level, easily-accessible metric will provide enough signal so as to drown out the bias. I'm not convinced that this is ever the case in social/political reality, and I'm almost completely convinced that it's never the case. But this is my particular bias. [2] This is somewhat analogous to Dijkstra's quip on testing being sufficiently only to prove the presence of bugs, not their absence. In a world of imperfect information, you can't be sure that you're right; you can only be sure that a given line of thinking is wrong. [3] c.f. Yudkowsky's revision of Hanlon's Razor

nkurz 10y ago

Consider the proposition of formalism - the idea that the stated public rules must, in fact, drive the expression of political power. I'm not familiar with this usage of "formalism". I searched your site, but didn't find other mentions. Your gloss helps[1], but is there a reference you could point to? It's pretty clear that we don't live in a world governed by formalism. Are you saying that we don't live in such a world, or that we can safely assume that almost everyone realizes that we do not live in such a world? The first seems arguably true, the second strikes me as likely false. we claim to favor individual fairness (the proposition that individuals should be judged on their own individual merits) yet in practice we push for equality of outcomes (the proposition that processes should make decisions that do not vary across certain reference classes) Your use of "we" here seems forced. You[2] are the only clear party to this "we", but since you offer this example as a contradiction, presumably you do not actually put yourself into both categories. I suspect there is actually a division between these groups, with few claiming to belong to both. political power in the world strives to have proportional representation of different racial groups in college I'm not sure if I disagree with this, or if I missed the beginning of a counterfactual. I'd agree that some people claim to strive for proportional representation of races in college. Some of these think of this as a goal in and of itself, some assume it's a convenient metric for "equal opportunity" or a related goal, and others find it beneficial to claim to be for this without doing any striving at all. But to say that "political power in the world strives" either seems at odds with the evidence --- which I think shows that there are still racial disparities in college attendance and outcomes both in the US and the world --- or implies that "political power" is a relatively weak force that is prevented from having its way by something stronger. Or maybe you are defining power differently than I would. I tend to a simplified Foucauldian worldview[3], where "power" is strong and "truth" is weak. In my view, the best way to understand what "power" is striving for is to ignore everything it claims and look instead at how the world actually behaves. While not always accurate, I think the actual behavior of the world is usually a better starting point in determining what "power" is striving for than listening to its consciously crafted claims. When the world does not behave in a way that most benefits those with power, it's likely we have misidentified the true holder of power, or have mistaken its stated goals for its true goals. As such, algorithms are a huge threat to existing power structures, because they both implement formalism and they also reveal the lies behind our existing stated principles. I liked the distinction that the authors of the linked article make between "syntactic transparency" and "algorithmic accountability". They claim that the real harm comes from "opaque algorithms", and that "interpretability" is something that should receive more emphasis. I think this is compatible with your theory, since an opaque algorithm is less likely to reveal the divorce between what is espoused and what is practiced, and an easily interpretable one is more threatening. What do you think? [1] In your linked post, you said "Unlike me in this blog post, they didn't gloss over the statistical test - instead, the used the one that came built into R." I wasn't sure what you meant here. Do you maybe mean "they did gloss over" (made it superficially lustrous by omitting essential details) or maybe that "didn't gloss" (failed to define their terms as with a glossary)? [2] I don't know if it's clear to the audience that 'yummyfajitas' and the 'chrisstucchio' of the linked blog are related. I think they are the same person, but maybe I'm missing a subtle distinction of persona? [3] http://www.powercube.net/other-forms-of-power/foucault-power-is-everywhere/ http://www.powercube.net/other-forms-of-power/foucault-power...