19 ms·

Moving beyond “algorithmic bias is a data problem”

69 points by mwexler 5y ago

salawat 5y ago
Ah... The ML scientist's "Metrics are harmless. People doing stupid things with metrics on the other...". Or if I'm being too subtle, this is the ML practitoner's "Guns don't kill people. People kill people." Which is absolutely right, I might add; but also a stop point and consideration that should be had before you go training something on sketchy data, and unleashing it on an unknowing populace.
- elcomet 5y ago
  That's not what the article is talking about. They are saying that data biases are a problem but not the only one: other things can amplify biases such as the model's architecture. So when working to remove biases, fixing the data might not be enough.
  - athrowaway3z 5y ago
    What salawat is probably saying is: There would be less issue with bias (for society), if the expectation/sales pitch was different. As an analogy: Somebody sells the idea of "The Unbreakable Rope" and it breaks after you hang 10kg on it. A blog post "Moving beyond material" goes in depth on how to process material. This is a good thing for people producing rope. However, for a layman who still sees people assuming "The Unbreakable Rope" is attainable, the content is going to be a little underwhelming.
trhway 5y ago
the point of the article is that algorithm design choices affect results on the long-tail of the dataset. Well, what is in the long-tail of the dataset is data bias. So, are we going to try to introduce algorithm bias to correct for the data bias?
zozbot234 5y ago
Obligatory reminder that there's an inherent tradeoff between bias and variance in any ML architecture. Try to create a "highly regular" model that gives predictable, less noisy results even with low-quality data, you'll just end up strengthening some implied prior and introducing bias. Try to remove inbuilt bias, you'll just make the model more data-dependent, and noisier with any given amount of data. You just can't win.
- jstx1 5y ago
  Also important to point out that the word bias has a very specific meaning in the context of statistics and machine learning and it isn't just a synonym for discrimination or something of that sort - https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff
  - commandlinefan 5y ago
    ... which seems not to be the definition the author of the linked article is using.
- rualca 5y ago
  > Obligatory reminder that there's an inherent tradeoff between bias and variance in any ML architecture. It should also be stressed that in this context "bias" is a kin to interpolating between data points where resolution in the training set is relatively low, thus leads to a simpler model that does not exactly match the test set. What this implies is that this "bias" does not reflect a preference, let alone an intention to favour one group over another. It just means that a combination of the test set being too sparse or with too few dimensions, the choice of model being too inflexible, and the problem domain having too much uncertainty at that resolution. Critics may single out the model, but the core of the problem actually lies on the data and the problem domain.
darawk 5y ago
> A surprisingly sticky belief is that a machine learning model merely reflects existing algorithmic bias in the dataset and does not itself contribute to harm. Why, despite clear evidence to the contrary, does the myth of the impartial model still hold allure for so many within our research community? Algorithms are not impartial, and some design choices are better than others. Recognizing how model design impacts harm opens up new mitigation techniques that are less burdensome than comprehensive data collection. Ah yes the classic "why do people keep insisting on X despite consistent proof of not X", with...zero citations. Well, let's read the paper and see if they make the case. > Even if we are able to label sensitive attributes at scale such as gender and race, algorithms can still leverage proxy variables to reconstruct the forbidden label. Data collection of even a limited number of protected attributes can be onerous. For example, it is hard to align on a standard taxonomy—categories attributed to race or gender are frequently encoded in inconsistent ways across datasets.2 Furthermore, procuring labels for these legally protected attributes is often perceived as intrusive leading to noisy or incomplete labels.3,4 > If we cannot guarantee we have fully addressed bias in data pipeline, the overall harm in a system is a product of the interactions between the data and our model design choices. Here, acknowledging the impact of model design bias can play an important role in curbing harm. Algorithms are not impartial, and some design choices are better than others. Recognizing how model design impacts harm opens up new mitigation techniques that are far less burdensome than comprehensive data collection. > We are well-versed in the connection between function choice and test-set accuracy because objective functions such as cross-entropy or mean squared error reflect our preference to optimize for high test-set accuracy. Standard loss functions do not explicitly encode preferences for other objectives we care about such as algorithmic bias, robustness, compactness, or privacy. However, just because these desiderata are not reflected does not mean they have ceased to exist. Turing award winner Donald Knuth said that computers “do exactly what they are told, no more and no less.” A model can fulfill an objective in many ways, while still violating the spirit of said objective. > Model design choices made to maximize test-set accuracy do not hold static other properties we care about such as robustness and fairness. On the contrary, training a parametric model is akin to having a fixed amount of materials to build a house with. If we decide to use more bricks building a bigger living room, we force the redistribution of the number of bricks available for all other rooms. In the same vein, when we prioritize one objective, whether that be test-set accuracy or additional criteria such as compactness and privacy, we inevitably introduce new trade-offs. This is literally just saying "Well yes all the bias starts out in the data, and so a race/gender-neutral model will encode those features, but in principle you could design a model expressly to avoid that, and since that hasn't been done, the models are racist too". Which is...a certain kind of take. But even if we accept the reasoning, it is not at all at odds with the common understanding of their starting sentence: "A surprisingly sticky belief is that a machine learning model merely reflects existing algorithmic bias in the dataset and does not itself contribute to harm.". Yes, people commonly believe the "bias is in the data". Models are designed to be mirrors, so they reflect whatever bias is in their training data. Yes, it's true you can design a distorted mirror that does not reflect certain attributes. But that would be inserting bias into a model to correct a bias that you found in the original data. It should be very clear that the root source of the bias here is the training data, not the model. We can and should build models that attempt to correct those biases, and we also can and should attempt to de-bias the data itself. But can we stop torturing the meaning of English sentences to support people's preferred narratives? The models are neutral. The data contains the bias. Convolutional neural nets do not inherently work better on white faces. Multi-layer perceptrons do not have beliefs about race and gender. One additional linguistic nitpick: > The belief that model design merely reflects algorithmic bias in the dataset can be partly ascribed to the difficulty of measuring interactions between all the variables we care about. There is no "algorithmic bias in the dataset". That is a contradiction in terms. The bias in the dataset is not "algorithmic". The algorithm is what processes the data. The dataset is just biased, it has not been biased algorithmically.
macleginn 5y ago
"A key takeaway is that our algorithms are not impartial." Of course they are, they just underperform on the data points from the long tail of the empirical distribution, as show basically all real-world examples from the paper. Underfitting on the samples from the long tale (aka blinding the model to ethically bad features) removes bias sensu strictu but increases variance, which leads to more errors, which is perceived as "bias" (did someone say catch 22?).
- geofft 5y ago
  The classic algorithm to solve the https://en.wikipedia.org/wiki/Stable_marriage_problem https://en.wikipedia.org/wiki/Stable_marriage_problem is a pretty straightforward and frankly introductory-level example of an algorithm which is not impartial (and not just because it ought to be called the stable heterosexual monogamous marriage problem). The algorithm outputs a matching between two sets A and B (e.g., bachelors and bachelorettes) where each member of set A has ranked each member of set B in preference order and vice versa. The output is called stable because there's no situation where any pair would prefer to be with each other than their current partners, e.g., there's never a case where Alice prefers Bob to her current partner and Bob prefers Alice to his current partner and are tempted to run away with each other. But a curious property of this algorithm is that the matching is always best for one group and always worst for the other. Of the possible answers to the problem, this algorithm an answer that is at least as good for set A as any other one, and at least as bad for set B as any other one. You can, of course, flip sets A and B when running the algorithm, and it will them be optimal for set B and pessimal for set A. This has absolutely nothing to do with ML or data fitting or anything. It's a deterministic, classical algorithm. And yet it is clearly partial, and you must choose when using it whether to be partial to set A or set B, to the men or to the women, to med school students or to their employers, etc. It would be silly to say that the algorithm is impartial and that it's solely the operator's choice to make it impartial - this particular algorithm forces the operator to make some choice. I think in the end you're agreeing with the author, who is saying that you must make a tradeoff of some sort, and the question is which tradeoff to choose. The author (and I) would describe that as the algorithm being partial, and the operator only being able to make some choice about what to do with that.
fouronnes3 5y ago
This belief has always been extremely cringe in my opinion, because it somehow implies that dataset engineering is not an important part of machine learning. Ensuring you are feeding good data to your model is a critical part of machine learning practice, but some researchers seem to hand wave it away as merely an implementation detail not worthy of their attention. "It's a data problem" is used as an excuse to be discharged of the moral responsability of the output bias, as if you were ethically responsible for the model only.
- darawk 5y ago
  I don't think "it's a data problem" is inherently used to hand-wave away the problem. I think it's used to locate the problem in the area where its most readily addressed. Designing algorithms to debias your data is hard, over-sampling under-represented groups is easier. I do think you're right that sometimes people use it that way, but that doesn't mean we should make up false narratives about biased models, either. It just means we should work on de-biasing the data, and also developing algorithms to help mitigate the learning of features we don't want weighted.
  - mjburgess 5y ago
    You are right that there are no algorithms which capture semantics, and therefore none which can be constructed not to introduce bias. In that sense it is a "data solution", but it is not a data problem. > de-biasing the data The issue isnt statistical bias. The issue is semantic: occurrence doesnt capture meaning. Negative terms and racial terms can co-occur more frequently in some text (eg., a biography of MLK) without the remedy being to "rebalance them with positive terms".
- rualca 5y ago
  > (...) as if you were ethically responsible for the model only. The model is the result and the outcome of the whole process. The model is supposed to be a reliable representation of reality that has acceptable accuracy within predetermined operational limits. If someone tries to generate models that interpolate and/or extrapolate data and the data it uses to seed the model is garbage then the model is garbage, and the person responsible for putting it together is the person responsible for the model failing to do its job. There is no separation between model and the data used to generate the model. Garbage in, garbage out. If a machine learning model uses garbage data then the model is garbage, and throwing machine learning buzzwords at the problem does not justify why garbage was sent in.
mjburgess 5y ago
Many here are in the throws of this "data mythology"... It is trivial for algorithms to introduce novel racism: consider processing text analysing minority causes. These texts contain statistical associations between politically sensitive terms, negative terms, (and so on). However they are not racist, are not born of any racist project, and most humans reading them would be moved to more positively regard racial terms. The meaning text express consists in what they want their readers to do. It is an activity of coordination (writer/reader); and it here it has no racist aspect. A machine if it should have learned anything at all from this text, should learn to associate minority terms with virtuous political projects; it should have acquired an understanding of the many types of associative relation: needing, wanting, opposing, supporting, trusting, advocating... Yet a machine performing an action based on statistical associations will not learn these; and so, will act prejudicially. It will merely expect terms to co-occur, wholly unable to determine why. We expect charity workers to be greedy? No, we expect them to morally oppose greed. We expect movies to smell like candy? No, we expect them to be sold together. (And so on.) It is somewhat alarming that statistical associations in terms in historical texts are being seen as characteristic of human communication, meaning, belief, association, ethics, action. I think it has led here to a great moral oversight: that absent the ability to understand text, machines here are introducing novel prejudice where none existed prior.
- davidgerard 5y ago
  > they are not racist "having racist effects" counts as a perfectly valid usage of the word. This is a cheap sophistry: 1. It's only racist if it was intended to be. 2. You can't read minds. 3. Therefore, you can't say "this is racist." This isn't a useful usage of the term, except for attempting to deflect people from calling out racist effects.
  - concordDance 5y ago
    I really wish the word "racist" had a single meaning.
    - user-the-name 5y ago
janto 5y ago
This field is a joke and should be mercilessly mocked. These manipulated models do not represent reality. I look forward to gaming the systems that are produced by these people.
elisbce 5y ago
Let's take the classic example of a "racist algorithm" Image recognition using neural nets and examine what's going on. Let's say we train the NN using equal number of human faces of all races, and animals faces. Let's say the trained neural net made some mistakes, including a few cases where black faces are recognized as gorilla faces. And this doesn't happen at all or as likely with white faces. The results are horrible, right? And people immediately start to point fingers to the training data and the algorithms, stating the training data is racially biased and/or the training algorithms or even the use of neural nets are racially biased. But is it really so? It's known that in order to take a black face photo with the same degree of details, the lighting condition and camera settings need to be adjusted. This is an effect purely due to physics. In other words, it could well be by nature, that recognizing black faces is harder than pale faces under the normal camera and scene settings. This is why you have night mode on your phones. It is just harder to take clear photos when less light gets into the camera. And this requires the camera and photography settings to be adjusted. So, the unwanted results here are still due to the input data. But neither the input data, nor the algorithms contain any racial bias towards the black people. The results might be merely due to the difference between dark faces and pale faces under the natural law of physics. These are unwanted results due to our social norms, but they are NOT racially biased or racist, because there is no such bias introduced or inherent during any part of the process. We could and should correct such unwanted results by introducing adjustments to the input data, like improving dark face photography and camera sensitivity. But we can't just label the input data, the algorithms and the people who designed these algorithms as "racist" or "racially-biased". There is zero racial bias that is man-made here. The race just coincide with the side-effect of photography. Likewise, there will be cases where the reverse is true, like white faces get unwanted results instead of dark faces. So, while we work towards improving the data quality and the algorithms, we must stop this trend of labeling or calling people and algorithms racists.
- deleted 5y ago
  [deleted]
- MichaelGroves 5y ago
  > So, the unwanted results here are still due to the input data. But neither the input data, nor the algorithms contain any racial bias towards the black people. The results might be merely due to the difference between dark faces and pale faces under the natural law of physics. These are unwanted results due to our social norms, but they are NOT racially biased or racist, because there is no such bias introduced or inherent during any part of the process. In this, and the reasoning above it, I think you are correct. Assuming we are right about this, what is the next step? You say the next step is to improve data collection, e.g. by creating better cameras. That seems a fine proposal to me, I support that, but I think there is more that might also be done. For instance, the use of ML models could be restricted or regulated in at least some contexts, until the problems with data collection are rectified. For instance, we could ban the police from using facial recognition models until the problems with data collection are solved. The bias is a side-effect of photography, not something intrinsic to the facial recognition algorithm, but restricting the use of that algorithm might nonetheless be a valid response to this circumstance.
blackbear_ 5y ago
Nah, I don't buy this. Minimizing test error on a socially unbiased dataset will always give a socially unbiased model. Ergo, algorithmic bias does not exist. End of story.
janto 5y ago
i.e. the ball is not under this cup.
narrator 5y ago
This reminds me of a previous crisis involving the intersection of far-left politics and technology after the Russian revolution. What to do about the problem of all scientific progress up to that point being the product of ideologically impure bourgeois capitalism? Stalin, being, for all his notorious faults, a technological pragmatist, had to actually get in the middle of it and straighten things out: "At one time there were 'Marxists' in our country who asserted that the railways left to us after the October Revolution were bourgeois railways, that it would be unseemly for us Marxists to use them, that they should be torn up and new, 'proletarian' railways built. For this they were nicknamed 'troglodytes'. It goes without saying that such a primitive-anarchist view of society, of classes, of language has nothing in common with Marxism. But it undoubtedly exists and continues to prevail in the minds of certain of our muddled comrades." - Stalin, Marxism and the Problem of Linguistics.[1] You could easily redo this for the 21st century: "At one time there were 'Social Justice Warriors' in our country who asserted that the data and algorithms left to us after the Critical Race Theory Revolution were systematically racist data and algorithms, that it would be unseemly for us Social Justice Warriors to use those data and algorithms, that those data and algorithms should be destroyed and new, 'anti-racist' data and algorithms be built. For this they were nicknamed the 'cancel mob'. It goes without saying that such a primitive-woke view of computing, of races, of machine learning has nothing in common with 'Social Justice Warrior Ideology'. But it undoubtedly exists and continues to prevail in the minds of certain of our muddled comrades." The Soviets almost failed to implement the atomic bomb because they had an ideological problem with relativity: "According to a story related by Mr. Holloway, Beria had asked Kurchatov shortly before the conference whether it was true that quantum mechanics and relativity theory were idealist and antimaterialist. Kurchatov reportedly replied that if relativity theory and quantum mechanics had to be rejected by Russian science, the atomic bomb would have to be rejected, too. According to another story in the book, Stalin phrased his decision to cancel this way: "Leave them [ the physicists ] in peace. We can always shoot them later." He could afford a charlatan like Lysenko in biology, but physics was another matter. Stalin relied on his physicists for the bomb -- and for Soviet status as a superpower. When his first atomic bomb was tested in August 1949, five months after the aborted conference, those scientists who would have been shot in the event of failure received the highest awards: Hero of Socialist Labor and so on, down the line."[2] The 21st century version: "Leave them [the data scientists] in peace. We can always cancel them later." I love my downvotes. Thank you :). [1] https://www.marxists.org/reference/archive/stalin/works/1950/jun/20.htm https://www.marxists.org/reference/archive/stalin/works/1950... [2] https://archive.nytimes.com/www.nytimes.com/books/98/12/06/specials/holloway-stalin.html https://archive.nytimes.com/www.nytimes.com/books/98/12/06/s...
ASpring 5y ago
I wrote about this exact topic a few years back: "Algorithmic Bias is Not Just Data Bias" (https://aaronlspringer.com/not-just-data-bias/ https://aaronlspringer.com/not-just-data-bias/). I think the author is generally correct but there is a lot of focus on algorithmic design and not on how we collectively decide what is fair and ethical for these algorithms to do. Right now it is totally up to the algorithm developer to articulate their version of "fair" and implement it however they see fit. I'm not convinced that is a responsibility that belongs to private corporations.
- fennecfoxen 5y ago
  > I'm not convinced that is a responsibility that belongs to private corporations. Private corporations are, by and large, the entities which execute their business using these algorithms, which their employees write. They are already responsible for business decisions whether made using computers or otherwise. Indeed, who else would possibly manage such a thing? This is tantamount to saying that private corporations should have no business deciding how to execute their business — definitely an opinion you can have, it's just that it's an incredibly statist-central-planning opinion the end.
  - bluesummers5651 5y ago
    One of the first papers I read in this area was very interesting in this regard (https://crim.sas.upenn.edu/sites/default/files/2017-1.0-Berk_FairnessCrimJustRisk.pdf https://crim.sas.upenn.edu/sites/default/files/2017-1.0-Berk...). I think the challenge is that a business (e.g. COMPAS) can certainly take a position on what definition of algorithmic fairness they want to enforce, but the paper mentions six different definitions of fairness, which are impossible to satisfy simultaneously unless base rates are the same across all groups (the "data problem"). Even the measurement of these base rates itself can be biased, such as over- or under-reporting of certain crimes. And even if you implement one definition, there's no guarantee that that is the kind of algorithmic fairness that the government/society/case law ends up interpreting as the formal mathematical instantiation of the written law. Moreover, this interpretation can change over time since laws, and for that matter, moral thinking, also change over time. I think the upshot to me is that businesses, whether it's one operating in criminal judicial risk assessment or advertising or whatever, don't really make obvious which definition (if any) of fairness that they are enforcing, and thus it becomes difficult to determine whether they are doing a good job at it.
mgraczyk 5y ago
The linked article is arguing against a straw man that I don't think many ML engineers and researchers actually believe. Whether or not loss functions and model calibration are part of the "data" or the "algorithm" is just a question of semantics. Nobody who knows anything about ML would argue, as is incorrectly suggested by this article, that the choice of loss function plays no role in producing bias or inequity. The actual argument that most closely resembles the straw man in this article goes something like this. "The general project of using deep neural networks is extremely flexible and powerful. It is possible to build datasets and train deep neural networks so that the biases in those models are understood and acceptable. When existing models show bias, there are usually engineering solutions that can remove the bias within the paradigm of deep learning." Counter arguments to this reasonable claim are much more difficult to defend. The research community rewards "whistleblowing" much more strongly than finding solutions, which is generally a good thing. But a nasty side effect is that the majority of algorithm fairness research is pessimistic, despite huge improvements and widespread belief in industry that these problems are solvable using known techniques.
dekhn 5y ago
I'm still trying to understand "algorithmic bias is an algorithm problem". Does that mean, if I select L1, I'm training a network which is less accurate against examples that have rare features, because the weights on those features will be forced to 0? I just want to understand if that's the rough idea or I'm far from the point. If my thinking is approximately correct, then I have a series of further questions and comments, but I think I must be understanding what shooker is saying.
- joiguru 5y ago
  The basic idea is as follows. Lets say you are building an ML model to decide whether to give someone insurance or not. Lets also assume your past behavior had some bias (say against some group). Now ML model trained on this past data will likely learn that bias. Part of modern ML focus is then to understand what bias exists in data, and how can we train models to use the data but somehow counteract that bias.
  - ramoz 5y ago
    A reference I like, based on your last point: https://www.frontiersin.org/articles/10.3389/fpsyg.2013.00504/full https://www.frontiersin.org/articles/10.3389/fpsyg.2013.0050...
  - dekhn 5y ago
    what you just described is a previous bias being encoded in the data. It's not algorithmic bias, because it's not encoded in the structure of the algorithm. Sara addresses that (data re-weighting) but says that's not all. I honestly don't think it can be what you're describing, or the debate is a very different one from what Sara and others in the "algorithmic bias exists and it is distinct from data bias" sense.
  - umvi 5y ago
    How do you tell if something is biased or not? Seems like the current system is "if people cry foul because it seems unfair, then the model is biased" which doesn't seem scientifically rigorous. This seems like a hard problem. For example, say that you have an ML model that decides whether someone will be a good sports athlete or not purely based on biometrics (blood oxygen level, blood pressure, BMI, reflex time, etc.). If the model starts predicting black people will be better athletes at higher rates than white people, is the ML model biased? Or is the reality that black people have higher-than-average advantageous physical characteristics? How do you tell the difference between bias and reality?
908B64B197 5y ago
Again with the "Ethics in AI". You can't say it publicly but ethics is a trend. It's trendy right now to apply it to AI because there's a lot of academic funding going into AI so fields where there's no value production (like ethics) will slap "AI" on their grant proposal. The point of ethics is to shame people or practices you don't like via some kind of cancel culture of peer pressure [0]. Of course, it's impossible to do it in court as you must actually prove things. So you get a large enough mob instead. There's mostly no point in debating with the ethics crowd (you really can't do that since their papers typically won't be reproducible). Or acknowledging their existence really. [0] https://syncedreview.com/2020/06/30/yann-lecun-quits-twitter-amid-acrimonious-exchanges-on-ai-bias/ https://syncedreview.com/2020/06/30/yann-lecun-quits-twitter...
- dekhn 5y ago
  Let's consider your comments applied to another ethical area: eugenics. That was a previous area where scientists did something which is now considered highly unethical (remove the sexual autonomy of people due to incorrect scientific judgements). Would you say the people who opposed eugenics were "shaming" the scientists, or helping "guide them to making better decisions?" I think there are many concerns around ethics in AI, but the majority of the players are not contributing (folks like Timnit, with the stochastic parrots paper, the sections on power consumption are completely technically wrong). Sara Hooker is a much better spokesperson for this, but I'm really struggling to see the point in her paper, beyond "we should identify existing data biases and make algorithmic changes to reduce them" which just sounds like "we need regularization that is the opposite of L1 and L2".
  - 908B64B197 5y ago
    It's interesting you had to pick illegal practices to attempt to make a point, while my point was that ethics was used to shame practices when the legal system couldn't be used because of a lack of rigor.
    - dekhn 5y ago
      eugenics wasn't illegal at the time. https://www.nytimes.com/2014/10/14/science/haunted-files-the-eugenics-record-office-recreates-a-dark-time-in-a-laboratorys-past.html https://www.nytimes.com/2014/10/14/science/haunted-files-the... When the Eugenics Record Office opened its doors in 1910, the founding scientists were considered progressives, intent on applying classic genetics to breeding better citizens. Funding poured in from the Rockefeller family and the Carnegie Institution. Charles Davenport, a prolific Harvard biologist, and his colleague, Harry H. Laughlin, led the charge. ... By the 1920s, the office had begun to influence the United States government. Laughlin testified before Congress, advocating forced sterilization and anti-immigration laws. Congress complied. The Immigration Act of 1924 effectively barred Eastern Europeans, Jews, Arabs and East Asians from entering the country. And, at the state level, thousands of people who were deemed unfit were sterilized.
mirker 5y ago
Have we even figured out what bias we care about? Race, gender, age, etc. are some potential problems, but is that it? These sorts of problems are often formulated theoretically (“suppose we want fairness with respect to variable Z”). It seems that half the battle is to figure out what to be fair with respect to. Often, the fairness variable in question isn’t even a feature, but is implicitly in the data (e.g., race in human photos). Therefore, the fairness space is potentially infinite. For example, maybe life insurance models are biased toward those predisposed to developing cancer. Maybe ads target those suffering depression. You can continue partitioning the space in this way forever, and, therefore, it seems that the algorithms are relatively straightforward if you could formalize the bias requirements. This is before you even consider the fairness variables interacting (e.g., age and gender and race), which requires potentially normalizing across exponentially growing feature combinations.
- mulmen 5y ago
  This is a philosophical gap that society in general needs to bridge. Data is truth. If we discover bias in data that tells us something about the world. I can't fix systemic racism with a SQL statement. But I can tell you where it happened. Society can decide what classes are protected and data can tell us when that happens. It is then up to the offending entity to change their behavior. In other words bias in data isn't unethical but bias in action can be.
  - whatshisface 5y ago
    I feel that talking about SQL in a way misses this issue - when data analysis is done by human beings, human virtues can restrain the worst excesses of self-interest. In contrast, a model trained blindly to maximize some kind of metric can commit essentially any sin in pursuit of that goal.
    - mulmen 5y ago
      A model is just another form of analysis. A SQL statement is already built on levels of abstraction. An “AI” model is only one more level. At the bottom it’s all ones and zeros. But no matter how high you make the tower there’s always a human at the top.
6gvONxR4sf7o 5y ago
I've had the best luck explaining this in terms of causal inference. We all know that it's really easy to screw up or invent connections between things when you use observational data instead of a randomized controlled trial. Observational data contains weird connections that you often can't tease apart, whether it's because there are important aspects of the mechanisms that were unrecorded (missing features), selection bias (missing data), or because regularization forces your hand in interesting ways. Generally for statisticians, this leads to bias in parameter space. For ML practicioners, it's more interesting/useful to regard this as bias in function space. There are useful lessons from the causal inference world too. There is a whole field of tools to try to get unbiased parameters/functions from observational data. There are even models that can guarantee unbiased parameters/functions under pretty reasonable assumptions and reasonable conditions. There is a rapidly developing field around learning unbiased (estimates of) functions from less "perfect" data.
bjornsing 5y ago
I have a really hard time following this kind of reasoning, partly because the word bias has very different meanings in social science and statistics. E.g. let’s say I have a statistical model that takes among other variables a person’s ethnicity and produces a credit score. If that model is statistically unbiased it would probably produce different credit scores depending on ethnicity. If we “fix” that by making the model insensitive to ethnicity, then it probably becomes statistically biased. Wouldn’t it be better to talk about “fairness” or something?
- Spivak 5y ago
  You’re viewing “fixing” from the wrong lens. There are infinite bits of information about a person that could potentially be used in a model to calculate a credit score. At all times in every model you’re choosing a subset of one’s facets and can build statistically unbiased models based on the data you make available. But when you talking actually fixing models like this you’re actually forced to correct the final result, not filter the data. Being blind to ethnicity doesn’t work because one’s ethnicity permeates (to different degrees, sure) every part of their lives. All the data is bad, everything is a statistically detectable proxy for ethnicity.
  - jhgb 5y ago
    But surely there's some kind of refining your inputs where all the paths converge to one result, much like the refinement of interval partitions in Riemann's integral converges to one value (for a certain class of functions). The bits of information may be infinite but there's some structure in them. I'm not sure that just because the optimal result (the one that uses all information) is something you don't like, the result is bad. Best thing you can say is that you're actually not predicting an existing credit score, but rather synthesizing some other indicator that is not an existing credit score.
- BoiledCabbage 5y ago
  Agreed, overloading of terms often makes topics much harder to grok. Quick summary, is there are many meanings of the term bias in ML. First let's define which the author is talking about. Some options are: 1. The idea of ignoring relevant features, ala bias-variance trade off. 2. Data bias, where data is trained on inputs / behavior that society would deem biased. Ex. A company frequently discriminates against group X, an ML model is trained on data / decisions this company has made and as a result encodes the same biases. 3. Features are used which are a proxy for other items. Ex. (pulling this out of the air), a repair shop charges a variable rate for repair. It takes in a number of inputs. It's illegal to offer higher prices to women for the same repair as men, but instead an ML model uses as input the customer's first name which ends up being nothing more than a proxy for their gender and introduces that bias. 4. A group of people A have a much smaller portion of the training data size in an ML model than group B. The model makes decisions for offering some good (ex. a loan). Because it's training data is larger on people from group B than group A it always has more confidence in its predictions from group B. The model is set to be risk adverse, so by default it rejects the majority of people from group A even though most of them would have been valid loans. (Where A and B are different across some societal class like religion, race, ethnicity, gender...) The author is talking about the fourth case. While yes there is over/under representation in the input data, the algorithm chosen to build the model encodes its own form of bias. A simple example everyone should be familiar with, using Mean Squared Error vs. Mean Absolute Error (or MBE) produce models that handle rarely seen features or data points very differently. This choice of loss function introduces algorithmic bias into a model. That's essentially what underlies the concept. Yes there is data bias, but what is being argued is that the solution isn't "find perfectly unbiased data" (because in many cases that's impossible), it's understand how the choice of algorithm will produce models that respond to and represent differently those data biases.
malshe 5y ago
I admit that I still don't know a single example of systematic AI bias in the absence of any data bias. I really want to improve my understanding of this topic. This article doesn't help either. It starts off with why we should address this bias without giving an example of it. In fact, the author skirts this issue altogether by first stating: "Here, we start out with a deceptively simple question: how does model design contribute to algorithmic bias?" and then moves into why this is an important issue! If you said "we start" then let's start indeed. Maybe I am not used to this style of writing?
- c1ccccc1 5y ago
  In the most general sense of the word bias, there can be biases that arise purely as a result of model architecture. For example, I was messing around with GANs about a year ago, and kept on getting checkerboard-like patterns in the generated images. This was a result of how I had set up the convolution layers: It was a bias introduced by model architecture. In terms of the "unfairness based on what group you belong to" kind of bias, that's almost always a result of bad input; you'd have to work really hard to make an inherently racist architecture. So the problem is introduced in the training data. But it could be solved either by getting better training data, or by making clever improvements to your algorithm to work around the limitations of the training data. Often changing the algorithm is easier than improving the data, which I think is the point this article is making. As an example of such a change, let's say we're designing an algorithm to estimate credit score, and we're worried about unfairness based on group membership. Let M be a variable representing what groups a person belongs to: race sex, etc, and let S be the output of the algorithm: that person's credit score. An easy thing to do would be to not allow M as an input to our algorithm. That way we can say that the output "does not depend on group membership". The problem is that we may still have proxies for M as inputs to the algorithm, and if the dataset is biased, that algorithm will still learn to use those proxies to discriminate based on group membership. A better method is surprisingly to keep M as an input during training of the model. Then, at inference time, average over M to get the final output of the model. That still erases the information about what group a particular person belongs to, but it also has the advantage that the model isn't incentivized to learn proxies for M during training, since it has access to M directly.
golemiprague 5y ago
If anything the system that made Obama to be a white man was right, he is half white so why do we assume the black half should define him?
a_square_peg 5y ago
I think talking about 'bias' in ML algorithms is a fundamentally incorrect anthropomorphic attribution. The only thing that ML does is mapping a set of inputs to an output according to a prior statistical distribution.
AndrewKemendo 5y ago
This is an important topic but the framing is not technically robust. Are convolutions "biased" because they are amplifying variance between adjacent pixels? Yes if your definition of biased is statistical, No if your definition of bias is social. I have $300 for someone who can make a convincing technical argument that the inception architecture [1] is socially biased, without actually training it on a dataset. Until then its the data or implementations that are biased. Maybe your trained model is biased - yep that's the data. Maybe you're cherry picking your algorithm to give you the best result on your chosen dataset - that's a biased implementation. More broadly though, the concept that "systems/people will make inferences on new data which align with patterns they see more frequently" is just a reality of intelligence at this point. Humans constantly fail at this. So yes we should be better and bias is real, to solve it we need to get way more technically accurate on what that bias consists of please. [1]https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43022.pdf https://static.googleusercontent.com/media/research.google.c...
cratermoon 5y ago
https://cjc-online.ca/index.php/journal/article/view/2196/2055 https://cjc-online.ca/index.php/journal/article/view/2196/20... "emulsions and digital camera design, the rendering of non-Caucasian skin toneswas highly deficient and required the development of compensatory practicesand technology improvements to redress its shortcomings. Using the emblem-atic “Shirley” norm reference card as a central metaphor reflecting the changingstate of race relations/aesthetics, this essay analytically traces the colour adjust-ment processes in the industries" https://sitn.hms.harvard.edu/flash/2020/racial-discrimination-in-face-recognition-technology/ https://sitn.hms.harvard.edu/flash/2020/racial-discriminatio... "All three algorithms performed the worst on darker-skinned females, with error rates up to 34% higher than for lighter-skinned males"

69 points by mwexler 5y ago

Ah... The ML scientist's "Metrics are harmless. People doing stupid things with metrics on the other...". Or if I'm being too subtle, this is the ML practitoner's "Guns don't kill people. People kill people." Which is absolutely right, I might add; but also a stop point and consideration that should be had before you go training something on sketchy data, and unleashing it on an unknowing populace.

elcomet 5y ago
That's not what the article is talking about. They are saying that data biases are a problem but not the only one: other things can amplify biases such as the model's architecture. So when working to remove biases, fixing the data might not be enough.
- athrowaway3z 5y ago
  What salawat is probably saying is: There would be less issue with bias (for society), if the expectation/sales pitch was different. As an analogy: Somebody sells the idea of "The Unbreakable Rope" and it breaks after you hang 10kg on it. A blog post "Moving beyond material" goes in depth on how to process material. This is a good thing for people producing rope. However, for a layman who still sees people assuming "The Unbreakable Rope" is attainable, the content is going to be a little underwhelming.

trhway 5y ago

the point of the article is that algorithm design choices affect results on the long-tail of the dataset. Well, what is in the long-tail of the dataset is data bias. So, are we going to try to introduce algorithm bias to correct for the data bias?

zozbot234 5y ago

Obligatory reminder that there's an inherent tradeoff between bias and variance in any ML architecture. Try to create a "highly regular" model that gives predictable, less noisy results even with low-quality data, you'll just end up strengthening some implied prior and introducing bias. Try to remove inbuilt bias, you'll just make the model more data-dependent, and noisier with any given amount of data. You just can't win.

jstx1 5y ago
Also important to point out that the word bias has a very specific meaning in the context of statistics and machine learning and it isn't just a synonym for discrimination or something of that sort - https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff
- commandlinefan 5y ago
  ... which seems not to be the definition the author of the linked article is using.
rualca 5y ago
> Obligatory reminder that there's an inherent tradeoff between bias and variance in any ML architecture. It should also be stressed that in this context "bias" is a kin to interpolating between data points where resolution in the training set is relatively low, thus leads to a simpler model that does not exactly match the test set. What this implies is that this "bias" does not reflect a preference, let alone an intention to favour one group over another. It just means that a combination of the test set being too sparse or with too few dimensions, the choice of model being too inflexible, and the problem domain having too much uncertainty at that resolution. Critics may single out the model, but the core of the problem actually lies on the data and the problem domain.

darawk 5y ago

> A surprisingly sticky belief is that a machine learning model merely reflects existing algorithmic bias in the dataset and does not itself contribute to harm. Why, despite clear evidence to the contrary, does the myth of the impartial model still hold allure for so many within our research community? Algorithms are not impartial, and some design choices are better than others. Recognizing how model design impacts harm opens up new mitigation techniques that are less burdensome than comprehensive data collection. Ah yes the classic "why do people keep insisting on X despite consistent proof of not X", with...zero citations. Well, let's read the paper and see if they make the case. > Even if we are able to label sensitive attributes at scale such as gender and race, algorithms can still leverage proxy variables to reconstruct the forbidden label. Data collection of even a limited number of protected attributes can be onerous. For example, it is hard to align on a standard taxonomy—categories attributed to race or gender are frequently encoded in inconsistent ways across datasets.2 Furthermore, procuring labels for these legally protected attributes is often perceived as intrusive leading to noisy or incomplete labels.3,4 > If we cannot guarantee we have fully addressed bias in data pipeline, the overall harm in a system is a product of the interactions between the data and our model design choices. Here, acknowledging the impact of model design bias can play an important role in curbing harm. Algorithms are not impartial, and some design choices are better than others. Recognizing how model design impacts harm opens up new mitigation techniques that are far less burdensome than comprehensive data collection. > We are well-versed in the connection between function choice and test-set accuracy because objective functions such as cross-entropy or mean squared error reflect our preference to optimize for high test-set accuracy. Standard loss functions do not explicitly encode preferences for other objectives we care about such as algorithmic bias, robustness, compactness, or privacy. However, just because these desiderata are not reflected does not mean they have ceased to exist. Turing award winner Donald Knuth said that computers “do exactly what they are told, no more and no less.” A model can fulfill an objective in many ways, while still violating the spirit of said objective. > Model design choices made to maximize test-set accuracy do not hold static other properties we care about such as robustness and fairness. On the contrary, training a parametric model is akin to having a fixed amount of materials to build a house with. If we decide to use more bricks building a bigger living room, we force the redistribution of the number of bricks available for all other rooms. In the same vein, when we prioritize one objective, whether that be test-set accuracy or additional criteria such as compactness and privacy, we inevitably introduce new trade-offs. This is literally just saying "Well yes all the bias starts out in the data, and so a race/gender-neutral model will encode those features, but in principle you could design a model expressly to avoid that, and since that hasn't been done, the models are racist too". Which is...a certain kind of take. But even if we accept the reasoning, it is not at all at odds with the common understanding of their starting sentence: "A surprisingly sticky belief is that a machine learning model merely reflects existing algorithmic bias in the dataset and does not itself contribute to harm.". Yes, people commonly believe the "bias is in the data". Models are designed to be mirrors, so they reflect whatever bias is in their training data. Yes, it's true you can design a distorted mirror that does not reflect certain attributes. But that would be inserting bias into a model to correct a bias that you found in the original data. It should be very clear that the root source of the bias here is the training data, not the model. We can and should build models that attempt to correct those biases, and we also can and should attempt to de-bias the data itself. But can we stop torturing the meaning of English sentences to support people's preferred narratives? The models are neutral. The data contains the bias. Convolutional neural nets do not inherently work better on white faces. Multi-layer perceptrons do not have beliefs about race and gender. One additional linguistic nitpick: > The belief that model design merely reflects algorithmic bias in the dataset can be partly ascribed to the difficulty of measuring interactions between all the variables we care about. There is no "algorithmic bias in the dataset". That is a contradiction in terms. The bias in the dataset is not "algorithmic". The algorithm is what processes the data. The dataset is just biased, it has not been biased algorithmically.

macleginn 5y ago

"A key takeaway is that our algorithms are not impartial." Of course they are, they just underperform on the data points from the long tail of the empirical distribution, as show basically all real-world examples from the paper. Underfitting on the samples from the long tale (aka blinding the model to ethically bad features) removes bias sensu strictu but increases variance, which leads to more errors, which is perceived as "bias" (did someone say catch 22?).

geofft 5y ago
The classic algorithm to solve the https://en.wikipedia.org/wiki/Stable_marriage_problem https://en.wikipedia.org/wiki/Stable_marriage_problem is a pretty straightforward and frankly introductory-level example of an algorithm which is not impartial (and not just because it ought to be called the stable heterosexual monogamous marriage problem). The algorithm outputs a matching between two sets A and B (e.g., bachelors and bachelorettes) where each member of set A has ranked each member of set B in preference order and vice versa. The output is called stable because there's no situation where any pair would prefer to be with each other than their current partners, e.g., there's never a case where Alice prefers Bob to her current partner and Bob prefers Alice to his current partner and are tempted to run away with each other. But a curious property of this algorithm is that the matching is always best for one group and always worst for the other. Of the possible answers to the problem, this algorithm an answer that is at least as good for set A as any other one, and at least as bad for set B as any other one. You can, of course, flip sets A and B when running the algorithm, and it will them be optimal for set B and pessimal for set A. This has absolutely nothing to do with ML or data fitting or anything. It's a deterministic, classical algorithm. And yet it is clearly partial, and you must choose when using it whether to be partial to set A or set B, to the men or to the women, to med school students or to their employers, etc. It would be silly to say that the algorithm is impartial and that it's solely the operator's choice to make it impartial - this particular algorithm forces the operator to make some choice. I think in the end you're agreeing with the author, who is saying that you must make a tradeoff of some sort, and the question is which tradeoff to choose. The author (and I) would describe that as the algorithm being partial, and the operator only being able to make some choice about what to do with that.

fouronnes3 5y ago

This belief has always been extremely cringe in my opinion, because it somehow implies that dataset engineering is not an important part of machine learning. Ensuring you are feeding good data to your model is a critical part of machine learning practice, but some researchers seem to hand wave it away as merely an implementation detail not worthy of their attention. "It's a data problem" is used as an excuse to be discharged of the moral responsability of the output bias, as if you were ethically responsible for the model only.

darawk 5y ago
I don't think "it's a data problem" is inherently used to hand-wave away the problem. I think it's used to locate the problem in the area where its most readily addressed. Designing algorithms to debias your data is hard, over-sampling under-represented groups is easier. I do think you're right that sometimes people use it that way, but that doesn't mean we should make up false narratives about biased models, either. It just means we should work on de-biasing the data, and also developing algorithms to help mitigate the learning of features we don't want weighted.
- mjburgess 5y ago
  You are right that there are no algorithms which capture semantics, and therefore none which can be constructed not to introduce bias. In that sense it is a "data solution", but it is not a data problem. > de-biasing the data The issue isnt statistical bias. The issue is semantic: occurrence doesnt capture meaning. Negative terms and racial terms can co-occur more frequently in some text (eg., a biography of MLK) without the remedy being to "rebalance them with positive terms".
rualca 5y ago
> (...) as if you were ethically responsible for the model only. The model is the result and the outcome of the whole process. The model is supposed to be a reliable representation of reality that has acceptable accuracy within predetermined operational limits. If someone tries to generate models that interpolate and/or extrapolate data and the data it uses to seed the model is garbage then the model is garbage, and the person responsible for putting it together is the person responsible for the model failing to do its job. There is no separation between model and the data used to generate the model. Garbage in, garbage out. If a machine learning model uses garbage data then the model is garbage, and throwing machine learning buzzwords at the problem does not justify why garbage was sent in.

mjburgess 5y ago

Many here are in the throws of this "data mythology"... It is trivial for algorithms to introduce novel racism: consider processing text analysing minority causes. These texts contain statistical associations between politically sensitive terms, negative terms, (and so on). However they are not racist, are not born of any racist project, and most humans reading them would be moved to more positively regard racial terms. The meaning text express consists in what they want their readers to do. It is an activity of coordination (writer/reader); and it here it has no racist aspect. A machine if it should have learned anything at all from this text, should learn to associate minority terms with virtuous political projects; it should have acquired an understanding of the many types of associative relation: needing, wanting, opposing, supporting, trusting, advocating... Yet a machine performing an action based on statistical associations will not learn these; and so, will act prejudicially. It will merely expect terms to co-occur, wholly unable to determine why. We expect charity workers to be greedy? No, we expect them to morally oppose greed. We expect movies to smell like candy? No, we expect them to be sold together. (And so on.) It is somewhat alarming that statistical associations in terms in historical texts are being seen as characteristic of human communication, meaning, belief, association, ethics, action. I think it has led here to a great moral oversight: that absent the ability to understand text, machines here are introducing novel prejudice where none existed prior.

davidgerard 5y ago
> they are not racist "having racist effects" counts as a perfectly valid usage of the word. This is a cheap sophistry: 1. It's only racist if it was intended to be. 2. You can't read minds. 3. Therefore, you can't say "this is racist." This isn't a useful usage of the term, except for attempting to deflect people from calling out racist effects.
- concordDance 5y ago
  I really wish the word "racist" had a single meaning.
  - user-the-name 5y ago

janto 5y ago

This field is a joke and should be mercilessly mocked. These manipulated models do not represent reality. I look forward to gaming the systems that are produced by these people.

elisbce 5y ago

Let's take the classic example of a "racist algorithm" Image recognition using neural nets and examine what's going on. Let's say we train the NN using equal number of human faces of all races, and animals faces. Let's say the trained neural net made some mistakes, including a few cases where black faces are recognized as gorilla faces. And this doesn't happen at all or as likely with white faces. The results are horrible, right? And people immediately start to point fingers to the training data and the algorithms, stating the training data is racially biased and/or the training algorithms or even the use of neural nets are racially biased. But is it really so? It's known that in order to take a black face photo with the same degree of details, the lighting condition and camera settings need to be adjusted. This is an effect purely due to physics. In other words, it could well be by nature, that recognizing black faces is harder than pale faces under the normal camera and scene settings. This is why you have night mode on your phones. It is just harder to take clear photos when less light gets into the camera. And this requires the camera and photography settings to be adjusted. So, the unwanted results here are still due to the input data. But neither the input data, nor the algorithms contain any racial bias towards the black people. The results might be merely due to the difference between dark faces and pale faces under the natural law of physics. These are unwanted results due to our social norms, but they are NOT racially biased or racist, because there is no such bias introduced or inherent during any part of the process. We could and should correct such unwanted results by introducing adjustments to the input data, like improving dark face photography and camera sensitivity. But we can't just label the input data, the algorithms and the people who designed these algorithms as "racist" or "racially-biased". There is zero racial bias that is man-made here. The race just coincide with the side-effect of photography. Likewise, there will be cases where the reverse is true, like white faces get unwanted results instead of dark faces. So, while we work towards improving the data quality and the algorithms, we must stop this trend of labeling or calling people and algorithms racists.

deleted 5y ago
[deleted]
MichaelGroves 5y ago
> So, the unwanted results here are still due to the input data. But neither the input data, nor the algorithms contain any racial bias towards the black people. The results might be merely due to the difference between dark faces and pale faces under the natural law of physics. These are unwanted results due to our social norms, but they are NOT racially biased or racist, because there is no such bias introduced or inherent during any part of the process. In this, and the reasoning above it, I think you are correct. Assuming we are right about this, what is the next step? You say the next step is to improve data collection, e.g. by creating better cameras. That seems a fine proposal to me, I support that, but I think there is more that might also be done. For instance, the use of ML models could be restricted or regulated in at least some contexts, until the problems with data collection are rectified. For instance, we could ban the police from using facial recognition models until the problems with data collection are solved. The bias is a side-effect of photography, not something intrinsic to the facial recognition algorithm, but restricting the use of that algorithm might nonetheless be a valid response to this circumstance.

blackbear_ 5y ago

Nah, I don't buy this. Minimizing test error on a socially unbiased dataset will always give a socially unbiased model. Ergo, algorithmic bias does not exist. End of story.

janto 5y ago

i.e. the ball is not under this cup.

narrator 5y ago

This reminds me of a previous crisis involving the intersection of far-left politics and technology after the Russian revolution. What to do about the problem of all scientific progress up to that point being the product of ideologically impure bourgeois capitalism? Stalin, being, for all his notorious faults, a technological pragmatist, had to actually get in the middle of it and straighten things out: "At one time there were 'Marxists' in our country who asserted that the railways left to us after the October Revolution were bourgeois railways, that it would be unseemly for us Marxists to use them, that they should be torn up and new, 'proletarian' railways built. For this they were nicknamed 'troglodytes'. It goes without saying that such a primitive-anarchist view of society, of classes, of language has nothing in common with Marxism. But it undoubtedly exists and continues to prevail in the minds of certain of our muddled comrades." - Stalin, Marxism and the Problem of Linguistics.[1] You could easily redo this for the 21st century: "At one time there were 'Social Justice Warriors' in our country who asserted that the data and algorithms left to us after the Critical Race Theory Revolution were systematically racist data and algorithms, that it would be unseemly for us Social Justice Warriors to use those data and algorithms, that those data and algorithms should be destroyed and new, 'anti-racist' data and algorithms be built. For this they were nicknamed the 'cancel mob'. It goes without saying that such a primitive-woke view of computing, of races, of machine learning has nothing in common with 'Social Justice Warrior Ideology'. But it undoubtedly exists and continues to prevail in the minds of certain of our muddled comrades." The Soviets almost failed to implement the atomic bomb because they had an ideological problem with relativity: "According to a story related by Mr. Holloway, Beria had asked Kurchatov shortly before the conference whether it was true that quantum mechanics and relativity theory were idealist and antimaterialist. Kurchatov reportedly replied that if relativity theory and quantum mechanics had to be rejected by Russian science, the atomic bomb would have to be rejected, too. According to another story in the book, Stalin phrased his decision to cancel this way: "Leave them [ the physicists ] in peace. We can always shoot them later." He could afford a charlatan like Lysenko in biology, but physics was another matter. Stalin relied on his physicists for the bomb -- and for Soviet status as a superpower. When his first atomic bomb was tested in August 1949, five months after the aborted conference, those scientists who would have been shot in the event of failure received the highest awards: Hero of Socialist Labor and so on, down the line."[2] The 21st century version: "Leave them [the data scientists] in peace. We can always cancel them later." I love my downvotes. Thank you :). [1] https://www.marxists.org/reference/archive/stalin/works/1950/jun/20.htm https://www.marxists.org/reference/archive/stalin/works/1950... [2] https://archive.nytimes.com/www.nytimes.com/books/98/12/06/specials/holloway-stalin.html https://archive.nytimes.com/www.nytimes.com/books/98/12/06/s...

ASpring 5y ago

I wrote about this exact topic a few years back: "Algorithmic Bias is Not Just Data Bias" (https://aaronlspringer.com/not-just-data-bias/ https://aaronlspringer.com/not-just-data-bias/). I think the author is generally correct but there is a lot of focus on algorithmic design and not on how we collectively decide what is fair and ethical for these algorithms to do. Right now it is totally up to the algorithm developer to articulate their version of "fair" and implement it however they see fit. I'm not convinced that is a responsibility that belongs to private corporations.

fennecfoxen 5y ago
> I'm not convinced that is a responsibility that belongs to private corporations. Private corporations are, by and large, the entities which execute their business using these algorithms, which their employees write. They are already responsible for business decisions whether made using computers or otherwise. Indeed, who else would possibly manage such a thing? This is tantamount to saying that private corporations should have no business deciding how to execute their business — definitely an opinion you can have, it's just that it's an incredibly statist-central-planning opinion the end.
- bluesummers5651 5y ago
  One of the first papers I read in this area was very interesting in this regard (https://crim.sas.upenn.edu/sites/default/files/2017-1.0-Berk_FairnessCrimJustRisk.pdf https://crim.sas.upenn.edu/sites/default/files/2017-1.0-Berk...). I think the challenge is that a business (e.g. COMPAS) can certainly take a position on what definition of algorithmic fairness they want to enforce, but the paper mentions six different definitions of fairness, which are impossible to satisfy simultaneously unless base rates are the same across all groups (the "data problem"). Even the measurement of these base rates itself can be biased, such as over- or under-reporting of certain crimes. And even if you implement one definition, there's no guarantee that that is the kind of algorithmic fairness that the government/society/case law ends up interpreting as the formal mathematical instantiation of the written law. Moreover, this interpretation can change over time since laws, and for that matter, moral thinking, also change over time. I think the upshot to me is that businesses, whether it's one operating in criminal judicial risk assessment or advertising or whatever, don't really make obvious which definition (if any) of fairness that they are enforcing, and thus it becomes difficult to determine whether they are doing a good job at it.

mgraczyk 5y ago

The linked article is arguing against a straw man that I don't think many ML engineers and researchers actually believe. Whether or not loss functions and model calibration are part of the "data" or the "algorithm" is just a question of semantics. Nobody who knows anything about ML would argue, as is incorrectly suggested by this article, that the choice of loss function plays no role in producing bias or inequity. The actual argument that most closely resembles the straw man in this article goes something like this. "The general project of using deep neural networks is extremely flexible and powerful. It is possible to build datasets and train deep neural networks so that the biases in those models are understood and acceptable. When existing models show bias, there are usually engineering solutions that can remove the bias within the paradigm of deep learning." Counter arguments to this reasonable claim are much more difficult to defend. The research community rewards "whistleblowing" much more strongly than finding solutions, which is generally a good thing. But a nasty side effect is that the majority of algorithm fairness research is pessimistic, despite huge improvements and widespread belief in industry that these problems are solvable using known techniques.

dekhn 5y ago

I'm still trying to understand "algorithmic bias is an algorithm problem". Does that mean, if I select L1, I'm training a network which is less accurate against examples that have rare features, because the weights on those features will be forced to 0? I just want to understand if that's the rough idea or I'm far from the point. If my thinking is approximately correct, then I have a series of further questions and comments, but I think I must be understanding what shooker is saying.

joiguru 5y ago
The basic idea is as follows. Lets say you are building an ML model to decide whether to give someone insurance or not. Lets also assume your past behavior had some bias (say against some group). Now ML model trained on this past data will likely learn that bias. Part of modern ML focus is then to understand what bias exists in data, and how can we train models to use the data but somehow counteract that bias.
- ramoz 5y ago
  A reference I like, based on your last point: https://www.frontiersin.org/articles/10.3389/fpsyg.2013.00504/full https://www.frontiersin.org/articles/10.3389/fpsyg.2013.0050...
- dekhn 5y ago
  what you just described is a previous bias being encoded in the data. It's not algorithmic bias, because it's not encoded in the structure of the algorithm. Sara addresses that (data re-weighting) but says that's not all. I honestly don't think it can be what you're describing, or the debate is a very different one from what Sara and others in the "algorithmic bias exists and it is distinct from data bias" sense.
- umvi 5y ago
  How do you tell if something is biased or not? Seems like the current system is "if people cry foul because it seems unfair, then the model is biased" which doesn't seem scientifically rigorous. This seems like a hard problem. For example, say that you have an ML model that decides whether someone will be a good sports athlete or not purely based on biometrics (blood oxygen level, blood pressure, BMI, reflex time, etc.). If the model starts predicting black people will be better athletes at higher rates than white people, is the ML model biased? Or is the reality that black people have higher-than-average advantageous physical characteristics? How do you tell the difference between bias and reality?

908B64B197 5y ago

Again with the "Ethics in AI". You can't say it publicly but ethics is a trend. It's trendy right now to apply it to AI because there's a lot of academic funding going into AI so fields where there's no value production (like ethics) will slap "AI" on their grant proposal. The point of ethics is to shame people or practices you don't like via some kind of cancel culture of peer pressure [0]. Of course, it's impossible to do it in court as you must actually prove things. So you get a large enough mob instead. There's mostly no point in debating with the ethics crowd (you really can't do that since their papers typically won't be reproducible). Or acknowledging their existence really. [0] https://syncedreview.com/2020/06/30/yann-lecun-quits-twitter-amid-acrimonious-exchanges-on-ai-bias/ https://syncedreview.com/2020/06/30/yann-lecun-quits-twitter...

dekhn 5y ago
Let's consider your comments applied to another ethical area: eugenics. That was a previous area where scientists did something which is now considered highly unethical (remove the sexual autonomy of people due to incorrect scientific judgements). Would you say the people who opposed eugenics were "shaming" the scientists, or helping "guide them to making better decisions?" I think there are many concerns around ethics in AI, but the majority of the players are not contributing (folks like Timnit, with the stochastic parrots paper, the sections on power consumption are completely technically wrong). Sara Hooker is a much better spokesperson for this, but I'm really struggling to see the point in her paper, beyond "we should identify existing data biases and make algorithmic changes to reduce them" which just sounds like "we need regularization that is the opposite of L1 and L2".
- 908B64B197 5y ago
  It's interesting you had to pick illegal practices to attempt to make a point, while my point was that ethics was used to shame practices when the legal system couldn't be used because of a lack of rigor.
  - dekhn 5y ago
    eugenics wasn't illegal at the time. https://www.nytimes.com/2014/10/14/science/haunted-files-the-eugenics-record-office-recreates-a-dark-time-in-a-laboratorys-past.html https://www.nytimes.com/2014/10/14/science/haunted-files-the... When the Eugenics Record Office opened its doors in 1910, the founding scientists were considered progressives, intent on applying classic genetics to breeding better citizens. Funding poured in from the Rockefeller family and the Carnegie Institution. Charles Davenport, a prolific Harvard biologist, and his colleague, Harry H. Laughlin, led the charge. ... By the 1920s, the office had begun to influence the United States government. Laughlin testified before Congress, advocating forced sterilization and anti-immigration laws. Congress complied. The Immigration Act of 1924 effectively barred Eastern Europeans, Jews, Arabs and East Asians from entering the country. And, at the state level, thousands of people who were deemed unfit were sterilized.

mirker 5y ago

Have we even figured out what bias we care about? Race, gender, age, etc. are some potential problems, but is that it? These sorts of problems are often formulated theoretically (“suppose we want fairness with respect to variable Z”). It seems that half the battle is to figure out what to be fair with respect to. Often, the fairness variable in question isn’t even a feature, but is implicitly in the data (e.g., race in human photos). Therefore, the fairness space is potentially infinite. For example, maybe life insurance models are biased toward those predisposed to developing cancer. Maybe ads target those suffering depression. You can continue partitioning the space in this way forever, and, therefore, it seems that the algorithms are relatively straightforward if you could formalize the bias requirements. This is before you even consider the fairness variables interacting (e.g., age and gender and race), which requires potentially normalizing across exponentially growing feature combinations.

mulmen 5y ago
This is a philosophical gap that society in general needs to bridge. Data is truth. If we discover bias in data that tells us something about the world. I can't fix systemic racism with a SQL statement. But I can tell you where it happened. Society can decide what classes are protected and data can tell us when that happens. It is then up to the offending entity to change their behavior. In other words bias in data isn't unethical but bias in action can be.
- whatshisface 5y ago
  I feel that talking about SQL in a way misses this issue - when data analysis is done by human beings, human virtues can restrain the worst excesses of self-interest. In contrast, a model trained blindly to maximize some kind of metric can commit essentially any sin in pursuit of that goal.
  - mulmen 5y ago
    A model is just another form of analysis. A SQL statement is already built on levels of abstraction. An “AI” model is only one more level. At the bottom it’s all ones and zeros. But no matter how high you make the tower there’s always a human at the top.

6gvONxR4sf7o 5y ago

I've had the best luck explaining this in terms of causal inference. We all know that it's really easy to screw up or invent connections between things when you use observational data instead of a randomized controlled trial. Observational data contains weird connections that you often can't tease apart, whether it's because there are important aspects of the mechanisms that were unrecorded (missing features), selection bias (missing data), or because regularization forces your hand in interesting ways. Generally for statisticians, this leads to bias in parameter space. For ML practicioners, it's more interesting/useful to regard this as bias in function space. There are useful lessons from the causal inference world too. There is a whole field of tools to try to get unbiased parameters/functions from observational data. There are even models that can guarantee unbiased parameters/functions under pretty reasonable assumptions and reasonable conditions. There is a rapidly developing field around learning unbiased (estimates of) functions from less "perfect" data.

bjornsing 5y ago

I have a really hard time following this kind of reasoning, partly because the word bias has very different meanings in social science and statistics. E.g. let’s say I have a statistical model that takes among other variables a person’s ethnicity and produces a credit score. If that model is statistically unbiased it would probably produce different credit scores depending on ethnicity. If we “fix” that by making the model insensitive to ethnicity, then it probably becomes statistically biased. Wouldn’t it be better to talk about “fairness” or something?

Spivak 5y ago
You’re viewing “fixing” from the wrong lens. There are infinite bits of information about a person that could potentially be used in a model to calculate a credit score. At all times in every model you’re choosing a subset of one’s facets and can build statistically unbiased models based on the data you make available. But when you talking actually fixing models like this you’re actually forced to correct the final result, not filter the data. Being blind to ethnicity doesn’t work because one’s ethnicity permeates (to different degrees, sure) every part of their lives. All the data is bad, everything is a statistically detectable proxy for ethnicity.
- jhgb 5y ago
  But surely there's some kind of refining your inputs where all the paths converge to one result, much like the refinement of interval partitions in Riemann's integral converges to one value (for a certain class of functions). The bits of information may be infinite but there's some structure in them. I'm not sure that just because the optimal result (the one that uses all information) is something you don't like, the result is bad. Best thing you can say is that you're actually not predicting an existing credit score, but rather synthesizing some other indicator that is not an existing credit score.
BoiledCabbage 5y ago
Agreed, overloading of terms often makes topics much harder to grok. Quick summary, is there are many meanings of the term bias in ML. First let's define which the author is talking about. Some options are: 1. The idea of ignoring relevant features, ala bias-variance trade off. 2. Data bias, where data is trained on inputs / behavior that society would deem biased. Ex. A company frequently discriminates against group X, an ML model is trained on data / decisions this company has made and as a result encodes the same biases. 3. Features are used which are a proxy for other items. Ex. (pulling this out of the air), a repair shop charges a variable rate for repair. It takes in a number of inputs. It's illegal to offer higher prices to women for the same repair as men, but instead an ML model uses as input the customer's first name which ends up being nothing more than a proxy for their gender and introduces that bias. 4. A group of people A have a much smaller portion of the training data size in an ML model than group B. The model makes decisions for offering some good (ex. a loan). Because it's training data is larger on people from group B than group A it always has more confidence in its predictions from group B. The model is set to be risk adverse, so by default it rejects the majority of people from group A even though most of them would have been valid loans. (Where A and B are different across some societal class like religion, race, ethnicity, gender...) The author is talking about the fourth case. While yes there is over/under representation in the input data, the algorithm chosen to build the model encodes its own form of bias. A simple example everyone should be familiar with, using Mean Squared Error vs. Mean Absolute Error (or MBE) produce models that handle rarely seen features or data points very differently. This choice of loss function introduces algorithmic bias into a model. That's essentially what underlies the concept. Yes there is data bias, but what is being argued is that the solution isn't "find perfectly unbiased data" (because in many cases that's impossible), it's understand how the choice of algorithm will produce models that respond to and represent differently those data biases.

malshe 5y ago

I admit that I still don't know a single example of systematic AI bias in the absence of any data bias. I really want to improve my understanding of this topic. This article doesn't help either. It starts off with why we should address this bias without giving an example of it. In fact, the author skirts this issue altogether by first stating: "Here, we start out with a deceptively simple question: how does model design contribute to algorithmic bias?" and then moves into why this is an important issue! If you said "we start" then let's start indeed. Maybe I am not used to this style of writing?

c1ccccc1 5y ago
In the most general sense of the word bias, there can be biases that arise purely as a result of model architecture. For example, I was messing around with GANs about a year ago, and kept on getting checkerboard-like patterns in the generated images. This was a result of how I had set up the convolution layers: It was a bias introduced by model architecture. In terms of the "unfairness based on what group you belong to" kind of bias, that's almost always a result of bad input; you'd have to work really hard to make an inherently racist architecture. So the problem is introduced in the training data. But it could be solved either by getting better training data, or by making clever improvements to your algorithm to work around the limitations of the training data. Often changing the algorithm is easier than improving the data, which I think is the point this article is making. As an example of such a change, let's say we're designing an algorithm to estimate credit score, and we're worried about unfairness based on group membership. Let M be a variable representing what groups a person belongs to: race sex, etc, and let S be the output of the algorithm: that person's credit score. An easy thing to do would be to not allow M as an input to our algorithm. That way we can say that the output "does not depend on group membership". The problem is that we may still have proxies for M as inputs to the algorithm, and if the dataset is biased, that algorithm will still learn to use those proxies to discriminate based on group membership. A better method is surprisingly to keep M as an input during training of the model. Then, at inference time, average over M to get the final output of the model. That still erases the information about what group a particular person belongs to, but it also has the advantage that the model isn't incentivized to learn proxies for M during training, since it has access to M directly.

golemiprague 5y ago

If anything the system that made Obama to be a white man was right, he is half white so why do we assume the black half should define him?

a_square_peg 5y ago

I think talking about 'bias' in ML algorithms is a fundamentally incorrect anthropomorphic attribution. The only thing that ML does is mapping a set of inputs to an output according to a prior statistical distribution.

AndrewKemendo 5y ago

This is an important topic but the framing is not technically robust. Are convolutions "biased" because they are amplifying variance between adjacent pixels? Yes if your definition of biased is statistical, No if your definition of bias is social. I have $300 for someone who can make a convincing technical argument that the inception architecture [1] is socially biased, without actually training it on a dataset. Until then its the data or implementations that are biased. Maybe your trained model is biased - yep that's the data. Maybe you're cherry picking your algorithm to give you the best result on your chosen dataset - that's a biased implementation. More broadly though, the concept that "systems/people will make inferences on new data which align with patterns they see more frequently" is just a reality of intelligence at this point. Humans constantly fail at this. So yes we should be better and bias is real, to solve it we need to get way more technically accurate on what that bias consists of please. [1]https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43022.pdf https://static.googleusercontent.com/media/research.google.c...

cratermoon 5y ago

https://cjc-online.ca/index.php/journal/article/view/2196/2055 https://cjc-online.ca/index.php/journal/article/view/2196/20... "emulsions and digital camera design, the rendering of non-Caucasian skin toneswas highly deficient and required the development of compensatory practicesand technology improvements to redress its shortcomings. Using the emblem-atic “Shirley” norm reference card as a central metaphor reflecting the changingstate of race relations/aesthetics, this essay analytically traces the colour adjust-ment processes in the industries" https://sitn.hms.harvard.edu/flash/2020/racial-discrimination-in-face-recognition-technology/ https://sitn.hms.harvard.edu/flash/2020/racial-discriminatio... "All three algorithms performed the worst on darker-skinned females, with error rates up to 34% higher than for lighter-skinned males"