23 ms·

Can anyone provide a simple explanation or example that illustrates why Yann was wrong? I’m highly predisposed to take his side in this, but I wonder if I could

• djohnston 6y ago
  i think its mostly miscomm.. he wasnt wrong in the narrow context of this particular model, what he said was almost certainly true. the concern is that, if interpreted as a general solution to the problem of racial bias in ML, its incomplete. after that it kind of devolves and everyone is talking past one another and flaming, its twitter after all
  • eric_b 6y ago
    To me it seemed like Timnit Gebru was spoiling for any reason to fight, and decided to make a big deal out of a factually correct and hard-to-misinterpret statement. I mean, he was clearly referring to the specific model.
    • whymauri 6y ago
      I'm in the group which was upset over the insinuation that ML Researchers should not care about fairness or ethics as much as ML Engineers. The distinction between researcher and engineer should not be grounds to care or not care about safe and ethical model building. This is especially true when the difference between either role is highly arbitrary and varies by organization or field. For the sake of ethical R&D, it's counter-productive to build a hierarchy of investment into the problem. Admittedly, the responsibility of end-results can differ, but the consensus that this ethical work is important should ideally be universal. That said, there should not be some ivory tower where you wash your hands of the ethical problems of your field.
      • eric_b 6y ago
        I don't understand what Timnit and these other people are really even advocating. She says "diverse datasets are not enough". Structural problems can't be ignored etc. Ok, so what is the solution then? Does she have a concrete set of steps or goals to address the problems she sees? Is there a list somewhere of things that would appease her, and in her mind make ML fair? Honestly interested.
        • whymauri 6y ago
          I can't speak for Timnit, but for me it would have sufficed to see Yann start with "I care about ethics and so should researchers and engineers." Instead it took so long to get there, so many "buts" and so many implications that researchers shouldn't be as invested, that I was disappointed. As for Timnit and LeCun: a user above notes that the epistemological frameworks they're using to analyze this problem are not aligned. I found that comment pretty eye-opening, honestly.
          • free_rms 6y ago
            Why does everything need 50 disclaimers before anyone can say anything. Shouldn't "I care about ethics" just be assumed? How many people do you know who would say "I don't care about ethics"?
            • whymauri 6y ago
              My experience in ML and computer science is that it cannot be assumed. Consider companies like Palantir and their Gotham system. Consider the facial recognition systems deployed in Detroit, which fail 96% of thee time. It especially cannot be assumed when you attempt to offload that responsibility to other people. Computer science programs around the country promise their young, bright-eyed undergrads that they'll change the world. Very few of them teach them the ethics they'll need to do that in a thoughtful way. The assumption that science and technology is inherently ethical has unfortunately led to dangerous ideas over the past 150 years. Some, like geographic determinism and eugenics, have directly led to the suffering of millions of people. I hope we tread carefully and take action when we see harmful models. [0] [0] https://twitter.com/SpringerNature/status/1275477365196566528 https://twitter.com/SpringerNature/status/127547736519656652...
      • unishark 6y ago
        > This is especially true when the difference between either role is highly arbitrary and varies by organization or field. I don't think this criticism is fair. Presumably if someone with the title "researcher" has a hand in actually doing what LeCun consider to be the engineer's role, LeCun would say to treat them as an engineer for the purposes of his argument.
        • whymauri 6y ago
          The distinction is fundamentally pointless because the lines are blurred and highly subjective. I'll repeat it again: >For the sake of ethical R&D, it's counter-productive to build a hierarchy of investment into the problem. Admittedly, the responsibility for end-results can differ, but the consensus that this ethical work is important should ideally be universal. That said, there should not be some ivory tower where you wash your hands of the ethical problems of your field. Suppose you're correct, should we define this threshold based on just LeCun's heuristic for defining it? Or would it likely be better to have a consensus that your role doesn't matter in acknowledging the important of fairness and ethics? Would you prefer the world's most prolific researchers being mindful of these issues, even subconsciously? Or to care less, perhaps very little, because it can be deferred to engineers? Because I would like for the field to unite against building harmful systems and to acknowledge the importance of this work throughout the academic hierarchy.
          • unishark 6y ago
            The concern in your quote is separate from the question of whether we can differentiate between engineers and researches. As for the ethical problems. Are you familiar with machine learning methods? It really is all about the data; that's not a dismissal, it's a fact about current technology. There are other kinds of A.I. which are not based entirely on raw data like this. Machine Learning has been simplistically described as "curve fitting", which I think isn't a bad description. So here you are arguing that the mathematician researching good ways to fit a smooth curve to a series of points in really high dimensions needs to somehow take into account what those points might represent in someone's use of the technology. It seems pretty unreasonable to me to require that they put ethical constraints on it.
    • cgearhart 6y ago
      The core distinction is understanding that bias in ML is not _just_ bias in the data. It may be true that we can reduce bias in this model by reducing bias in the training data, but there is a deeper, more fundamental problem of bias that will not be solved just by changing the training data. Marginalizing the discussion by pointing out that this case would benefit from less biased data is unproductive.
      • eric_b 6y ago
        I asked another commenter as well, but what are the proposed solutions then? People are obviously upset about ML and bias, is there a place I can get a summary of actionable next steps to lessen bias in ML?
        • cgearhart 6y ago
          There’s not one neat trick to make it go away. There have been a number of fairness and bias workshops and forums in recent ML conferences. There’s also a growing podcast and book collection on the topic. Timnit Gebru (mentioned in the OP article) has published and participated in a bunch, maybe start there.
          • Nasrudith 6y ago
            To be frank phrasing it like that makes it sound like Gebru has an ulterior motive in trying to cash in books/speakers fees/consulting. Even if sincere good faith and he has real expertise is assumed that approach kind of raises several "huckster alert" red flags.
      • AlexCoventry 6y ago
        What are some examples of racial bias in ML models which cannot be solved by just changing the training data?
      • unishark 6y ago
        > Marginalizing the discussion by pointing out that this case would benefit from less biased data is unproductive. Except for the part where it provides an actual workable solution to the problem at hand. To me this is an argument between completely different mindsets, one that restricts itself to provable facts and one which restricts itself to political agendas. I don't see how the latter can also work in facts. Or belongs in a technical research discussion at all frankly. You want to make laws that force companies to produce identical/equivalent outcomes for every race somehow? Just go lobby for it. Perhaps it's a good idea. You aren't going to reprogram mathematicians to think in political terms instead of mathematical terms.
    • curiousgal 6y ago
      Just browsing her Twitter made me cringe and I am a female PoC. Her response to every argument is "Ah yes the problem with the world is a PoC doing this and that" when the original argument hasn't even suggested that. Recently seems in support of a black scholar who cried racism because her non peer-reviewed work that was only posted on arxiv wasn't cited in a lecture on GANs ... Voicing these opinions would probably label me as racist in their book ironically.
      • dmix 6y ago
        > Voicing these opinions would probably label me as racist in their book ironically. Don't worry, Twitter isn't real life. People have become experts at shouting down people who disagree with them on that platform, they aren't seeking proper arguments and make disingenuous attempts to present them as honest debates. But fortunately what's popular on Twitter doesn't translate to the average population. Plenty of completely fringe ideas get 50-100k likes/retweets. It mostly just represents the voices of various super-niches living in bubbles.
        • blululu 6y ago
          >>Don't worry, Twitter isn't real life. For the most part yes, but President Trump's behavior on Twitter can have serious consequences in the real world. I think that the behavioral norms of social media are penetrating deeper and deeper into culture.
          • dmix 6y ago
            You seem to be taking "not real life" too literally. I'm not saying "nothing on Twitter affects real life". The point is that it's often a poor representation of real life. Small but highly vocal groups can have a seemingly loud and powerful voice. Yet the results of polls and other public signals (even election outcomes) are frequent reminders that what is gospel on Twitter is often detached from 'real life'. Using a president of a major country is a poor example in this context. But if anything Trump being one of the first major Twitter users strongly reinforces my point. Prior to election he tweeted plenty of things most mainstream US republicans wouldn't touch with a 10 foot poll. Let alone what an average American would say IRL (even right leaning ones). Not to mention Twitter is a global platform so conversation around local politics can be heavily skewed by people not even in the country. But otherwise I agree, it is infesting real life far more frequently these days. And it is worrying. Despite everything I said above, big corporations, the media, politicians, etc can't seem to make this distinction and take what is popular there as a direct reflection of the general public. And it creates a negative reinforcing spiral.
    • ameen 6y ago
      As a POC, it is exhausting to have folks like Timnit spin everything into a problem devoid of a solution. I hope we take a scientific approach to problems and solve it rather than driving away folks who can help in solving the problem.
      • thegayngler 6y ago
        As a POC, it bothers me that someone from the "corporate(or educated) class" of POCs always coming in to criticize those POC who have a legitimate concern about some injustice they are seeing. In this case Timmits concern is entirely valid and does not have a solution currently other than do not use ML for some applications. We benefit from Timmit and others voicing these valid concerns. Engineers always want to base everything on the data to take themselves out of what is being asked of them. It is not always possible to reduce problems to "the data". POC in particular should not be quiet when it comes to some of the issues around the questionable use of ML in relation to race and issues that are ultimately surrounding race.
      • newen 6y ago
        It makes sense once you realize she is only using this to get an executive position in some institute for removing racial biases in AI and be set for life.
  • lsh123 6y ago
    Could you please explain why balanced training data is not enough?
    • dlp211 6y ago
      What exactly is balanced training data, who decides it is balanced?
      • claudeganon 6y ago
        Yes, the problem in all these discussions is that unless you engage with them on broader epistemological grounds, you’re just occluding the a priori biases. Engineering, as a discipline, has an unfortunate history of not wanting to engage with the social and political conditions under which its work occurs, but that becomes entirely untenable for how a lot of ML is put to use (if you’re trying to be honest anyway).
        • jcims 6y ago
          >Engineering, as a discipline, has an unfortunate history of not wanting to engage with the social and political conditions under which its work occur Are there any industries that actually do this well?
      • lsh123 6y ago
        Right, probably it’s better to look at the model and call it “fair model” as long as the model output error on a given class of inputs is inversely proportional to the percent of this class in training data (eg more training data of the same class means less error and less training data means more error). I can’t recall a single ML model which was not fair though I mostly deal with texts, not images.
    • baddox 6y ago
      What actual task are you performing that requires you to generate a high-resolution photo of a face given a low-resolution photo of a face? If it’s just for fun, then sure, bias may not matter by definition. But if it’s used to make decisions that have real consequences for people’s lives, then it sounds like a really bad idea no matter what the training set is.
• baddox 6y ago
  My takeaway is that people disagree that the problem is simply the choice of dataset, but rather, they think that the problem is using ML at all to perform certain tasks (including, perhaps, making important decisions based on a person’s appearance).
  • darawk 6y ago
    I agree that some people appear to believe this. But what is the justification for it? I don't really get how it cashes out in concrete harm, assuming you de-bias the datasets. To be clear, I do get arguments that ML might increase the efficiency of policing, and there is some inherent tension between the efficiency of the state and freedom, to the extent that imperfect enforcement is a tacit feature of our legal system. But, this is not an inherently racial point.
    • whymauri 6y ago
      Using appearance and superficial features to predict personality or aptitude is phrenology.
      • darawk 6y ago
        Phrenology was bad because it was scientifically invalid. I don't see any problem with predicting traits from appearances, at least, not intrinsically if the data is accurate.
        • whymauri 6y ago
          There will be no accurate data for this because traits are subjective. Unlike other subjective predictive modeling, like sentiment analysis, I cannot feasibly think of a use-case for this that is not deeply disturbing. I really don't want my field to regress towards 19th-century pseudoscience when we can empower other scientific advancements that will yield immediate benefits.
          • darawk 6y ago
• TAForObvReasons 6y ago
  Yann was "technically correct" in that the nature of the results are highly driven by the nature of the dataset you use. However, that framing of the discussion can easily be interpreted as absolving the ML community from the ethical discussions around the nature of the datasets. Is it acceptable for the community to hand-wave these issues because they are "dataset problems"? For what it's worth, this is a much larger debate that is happening in many fields. For example, a bunch of decisions around car safety were based on crash tests. Those crash tests are based on dummies that were designed around male body forms and thus don't really test how women would handle the crashes. To quote the author of Invisible Women: > As a result, if they are involved in a car crash, women are more likely to be injured – 47 per cent more likely to be seriously injured and 17 per cent more likely to die. This general questioning about the implications of bias in the datasets is happening in many fields
  • laretluval 6y ago
    > As a result, How was causation established?
• andrewflnr 6y ago
  I think Yann and his critics are focusing on different questions. IIUC Yann is looking at the narrow engineering perspective of how to build an fair ML system. Gebru et al seem to be talking about a much more abstract, societal trend-oriented, "let's think about if we should not just if we can" type of question, though it's still pretty fuzzy to me. Anyway, the epistemological standards of those questions are basically incompatible (incommensurable?), so discussion is impossible unless they agree on which one they're trying to answer. I'm starting to notice this kind of thing a lot.
  • bhouston 6y ago
    If that is the case they should notice they are talking past each other and reset the conversation... But I guess that is very hard/impossible once it gets personal. Thus I think the fact that it became personal is the main problem...
    • dgellow 6y ago
      Twitter doesn’t help finding out the misunderstanding and do a reset, you have arguments/counter-arguments spread between thousands of messages with no overview of what is being said, what are the core ideas being discussed.
  • mlyle 6y ago
    I think there's two different issues here. One, if datasets are biased --- if you build your system to only work on white males --- then it may have suboptimal results for other groups. This is a common problem: you use your company's faces, or college students enrolling in data gathering exercises, etc, who are not representative of the population at large. We can fix this by being careful about dataset bias. But the second issue hits right at the heart of a major societal problem/debate. When we use AI to make decisions about people, will the system become racist--- even with representative datasets? If you train something to predict, say, odds to default on a loan-- will it figure out things that correlate to race and be making mostly racial decisions? Different races do have different default rates, but we've decided as a society that it is unfair to use race to determine an individual's probability of default. But if we choose things that are correlates of both default rates and race, when is that fair and when is it just veiled racism (redlining)? What things are measuring a causal relationship and what things are just racism in disguise? This second problem is much worse with ML, because we have the ability to accept a whole bunch more things into our models and explaining the rationale of why decision are made is much harder. And of course, the first problem-- both bias in datasets during use and biases during research and development -- makes the second problem worse. It can't even necessarily be addressed by broadening the dataset and retraining: if, in this case, you do your research and training with just white faces, and report positive results, it may not generalize to work as well for everyone with a broader dataset.
• lowbloodsugar 6y ago
  Guns don't kill people, its the people who keep loading them with bullets and shooting them at people. ML isn't biased, its the people who keep loading them with biased data, and then running them against photos of people.
  • laretluval 6y ago
    This was the kind of reasoning that led gun makers to develop safety mechanisms. It isn't clear what a similar safety mechanism would look like for an ML algorithm. Modern ML algorithms are nothing more than generic pattern extractors. What would a safety mechanism look like?
    • karpierz 6y ago
      And led to government regulation of distributing guns. Have government grants for ML research be contingent on training on diverse datasets. Fund research into more diverse benchmarks and push them as the goal to beat.
  • int_19h 6y ago
    It is unethical (and, in many places, a crime) to sell a firearm to somebody whom you know, or reasonably suspect, intends to use it for criminal purposes.
  • mempko 6y ago
    Yes, but certainly guns are designed to kill people. Just like an AI.system might be designed to catch "criminals"
• IfOnlyYouKnew 6y ago
  Part of the problem is that "it's just the dataset" is being used as an excuse (witness the "just"). It doesn't really matter to possible victims of, say, the use of AI in law enforcement where exactly the problem originates. I doubt that anybody was accusing the creators of that upsampling model to have intentionally tweaked the model to default to white people. In that sense, YLC is attacking a straw man. Given the publicity such problems have gained in the community, one would expect publishers of any model to verify it doesn't fail with the most obvious examples. Not doing so is negligent at best. If we lack datasets to train AI that doesn't spectacularly fail any test for racial biases, we lack datasets for anything that could be considered fit for use. At that point it doesn't matter if the model is flawed, or if it's "just" the available data. Such models shouldn't be published, and we should instead invest in either better data, or come up with better methods of training. (And, as a minor point, his idea that Senegal is representative of "Africa" as a whole is also... let's say "unfortunate")
  • tinyhouse 6y ago
    > (And, as a minor point, his idea that Senegal is representative of "Africa" as a whole is also... let's say "unfortunate") Senegal was just an example he gave in a tweet. No need to be so petty on every word.
    • guerrilla 6y ago
      It's not the words, it's the idea that Senegal is representative of darker people and could produce the correct results. It's an indication that he still doesn't understand.
      • madaxe_again 6y ago
        Isn’t “Africa” just as bad, if not worse? Not all black people are Africans or have African heritage. Far from it, in fact. Also: darker? Darker than what? Are you taking “white” as your baseline? I mean, if you’re going to throw stones about how “you don’t understand”, you could try having a rational point.
• deleted 6y ago
  [deleted]
• karpierz 6y ago
  Saying that the issue with the model was biased data is like saying the issue with obesity in America is that people eat too much. It's ignoring the broader context of the issue, in favour of a simple answer of "well if people ate less McDonald's, there wouldn't be an obesity crisis."
  • jmeister 6y ago
    No. It’s easy to fix bias in the data for a model developer, but one can’t control other people’s dietary choices.
    • atq2119 6y ago
      If it's so easy to fix the bias, then why isn't it fixed, always? That should be the expectation. More likely, the reason that bias isn't routinely fixed is that it isn't easy, and these kinds of biases do make it into production systems. Which makes it a net positive in my view that the occasional shitstorm reminds society of this fact. Can that be unpleasant for AI researchers? Sure... but if it bothers then, then perhaps they could focus their research on trying to fix the problem? Physicists had and have unpleasant conversations about their moral responsibility for nuclear weapons. Other fields of research should take their moral responsibilities seriously as well (not just AI research, by the way).
    • kennywinker 6y ago
      But can you control other model developers dataset choices? ML tools are developed, and filter down into the general developer population where they are used without fully comprehending the biases they contain or can contain if used incorrectly. I'm pretty partial to the idea that ML is a systemic bias footgun. VERY hard to not use incorrectly, and with potentially HUGE social repercussions. E.g. the youtube funnel towards extremism that was well documented a few years ago - visitors start on some innocuous video, and get recommended more and more extreme things, funnelling traffic towards the alt-right in an unhealthy way.
      • TrackerFF 6y ago
• nmfisher 6y ago
  TLDR: LeCun - ML is biased when datasets are biased. But unlike deploying to real world problems, I don't see any ethical obligation to use "unbiased" datasets for pure research or tinkering with models. Gebru - This is wrong, this is hurtful to marginalized people and you need to listen to them. Watch my tutorial for an explanation. Headlines from tutorial (that Gebru didn't even link herself): The CV community is largely homogenous and has very few black people. Here's a bunch of startups that purport to use CV to predict IQ, hiring, etc. Marginalized people don't work on these platforms and there's no legal vetting for fairness before these platforms are deployed. Facial analysis has the highest rate of inaccuracy (gender classification) on fair-skinned men (?) and dark-skinned women. Datasets are usually white/male. Most object detection models are biased towards Western concepts (e.g. marriage). Crash test dummies are representative of males, so women and children are overrepresented in car crash injuries. Nearest neighbor image search is a unfair because of automation bias and surveillance bias. China is using face detection for surveilling ethnic minorities. Amazon's face recognition sold to police had the same biases (greater difficulty distinguishing between black women). Now, I largely agree with what Gebru said in the tutorial. So does LeCun, who explicitly agreed a number of times that biased datasets/models should never be used for deployed solutions. But it's a huge leap in logic to then demand that every research dataset be "unbiased". It's like criticizing someone for using exclusively male Lego figures to storyboard a movie shoot, or if I attacked a Chinese researcher because they only used Chinese faces to train a generative model, and none the outputs looked anything like me. That being said, I'm open to being convinced if she had made any effort to show/prove that "use of biased datasets in research" is correlated with "biased outcomes in real world production deployments". But she didn't, which is why her criticism of LeCun smacks of cheap point-scoring rather than genuine debate (a criticism I made of Twitter generally the last time this topic came up).
  • karpierz 6y ago
    Do you believe that industry uses pre-trained model that researchers release? Do you believe that industry uses pre-made datasets that researchers promote in their work? Would yes to the above two question be sufficient to show "use of biased datasets in research" is correlated with "biased outcomes in real world production deployments"?
• cosmic_ape 6y ago
  I haven't followed the original debate, so not sure if this was a part of the argument against LeCun, but it could be: The issue is LeCun makes an argument about fundamental research, but is not exactly a fundamental researcher, and does not necessarily represent fundamental research. As an analogy, if you are a researcher of general chemistry, persumably there is no issue. However, if your research is specifically about chemistry for improving bullets, and you produce working prototypes, then some might say you should be subject to some regulation as a part of the arms industry. I'm not saying this is the right thing, just that such a point could be made. LeCun is arguably much more the second kind of researcher than the first. The research he represents is "better ways to recognize faces", not "statistical properties of natural images". To take another example, there is an enormous amount of regulation in, say, medical research. And in this case there are good reasons for that. Gebru could be possibly arguing for something similar in say face recognition.
• Kapura 6y ago
  This is fundamentally a forest/trees type of situation. LeCun sees the issue with this one model, says, "If this had been trained on a different sample it wouldn't have this issue," and stops his train of reasoning there. The problem he is seeing is the mis-trained model, and nothing more. But the problem is larger than this single model, because this issue (or similar ones) are pervasive in the fields in which AI are being employed. If a neural net is helping a court hand down sentences, it is going to be trained on historical sentencing data, and will in turn reflect the biases present in that data. If you are still only seeing the one tree, you say "well we must correct for the historical bias," and absolve yourself of thinking of the larger problem. That forest problem is that we will always be feeding these algorithms biased inputs, unless we do the work to understand social biases and attempt to rectify them.
  • visarga 6y ago
    The court example was good. Trying to make a list of possible domains where social bias is a factor: - court risk assessments - loan risk assessment - job application pre-screening - law enforcement face recognition - medical scan interpretation ... So, in essence: justice, banking, jobs, policing and health care. Anything else? Seems like social bias can only affect a small proportion of the ML application domains.
    • trobertson 6y ago
      The classic (multi-decade problem repeated again, and again, and again...) is face detection. Face detection models continue to get trained by primarily white developers/researches who continue to use primarily white people as their training set. And then they're shocked when their model performs terribly for everyone who isn't of European decent. This case is so pervasive that it gets taught in basically every university that teaches ML - and yet those students go on to repeat the exact problem they were educated against.
      • visarga 6y ago
        > Face detection models continue to get trained by primarily white developers/researches who continue to use primarily white people as their training set. I guess you didn't watch Asian developments in face recognition closely in the last few years.