9 ms·

On the dangers of stochastic parrots: Can language models be too big? (2021)

73 points by Schiphol 4y ago

larve 4y ago
I just finished working my way through this this morning. The literature list is quite interesting and gives a lot of pointers for people who want to walk the line between overblown hype and doomsday scenarios.
xiaolingxiao 4y ago
I believe this is the papers that got timnit and mmitchel fired from google, followed by a protracted media/legal campaign against google and vice versa.
- deleted 4y ago
  [deleted]
  - deleted 4y ago
    [deleted]
- freyr 4y ago
  I suspect it was Timnit’s behavior after the paper didn’t pass internal review that actually got her fired (issuing an ultimatum and threatening to resign unless the company met her demands; telling her coworkers to stop writing documents because their work didn’t matter; insinuations of racist/misogynistic treatment from leadership when she didn’t get her way).
  - visarga 4y ago
    I think it was a well calculated career move, she wanted fame, she got what she wanted. Now she's leading a new research institute > We are an interdisciplinary and globally distributed AI research institute rooted in the belief that AI is not inevitable, its harms are preventable, and when its production and deployment include diverse perspectives and deliberate processes it can be beneficial. Our research reflects our lived experiences and centers our communities. https://www.dair-institute.org/about https://www.dair-institute.org/about
  - tlringer 4y ago
    It was Megan Kacholia, who had put Timnit Gebru and others close to her down for a long time constantly within Google, always talking down and being condescending and rude, failing to respect Timnit in how she confronted Timnit about the paper (which she was ordered to retract by way of not Google's normal paper review process, but by a then-newly-implemented and since retracted secondary "sensitive topics review" process, due to a combination of actual mistakes like the environment numbers, and also Google being too afraid of reputational damage for her discussion of the very real and tangible harms of LLMs). Timnit tried to raise this to Jeff Dean to get help (Jeff was Megan's manager at the time). Jeff completely misunderstood what she was asking for, and instead sent some response about the environment numbers being incorrect (and they are, but that doesn't at all justify the way Timnit was treated). Not beginning to imagine that Jeff could have missed this signal, Timnit responded sarcastically. Jeff didn't pick up on the sarcasm and thought all was good. Timnit then reacted by describing her frustrations with how she was treated in an internal diversity mailing list. She also emailed Megan Kacholia with a number of demands, mostly to be treated reasonably. Appalled at how she and her coauthors were treated, she refused to retract the paper. Megan reacted by taking her note that she would work on a resignation date if demands were not met in combination with Timnit's email completely pedantically and out of context, using them as an excuse to fire her by rushing her out, without allowing her to follow the actual resignation process. She also acted over Timnit's manager's head (Samy Bengio), who was so annoyed he later quit. (Megan cc'd Jeff, but hadn't spoken to Jeff about any of this, and was acting on her own.) Interestingly, Timnit's email to the diversity list was so resonant that several of the changes it asked for in how Google approaches diversity were enacted after her firing. But Megan and Google's official line on all of this chose to obsess over Timnit's rhetorical devices and take them literally instead, using an email to a diversity list about diversity against her. People are still too afraid to talk about diversity on diversity lists, now, because of Google using that email against her. Google reacted by gaslighting Timnit to protect its ass. After Timnit Tweeted that Jeff had fired her (Timnit probably really thought that Jeff and Megan had spoken to each other before Megan had sent that email), Jeff participated in this part in public, on Twitter, with a lot of serious consequences for Timnit and others, without considering power dynamics. (Jeff suffered a lot on Twitter, too, but that doesn't excuse not considering power dynamics in such a consequential way on such a consequential medium.) Timnit and others, including me, were harassed and threatened because of this, by way of third-party harassers. I was not even involved on the paper, just proximal damage. I was afraid for my life honestly. Meg Mitchell, feeling lost, having seen the truth of how Timnit was treated internally, and trusting Jeff to protect her, tried to put together some things for Jeff to get him to see how Timnit was mistreated. She panicked and backed them up on her personal email because she was afraid of retaliation from Google (a reasonable fear---doing any diversity or community work at Google that at all challenges the status quo IME gets you retaliatorily reported to PeopleOps, who then try to get you in trouble and read your private communications and so on). She was transparent about doing this and gave instructions for Google to remove her personal copy if needed. Sundar Pichai fired her and then comms smeared her publicly with outright lies. She was harassed and threatened for this, too, and a number of places refused to hire her because of Google's treatment of her. Really tangible damage emotionally, financially, and reputationally. Out of fear of being sued, Google's comms and legal departments reacted by continuing to censor and gaslight. Sundar was extremely complicit in this, too. Megan was moved out of Research, but not much else happened; she continues to send monthly emails about diversity, as if her continued contact with Research is not actively harmful to diversity. So sick of internet people speculating about this without knowing anything about the situation. Sorry if I broke anyone's trust here. Just can't deal with this incorrect speculation anymore. (I have extremely thorough information about this, but to those directly involved, please feel free to correct me about any details I got wrong, or about important details I omitted.)
- oh_sigh 4y ago
  A small correction: this paper didn't get her fired, her reaction to feedback on this paper got her fired. Note to all: if you give an employer an ultimatum "do X or I resign", don't be surprised if they accept your resignation.
  - deleted 4y ago
    [deleted]
platypii 4y ago
This paper is embarrassingly bad. It's really just an opinion piece where the authors rant about why they don't like large language models. There is no falsifiable hypothesis to be found in it. I think this paper will age very poorly, as LLMs continue to improve and our ability to guide them (such as with RLHF) improves.
- deleted 4y ago
  [deleted]
- jasmer 4y ago
  This is ok. 90% of research is creative thinking, dialogue. One idea creates the next, some are a foil, some are dead ends. As long as there are not outrageous claims being made for 'hard evidence' where there is none, it's fine. Maybe the format isn't fully appropriate but the content is. Most good things come about in a non-linear process which involves provocation along the line somewhere.
  - janalsncm 4y ago
    I expect science to have a hypothesis which can be falsified. Otherwise it’s just opining on a topic. Otherwise we could just call this HN thread “research”.
    - joshuamorton 4y ago
      Position papers are exceedingly common. Common enough that there's a term for them.
    - jasmer 4y ago
      Science is 80% discussion, opining, thinking, reading etc.
      - janalsncm 4y ago
        A lot of areas are made up of those things. Theology, for example. What sets science apart is testable hypotheses.
monkaiju 4y ago
The problems with LLM are numerous but whats really wild to me is that even as they get better at fairly trivial tasks the advertising gets more and more out of hand. These machine dont think, and they dont understand, but people like the CEO of OpenAI allude to them doing just that, obviously so the hype can make them money.
- amelius 4y ago
  Could be the sign of the next AI winter.
- visarga 4y ago
  > These machine dont think, and they dont understand But they do solve many tasks correctly, even problems with multiple steps and new tasks for which they got no specific training. They can combine skills in new ways on demand. Call it what you want.
  - choeger 4y ago
    They don't. Solve tasks, I mean. There's not a single task you can throw at them and rely on the answer. Could they solve tasks? Potentially. But how would we ever know that we could trust them? With humans we not only have millennia of collective experience when it comes to tasks, judging the result, and finding bullshitters. Also, we can retrain a human on the spot and be confident they won't immediately forget something important over that retraining. If we ever let a model produce important decisions, I'd imagine we'd want to certify it beforehand. But that excludes improvements and feedback - the certified software should better not change. If course, a feedback loop could involve recertification, but that means that the certification process itself needs to be cheap. And all that doesn't even take into account the generalized interface: How can we make sure that a model is aware of its narrow purpose and doesn't answer to tasks outside of that purpose? I think all these problems could eventually be overcome, but I don't see much effort put into such a framework to actually make models solve tasks.
    - pedrosorio 4y ago
weeksie 4y ago
This was mostly political guff about environmentalism and bias, but one thing I didn't know was that apparently larger models make it easier to extract training data. > Finally, we note that there are risks associated with the fact that LMs with extremely large numbers of parameters model their training data very closely and can be prompted to output specific information from that training data. For example, [28] demonstrate a methodology for extracting personally identifiable information (PII) from an LM and find that larger LMs are more susceptible to this style of attack than smaller ones. Building training data out of publicly available documents doesn’t fully mitigate this risk: just because the PII was already available in the open on the Internet doesn’t mean there isn’t additional harm in collecting it and providing another avenue to its discovery. This type of risk differs from those noted above because it doesn’t hinge on seeming coherence of synthetic text, but the possibility of a sufficiently motivated user gaining access to training data via the LM. In a similar vein, users might query LMs for ‘dangerous knowledge’ (e.g. tax avoidance advice), knowing that what they were getting was synthetic and therefore not credible but nonetheless representing clues to what is in the training data in order to refine their own search queries Shame they only gave that one graf. I'd like to know more about this. Again, miss me with the political garbage about "dangerous knowledge", the most concerning thing is the PII leakage as far as I can tell.
- deleted 4y ago
  [deleted]
- srvmshr 4y ago
  That is sort of understood facts with even models like Copilot & ChatGPT. With the amount of information we are generally churning, all PII may not get scrubbbed. And these LLMs often could be running on unsanitized data - like a cache of Web on Archive.org, Getty images & the likes. I feel this is a unavoidable consequence of using LLM. We cannot ensure all data is free from any markers. I am not a expert on databases/data engineering so please take it as an informed opinion
  - weeksie 4y ago
    Copilot has a ton of well publicised examples of verbatim code being used, but I didn't realize that it was as trivial as all that to go plumbing for it directly.
sthatipamala 4y ago
This paper is the product of a failed model of AI safety, in which dedicated safety advocates act as a public ombudsman with an adversarial relationship with their employer. It's baffling to me why anyone thought that would be sustainable. Compare this to something like RLHF[0] which has acheived far more for aligning models toward being polite and non-evil. (This is the technique that helps ChatGPT decline to answer questions like "how to make a bomb?") There's still a lot of work to be done and the real progress will be made by researchers who implement systems in collaboration with their colleagues and employers. [0] https://openai.com/blog/instruction-following/ https://openai.com/blog/instruction-following/
- generalizations 4y ago
  > The resulting InstructGPT models are much better at following instructions than GPT-3. They also make up facts less often, and show small decreases in toxic output generation. Our labelers prefer outputs from our 1.3B InstructGPT model over outputs from a 175B GPT-3 model, despite having more than 100x fewer parameters. I wonder if anyone's working on public models of this size. Looking forward to when we can selfhost ChatGPT.
  - lumost 4y ago
    This is going to happen alot over the next few years. One can fine tune GPT-2 medium on an RTX2070. Training GPT-2 medium from scratch can be done for $162 on vast.ai. The newer H100/Trainium/Tensorcore chips will bring the price down even further. I suspect if one wanted to fully replicate ChatGPT from scratch it would take ~1-2 million including label acquisition. You probably only require ~200-500k in compute. The next few years are going to be wild!
    - generalizations 4y ago
      These things have reached the tipping point where they provide significant utility to a significant portion of the computer scientists working on making these things. Could be that the coming iterations of these new tools will make it increasingly easy to write the code for the next iterations of these tools. I wonder if this is the first rumblings of the singularity.
srvmshr 4y ago
I am of the general understanding that this paper became less about the LLMs & more of a insinuating hit piece against Alphabet. At least, some of the controversial nuggets got Gebru (and later M Mitchell) fired. From a technical standpoint, there is little new stuff that I found this paper offered in understanding why LLMs can have unpredictable nature, or what degree of data will get exposed by clever hacks (or if there are systematic ways to go about it). It sounded more like a collection of verifiable anecdotes for easy consumption (which can be a good thing by itself if you want capsule understanding in a non-technical way)
- deleted 4y ago
  [deleted]
  - deleted 4y ago
    [deleted]
- visarga 4y ago
  It was activism masquerading as science. Many researches noted that positives and negatives were not presented in a balanced way. New approaches and efforts were not credited.
  - srvmshr 4y ago
    I haven't kept track but the activism of the trio could be severe sometimes. (Anecdotally, I have faced a bite-sized brunt: When discussion surrounding this paper was going on in Twitter, I had mentioned in my timeline (in a neutral tone) that "dust needed to settle to understand what was going wrong". This was unfortunately picked up & RTed by Gebru & the mob responded by name-calling, threatening DMs accusing me of racism/misogyny etc, and one instance of a call to my employer asking to terminate me - all for that one single tweet. I don't want confrontations - not my forte to deal.)
    - deleted 4y ago
      [deleted]
2bitencryption 4y ago
Pure speculation ahead- The other day on Hacker News, there was that article about how scientists could not tell GPT-generated paper abstracts from real ones. Which makes me think- abstracts for scientific papers are high-effort. The corpus of scientific abstracts would understandably have a low count of "garbage" compared to, say, Twitter posts or random blogs. That's not to say that all scientific abstracts are amazing, just that their goal is to sound intelligent and convincing, while probably 60% of the junk fed into GPT is simply clickbait and junk content padded to fit some publisher's SEO requirements. In other words, ask GPT to generate an abstract, and I would expect it to be quite good. Ask it to generate a 5-paragraph essay about Huckleberry Finn, and I would expect it to be the same quality as the corpus- that is to say, high-school English students. So now that we know these models can learn many one-shot tasks, perhaps some cleanup of the training data is required to advance. Imagine GPT trained ONLY on the library of congress, without the shitty travel blogs or 4chan rants.
- williamcotton 4y ago
  The science is in the reproduction of the methodology, not in the abstract… in fact, a lot of garbage publications with catchy abstracts built on a shaky foundation sounds like one of the issues that plagues contemporary science. That people would stop finding abstracts useful seems a good thing!
- JPLeRouzic 4y ago
  > "The corpus of scientific abstracts would understandably have a low count of "garbage" compared to, say, Twitter posts or random blogs" That's certainly true, but it's not by a so large margin, at least in biology. For example in ALS (a neurodegenerative disease) there is a real breakthrough perhaps every two years, but most papers about ALS (thousands every year) look like they describe something very important. Similarly for ALZforum the most recent "milestone" paper about Alzheimer disease was in 2012, yet in 2022 alone there were more than 16K papers! So the ratio signal on noise is close to zero. https://www.alzforum.org/papers?type%5Bmilestone%5D=milestone&page=1 https://www.alzforum.org/papers?type%5Bmilestone%5D=mileston... https://pubmed.ncbi.nlm.nih.gov/?term=alzheimer%27s+disease&sort=date https://pubmed.ncbi.nlm.nih.gov/?term=alzheimer%27s+disease&...
hardwaregeek 4y ago
I'm still midway through the paper, but I gotta say, I'm a little surprised at the contrast between the contents of the paper and how people have described it on HN. I don't agree with everything that is said, but there are some interesting points made about the data used to train the models, such as it capturing bias (I would certainly question the methodology of using reddit as a large source of training data), and that bias being amplified by filtering algorithms that produce the even larger datasets used for modern LLMs. The section about environmental impact might not hit home for everyone, but it is valid to raise issues around the compute usage involved in training these models. First, because it limits this training to companies who can spend millions of dollars on compute, and second because if we want to scale up models, efficiency is probably a top goal. What really confuses me here is how this paper is somehow outside the realm of valid academic discourse. Yes, it is steeped in activist, social justice language. Yes, it has a different perspective than most CS papers. But is that wrong? Is that enough of a sin to warrant such a response that this paper has received? I'll need to finish the paper to fully judge, but I'm leaning towards no, it is not enough of a sin.
- AbrahamParangi 4y ago
  I guess if one aspires to academic discourse about Truth rather than about Activism or about Power then this will seem somewhat like motivated reasoning.
  - wizzwizz4 4y ago
    If you want academic discourse about truth, become a philosopher. Activism and power are just as real as trees and language models. Discussing them is discussing the world. Your concern about motivated reasoning is a valid one – but motivated reasoning is different to persuasive language. If anything, persuasive language highlights the places you should be looking for motivated reasoning (but it also makes it harder to spot, for some people; I don't recommend using persuasive language in your own papers).
    - AbrahamParangi 4y ago
SinePost 4y ago
This is a great paper to introduce someone to the potential ethical issues that large language models have. A concern I have with these models that the paper only dusts upon is the notion of "truth" when the machine has no method to determine it. Do we admit defeat for what the machine does not have a very high confidence in an answer, or go with a popular opinion/interpretation? These are epistemological issues that I don't see being resolved anytime soon.
Roark66 4y ago
[flagged]
kinston 4y ago
People seem to miss the specific thing that was controversial about this paper: The paper gives inaccurate estimates of the carbon dioxide emissions that result from machine learning. My understanding is that this is the main reason why Google wanted the paper pulled. You can't have a paper by Google authors saying incorrect things about how much CO2 Google is emitting. The authors refused. They argued that they gave a citation for their numbers, and it's not their problem if the paper they cite is inaccurate. Eventually Google did publish a response: https://arxiv.org/abs/2204.05149 https://arxiv.org/abs/2204.05149 But this got totally buried...

73 points by Schiphol 4y ago

I just finished working my way through this this morning. The literature list is quite interesting and gives a lot of pointers for people who want to walk the line between overblown hype and doomsday scenarios.

xiaolingxiao 4y ago

I believe this is the papers that got timnit and mmitchel fired from google, followed by a protracted media/legal campaign against google and vice versa.

deleted 4y ago
[deleted]
- deleted 4y ago
  [deleted]
freyr 4y ago
I suspect it was Timnit’s behavior after the paper didn’t pass internal review that actually got her fired (issuing an ultimatum and threatening to resign unless the company met her demands; telling her coworkers to stop writing documents because their work didn’t matter; insinuations of racist/misogynistic treatment from leadership when she didn’t get her way).
- visarga 4y ago
  I think it was a well calculated career move, she wanted fame, she got what she wanted. Now she's leading a new research institute > We are an interdisciplinary and globally distributed AI research institute rooted in the belief that AI is not inevitable, its harms are preventable, and when its production and deployment include diverse perspectives and deliberate processes it can be beneficial. Our research reflects our lived experiences and centers our communities. https://www.dair-institute.org/about https://www.dair-institute.org/about
- tlringer 4y ago
  It was Megan Kacholia, who had put Timnit Gebru and others close to her down for a long time constantly within Google, always talking down and being condescending and rude, failing to respect Timnit in how she confronted Timnit about the paper (which she was ordered to retract by way of not Google's normal paper review process, but by a then-newly-implemented and since retracted secondary "sensitive topics review" process, due to a combination of actual mistakes like the environment numbers, and also Google being too afraid of reputational damage for her discussion of the very real and tangible harms of LLMs). Timnit tried to raise this to Jeff Dean to get help (Jeff was Megan's manager at the time). Jeff completely misunderstood what she was asking for, and instead sent some response about the environment numbers being incorrect (and they are, but that doesn't at all justify the way Timnit was treated). Not beginning to imagine that Jeff could have missed this signal, Timnit responded sarcastically. Jeff didn't pick up on the sarcasm and thought all was good. Timnit then reacted by describing her frustrations with how she was treated in an internal diversity mailing list. She also emailed Megan Kacholia with a number of demands, mostly to be treated reasonably. Appalled at how she and her coauthors were treated, she refused to retract the paper. Megan reacted by taking her note that she would work on a resignation date if demands were not met in combination with Timnit's email completely pedantically and out of context, using them as an excuse to fire her by rushing her out, without allowing her to follow the actual resignation process. She also acted over Timnit's manager's head (Samy Bengio), who was so annoyed he later quit. (Megan cc'd Jeff, but hadn't spoken to Jeff about any of this, and was acting on her own.) Interestingly, Timnit's email to the diversity list was so resonant that several of the changes it asked for in how Google approaches diversity were enacted after her firing. But Megan and Google's official line on all of this chose to obsess over Timnit's rhetorical devices and take them literally instead, using an email to a diversity list about diversity against her. People are still too afraid to talk about diversity on diversity lists, now, because of Google using that email against her. Google reacted by gaslighting Timnit to protect its ass. After Timnit Tweeted that Jeff had fired her (Timnit probably really thought that Jeff and Megan had spoken to each other before Megan had sent that email), Jeff participated in this part in public, on Twitter, with a lot of serious consequences for Timnit and others, without considering power dynamics. (Jeff suffered a lot on Twitter, too, but that doesn't excuse not considering power dynamics in such a consequential way on such a consequential medium.) Timnit and others, including me, were harassed and threatened because of this, by way of third-party harassers. I was not even involved on the paper, just proximal damage. I was afraid for my life honestly. Meg Mitchell, feeling lost, having seen the truth of how Timnit was treated internally, and trusting Jeff to protect her, tried to put together some things for Jeff to get him to see how Timnit was mistreated. She panicked and backed them up on her personal email because she was afraid of retaliation from Google (a reasonable fear---doing any diversity or community work at Google that at all challenges the status quo IME gets you retaliatorily reported to PeopleOps, who then try to get you in trouble and read your private communications and so on). She was transparent about doing this and gave instructions for Google to remove her personal copy if needed. Sundar Pichai fired her and then comms smeared her publicly with outright lies. She was harassed and threatened for this, too, and a number of places refused to hire her because of Google's treatment of her. Really tangible damage emotionally, financially, and reputationally. Out of fear of being sued, Google's comms and legal departments reacted by continuing to censor and gaslight. Sundar was extremely complicit in this, too. Megan was moved out of Research, but not much else happened; she continues to send monthly emails about diversity, as if her continued contact with Research is not actively harmful to diversity. So sick of internet people speculating about this without knowing anything about the situation. Sorry if I broke anyone's trust here. Just can't deal with this incorrect speculation anymore. (I have extremely thorough information about this, but to those directly involved, please feel free to correct me about any details I got wrong, or about important details I omitted.)
oh_sigh 4y ago
A small correction: this paper didn't get her fired, her reaction to feedback on this paper got her fired. Note to all: if you give an employer an ultimatum "do X or I resign", don't be surprised if they accept your resignation.
- deleted 4y ago
  [deleted]

platypii 4y ago

This paper is embarrassingly bad. It's really just an opinion piece where the authors rant about why they don't like large language models. There is no falsifiable hypothesis to be found in it. I think this paper will age very poorly, as LLMs continue to improve and our ability to guide them (such as with RLHF) improves.

deleted 4y ago
[deleted]
jasmer 4y ago
This is ok. 90% of research is creative thinking, dialogue. One idea creates the next, some are a foil, some are dead ends. As long as there are not outrageous claims being made for 'hard evidence' where there is none, it's fine. Maybe the format isn't fully appropriate but the content is. Most good things come about in a non-linear process which involves provocation along the line somewhere.
- janalsncm 4y ago
  I expect science to have a hypothesis which can be falsified. Otherwise it’s just opining on a topic. Otherwise we could just call this HN thread “research”.
  - joshuamorton 4y ago
    Position papers are exceedingly common. Common enough that there's a term for them.
  - jasmer 4y ago
    Science is 80% discussion, opining, thinking, reading etc.
    - janalsncm 4y ago
      A lot of areas are made up of those things. Theology, for example. What sets science apart is testable hypotheses.

monkaiju 4y ago

The problems with LLM are numerous but whats really wild to me is that even as they get better at fairly trivial tasks the advertising gets more and more out of hand. These machine dont think, and they dont understand, but people like the CEO of OpenAI allude to them doing just that, obviously so the hype can make them money.

amelius 4y ago
Could be the sign of the next AI winter.
visarga 4y ago
> These machine dont think, and they dont understand But they do solve many tasks correctly, even problems with multiple steps and new tasks for which they got no specific training. They can combine skills in new ways on demand. Call it what you want.
- choeger 4y ago
  They don't. Solve tasks, I mean. There's not a single task you can throw at them and rely on the answer. Could they solve tasks? Potentially. But how would we ever know that we could trust them? With humans we not only have millennia of collective experience when it comes to tasks, judging the result, and finding bullshitters. Also, we can retrain a human on the spot and be confident they won't immediately forget something important over that retraining. If we ever let a model produce important decisions, I'd imagine we'd want to certify it beforehand. But that excludes improvements and feedback - the certified software should better not change. If course, a feedback loop could involve recertification, but that means that the certification process itself needs to be cheap. And all that doesn't even take into account the generalized interface: How can we make sure that a model is aware of its narrow purpose and doesn't answer to tasks outside of that purpose? I think all these problems could eventually be overcome, but I don't see much effort put into such a framework to actually make models solve tasks.
  - pedrosorio 4y ago

weeksie 4y ago

This was mostly political guff about environmentalism and bias, but one thing I didn't know was that apparently larger models make it easier to extract training data. > Finally, we note that there are risks associated with the fact that LMs with extremely large numbers of parameters model their training data very closely and can be prompted to output specific information from that training data. For example, [28] demonstrate a methodology for extracting personally identifiable information (PII) from an LM and find that larger LMs are more susceptible to this style of attack than smaller ones. Building training data out of publicly available documents doesn’t fully mitigate this risk: just because the PII was already available in the open on the Internet doesn’t mean there isn’t additional harm in collecting it and providing another avenue to its discovery. This type of risk differs from those noted above because it doesn’t hinge on seeming coherence of synthetic text, but the possibility of a sufficiently motivated user gaining access to training data via the LM. In a similar vein, users might query LMs for ‘dangerous knowledge’ (e.g. tax avoidance advice), knowing that what they were getting was synthetic and therefore not credible but nonetheless representing clues to what is in the training data in order to refine their own search queries Shame they only gave that one graf. I'd like to know more about this. Again, miss me with the political garbage about "dangerous knowledge", the most concerning thing is the PII leakage as far as I can tell.

deleted 4y ago
[deleted]
srvmshr 4y ago
That is sort of understood facts with even models like Copilot & ChatGPT. With the amount of information we are generally churning, all PII may not get scrubbbed. And these LLMs often could be running on unsanitized data - like a cache of Web on Archive.org, Getty images & the likes. I feel this is a unavoidable consequence of using LLM. We cannot ensure all data is free from any markers. I am not a expert on databases/data engineering so please take it as an informed opinion
- weeksie 4y ago
  Copilot has a ton of well publicised examples of verbatim code being used, but I didn't realize that it was as trivial as all that to go plumbing for it directly.

sthatipamala 4y ago

This paper is the product of a failed model of AI safety, in which dedicated safety advocates act as a public ombudsman with an adversarial relationship with their employer. It's baffling to me why anyone thought that would be sustainable. Compare this to something like RLHF[0] which has acheived far more for aligning models toward being polite and non-evil. (This is the technique that helps ChatGPT decline to answer questions like "how to make a bomb?") There's still a lot of work to be done and the real progress will be made by researchers who implement systems in collaboration with their colleagues and employers. [0] https://openai.com/blog/instruction-following/ https://openai.com/blog/instruction-following/

generalizations 4y ago
> The resulting InstructGPT models are much better at following instructions than GPT-3. They also make up facts less often, and show small decreases in toxic output generation. Our labelers prefer outputs from our 1.3B InstructGPT model over outputs from a 175B GPT-3 model, despite having more than 100x fewer parameters. I wonder if anyone's working on public models of this size. Looking forward to when we can selfhost ChatGPT.
- lumost 4y ago
  This is going to happen alot over the next few years. One can fine tune GPT-2 medium on an RTX2070. Training GPT-2 medium from scratch can be done for $162 on vast.ai. The newer H100/Trainium/Tensorcore chips will bring the price down even further. I suspect if one wanted to fully replicate ChatGPT from scratch it would take ~1-2 million including label acquisition. You probably only require ~200-500k in compute. The next few years are going to be wild!
  - generalizations 4y ago
    These things have reached the tipping point where they provide significant utility to a significant portion of the computer scientists working on making these things. Could be that the coming iterations of these new tools will make it increasingly easy to write the code for the next iterations of these tools. I wonder if this is the first rumblings of the singularity.

srvmshr 4y ago

I am of the general understanding that this paper became less about the LLMs & more of a insinuating hit piece against Alphabet. At least, some of the controversial nuggets got Gebru (and later M Mitchell) fired. From a technical standpoint, there is little new stuff that I found this paper offered in understanding why LLMs can have unpredictable nature, or what degree of data will get exposed by clever hacks (or if there are systematic ways to go about it). It sounded more like a collection of verifiable anecdotes for easy consumption (which can be a good thing by itself if you want capsule understanding in a non-technical way)

deleted 4y ago
[deleted]
- deleted 4y ago
  [deleted]
visarga 4y ago
It was activism masquerading as science. Many researches noted that positives and negatives were not presented in a balanced way. New approaches and efforts were not credited.
- srvmshr 4y ago
  I haven't kept track but the activism of the trio could be severe sometimes. (Anecdotally, I have faced a bite-sized brunt: When discussion surrounding this paper was going on in Twitter, I had mentioned in my timeline (in a neutral tone) that "dust needed to settle to understand what was going wrong". This was unfortunately picked up & RTed by Gebru & the mob responded by name-calling, threatening DMs accusing me of racism/misogyny etc, and one instance of a call to my employer asking to terminate me - all for that one single tweet. I don't want confrontations - not my forte to deal.)
  - deleted 4y ago
    [deleted]

2bitencryption 4y ago

Pure speculation ahead- The other day on Hacker News, there was that article about how scientists could not tell GPT-generated paper abstracts from real ones. Which makes me think- abstracts for scientific papers are high-effort. The corpus of scientific abstracts would understandably have a low count of "garbage" compared to, say, Twitter posts or random blogs. That's not to say that all scientific abstracts are amazing, just that their goal is to sound intelligent and convincing, while probably 60% of the junk fed into GPT is simply clickbait and junk content padded to fit some publisher's SEO requirements. In other words, ask GPT to generate an abstract, and I would expect it to be quite good. Ask it to generate a 5-paragraph essay about Huckleberry Finn, and I would expect it to be the same quality as the corpus- that is to say, high-school English students. So now that we know these models can learn many one-shot tasks, perhaps some cleanup of the training data is required to advance. Imagine GPT trained ONLY on the library of congress, without the shitty travel blogs or 4chan rants.

williamcotton 4y ago
The science is in the reproduction of the methodology, not in the abstract… in fact, a lot of garbage publications with catchy abstracts built on a shaky foundation sounds like one of the issues that plagues contemporary science. That people would stop finding abstracts useful seems a good thing!
JPLeRouzic 4y ago
> "The corpus of scientific abstracts would understandably have a low count of "garbage" compared to, say, Twitter posts or random blogs" That's certainly true, but it's not by a so large margin, at least in biology. For example in ALS (a neurodegenerative disease) there is a real breakthrough perhaps every two years, but most papers about ALS (thousands every year) look like they describe something very important. Similarly for ALZforum the most recent "milestone" paper about Alzheimer disease was in 2012, yet in 2022 alone there were more than 16K papers! So the ratio signal on noise is close to zero. https://www.alzforum.org/papers?type%5Bmilestone%5D=milestone&page=1 https://www.alzforum.org/papers?type%5Bmilestone%5D=mileston... https://pubmed.ncbi.nlm.nih.gov/?term=alzheimer%27s+disease&sort=date https://pubmed.ncbi.nlm.nih.gov/?term=alzheimer%27s+disease&...

hardwaregeek 4y ago

I'm still midway through the paper, but I gotta say, I'm a little surprised at the contrast between the contents of the paper and how people have described it on HN. I don't agree with everything that is said, but there are some interesting points made about the data used to train the models, such as it capturing bias (I would certainly question the methodology of using reddit as a large source of training data), and that bias being amplified by filtering algorithms that produce the even larger datasets used for modern LLMs. The section about environmental impact might not hit home for everyone, but it is valid to raise issues around the compute usage involved in training these models. First, because it limits this training to companies who can spend millions of dollars on compute, and second because if we want to scale up models, efficiency is probably a top goal. What really confuses me here is how this paper is somehow outside the realm of valid academic discourse. Yes, it is steeped in activist, social justice language. Yes, it has a different perspective than most CS papers. But is that wrong? Is that enough of a sin to warrant such a response that this paper has received? I'll need to finish the paper to fully judge, but I'm leaning towards no, it is not enough of a sin.

AbrahamParangi 4y ago
I guess if one aspires to academic discourse about Truth rather than about Activism or about Power then this will seem somewhat like motivated reasoning.
- wizzwizz4 4y ago
  If you want academic discourse about truth, become a philosopher. Activism and power are just as real as trees and language models. Discussing them is discussing the world. Your concern about motivated reasoning is a valid one – but motivated reasoning is different to persuasive language. If anything, persuasive language highlights the places you should be looking for motivated reasoning (but it also makes it harder to spot, for some people; I don't recommend using persuasive language in your own papers).
  - AbrahamParangi 4y ago

SinePost 4y ago

This is a great paper to introduce someone to the potential ethical issues that large language models have. A concern I have with these models that the paper only dusts upon is the notion of "truth" when the machine has no method to determine it. Do we admit defeat for what the machine does not have a very high confidence in an answer, or go with a popular opinion/interpretation? These are epistemological issues that I don't see being resolved anytime soon.

Roark66 4y ago

[flagged]

kinston 4y ago

People seem to miss the specific thing that was controversial about this paper: The paper gives inaccurate estimates of the carbon dioxide emissions that result from machine learning. My understanding is that this is the main reason why Google wanted the paper pulled. You can't have a paper by Google authors saying incorrect things about how much CO2 Google is emitting. The authors refused. They argued that they gave a citation for their numbers, and it's not their problem if the paper they cite is inaccurate. Eventually Google did publish a response: https://arxiv.org/abs/2204.05149 https://arxiv.org/abs/2204.05149 But this got totally buried...