8 ms·

This reads to me like Google felt that the paper painted some of their other technologies in a poor light, and wanted to insert language that made them look bet

by Imnimo 6y ago

This reads to me like Google felt that the paper painted some of their other technologies in a poor light, and wanted to insert language that made them look better. The way he describes their objections, they strike me as the sort of thing that is routinely addressed in the camera-ready version of papers by adding a few lines to the related work section. Not the sort of thing that a conference reviewer or an internal reviewer would reject a paper over. Previously, we only had one side of this story. But if this is Dean's best spin on Google's side of the story, I'm very tempted to conclude they're in the wrong here. Obviously I don't have all the information, but the information I do have feels consistent with the idea that someone important at Google didn't like Gebru's paper for corporate-political (meaning making Google look good, as opposed to political-political) reasons, they tried to get Gebru to play ball, she refused, and now they have to back-project a justification in the name of "scientific integrity". Unfortunately, I think this is a story where most people's opinions about who's in the right will be more informed by their previous opinions about Gebru and Dean than the narrower question of what happened with this particular paper. I'm probably even guilty of that to some extent myself, given that I'm a fan of some of Gebru's previous work.

asciident 6y ago
I've seen these review comments before and I don't know if we can say from our position whether it's corrections that can be made for camera ready or not. If a paper does not analyze the improvements from recent work, and just older work that has been surpassed (deemed inefficient by Dean), the new reanalysis might not be as favorable to the results as the paper proposes which means the paper is moot. Or the second point is about bias in language models, but it sounds like these issues are mitigated in recent research, which means people are already aware and have solved a bulk of the issue being described in the paper. But certainly it's possible that the paper's contributions stands strong even after accounting for the recent work that Dean mentions. In that case, it could be corrected for camera ready. But my point is that we can't tell right now without seeing the paper, and the relevant research that was omitted.
- whimsicalism 6y ago
  But this review process had never been used for introducing comments of this sort to a paper ever before. Plus, there was no suggestion of change, just retraction.
  - lokar 6y ago
    > But this review process had never been used for introducing comments of this sort to a paper ever before How do you know that?
    - pseudalopex 6y ago
      "My submissions were always checked for disclosure of sensitive material, never for the quality of the literature review."[1] "In all of my time at Google AI, I never heard of pubapproval being used for peer review or to critique the scientific rigor of the work. It was never used as a journal, it was an afterthought that folks on my team would usually clear only hours before important deadlines. We like to leave peer review to the conferences/journals' existing process to weed out bad papers; why duplicate that work internally?"[2] [1] https://twitter.com/le_roux_nicolas/status/1334601960972906496 https://twitter.com/le_roux_nicolas/status/13346019609729064... [2] https://news.ycombinator.com/item?id=25307456 https://news.ycombinator.com/item?id=25307456
      - lokar 6y ago
        Huh. When I was there I certainly saw review for quality / novel ideas.
skissane 6y ago
> they strike me as the sort of thing that is routinely addressed in the camera-ready version of papers by adding a few lines to the related work section What I don't understand is why in the discussion nobody proposed amending the paper rather than withdrawing it. If Dean's issue was it didn't cite papers X,Y and Z, rather than demand a withdrawal, why didn't he just demand "I want you to amend the paper to add cites to X,Y,Z". And then, if Gebru and her coauthors were willing to add those cites, that would resolve it. Indeed, from what I understand, "you should add a cite to X" is common peer review feedback, and a lot of papers get citations added due to requests from peer reviewers. So this isn't hugely different from that scenario. It would seem that withdrawal over this issue would only make sense if Gebru and her coauthors refused to amend the paper to add the requested citations, but I haven't heard anything saying that she did refuse to do so. It isn't clear if the alternative solution of amending rather than withdrawing was ever brought up in the discussion by either party. Not that I'm a researcher or anything, but if I was, and a superior told me "we need you to withdraw your paper because it doesn't cite X,Y,Z", my immediate response would be "How about I add the citations you are requesting and resubmit it with those additions?"
- pseudalopex 6y ago
  Gebru complained about not being told what the criticisms were. That implies she was willing to consider feedback.
  - tfehring 6y ago
    More specifically, she stated in her original email [0]: >Have you ever heard of someone getting “feedback” on a paper through a privileged and confidential document to HR? Does that sound like a standard procedure to you or does it just happen to people like me who are constantly dehumanized? So she did get feedback of some kind, though my reading is that it was probably vague legalese that she reasonably deemed insufficient. [0] https://www.platformer.news/p/the-withering-email-that-got-an-ethical https://www.platformer.news/p/the-withering-email-that-got-a...
    - 6y ago
jonahbenton 6y ago
Actually, the opposite. Speaking personally, I have been a fan boy of Dean's work on a number of fronts for a decade. I had awareness of her work but more superficially, and saw it as largely pissing into the wind. Reading hers more closely now, her papers and her communication, she has my complete respect. I better understand the way in which she was scaling this incredibly difficult mountain. Deeply, deeply impressive. And Jeff has lost mine. Even if his comments are not just bad faith, his blindness borders on incompetence for the role he has. Yet another clever, blind white man. I hope a position in the Biden administration can be found for her, and her vision can have a larger societal impact.

by Imnimo 6y ago

asciident 6y ago

I've seen these review comments before and I don't know if we can say from our position whether it's corrections that can be made for camera ready or not. If a paper does not analyze the improvements from recent work, and just older work that has been surpassed (deemed inefficient by Dean), the new reanalysis might not be as favorable to the results as the paper proposes which means the paper is moot. Or the second point is about bias in language models, but it sounds like these issues are mitigated in recent research, which means people are already aware and have solved a bulk of the issue being described in the paper. But certainly it's possible that the paper's contributions stands strong even after accounting for the recent work that Dean mentions. In that case, it could be corrected for camera ready. But my point is that we can't tell right now without seeing the paper, and the relevant research that was omitted.

whimsicalism 6y ago
But this review process had never been used for introducing comments of this sort to a paper ever before. Plus, there was no suggestion of change, just retraction.
- lokar 6y ago
  > But this review process had never been used for introducing comments of this sort to a paper ever before How do you know that?
  - pseudalopex 6y ago
    "My submissions were always checked for disclosure of sensitive material, never for the quality of the literature review."[1] "In all of my time at Google AI, I never heard of pubapproval being used for peer review or to critique the scientific rigor of the work. It was never used as a journal, it was an afterthought that folks on my team would usually clear only hours before important deadlines. We like to leave peer review to the conferences/journals' existing process to weed out bad papers; why duplicate that work internally?"[2] [1] https://twitter.com/le_roux_nicolas/status/1334601960972906496 https://twitter.com/le_roux_nicolas/status/13346019609729064... [2] https://news.ycombinator.com/item?id=25307456 https://news.ycombinator.com/item?id=25307456
    - lokar 6y ago
      Huh. When I was there I certainly saw review for quality / novel ideas.

skissane 6y ago

> they strike me as the sort of thing that is routinely addressed in the camera-ready version of papers by adding a few lines to the related work section What I don't understand is why in the discussion nobody proposed amending the paper rather than withdrawing it. If Dean's issue was it didn't cite papers X,Y and Z, rather than demand a withdrawal, why didn't he just demand "I want you to amend the paper to add cites to X,Y,Z". And then, if Gebru and her coauthors were willing to add those cites, that would resolve it. Indeed, from what I understand, "you should add a cite to X" is common peer review feedback, and a lot of papers get citations added due to requests from peer reviewers. So this isn't hugely different from that scenario. It would seem that withdrawal over this issue would only make sense if Gebru and her coauthors refused to amend the paper to add the requested citations, but I haven't heard anything saying that she did refuse to do so. It isn't clear if the alternative solution of amending rather than withdrawing was ever brought up in the discussion by either party. Not that I'm a researcher or anything, but if I was, and a superior told me "we need you to withdraw your paper because it doesn't cite X,Y,Z", my immediate response would be "How about I add the citations you are requesting and resubmit it with those additions?"

pseudalopex 6y ago
Gebru complained about not being told what the criticisms were. That implies she was willing to consider feedback.
- tfehring 6y ago
  More specifically, she stated in her original email [0]: >Have you ever heard of someone getting “feedback” on a paper through a privileged and confidential document to HR? Does that sound like a standard procedure to you or does it just happen to people like me who are constantly dehumanized? So she did get feedback of some kind, though my reading is that it was probably vague legalese that she reasonably deemed insufficient. [0] https://www.platformer.news/p/the-withering-email-that-got-an-ethical https://www.platformer.news/p/the-withering-email-that-got-a...
  - 6y ago

jonahbenton 6y ago

Actually, the opposite. Speaking personally, I have been a fan boy of Dean's work on a number of fronts for a decade. I had awareness of her work but more superficially, and saw it as largely pissing into the wind. Reading hers more closely now, her papers and her communication, she has my complete respect. I better understand the way in which she was scaling this incredibly difficult mountain. Deeply, deeply impressive. And Jeff has lost mine. Even if his comments are not just bad faith, his blindness borders on incompetence for the role he has. Yet another clever, blind white man. I hope a position in the Biden administration can be found for her, and her vision can have a larger societal impact.