5 ms·

> Google continues to peddle unsubstantiated snake oil I read your comment, but I'm not following -- or maybe I disagree with it -- I'm not sure yet. "Snake o

by xpe 2y ago

> Google continues to peddle unsubstantiated snake oil I read your comment, but I'm not following -- or maybe I disagree with it -- I'm not sure yet. "Snake oil" is an emotionally loaded term that raises the temperature of the conversation. That usually makes having a conversation harder. From my point of view, AlphaGo, AlphaZero, AlphaFold were significant achievements. Agree? Are you claiming that AlphaChip is not? Are you claiming they are perpetrating some kind of deception or exaggeration? Your numbered points seem like valid criticisms (I haven't evaluated them closely), but even if true, I don't see how they support your "snake oil" claim.

griomnib 2y ago
They have literally been caught faking AI demos, they brought distrust on themselves.
- rajup 2y ago
  Really not sure how you’re conflating product demos which are known to be pie in the sky across the industry (not just Google) with peer reviewed research published in journals. Super basic distinction imho.
  - stackghost 2y ago
    >peer reviewed research published in journals Peer review doesn't mean as much as Elsevier would like you to believe. Plenty of peer-reviewed research is absolute trash.
    - throwaway2037 2y ago
      All of the highest impact papers authored by DeepMind and Google Brain have appeared in Nature, which is the gold standard for peer-reviewed natural science research. What exactly are you trying to claim about Google's peer-reviewed papers?
      - stackghost 2y ago
        Nature is just as susceptible to the perverse incentives at play in the academic publishing market as anyone else, and has had their share of controversies over the years including having to retract papers after they were found to be bogus. In and of itself, "Being published in a peer reviewed journal" does not place the contents of a paper beyond reproach or criticism.
      - gabegobblegoldi 2y ago
        Peer review is not designed to combat fraud.
      - nautilius 2y ago
        From personal experience: in Nature Communications the handling editor and editor in chief absolutely do intervene, in my example to suppress a proper lit review that would have revealed the paper under review as much less innovative than claimed.
11101010001100 2y ago
Their material discovery paper turned out to have negligible significance.
- xpe 2y ago
  If so, does this qualify as “snake oil”? What do you mean? Snake oil requires exaggeration and deception. Fair? If a paper / experiment is done with intellectual honesty, great! If it doesn’t make a big splash, fine.
  - 11101010001100 2y ago
    The paper is more or less a dead end. If there is another name you want to call it, by all means.
    - xpe 2y ago
      /[01]{8,}/: I was hoping to have a conversation. This is why I asked questions. Any responses to them? Looking up the thread, you can see the context. Many of us pushed back against vague claims that AlphaChip was "snake oil". Like good engineers, we split apart the problem into clearer concepts. The "snake oil" proponents did not offer compelling replies, did they? Instead, they retreated to irrelevant points that have no bearing on making sense of the "snake oil" claim. Sometimes technical people forget to bring their "debugging" skills to bear on conversations. There is a metaphorical connection; good debuggers would disambiguate terms, decompose the problem, answer questions, find cruxes, synthesize, find clearer terms, generate alternative explanations, and so on.
      - 11101010001100 2y ago
        MRS is this week, you can go and join the conversations with people at the metal level. Probably even talk to the authors themselves!
  - sanxiyn 2y ago
    I think the paper was probably done honestly, but also very poorly. They claimed synthesis of 36 new materials. When reviewed, for 24/36 "the predicted structure has ordered cations but there is no evidence for order, and a known, disordered version of the compound exists". In fact, with other errors, 36/36 claims were doubtful. This reflects badly for authors and worse for peer review process of Nature. https://x.com/Robert_Palgrave/status/1744383962913394758 https://x.com/Robert_Palgrave/status/1744383962913394758
bsder 2y ago
> From my point of view, AlphaGo, AlphaZero, AlphaFold were significant achievements. These things you mentioned had obvious benchmarks that were easily surpassed by the appropriate "AI". The evidence that they were better wasn't just significant, it was obvious. This leaves the fact that with what appears to be maximal cooking of the books, the only thing AlphaChip seems to be able to beat is human, manual placement and not anything algorithmic--even from many, many generations ago. Trying to pass that off as a significant "advance" in a "scientific publication" borders on scientific fraud and should definitely be called out. The problem here is that I am certain that this is wired to the career trajectories of "Very Important People(tm)" and the fact that it essentially failed miserably is simply not politically allowed. If they want to lie, they can do that in press releases. If they want published in something reputable, they should have to be able to provide proper evidence for replication. And, if they can't do that, well, that's an answer itself, no?
- xpe 2y ago
  > Trying to pass that off as a significant "advance" in a "scientific publication" borders on scientific fraud and should definitely be called out. If true, your stated concerns with the AlphaChip paper -- selective benchmarking and potential overselling of results - reflect poor scientific practice and possible intellectual dishonesty. This does not constitute scientific fraud, which occurs when the underlying method/experiment/rules are faked. If the paper has issues with how it positions and contextualizes its contribution, criticism is warranted, sure. But don't confuse this with "scientific fraud". Some context: for as long as benchmark suites have existed, people rightly comment on which benchmarks should be included and how they should be weighted.
- xpe 2y ago
  > "scientific publication" These air quotes suggests the commenter above doesn't think the paper qualifies a scientific publication. Such a characterization is unfair. When I read the Nature article titled "Addendum: A graph placement methodology for fast chip design" [1], I see writing that more than meets the bar for a scientific publication. For example: > Since publication, we have open-sourced a software repository [21] to fully reproduce the methods described in our paper. External researchers can use this repository to pre-train on a variety of chip blocks and then apply the pre-trained model to new blocks, as was done in our original paper. As part of this addendum, we are also releasing a model checkpoint pre-trained on 20 TPU blocks [22]. For best results, however, we continue to recommend that developers pre-train on their own in-distribution blocks [18], and provide a tutorial on how to perform pre-training with our open-source repository [23]. [1]: https://www.nature.com/articles/s41586-024-08032-5 https://www.nature.com/articles/s41586-024-08032-5 [18]: Yue, S. et al. Scalability and generalization of circuit training for chip floorplanning. In Proc. 2022 International Symposium on Physical Design 65–70 (2022). [21]: Guadarrama, S. et al. Circuit Training: an open-source framework for generating chip floor plans with distributed deep reinforcement learning. GitHub https://github.com/google-research/circuit_training https://github.com/google-research/circuit_training (2021). [23]: Guadarrama, S. et al. Pre-training. GitHub https://github.com/google-research/circuit_training/blob/main/docs/PRETRAINING.md https://github.com/google-research/circuit_training/blob/mai... (2021).
seanhunter 2y ago
Well here’s one exaggeration that was pretty obvious to me straight away as a somewhat disinterested observer. In her status on X Anna Goldie says [1] “ AlphaChip was one of the first RL methods deployed to solve a real-world engineering problem”. This seems very clearly untrue- for example here’s a real-world engineering use of reinforcement learning by google AI themselves from 6 years ago [2] which if you use Anna Goldie’s own timeline is 2 years before alphachip. [1] https://x.com/annadgoldie/status/1858531756506558688 https://x.com/annadgoldie/status/1858531756506558688 [2] https://youtu.be/W4joe3zzglU?si=mFvZq8gEI6LeEQdC https://youtu.be/W4joe3zzglU?si=mFvZq8gEI6LeEQdC
- wholehog 2y ago
  That is definitely a cool project, but I don't see how it contradicts "one of the first RL methods deployed to solve a real-world engineering problem". "One of the first" does not mean literally the first ever.
  - seanhunter 2y ago
    Agreed but if someone at your own company did it two years before you in the context of something that recent it’s stretching credibility to say you were one of the first.
    - wholehog 2y ago
      I mean, I think second is still "one of the first?" And, no offense to this project, but I don't know of it being used in a real industrial setting, whereas AlphaChip was used in TPU.
      - seanhunter 2y ago
        Yes. The sky is also blue? However it's hard to see how being provably 2 years behind the first even in your own company in an incredibly hot area that people are doing tons of work in makes you suddenly second. By that logic I might still be in time to claim the silver for the 100m at the Paris olympics if I pop over there in the next 18 months or so. I can see you created this account just to comment on this thread so I'm sure you have more inside information than I do given that I'm really not connected to this in any way. Enjoy your work at Google Research. I think you guys do cool stuff. It's a shame in my opinion that you choose to damage your credibility by making (and defending) such obviously false claims rather than concentrating on the genuinely innovative work you have done advancing the field.

by xpe 2y ago

griomnib 2y ago

They have literally been caught faking AI demos, they brought distrust on themselves.

rajup 2y ago
Really not sure how you’re conflating product demos which are known to be pie in the sky across the industry (not just Google) with peer reviewed research published in journals. Super basic distinction imho.
- stackghost 2y ago
  >peer reviewed research published in journals Peer review doesn't mean as much as Elsevier would like you to believe. Plenty of peer-reviewed research is absolute trash.
  - throwaway2037 2y ago
    All of the highest impact papers authored by DeepMind and Google Brain have appeared in Nature, which is the gold standard for peer-reviewed natural science research. What exactly are you trying to claim about Google's peer-reviewed papers?
    - stackghost 2y ago
      Nature is just as susceptible to the perverse incentives at play in the academic publishing market as anyone else, and has had their share of controversies over the years including having to retract papers after they were found to be bogus. In and of itself, "Being published in a peer reviewed journal" does not place the contents of a paper beyond reproach or criticism.
    - gabegobblegoldi 2y ago
      Peer review is not designed to combat fraud.
    - nautilius 2y ago
      From personal experience: in Nature Communications the handling editor and editor in chief absolutely do intervene, in my example to suppress a proper lit review that would have revealed the paper under review as much less innovative than claimed.

11101010001100 2y ago

Their material discovery paper turned out to have negligible significance.

xpe 2y ago
If so, does this qualify as “snake oil”? What do you mean? Snake oil requires exaggeration and deception. Fair? If a paper / experiment is done with intellectual honesty, great! If it doesn’t make a big splash, fine.
- 11101010001100 2y ago
  The paper is more or less a dead end. If there is another name you want to call it, by all means.
  - xpe 2y ago
    /[01]{8,}/: I was hoping to have a conversation. This is why I asked questions. Any responses to them? Looking up the thread, you can see the context. Many of us pushed back against vague claims that AlphaChip was "snake oil". Like good engineers, we split apart the problem into clearer concepts. The "snake oil" proponents did not offer compelling replies, did they? Instead, they retreated to irrelevant points that have no bearing on making sense of the "snake oil" claim. Sometimes technical people forget to bring their "debugging" skills to bear on conversations. There is a metaphorical connection; good debuggers would disambiguate terms, decompose the problem, answer questions, find cruxes, synthesize, find clearer terms, generate alternative explanations, and so on.
    - 11101010001100 2y ago
      MRS is this week, you can go and join the conversations with people at the metal level. Probably even talk to the authors themselves!
- sanxiyn 2y ago
  I think the paper was probably done honestly, but also very poorly. They claimed synthesis of 36 new materials. When reviewed, for 24/36 "the predicted structure has ordered cations but there is no evidence for order, and a known, disordered version of the compound exists". In fact, with other errors, 36/36 claims were doubtful. This reflects badly for authors and worse for peer review process of Nature. https://x.com/Robert_Palgrave/status/1744383962913394758 https://x.com/Robert_Palgrave/status/1744383962913394758

bsder 2y ago

> From my point of view, AlphaGo, AlphaZero, AlphaFold were significant achievements. These things you mentioned had obvious benchmarks that were easily surpassed by the appropriate "AI". The evidence that they were better wasn't just significant, it was obvious. This leaves the fact that with what appears to be maximal cooking of the books, the only thing AlphaChip seems to be able to beat is human, manual placement and not anything algorithmic--even from many, many generations ago. Trying to pass that off as a significant "advance" in a "scientific publication" borders on scientific fraud and should definitely be called out. The problem here is that I am certain that this is wired to the career trajectories of "Very Important People(tm)" and the fact that it essentially failed miserably is simply not politically allowed. If they want to lie, they can do that in press releases. If they want published in something reputable, they should have to be able to provide proper evidence for replication. And, if they can't do that, well, that's an answer itself, no?

xpe 2y ago
> Trying to pass that off as a significant "advance" in a "scientific publication" borders on scientific fraud and should definitely be called out. If true, your stated concerns with the AlphaChip paper -- selective benchmarking and potential overselling of results - reflect poor scientific practice and possible intellectual dishonesty. This does not constitute scientific fraud, which occurs when the underlying method/experiment/rules are faked. If the paper has issues with how it positions and contextualizes its contribution, criticism is warranted, sure. But don't confuse this with "scientific fraud". Some context: for as long as benchmark suites have existed, people rightly comment on which benchmarks should be included and how they should be weighted.
xpe 2y ago
> "scientific publication" These air quotes suggests the commenter above doesn't think the paper qualifies a scientific publication. Such a characterization is unfair. When I read the Nature article titled "Addendum: A graph placement methodology for fast chip design" [1], I see writing that more than meets the bar for a scientific publication. For example: > Since publication, we have open-sourced a software repository [21] to fully reproduce the methods described in our paper. External researchers can use this repository to pre-train on a variety of chip blocks and then apply the pre-trained model to new blocks, as was done in our original paper. As part of this addendum, we are also releasing a model checkpoint pre-trained on 20 TPU blocks [22]. For best results, however, we continue to recommend that developers pre-train on their own in-distribution blocks [18], and provide a tutorial on how to perform pre-training with our open-source repository [23]. [1]: https://www.nature.com/articles/s41586-024-08032-5 https://www.nature.com/articles/s41586-024-08032-5 [18]: Yue, S. et al. Scalability and generalization of circuit training for chip floorplanning. In Proc. 2022 International Symposium on Physical Design 65–70 (2022). [21]: Guadarrama, S. et al. Circuit Training: an open-source framework for generating chip floor plans with distributed deep reinforcement learning. GitHub https://github.com/google-research/circuit_training https://github.com/google-research/circuit_training (2021). [23]: Guadarrama, S. et al. Pre-training. GitHub https://github.com/google-research/circuit_training/blob/main/docs/PRETRAINING.md https://github.com/google-research/circuit_training/blob/mai... (2021).

seanhunter 2y ago

Well here’s one exaggeration that was pretty obvious to me straight away as a somewhat disinterested observer. In her status on X Anna Goldie says [1] “ AlphaChip was one of the first RL methods deployed to solve a real-world engineering problem”. This seems very clearly untrue- for example here’s a real-world engineering use of reinforcement learning by google AI themselves from 6 years ago [2] which if you use Anna Goldie’s own timeline is 2 years before alphachip. [1] https://x.com/annadgoldie/status/1858531756506558688 https://x.com/annadgoldie/status/1858531756506558688 [2] https://youtu.be/W4joe3zzglU?si=mFvZq8gEI6LeEQdC https://youtu.be/W4joe3zzglU?si=mFvZq8gEI6LeEQdC

wholehog 2y ago
That is definitely a cool project, but I don't see how it contradicts "one of the first RL methods deployed to solve a real-world engineering problem". "One of the first" does not mean literally the first ever.
- seanhunter 2y ago
  Agreed but if someone at your own company did it two years before you in the context of something that recent it’s stretching credibility to say you were one of the first.
  - wholehog 2y ago
    I mean, I think second is still "one of the first?" And, no offense to this project, but I don't know of it being used in a real industrial setting, whereas AlphaChip was used in TPU.
    - seanhunter 2y ago
      Yes. The sky is also blue? However it's hard to see how being provably 2 years behind the first even in your own company in an incredibly hot area that people are doing tons of work in makes you suddenly second. By that logic I might still be in time to claim the silver for the 100m at the Paris olympics if I pop over there in the next 18 months or so. I can see you created this account just to comment on this thread so I'm sure you have more inside information than I do given that I'm really not connected to this in any way. Enjoy your work at Google Research. I think you guys do cool stuff. It's a shame in my opinion that you choose to damage your credibility by making (and defending) such obviously false claims rather than concentrating on the genuinely innovative work you have done advancing the field.