3 ms·

> From my point of view, AlphaGo, AlphaZero, AlphaFold were significant achievements. These things you mentioned had obvious benchmarks that were easily surpas

by bsder 2y ago

> From my point of view, AlphaGo, AlphaZero, AlphaFold were significant achievements. These things you mentioned had obvious benchmarks that were easily surpassed by the appropriate "AI". The evidence that they were better wasn't just significant, it was obvious. This leaves the fact that with what appears to be maximal cooking of the books, the only thing AlphaChip seems to be able to beat is human, manual placement and not anything algorithmic--even from many, many generations ago. Trying to pass that off as a significant "advance" in a "scientific publication" borders on scientific fraud and should definitely be called out. The problem here is that I am certain that this is wired to the career trajectories of "Very Important People(tm)" and the fact that it essentially failed miserably is simply not politically allowed. If they want to lie, they can do that in press releases. If they want published in something reputable, they should have to be able to provide proper evidence for replication. And, if they can't do that, well, that's an answer itself, no?

xpe 2y ago
> Trying to pass that off as a significant "advance" in a "scientific publication" borders on scientific fraud and should definitely be called out. If true, your stated concerns with the AlphaChip paper -- selective benchmarking and potential overselling of results - reflect poor scientific practice and possible intellectual dishonesty. This does not constitute scientific fraud, which occurs when the underlying method/experiment/rules are faked. If the paper has issues with how it positions and contextualizes its contribution, criticism is warranted, sure. But don't confuse this with "scientific fraud". Some context: for as long as benchmark suites have existed, people rightly comment on which benchmarks should be included and how they should be weighted.
xpe 2y ago
> "scientific publication" These air quotes suggests the commenter above doesn't think the paper qualifies a scientific publication. Such a characterization is unfair. When I read the Nature article titled "Addendum: A graph placement methodology for fast chip design" [1], I see writing that more than meets the bar for a scientific publication. For example: > Since publication, we have open-sourced a software repository [21] to fully reproduce the methods described in our paper. External researchers can use this repository to pre-train on a variety of chip blocks and then apply the pre-trained model to new blocks, as was done in our original paper. As part of this addendum, we are also releasing a model checkpoint pre-trained on 20 TPU blocks [22]. For best results, however, we continue to recommend that developers pre-train on their own in-distribution blocks [18], and provide a tutorial on how to perform pre-training with our open-source repository [23]. [1]: https://www.nature.com/articles/s41586-024-08032-5 https://www.nature.com/articles/s41586-024-08032-5 [18]: Yue, S. et al. Scalability and generalization of circuit training for chip floorplanning. In Proc. 2022 International Symposium on Physical Design 65–70 (2022). [21]: Guadarrama, S. et al. Circuit Training: an open-source framework for generating chip floor plans with distributed deep reinforcement learning. GitHub https://github.com/google-research/circuit_training https://github.com/google-research/circuit_training (2021). [23]: Guadarrama, S. et al. Pre-training. GitHub https://github.com/google-research/circuit_training/blob/main/docs/PRETRAINING.md https://github.com/google-research/circuit_training/blob/mai... (2021).

by bsder 2y ago

xpe 2y ago

> Trying to pass that off as a significant "advance" in a "scientific publication" borders on scientific fraud and should definitely be called out. If true, your stated concerns with the AlphaChip paper -- selective benchmarking and potential overselling of results - reflect poor scientific practice and possible intellectual dishonesty. This does not constitute scientific fraud, which occurs when the underlying method/experiment/rules are faked. If the paper has issues with how it positions and contextualizes its contribution, criticism is warranted, sure. But don't confuse this with "scientific fraud". Some context: for as long as benchmark suites have existed, people rightly comment on which benchmarks should be included and how they should be weighted.

xpe 2y ago

> "scientific publication" These air quotes suggests the commenter above doesn't think the paper qualifies a scientific publication. Such a characterization is unfair. When I read the Nature article titled "Addendum: A graph placement methodology for fast chip design" [1], I see writing that more than meets the bar for a scientific publication. For example: > Since publication, we have open-sourced a software repository [21] to fully reproduce the methods described in our paper. External researchers can use this repository to pre-train on a variety of chip blocks and then apply the pre-trained model to new blocks, as was done in our original paper. As part of this addendum, we are also releasing a model checkpoint pre-trained on 20 TPU blocks [22]. For best results, however, we continue to recommend that developers pre-train on their own in-distribution blocks [18], and provide a tutorial on how to perform pre-training with our open-source repository [23]. [1]: https://www.nature.com/articles/s41586-024-08032-5 https://www.nature.com/articles/s41586-024-08032-5 [18]: Yue, S. et al. Scalability and generalization of circuit training for chip floorplanning. In Proc. 2022 International Symposium on Physical Design 65–70 (2022). [21]: Guadarrama, S. et al. Circuit Training: an open-source framework for generating chip floor plans with distributed deep reinforcement learning. GitHub https://github.com/google-research/circuit_training https://github.com/google-research/circuit_training (2021). [23]: Guadarrama, S. et al. Pre-training. GitHub https://github.com/google-research/circuit_training/blob/main/docs/PRETRAINING.md https://github.com/google-research/circuit_training/blob/mai... (2021).