4 ms·
You should publish a response paper and get them to retract their paper if it has major flaws.
by bingbingbing777 2y ago
You should publish a response paper and get them to retract their paper if it has major flaws.
- karalala 2y agoIts xlstm contradicting existing peer reviewed papers lmao. Either xlstm should fix their benchmarks or existing peer reviewed papers should retract. RWKV-v6 > RWKV-v5 > RWKV-v4, not the other way round obviously. HGRN 8 ppl worse than baseline transformers? NIPS 2023 spotlight paper btw.
- logicchains 2y agoI thought it was common knowledge that architecture comparisons in papers aren't worth the paper they're printed on; there are so many ways to deliberately or accidentally structure things to favour one architecture over the others. Ultimately the lmsys chatpot arena will be the final judge.
- karalala 2y agoTrue, but they normally arent this far off. HGRN claims that they outperform transformer for 1B parameter model trained on the pile. HGRN performing 8ppl worse suggests that its useless.
- AIsore 2y agoMy experience - many are far off and most of the time published tables of different papers are hard to compare. If you make the assertion here of these results to be flawed, I would like to see more substance (code, reproduction,...). And for balance, for the same reason, hard to verify the accuracy of these results without further insight.
- logicchains 2y agoSo many papers play tricks with the learning rate schedule: https://arxiv.org/abs/2307.06440 https://arxiv.org/abs/2307.06440
- AIsore 2y agoAre you saying this is obvious because people have published the exact same benchmarks which are 100% comparable in journals? If so where are they? I have seen quite a few published benchmarks that could not quite be reproduced, tbh. So, again, what makes this "obvious" to you?