3 ms·
Honestly, these complaints (other than 4) apply to the vast majority of papers. #4 is just false. It has already been viewed by other lab members (peers) and op
by godelski 4y ago
Honestly, these complaints (other than 4) apply to the vast majority of papers. #4 is just false. It has already been viewed by other lab members (peers) and open publication is peer reviewing. The "peer review system" (publishing to conferences/journals) is relatively new and I think ML demonstrates all the problems with the system (yay hype).
Novelty is especially a joke. ViTs are "just" NLP encoding transformers. T2I models are "just" NLP models connected to generative models. Diffusion models are "just" whitening models. GPT3 is just GPT2 with more layers and more data which is just GPT with more layers and more data. We can go even deeper if we pull from math and physics works. But that doesn't mean these works haven't been highly fruitful and useful. I'm happy all of these have been published.
> because of the hype around LLMs
I too hate the hype, but it is often bimodal. There are people who are far too critical and people who are far too accepting. The harm is not preprints or people reading papers, the harm is people who have no business/qualifications evaluating works confidently spouting out critiques. It is people not understanding that researchers are just critical of one another's work by default and that doesn't mean it shouldn't have been published.
It is well known that reviewers are good at identifying bad papers but not good at identifying good papers[0,1]. Which let's be honest, that means reviewers just have high reject rates in a noisy system. Making publication as a metric for merit a highly noisy one at best.
As for the paper:
Many LLMs and large models are using attention approximations. Nor is the kNN technique particularly new. My main complaints are a lack of comparisons for Figure 3 and 4, but I'm not a NLP person so I don't even know if there's some other good works that can compare better (BART is a common baseline). But generative models are (unfortunately not notoriously known) extremely difficult to evaluate. Paper seems fine to me. It is useful to the community. I don't like the name either, but their input is limited by computer memory, not the model. I would want to see more on this. Not a NLP person all I can say is that this looks neither like a strong reject nor a strong accept. I'll leave it to the community to determine if they want more experiments for the conference publication or not but the work seems useful.
[0] https://inverseprobability.com/talks/notes/the-neurips-experiment-snsf.html https://inverseprobability.com/talks/notes/the-neurips-exper...
[1] https://arxiv.org/abs/2109.09774 https://arxiv.org/abs/2109.09774