4 ms·
There's nothing really wrong with a preprint making it to the top - there can be genuinely good work that stays in preprint for quite some time. I believe the o
by chaxor 3y ago
There's nothing really wrong with a preprint making it to the top - there can be genuinely good work that stays in preprint for quite some time.
I believe the original ELMo work that spurred the Sesame street gang is still in preprint despite its importance in NLP (:shocked Pikachu face: not a transformer?!).
But yes, you're correct in this instance that it's not necessarily 'huge news' since it is highly similar to a long list of the Reformer (LSH-based), Performer (FAVOR**), FNet (Fourier-based), Routing Transformer, Sparse Transformer, Longformer (task specific sparse), blockbert, XLNet/xfmr-xl (slide + relative PE), BP-Transformer (binary partition), BigBird (global and rand attn), RWKV which is..., etc.
** FAVOR actually is innovative and different in this space, but towards similar ends anyway
- visarga 3y agoHow come you know the efficient-transformers family, when I ask questions about transformers in ML interviews nobody has heard of them. Can't figure out why it's not common knowledge. For years all the transformer papers were about reducing O( N^2 )
- chaxor 3y agoThe reason they don't know them is they're not serious researchers or practitioners of ML - it's as simple as that. Anyone in this area should have this exceedingly basic common knowledge.
- f_devd 3y agoTo be fair ML is (used to be?) pretty broad, so unless someone is actively keeping up with the sota in the high-data sequence modeling area it's quite possible to miss. I know ML teams which were entirely made up of OSML practicioners, because that was the most commonly useful until recently.
- Nimitz14 3y agoWhy learn something noone is using.
- visarga 3y agoOh I didn't test their in-depth knowledge, just checking to know if they were aware people were writing hundreds of papers on this topic, and what impact would have a sub-quadratic attention that works as well as the regular one. Fishing to see if they remember at least one approach or two.
- chaxor 3y agoMany people are using it, but people that are relying on OpenAI to do all of the work for them aren't looking deep enough to realize it.
- Nimitz14 3y agoCan you point to a single major project that does? AFAIK it always leads significant regressions in real world use cases, which is why no major repo uses them.
- zwaps 3y agoTwo reasons: They were only recently implemented in large production models and are not part of ye-standard-ML-coursera. And I mean, for half the papers claiming that their particular efficiency variant reduces O(n^2) to whatever without performance loss, we found that in practice it ain't quite so shiny. Anyone who has been for whatever reason reading the papers since 2017 has invariably read dozens of these papers. Anyone who has heard of GPT-x in 202x and started from there probably didn't. This will likely change with implementation of memory retrieval, some form of linear attention etc. in many productions models, and the democratization of some decoder models... although I have been thinking this for a while. Don't get me wrong, you want to hire the people who know these papers, especially if they started after 2017 :-)
- chaxor 3y agoI have a team of bioinformaticians with very little ML knowledge, but they know of these basic papers... So yeah, if someone claims to be in ML and aren't aware of this foundational knowledge... they're not to be taken seriously.