4 ms·
While I generally agree to some extent: roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative Researche
by The_rationalist 6y ago
While I generally agree to some extent:
roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative
Researchers take inspirations from existing models of course and some BERT derivatives are trivial.
However, XLnet is in it's own league, while the author (a genius chinese student) was inspired by BERT it is one of the few SOTA pré trained models to be not based on BERT and is actually an auto regressive one! Such difference allow it to be better at many things as it doesn't has to corrupt the tokens (from my shallow understanding).
This model is two years old but is still sadly the one that ranks the most SOTA in key tasks e.g dependency parsing. And after all those time nobody cared enough to even test it on other foundational tasks (which is extremely sad and pathetic) like e.g coreference resolution.
Sadly because of conformism effects almost zero researcher has created XLnet derivatives. Almost all researchers continue to search in the local minima that is BERT, which I find, immensely ironic.
While ad hoc empirical fine tuning is a big part of improving sota, mathematical genius can still enable revolutions e.g this recent alternative to classical backpropagation that is 300X faster with low accuracy loss
https://paperswithcode.com/paper/zorb-a-derivative-free-backpropagation https://paperswithcode.com/paper/zorb-a-derivative-free-back...
- throwaway189262 6y agoInteresting, but in not sure you're completely right about XLNET. I heard it takes an absurd amount of resources to train. Even more than the BERT variations. And this is likely why there's not a ton of interest in it
- The_rationalist 6y agohttps://github.com/renatoviolin/xlnet https://github.com/renatoviolin/xlnet XLnet running on very low end hardware (a single 8GB 2080 non ti) significantly outperform BERT large on e.g the reference question answering benchmarck: SQUAD 2 86% vs 81% Nobody has even tried to create a spanXLnet (akin to spanBERT) How many years will be wasted before researchers get out of the BERT local minima? I'm a afraid it might last a decade
- Der_Einzige 6y agoNot sure why you're being downvoted. I was about to swoop in and mention that the top level comment was wrong about XLNet being some bert based model but you beat me to it. Sometimes HN is full of people who think they know what they're talking about but just don't. This is one of those times.
- The_rationalist 6y agoHonestly my comments that have negative Karma have generally much more useful truths than my comments that have positive Karma, this is almost systemic. This show how low quality the HN community is, epistemologically speaking. There are much less lazy communities out there like lesswrong.com but unfortunately they don't talk much about computer science