4 ms·
Then again, if you don't have access to giant compute clusters you can't test this, so it's either a blog post or nothing. I believe the outlier problem that th
by WithinReason 3y ago
Then again, if you don't have access to giant compute clusters you can't test this, so it's either a blog post or nothing. I believe the outlier problem that this solves only appears for very large models.
- janalsncm 3y agoThat isn’t true at all. Train a smaller model on a smaller dataset. You can even train on your laptop. It’s definitely feasible. This is just a proof of concept, it doesn’t need to beat state of the art.
- WithinReason 3y agoMaybe I edited my comment too late.
- janalsncm 3y ago> I believe the outlier problem that this solves only appears for very large models. Any reason to believe this? The author never mentioned it, and I can’t think of any other a priori reason why it should be true.
- WithinReason 3y agoSee figure 1: https://arxiv.org/pdf/2208.07339.pdf https://arxiv.org/pdf/2208.07339.pdf Outliers appear at model size 6.7B and are not present at 2.7B
- janalsncm 3y agoSure, emergent properties can arise as parameters increase. Everyone knows that. That’s a much less specific claim than to say that the benefit of modifying softmax can only arise as an emergent property after N parameters, and therefore the benefit can only be evaluated on models above a certain size. To my understanding the author of TFA isn’t suggesting the same issue as the one in your linked paper.
- WithinReason 3y agoThe second heading in the TFA is "It’s All About Outliers"
- PoignardAzur 3y ago6.7B isn't "needs a datacenter" scale.
- WithinReason 3y agoIt's in the million dollar range. XLnet which is a 1.3B model cost $245,000 to train for example.