For what it's worth, I think the lack of comments is due to the highly academic nature of the experiment displayed in TFA. LLMs are all of the rage now, and th

For what it's worth, I think the lack of comments is due to the highly academic nature of the experiment displayed in TFA. LLMs are all of the rage now, and the pre-training techniques used for BERT ended up laying the foundation for what became GPT pre-training runs. But I would assume even among HN users, there really are only a small handful of people interacting with this side of the NLP craze on a regular basis. Compare that with the amount of users who probably have at least dabbled in using a programming language, and suddenly you see why the opinions fly everywhere. A lot of the practical aspects of training neural nets are still more "I'm doing it this way because Radford et al did it" or "when you picture the problem in three dimensions, this generally seems like it should work" instead of "This was demonstrated to be the best way and I can show you it's the best way from first principles." because quite frankly that's the best MO we have when dealing with the statistical principles guiding billions of matrix multiplies for trillions of different token sequences.