There are basically no people who have done pre-training of LLMs. What you're describing is absolutely the case, but everyone not in a top 100 AI lab would hav

There are basically no people who have done pre-training of LLMs. What you're describing is absolutely the case, but everyone not in a top 100 AI lab would have never had the displeasure of having to deal with that. That said I strongly suggest everyone follow along with a GPT2 tutorial like: https://github.com/karpathy/minGPT https://github.com/karpathy/minGPT to get a feel for what the big boys have to deal with. Any $100,000 training run can end in failure for no reason what so ever because you just got unlucky with your random seed. If you want something that is both SOTA and cheap: https://huggingface.co/blog/modernbert https://huggingface.co/blog/modernbert you can get from pre-training to fine tuning for a couple hundred dollars with a model that is better than anything available to the public. Bert models are so under researched that you legitimately push the state of the art for them on a work station. Also feel free to drop a line at my profile's address.