What is your desired objective and what are you going to train on? There are plenty of publicly available model checkpoints so you don’t have to start from scra

What is your desired objective and what are you going to train on? There are plenty of publicly available model checkpoints so you don’t have to start from scratch with large GPU clusters. There’s no point in repeating the pretraining that has already been done. I’m not sure that compute is still the bottleneck right now, it’s fairly cheap to train LLMs. Many optimizations like DeepSpeed dramatically reduce computation requirements / increase throughout (I.e. 13b model on a single GPU). If we’ve learned anything from the massive LLMs like PaLM it’s that scaling autoregressive models to infinity has diminishing returns and is not feasible to implement in inference infra. Google themselves acknowledge this resource limitation in the Med-PaLM paper when they discuss fine tuning a 450b parameter model. We’re really in more of a dataset and training task era of AI/NLP gains. Scaling masked the issues of poor quality training data and conventional language modelling up to a certain point but we’re starting to see the problems with that (hallucination) in all of the big (Galactica is an example of the limitations in this). OpenAI’s main advantage is that they paid humans to build a large labelled dataset for their RL objective. They’re offering ChatGPT for free to get more training data.