One of the key things OpenAI does is very small scale model experiments. You can iterate a lot faster if your training run takes on the order of minutes. So v

One of the key things OpenAI does is very small scale model experiments. You can iterate a lot faster if your training run takes on the order of minutes. So vet ideas there, only allow proven ideas to be part of the longer training run. ML model training has become the new "my code's compiling" except now it takes weeks in many cases. The train then chill mindset that some ML engineers have is not they way to make rapid progress.