Wow, they hot-swapped activation functions (GELU -> RELU) during training. They are indeed very similar activation functions, but it's kinda crazy to me that y

Wow, they hot-swapped activation functions (GELU -> RELU) during training. They are indeed very similar activation functions, but it's kinda crazy to me that you can make that kind of a change to a model while it's training, preserving all weights and other state, and just keep going. They changed weight clipping thresholds on the fly too. They also swapped out the optimizer several times from what I can tell, switching between Adam, "Fake SGD", and "Vanilla SGD" multiple times. Even without the huge amounts of hardware/driver issues they seemed to be having with the GPUs in their big training cluster(s), this puts into perspective how hard it is to train enormous models like this. Many of the failures don't have an immediately obvious cause. Plus, there aren't all that many places out there doing training at this scale so I imagine many of these things need to get figured out on their own.