I've started reading from bottom and haven't read the whole thing yet. But their default action as stated in their log when face exploding gradients or unstable

I've started reading from bottom and haven't read the whole thing yet. But their default action as stated in their log when face exploding gradients or unstable training is to just roll back a checkpoint and lower lr. Other proposed actions such as clamping activations are also just pretty standard things to try. I guess since their goal is to just be able to have a trained model it doesn't really matter. But it doesn't seem to be a easily reproducible process, and like i said a bit hacky in my opinion