It's pretty incredible how robust neural networks are to even things like architectural changes mid-run. The team at Meta when training their own copy of GPT-3

It's pretty incredible how robust neural networks are to even things like architectural changes mid-run. The team at Meta when training their own copy of GPT-3 (called OPT-3) even changed optimizers mid-run (from AdamW -> SGD -> AdamW, https://arxiv.org/pdf/2205.01068 https://arxiv.org/pdf/2205.01068 section 2.5)!