Recent pre-print from Intel Habana labs regarding training instability they encountered when training a Llama model at FP8, traced to the use of the SwiGLU from

Recent pre-print from Intel Habana labs regarding training instability they encountered when training a Llama model at FP8, traced to the use of the SwiGLU from Noam Shazeer's 2020 paper: https://arxiv.org/abs/2002.05202 https://arxiv.org/abs/2002.05202