NaNs are not only possible by design, but are extremely common. Training of LLMs involve many tricks about how to deal with training steps that result in NaNs.

NaNs are not only possible by design, but are extremely common. Training of LLMs involve many tricks about how to deal with training steps that result in NaNs. Quantisation of LLMs also require dealing with huge outlier values.