I love the idea of ReZero, basically using a trainable parameter, alpha, in residual layers like this: Deep Network | xi+1 = F(xi)

I love the idea of ReZero, basically using a trainable parameter, alpha, in residual layers like this: Deep Network | xi+1 = F(xi) Residual Network | xi+1 = xi + F(xi) Deep Network + Norm | xi+1 = Norm(F(xi)) Residual Network + Pre-Norm | xi+1 = xi + F(Norm(xi)) Residual Network + Post-Norm | xi+1 = Norm(xi + F(xi)) ReZero | xi+1 = xi + αi F(xi) However, I haven't actually seen this used in practice. The papers we have on Gemma and Llama all still seem to be using layer norms. Am I missing something?