They describe Transformer++ as "A Transformer with an improved architecture, namely rotary positional encodings and SwiGLU MLP", no linear bias terms and RMSNor

They describe Transformer++ as "A Transformer with an improved architecture, namely rotary positional encodings and SwiGLU MLP", no linear bias terms and RMSNorm instead of LayerNorm. But modern transformers have many more tricks than that. Such as pre-norm, sparse better use of residual layers, sparse attention masks and so on.