It's to the point now where using a basic LSTM is almost discouraged compared to using a Transformer. GPT-2 wouldn't have been possible without these recent inn

It's to the point now where using a basic LSTM is almost discouraged compared to using a Transformer. GPT-2 wouldn't have been possible without these recent innovations. So, but that "point" could just be that being a published academic means always using the latest thing. That doesn't demonstrate how much better these are, much less demonstrate that they represent fundamental steps forward.