I don't think we'll need to completely replace the transformer, we just need better, more annotated data sets and a really good solution to large context window

I don't think we'll need to completely replace the transformer, we just need better, more annotated data sets and a really good solution to large context windows. Data set annotation has a huge impact on model output, you can see this for yourself by comparing SDXL to Stable Diffusion 1.5, the model is larger but the real juice comes from using aesthetics scores for images during training to bias output towards things that look good - compare it to other large models like DALL-E or Imagen and you'll see parameter size is not the most important thing. With larger context windows, we can use traditional search algorithms to cherry pick an educational context for the LLM for few shot learning. If that's paired with more, better and annotated data, that will take us quite far.