Even though GPT-4 "fixes" many of the problems pointed out in this blog post, I question whether or not some of the underlying issues associated with transforme

Even though GPT-4 "fixes" many of the problems pointed out in this blog post, I question whether or not some of the underlying issues associated with transformers (and the attention mechanism in particular) can be overcome with with more sparse attention and deeper/wider models. e.g. GPT-3.5 might not be able to write simple poems correctly, but does GPT-4 solve the disease or the symptom (by masking it with a bigger model and more data)? I'm not entirely convinced.