What you observe there is context limitation. GPT-3 as is, has a fixed length context window which limits its capacity to remember and be coherent across long h

What you observe there is context limitation. GPT-3 as is, has a fixed length context window which limits its capacity to remember and be coherent across long horizons. A hierarchical version that contained summaries e.g. for previous sentences, paragraphs, and so on, could in theory, behave much better. This is the more "sophisticated" approach. The brute force approach says to just throw more compute, larger contexts and see what happens.