> One thing I can say for sure is that GPT-3 has a known bias where it only views the text within a certain window. GPT-3 is physically unable to "read" a book,

> One thing I can say for sure is that GPT-3 has a known bias where it only views the text within a certain window. GPT-3 is physically unable to "read" a book, it can only use a certain window of text in order to issue its "most likely continuations". Therefore, anything outside of that window is as good as something that never happened from its point of view. This description reminds me of simple Markov chains. You just ingest a bunch of text taking a window of, say, 10 characters and recording all the possible continuations thereof. So you might get [This remind] => "s" or such. Then you reverse that by picking a starting node and spinning text by picking a random continuation as you slide the window 'forward' to output.