Perhaps the process was an initial c. 1PB then sampled down to the TBs. Text is extremely lightweight, so I suppose everything ever written is at most 1-10PB.

Perhaps the process was an initial c. 1PB then sampled down to the TBs. Text is extremely lightweight, so I suppose everything ever written is at most 1-10PB. This is one of the illusions of text-generative NNs: a 0.5TB weight set is basically enough to store every book. Making claims to "out-sample generalisation" extremely suspicious, and indeed, fairly obviously false. eg., Ask ChatGPT to write tic-tak-toe in javascript and you get a working game; as it to write duck-hunt and you dont.