3 ms·
> 100s PB Unrelated and I know it is just a representative number but I have seen the training data to be assumed something in this range few times. Entire tra
by YetAnotherNick 3y ago
> 100s PB
Unrelated and I know it is just a representative number but I have seen the training data to be assumed something in this range few times. Entire training set of ChatGPT is almost surely less than a TB or two with compression which is 5 orders of magnitude lower. I believe that such efficient representation of text is one of the biggest reason why text models are working so well but image understanding models are not.
- mjburgess 3y agoPerhaps the process was an initial c. 1PB then sampled down to the TBs. Text is extremely lightweight, so I suppose everything ever written is at most 1-10PB. This is one of the illusions of text-generative NNs: a 0.5TB weight set is basically enough to store every book. Making claims to "out-sample generalisation" extremely suspicious, and indeed, fairly obviously false. eg., Ask ChatGPT to write tic-tak-toe in javascript and you get a working game; as it to write duck-hunt and you dont.