3 ms·

Where are you getting these numbers for GPT-4/5 resource requirements from? I’ve heard of diminishing returns happening though not to the degree you’re describ

Where are you getting these numbers for GPT-4/5 resource requirements from? I’ve heard of diminishing returns happening though not to the degree you’re describing.

PaulHoule 5y ago
The numbers don't matter. The fact that it will converge towards an asymptote is qualitative, not quantitative because the structure of the system is wrong for the problem. If the people who were doing this research were physicists you could excuse them, but they are computer scientists and should know better. It's a basic problem that GPT-x allocates the same amount of resources despite the complexity of the problem. Some problems in programming and language understanding have a SAT/SMT-like structure (e.g. for each pronoun solve for what the pronoun refers to, figure out what the subject/objects of all the verbs are, etc.) which scales in a certain way. Other problems in "understanding" require necessarily that you try one interpretation, work through the consequences of that interpretation until you find a dead end, back up to the place where you failed, try something else, test it until done. Some of these problems that you might want to "solve" with GPT-x might really be problems that can't be solved in a finite amount of time. Something like GPT-x with some extra structural features might be able to rise to some of these challenges but we are not hearing about that, we are hearing that adding more nodes and more training data and training for longer is the way to salvation. (So far as "diminishing returns" matter the question is "does the system ever get good enough that it can be let off the leash?" and my answer is no.)
MauranKilom 5y ago
Well, we are already hitting pretty strong limits in terms of corpus size. For reference, GPT-3 has 175 billion parameters. That's a few TB (uncompressed). GPT-3 was trained on 45 TB of text. Most of it comes from 8 years of web crawling, and a significant chunk of the remainder comes from online book corpora [0]. At some point you just run out of human-written text to train on. You can't just keep making the model bigger (by orders of magnitude I mean) without scaling the input data the same. And at some point we just run out of the latter. [0]: https://in.springboard.com/blog/openai-gpt-3/ https://in.springboard.com/blog/openai-gpt-3/

Where are you getting these numbers for GPT-4/5 resource requirements from? I’ve heard of diminishing returns happening though not to the degree you’re describing.

PaulHoule 5y ago

The numbers don't matter. The fact that it will converge towards an asymptote is qualitative, not quantitative because the structure of the system is wrong for the problem. If the people who were doing this research were physicists you could excuse them, but they are computer scientists and should know better. It's a basic problem that GPT-x allocates the same amount of resources despite the complexity of the problem. Some problems in programming and language understanding have a SAT/SMT-like structure (e.g. for each pronoun solve for what the pronoun refers to, figure out what the subject/objects of all the verbs are, etc.) which scales in a certain way. Other problems in "understanding" require necessarily that you try one interpretation, work through the consequences of that interpretation until you find a dead end, back up to the place where you failed, try something else, test it until done. Some of these problems that you might want to "solve" with GPT-x might really be problems that can't be solved in a finite amount of time. Something like GPT-x with some extra structural features might be able to rise to some of these challenges but we are not hearing about that, we are hearing that adding more nodes and more training data and training for longer is the way to salvation. (So far as "diminishing returns" matter the question is "does the system ever get good enough that it can be let off the leash?" and my answer is no.)

MauranKilom 5y ago

Well, we are already hitting pretty strong limits in terms of corpus size. For reference, GPT-3 has 175 billion parameters. That's a few TB (uncompressed). GPT-3 was trained on 45 TB of text. Most of it comes from 8 years of web crawling, and a significant chunk of the remainder comes from online book corpora [0]. At some point you just run out of human-written text to train on. You can't just keep making the model bigger (by orders of magnitude I mean) without scaling the input data the same. And at some point we just run out of the latter. [0]: https://in.springboard.com/blog/openai-gpt-3/ https://in.springboard.com/blog/openai-gpt-3/