5 ms·

you are in denial if you are doubting effectiveness of Codex. Keep in mind that this was just beta. If you saw how GPT-2 improved to GPT-3 in a year, it's eas

by treme 5y ago

you are in denial if you are doubting effectiveness of Codex. Keep in mind that this was just beta. If you saw how GPT-2 improved to GPT-3 in a year, it's easy to see where this is gonna go over next few iterations. It's a 2007 Iphone level catalyst that's going to dramatically shift the landscape

PaulHoule 5y ago
And GPT-4 will almost work when it is powered by a Dyson sphere and GPT-5 would require more text to train than exists in all the planets of the universe. All of those things are structurally inadequate to solve the problem in front of them and just make up for it for the same reason ELIZA seemed intelligent... People are willing to believe. You could put fantastically more resources into that approach and find you're approaching an asymptote. It's the deadliest trap in advanced technology development and it happens when you ignore the first law of cybernetics. Anyone who's been a practitioner in the software field has experienced that repairing mistakes from a program written by somebody clueless is almost always vastly more expensive than writing it correctly to begin with -- I remember helping a friend "cheat" at CS 101 by stealing somebody else's homework, finding the program was wrong and putting a lot of effort into debugging it and fixing it, never mind changing the symbol names and taking other measures to hide the origin of the program. It might be my karma, but fixing the program I stole turned out to be excellent preparation for a career in software dev.
- qualudeheart 5y ago
  Where are you getting these numbers for GPT-4/5 resource requirements from? I’ve heard of diminishing returns happening though not to the degree you’re describing.
  - PaulHoule 5y ago
    The numbers don't matter. The fact that it will converge towards an asymptote is qualitative, not quantitative because the structure of the system is wrong for the problem. If the people who were doing this research were physicists you could excuse them, but they are computer scientists and should know better. It's a basic problem that GPT-x allocates the same amount of resources despite the complexity of the problem. Some problems in programming and language understanding have a SAT/SMT-like structure (e.g. for each pronoun solve for what the pronoun refers to, figure out what the subject/objects of all the verbs are, etc.) which scales in a certain way. Other problems in "understanding" require necessarily that you try one interpretation, work through the consequences of that interpretation until you find a dead end, back up to the place where you failed, try something else, test it until done. Some of these problems that you might want to "solve" with GPT-x might really be problems that can't be solved in a finite amount of time. Something like GPT-x with some extra structural features might be able to rise to some of these challenges but we are not hearing about that, we are hearing that adding more nodes and more training data and training for longer is the way to salvation. (So far as "diminishing returns" matter the question is "does the system ever get good enough that it can be let off the leash?" and my answer is no.)
  - MauranKilom 5y ago
    Well, we are already hitting pretty strong limits in terms of corpus size. For reference, GPT-3 has 175 billion parameters. That's a few TB (uncompressed). GPT-3 was trained on 45 TB of text. Most of it comes from 8 years of web crawling, and a significant chunk of the remainder comes from online book corpora [0]. At some point you just run out of human-written text to train on. You can't just keep making the model bigger (by orders of magnitude I mean) without scaling the input data the same. And at some point we just run out of the latter. [0]: https://in.springboard.com/blog/openai-gpt-3/ https://in.springboard.com/blog/openai-gpt-3/
- scelerat 5y ago
  And yet today, the descendants of ELIZA provide first-line customer support for thousands of companies.
  - twic 5y ago
    Extremely poorly. Companies use them because they're cheap, not because they're good.

by treme 5y ago

PaulHoule 5y ago

And GPT-4 will almost work when it is powered by a Dyson sphere and GPT-5 would require more text to train than exists in all the planets of the universe. All of those things are structurally inadequate to solve the problem in front of them and just make up for it for the same reason ELIZA seemed intelligent... People are willing to believe. You could put fantastically more resources into that approach and find you're approaching an asymptote. It's the deadliest trap in advanced technology development and it happens when you ignore the first law of cybernetics. Anyone who's been a practitioner in the software field has experienced that repairing mistakes from a program written by somebody clueless is almost always vastly more expensive than writing it correctly to begin with -- I remember helping a friend "cheat" at CS 101 by stealing somebody else's homework, finding the program was wrong and putting a lot of effort into debugging it and fixing it, never mind changing the symbol names and taking other measures to hide the origin of the program. It might be my karma, but fixing the program I stole turned out to be excellent preparation for a career in software dev.

qualudeheart 5y ago
Where are you getting these numbers for GPT-4/5 resource requirements from? I’ve heard of diminishing returns happening though not to the degree you’re describing.
- PaulHoule 5y ago
  The numbers don't matter. The fact that it will converge towards an asymptote is qualitative, not quantitative because the structure of the system is wrong for the problem. If the people who were doing this research were physicists you could excuse them, but they are computer scientists and should know better. It's a basic problem that GPT-x allocates the same amount of resources despite the complexity of the problem. Some problems in programming and language understanding have a SAT/SMT-like structure (e.g. for each pronoun solve for what the pronoun refers to, figure out what the subject/objects of all the verbs are, etc.) which scales in a certain way. Other problems in "understanding" require necessarily that you try one interpretation, work through the consequences of that interpretation until you find a dead end, back up to the place where you failed, try something else, test it until done. Some of these problems that you might want to "solve" with GPT-x might really be problems that can't be solved in a finite amount of time. Something like GPT-x with some extra structural features might be able to rise to some of these challenges but we are not hearing about that, we are hearing that adding more nodes and more training data and training for longer is the way to salvation. (So far as "diminishing returns" matter the question is "does the system ever get good enough that it can be let off the leash?" and my answer is no.)
- MauranKilom 5y ago
  Well, we are already hitting pretty strong limits in terms of corpus size. For reference, GPT-3 has 175 billion parameters. That's a few TB (uncompressed). GPT-3 was trained on 45 TB of text. Most of it comes from 8 years of web crawling, and a significant chunk of the remainder comes from online book corpora [0]. At some point you just run out of human-written text to train on. You can't just keep making the model bigger (by orders of magnitude I mean) without scaling the input data the same. And at some point we just run out of the latter. [0]: https://in.springboard.com/blog/openai-gpt-3/ https://in.springboard.com/blog/openai-gpt-3/
scelerat 5y ago
And yet today, the descendants of ELIZA provide first-line customer support for thousands of companies.
- twic 5y ago
  Extremely poorly. Companies use them because they're cheap, not because they're good.