4 ms·

> The difference between GPT3 and ChatGPT, to me, suggests a lot. Could you be more specific about what those differences are exactly? I'm curious how closely

> The difference between GPT3 and ChatGPT, to me, suggests a lot. Could you be more specific about what those differences are exactly? I'm curious how closely you've watched this space because we're basically exactly where everyone predicted we'd be back in 2018. I haven't seen anything yet that comes as a surprise in terms of the progress we're making. I think the public has the wrong impression of the technology because without seeing the growth in the space it appears as this is a sudden advancement rather than an incremental improvement and you can easily get the wrong impression about future progress extrapolating from an incomplete data set. > Skynet is not out of the question. Lots of reasons for trepidation. It's out of the question with this line of technology. This is not artificial general intelligence and there's no reasonable pathway for it to become AGI. I've used this analogy before but right now it's like we're at the dawn of flight and you're talking about going to the moon. No amount of incremental improvements to an airplane will get you a space traveling vehicle. You need a different technology entirely (a rocket) to make that happen.

ctoth 4y ago
I have been watching this space since Skip Thought Vectors in 2015. No. No one in 2018 suspected that large language models would smoothly scale up by simply increasing the number of parameters. There is no clear and obvious path from Attention is All you Need to InstructGPT, which just came out last February without hindsight. Point at a single person, let alone "everyone" who predicted we would have AI-based coding assistance and be integrating this technology into a search engine by 2023. Anyone at all. I'd love to read a paper or even a blog post from 2018 predicting half the things that work now. ! You can't. I've seen some hardcore goalpost moving before, but nothing as obviously provably wrong as "we're basically exactly where everyone predicted we'd be back in 2018." Is this on the path to AGI? I doubt it. You need some sort of actor-critic component likely, though the RLHF stuff is working way better than it has any right to and already is far more agentic than the pure dumb language model of a year ago.
- CSMastermind 4y ago
  > No one in 2018 suspected that large language models would smoothly scale up by simply increasing the number of parameters. https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf https://d4mucfpksywv.cloudfront.net/better-language-models/l... That was kind of the entire point of GPT-2. Computerphile summed it up pretty well on GPT-3's release: https://youtu.be/_8yVOC4ciXc https://youtu.be/_8yVOC4ciXc Here's some quotes from that video: "The thing about gpt2 is just that it was much bigger than anything that came before. It was more parameters and was kind of the point of that paper." ... "They made gpt2 because the curve wasn't leveling off. We've gone 117 times bigger than gpt2 and they're still not leveling off."

ctoth 4y ago

I have been watching this space since Skip Thought Vectors in 2015. No. No one in 2018 suspected that large language models would smoothly scale up by simply increasing the number of parameters. There is no clear and obvious path from Attention is All you Need to InstructGPT, which just came out last February without hindsight. Point at a single person, let alone "everyone" who predicted we would have AI-based coding assistance and be integrating this technology into a search engine by 2023. Anyone at all. I'd love to read a paper or even a blog post from 2018 predicting half the things that work now. ! You can't. I've seen some hardcore goalpost moving before, but nothing as obviously provably wrong as "we're basically exactly where everyone predicted we'd be back in 2018." Is this on the path to AGI? I doubt it. You need some sort of actor-critic component likely, though the RLHF stuff is working way better than it has any right to and already is far more agentic than the pure dumb language model of a year ago.

CSMastermind 4y ago
> No one in 2018 suspected that large language models would smoothly scale up by simply increasing the number of parameters. https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf https://d4mucfpksywv.cloudfront.net/better-language-models/l... That was kind of the entire point of GPT-2. Computerphile summed it up pretty well on GPT-3's release: https://youtu.be/_8yVOC4ciXc https://youtu.be/_8yVOC4ciXc Here's some quotes from that video: "The thing about gpt2 is just that it was much bigger than anything that came before. It was more parameters and was kind of the point of that paper." ... "They made gpt2 because the curve wasn't leveling off. We've gone 117 times bigger than gpt2 and they're still not leveling off."