2 ms·

Meh. The difference between GPT-2 and GPT-4 is not as revolutionary as you're making it seem. The ability to process images and text as input is impressive, and

by timr 4y ago

Meh. The difference between GPT-2 and GPT-4 is not as revolutionary as you're making it seem. The ability to process images and text as input is impressive, and also an evolutionary combination of things that were happening separately before.

crop_rotation 4y ago
It is revolutionary for many. GPT2 was just another "looks cool" thing, GPT4 is just so jaw dropping that I am scared what GPT5 would look like.
- famouswaffles 4y ago
  Yeah lol. I didn't want to argue further because hey it's his/her opinion. But saying GPT-2 to 4 in 4 years isn't revolutionary just seems contrarian to me. Bespoke NLP of any kind is out the door with a single model. That's fucking crazy.
  - timr 4y ago
    Yours is also just an opinion. But having a baseline understanding of what came before is important to having an informed opinion. To that end: GPT is not the first tech to combine "NLP" into a single model. For example, from 2019 [1]. This stuff is evolutionary. [1] https://pub.towardsai.net/unified-language-model-pre-training-for-natural-language-understanding-and-generation-f87dc226aa2 https://pub.towardsai.net/unified-language-model-pre-trainin...
    - famouswaffles 4y ago
      You linked a paper that came months after the release of GPT-2. It is also a modified transformer, the basis of most LLMs today and an architecture that was released only two years prior. and this is without mentioning that NLP is far from the only things LLMs do/excel at. Frankly, nothing you have said so far has made me think your opinion is more informed than mine.
      - timr 4y ago
        No. I linked to a blog article which mentions many different papers, many of which were from 2018 or before. ELMO, BERT, UNILM...all examples of multi-target NLP models that pre-dated GPT. I didn't dig this up because I thought the date on the webpage was some kind of rebuttal. I picked the first article I saw that mentioned models I knew about, with dates. > It is also a modified transformer Well, yes. Exactly. We're in the echo of that work, which was the core advancement that happened circa 2017. As I keep saying, it only seems revolutionary if you haven't been paying attention to the last 5 years.
famouswaffles 4y ago
If you say so
rvz 4y ago
Absolutely. This. This 'revolution' is quite fuelled by clever marketing and hype generated by Microsoft attempting to overthrow Google crown for better search. Unfortunately, that use-case has fallen flat on its face. With hallucinations, bullshitting and regurgitating falsehoods and it continues to be unable to reason transparently and like all neural networks are just black-boxes with limited explanation of it's decisions, still requiring humans to review the output just like 'AI systems' still requiring humans to sit behind the wheel of a so-called robo-taxi 'AI' driver. Sounds like typical AI snake-oil. The only 'safe' use-case for LLMs or even GPT-3 is 'summarization'. Everything else is complete hype; devoid of any reason to even justify the euphoric mania of one section of 'AI'.
- adamsmith143 4y ago
  I really doubt you understand LLMs at all if you think all they are good for is summarization. You will be shouting at your wall while the world changes outside.
  - rvz 4y ago
    > I really doubt you understand LLMs at all if you think all they are good for is summarization. A 'counter claim' without giving examples of refuting mine can be dismissed without any further explanation, since my point still remains un-refuted. Summarization of existing text is the only safe killer use-case for LLMs than the rest of the so-called 'applications' and the hype of applying it to everything, where it makes little sense other than attempting to jump on another hype train. I predict that the majority of these ChatGPT-enabled startups will be closing down in the long run, due to no product market fit, easily running out of money or another bigger company over-taking them.
    - lostmsu 4y ago
      There are samples all around, including it being able to solve newly generated logical puzzles. There was quite a bit about variations of river crossing.
    - adamsmith143 4y ago
adamsmith143 4y ago
I'm sorry but I find this take to be absurd. Across every metric GPT3 was a massive update past GPT2 and GPT4 soundly beats GPT3 by similar margins. Beyond OpenAI, AlphaGo was a massive milestone, people thought it would take decades until AI go was at GM levels, AlphaZero blitzed AlphaGo and MuZero blitzes both of them. AlphaFold was also a massive breakthough in protein folding. Idk what your "PhD" was in but everyone I know in the field is struggling to keep up with the pace of research so unless you are working on "Expert Systems" and trying to convince everyone that the past 10 years of DNN progress has been irrelevant I don't understand how you can have this point of view.
- version_five 4y ago
  Lol
- timr 4y ago
  > AlphaFold was also a massive breakthough in protein folding. My PhD was applying simpler ML models to problems in structural biology (this was a while ago, so transformers and other large neural models didn't exist then), including protein structure prediction. Most recently I worked in applying generative models and neural network classifiers to problems in drug discovery. I have also worked in IR (search) and NLP, but again with older technologies. The AlphaFold example is apt: yes, it outperformed methods at CASP. It was indeed a dramatic leap relative to those methods...which didn't work very well. The claims from laypeople that "protein folding is a solved problem" is total bunk, and yet we see that constantly. If you don't know the field, you will be fundamentally misled by statements like "breakthrough". For other areas (e.g. NLP), the definition of "progress" is also well quantified, and is not accurately captured by the breathless hype that surrounds this technology. Are these large transformers a dramatic improvement? Absolutely. No question. Are they the end of white-collar labor? No. That's ridiculous. Has this been an evolutionary change over the last decade or so? Not a smooth curve, but yes. The world didn't suddenly change overnight. > Idk what your "PhD" was in but everyone I know in the field is struggling to keep up with the pace of research so unless you are working on "Expert Systems" and trying to convince everyone that the past 10 years of DNN progress has been irrelevant Except, I'm not trying to convince anyone of that. I'm saying that these models are evolutionary, progress in research is always a form of punctuated equilibrium, and picking random points of progress across fields and extrapolating to the infinite, glorious future doesn't work well. That's what people are doing today.

by timr 4y ago

crop_rotation 4y ago

It is revolutionary for many. GPT2 was just another "looks cool" thing, GPT4 is just so jaw dropping that I am scared what GPT5 would look like.

famouswaffles 4y ago
Yeah lol. I didn't want to argue further because hey it's his/her opinion. But saying GPT-2 to 4 in 4 years isn't revolutionary just seems contrarian to me. Bespoke NLP of any kind is out the door with a single model. That's fucking crazy.
- timr 4y ago
  Yours is also just an opinion. But having a baseline understanding of what came before is important to having an informed opinion. To that end: GPT is not the first tech to combine "NLP" into a single model. For example, from 2019 [1]. This stuff is evolutionary. [1] https://pub.towardsai.net/unified-language-model-pre-training-for-natural-language-understanding-and-generation-f87dc226aa2 https://pub.towardsai.net/unified-language-model-pre-trainin...
  - famouswaffles 4y ago
    You linked a paper that came months after the release of GPT-2. It is also a modified transformer, the basis of most LLMs today and an architecture that was released only two years prior. and this is without mentioning that NLP is far from the only things LLMs do/excel at. Frankly, nothing you have said so far has made me think your opinion is more informed than mine.
    - timr 4y ago
      No. I linked to a blog article which mentions many different papers, many of which were from 2018 or before. ELMO, BERT, UNILM...all examples of multi-target NLP models that pre-dated GPT. I didn't dig this up because I thought the date on the webpage was some kind of rebuttal. I picked the first article I saw that mentioned models I knew about, with dates. > It is also a modified transformer Well, yes. Exactly. We're in the echo of that work, which was the core advancement that happened circa 2017. As I keep saying, it only seems revolutionary if you haven't been paying attention to the last 5 years.

famouswaffles 4y ago

If you say so

rvz 4y ago

Absolutely. This. This 'revolution' is quite fuelled by clever marketing and hype generated by Microsoft attempting to overthrow Google crown for better search. Unfortunately, that use-case has fallen flat on its face. With hallucinations, bullshitting and regurgitating falsehoods and it continues to be unable to reason transparently and like all neural networks are just black-boxes with limited explanation of it's decisions, still requiring humans to review the output just like 'AI systems' still requiring humans to sit behind the wheel of a so-called robo-taxi 'AI' driver. Sounds like typical AI snake-oil. The only 'safe' use-case for LLMs or even GPT-3 is 'summarization'. Everything else is complete hype; devoid of any reason to even justify the euphoric mania of one section of 'AI'.

adamsmith143 4y ago
I really doubt you understand LLMs at all if you think all they are good for is summarization. You will be shouting at your wall while the world changes outside.
- rvz 4y ago
  > I really doubt you understand LLMs at all if you think all they are good for is summarization. A 'counter claim' without giving examples of refuting mine can be dismissed without any further explanation, since my point still remains un-refuted. Summarization of existing text is the only safe killer use-case for LLMs than the rest of the so-called 'applications' and the hype of applying it to everything, where it makes little sense other than attempting to jump on another hype train. I predict that the majority of these ChatGPT-enabled startups will be closing down in the long run, due to no product market fit, easily running out of money or another bigger company over-taking them.
  - lostmsu 4y ago
    There are samples all around, including it being able to solve newly generated logical puzzles. There was quite a bit about variations of river crossing.
  - adamsmith143 4y ago

adamsmith143 4y ago

I'm sorry but I find this take to be absurd. Across every metric GPT3 was a massive update past GPT2 and GPT4 soundly beats GPT3 by similar margins. Beyond OpenAI, AlphaGo was a massive milestone, people thought it would take decades until AI go was at GM levels, AlphaZero blitzed AlphaGo and MuZero blitzes both of them. AlphaFold was also a massive breakthough in protein folding. Idk what your "PhD" was in but everyone I know in the field is struggling to keep up with the pace of research so unless you are working on "Expert Systems" and trying to convince everyone that the past 10 years of DNN progress has been irrelevant I don't understand how you can have this point of view.

version_five 4y ago
Lol
timr 4y ago
> AlphaFold was also a massive breakthough in protein folding. My PhD was applying simpler ML models to problems in structural biology (this was a while ago, so transformers and other large neural models didn't exist then), including protein structure prediction. Most recently I worked in applying generative models and neural network classifiers to problems in drug discovery. I have also worked in IR (search) and NLP, but again with older technologies. The AlphaFold example is apt: yes, it outperformed methods at CASP. It was indeed a dramatic leap relative to those methods...which didn't work very well. The claims from laypeople that "protein folding is a solved problem" is total bunk, and yet we see that constantly. If you don't know the field, you will be fundamentally misled by statements like "breakthrough". For other areas (e.g. NLP), the definition of "progress" is also well quantified, and is not accurately captured by the breathless hype that surrounds this technology. Are these large transformers a dramatic improvement? Absolutely. No question. Are they the end of white-collar labor? No. That's ridiculous. Has this been an evolutionary change over the last decade or so? Not a smooth curve, but yes. The world didn't suddenly change overnight. > Idk what your "PhD" was in but everyone I know in the field is struggling to keep up with the pace of research so unless you are working on "Expert Systems" and trying to convince everyone that the past 10 years of DNN progress has been irrelevant Except, I'm not trying to convince anyone of that. I'm saying that these models are evolutionary, progress in research is always a form of punctuated equilibrium, and picking random points of progress across fields and extrapolating to the infinite, glorious future doesn't work well. That's what people are doing today.