3 ms·

I'm sorry but I find this take to be absurd. Across every metric GPT3 was a massive update past GPT2 and GPT4 soundly beats GPT3 by similar margins. Beyond Open

I'm sorry but I find this take to be absurd. Across every metric GPT3 was a massive update past GPT2 and GPT4 soundly beats GPT3 by similar margins. Beyond OpenAI, AlphaGo was a massive milestone, people thought it would take decades until AI go was at GM levels, AlphaZero blitzed AlphaGo and MuZero blitzes both of them. AlphaFold was also a massive breakthough in protein folding. Idk what your "PhD" was in but everyone I know in the field is struggling to keep up with the pace of research so unless you are working on "Expert Systems" and trying to convince everyone that the past 10 years of DNN progress has been irrelevant I don't understand how you can have this point of view.

version_five 4y ago
Lol
timr 4y ago
> AlphaFold was also a massive breakthough in protein folding. My PhD was applying simpler ML models to problems in structural biology (this was a while ago, so transformers and other large neural models didn't exist then), including protein structure prediction. Most recently I worked in applying generative models and neural network classifiers to problems in drug discovery. I have also worked in IR (search) and NLP, but again with older technologies. The AlphaFold example is apt: yes, it outperformed methods at CASP. It was indeed a dramatic leap relative to those methods...which didn't work very well. The claims from laypeople that "protein folding is a solved problem" is total bunk, and yet we see that constantly. If you don't know the field, you will be fundamentally misled by statements like "breakthrough". For other areas (e.g. NLP), the definition of "progress" is also well quantified, and is not accurately captured by the breathless hype that surrounds this technology. Are these large transformers a dramatic improvement? Absolutely. No question. Are they the end of white-collar labor? No. That's ridiculous. Has this been an evolutionary change over the last decade or so? Not a smooth curve, but yes. The world didn't suddenly change overnight. > Idk what your "PhD" was in but everyone I know in the field is struggling to keep up with the pace of research so unless you are working on "Expert Systems" and trying to convince everyone that the past 10 years of DNN progress has been irrelevant Except, I'm not trying to convince anyone of that. I'm saying that these models are evolutionary, progress in research is always a form of punctuated equilibrium, and picking random points of progress across fields and extrapolating to the infinite, glorious future doesn't work well. That's what people are doing today.
- adamsmith143 4y ago
  You're making a different argument to what you started the thread with. > Having worked in the space on-and-off since my PhD, the pace is not "overwhelming". You just started paying attention to it. >ChatGPT and Stable Diffusion are good, but they're incremental gains over what came before, which were incremental gains over what came before that >The AlphaFold example is apt: yes, it outperformed methods at CASP. It was indeed a dramatic leap >Are these large transformers a dramatic improvement? Absolutely. No question. >Has this been an evolutionary change over the last decade or so? Not a smooth curve, but yes. The world didn't suddenly change overnight. Starting with AlexNet there have been numerous discontinuous improvements in CV, NLP, Game Playing, etc. It's just not true to say otherwise.
  - timr 4y ago
    > You're making a different argument to what you started the thread with. No, I'm not. The field has been making gradual improvement for a long time, with transformers (ca 2017) being a notable jump forward. Research is always a form of punctuated equilibrium. > Starting with AlexNet there have been numerous discontinuous improvements in CV, NLP, Game Playing, etc. It's just not true to say otherwise. Nor did I. I said that these models have been advancing gradually for a long time. The fact that particular steps can be characterized as "jumps" when viewed in isolation is not a rebuttal.

version_five 4y ago

Lol

timr 4y ago

> AlphaFold was also a massive breakthough in protein folding. My PhD was applying simpler ML models to problems in structural biology (this was a while ago, so transformers and other large neural models didn't exist then), including protein structure prediction. Most recently I worked in applying generative models and neural network classifiers to problems in drug discovery. I have also worked in IR (search) and NLP, but again with older technologies. The AlphaFold example is apt: yes, it outperformed methods at CASP. It was indeed a dramatic leap relative to those methods...which didn't work very well. The claims from laypeople that "protein folding is a solved problem" is total bunk, and yet we see that constantly. If you don't know the field, you will be fundamentally misled by statements like "breakthrough". For other areas (e.g. NLP), the definition of "progress" is also well quantified, and is not accurately captured by the breathless hype that surrounds this technology. Are these large transformers a dramatic improvement? Absolutely. No question. Are they the end of white-collar labor? No. That's ridiculous. Has this been an evolutionary change over the last decade or so? Not a smooth curve, but yes. The world didn't suddenly change overnight. > Idk what your "PhD" was in but everyone I know in the field is struggling to keep up with the pace of research so unless you are working on "Expert Systems" and trying to convince everyone that the past 10 years of DNN progress has been irrelevant Except, I'm not trying to convince anyone of that. I'm saying that these models are evolutionary, progress in research is always a form of punctuated equilibrium, and picking random points of progress across fields and extrapolating to the infinite, glorious future doesn't work well. That's what people are doing today.

adamsmith143 4y ago
You're making a different argument to what you started the thread with. > Having worked in the space on-and-off since my PhD, the pace is not "overwhelming". You just started paying attention to it. >ChatGPT and Stable Diffusion are good, but they're incremental gains over what came before, which were incremental gains over what came before that >The AlphaFold example is apt: yes, it outperformed methods at CASP. It was indeed a dramatic leap >Are these large transformers a dramatic improvement? Absolutely. No question. >Has this been an evolutionary change over the last decade or so? Not a smooth curve, but yes. The world didn't suddenly change overnight. Starting with AlexNet there have been numerous discontinuous improvements in CV, NLP, Game Playing, etc. It's just not true to say otherwise.
- timr 4y ago
  > You're making a different argument to what you started the thread with. No, I'm not. The field has been making gradual improvement for a long time, with transformers (ca 2017) being a notable jump forward. Research is always a form of punctuated equilibrium. > Starting with AlexNet there have been numerous discontinuous improvements in CV, NLP, Game Playing, etc. It's just not true to say otherwise. Nor did I. I said that these models have been advancing gradually for a long time. The fact that particular steps can be characterized as "jumps" when viewed in isolation is not a rebuttal.