5 ms·

It boggles my mind how people can think this when Alexnet stepped on the scene over a decade ago and not much has fundamentally improved since then. OpenAI just

It boggles my mind how people can think this when Alexnet stepped on the scene over a decade ago and not much has fundamentally improved since then. OpenAI just said they’re not planning on training GPT-5 for awhile because the scaling trick has hit diminishing returns. This isn’t “AGI is 2 years away”, it’s we’ve milked this advancement as best we can and bigger advancements are going to need a fundamental paradigm shift which usually doesn’t happen overnight. Not to say GPT-4 or vision models aren’t amazingly useful, but they’re no silver bullet. I’ve been using GPT-3 for awhile now and it makes a lot of fundamentally stupid mistakes.

est31 3y ago
What about Transformers? And GPT-3 is fundamental improvement to earlier LLMs. Same goes for speech recognition which went from highly parametric to end to end in the last decade. Many other things like stable diffusion, TTS, etc. It's like github's UI. If you use it daily, you maybe notice a little change here or there, but it doesn't disrupt you much. But now if one checks github screenshots from 10 years ago they look so different. Say for example Youtube's auto transcription feature. I remember when it launched, it wasn't that bad but got many things wrong. Now it's really good, still making mistakes but gotten much better.
dave_sullivan 3y ago
"Not much" has fundamentally improved since Alexnet??? Honestly half the comments in this thread boggle my mind. What is even the word for someone that is shown something amazing that they don't understand and then dismisses it on completely superficial grounds? Like if I go back in time and show someone a computer and they think it's just a typewriter and who ever needed one of those anyway? It's like ignorance but in the most willfull, dismissive way possible. Gross.
- flangola7 3y ago
  The straight denialism around what transformer models can do has been very disappointing, though maybe not unsurprising. Acknowledging what that means acknowledging that your world is about to soon change in massive and unpredictable ways and the future you had planned and expected is not going to happen.
- rjh29 3y ago
  And the entire point of this article is to show what has changed since AlexNet (spoiler: a lot!), let alone GPT-3/DALL-E/Midjourney/Whisper/ChatGPT/Imagen/Imagen Video which aren't even in this article! I wonder if what we're seeing is anxiety - that AIs are going to take my job, or create huge problems with spam/deepfakes - manifesting in frankly ignorant, jaded comments that completely ignore the huge positive potential.
- chaxor 3y ago
  People have been rewarded in the past for being skeptical, and have learned this trick to a detriment to their own intellect. It is a very easy trap to confuse skepticism with expertise on the internet, and HN is one of the worst about this - most of the comments that feign skepticism barely even understand the topic they are commenting upon anymore, and are simply generating skeptical words because that's what has rewarded them in the past. There is also unsubstantiated hype, which is not helpful either. Occasionally someone actually has read a few hundred fundamental papers on ML and can give an actual educated response, but it is quite rare. Typically they don't feign skepticism, but rather notice there are noteworthy improvements provided by metaRL and RLHF, etc.
  - jackblemming 3y ago
    I work in vision. Go look up the imagenet leaderboard. Look at the results of Alexnet vs the top result today. The trend is a log line. The top contending architectures still include CNNs trained on backprop, they’ve just had a decade of tricks applied to eek out some improvements. The transformer based vision models aren’t much better. Talk to any machine learning expert and they’ll tell you the math and fundamentals haven’t really changed since the 90s, we’ve just gotten better at scaling. Transformers came onto the scene half a decade ago and we could scale them much better than CNNs, but like CNNs of today, we’ve hit the diminishing returns limit. Maybe look at actual data instead of being dismissive to different opinions.
    - olddustytrail 3y ago
      Ok, I checked. Alexnet 63%. Top rated 91%. That's a big difference. And what did you expect other than a log curve. The maximum is obviously 100%.
      - jackblemming 3y ago
        So interestingly, you can actually have linear or exponential curves on your way from 0 to 100. And you completely ignored how the basic building blocks and algorithms are more or less the same. I think I'm done discussing with non-experts.
- jackblemming 3y ago
  Vision models are still CNNs trained with backprop. Now they’ve begun to incorporate transformers into vision task, but the results aren’t x10 better. Please explain to me how vision models today are massively different than Alexnet and not just a bunch of slight optimizations and tricks to eek out some marginal accuracy improvements.
  - versteegen 3y ago
    If you're talking about vision, fine. But you were replying to a comment about ChatGPT and then brought up AGI. No offence, but although some vision tasks are AI-hard, in general vision has minimal to do with AGI. That's why I quit the field. Transformers in vision may not be very interesting, but they certainly are a breakthrough in language.
deleted 3y ago
[deleted]

est31 3y ago

What about Transformers? And GPT-3 is fundamental improvement to earlier LLMs. Same goes for speech recognition which went from highly parametric to end to end in the last decade. Many other things like stable diffusion, TTS, etc. It's like github's UI. If you use it daily, you maybe notice a little change here or there, but it doesn't disrupt you much. But now if one checks github screenshots from 10 years ago they look so different. Say for example Youtube's auto transcription feature. I remember when it launched, it wasn't that bad but got many things wrong. Now it's really good, still making mistakes but gotten much better.

dave_sullivan 3y ago

"Not much" has fundamentally improved since Alexnet??? Honestly half the comments in this thread boggle my mind. What is even the word for someone that is shown something amazing that they don't understand and then dismisses it on completely superficial grounds? Like if I go back in time and show someone a computer and they think it's just a typewriter and who ever needed one of those anyway? It's like ignorance but in the most willfull, dismissive way possible. Gross.

flangola7 3y ago
The straight denialism around what transformer models can do has been very disappointing, though maybe not unsurprising. Acknowledging what that means acknowledging that your world is about to soon change in massive and unpredictable ways and the future you had planned and expected is not going to happen.
rjh29 3y ago
And the entire point of this article is to show what has changed since AlexNet (spoiler: a lot!), let alone GPT-3/DALL-E/Midjourney/Whisper/ChatGPT/Imagen/Imagen Video which aren't even in this article! I wonder if what we're seeing is anxiety - that AIs are going to take my job, or create huge problems with spam/deepfakes - manifesting in frankly ignorant, jaded comments that completely ignore the huge positive potential.
chaxor 3y ago
People have been rewarded in the past for being skeptical, and have learned this trick to a detriment to their own intellect. It is a very easy trap to confuse skepticism with expertise on the internet, and HN is one of the worst about this - most of the comments that feign skepticism barely even understand the topic they are commenting upon anymore, and are simply generating skeptical words because that's what has rewarded them in the past. There is also unsubstantiated hype, which is not helpful either. Occasionally someone actually has read a few hundred fundamental papers on ML and can give an actual educated response, but it is quite rare. Typically they don't feign skepticism, but rather notice there are noteworthy improvements provided by metaRL and RLHF, etc.
- jackblemming 3y ago
  I work in vision. Go look up the imagenet leaderboard. Look at the results of Alexnet vs the top result today. The trend is a log line. The top contending architectures still include CNNs trained on backprop, they’ve just had a decade of tricks applied to eek out some improvements. The transformer based vision models aren’t much better. Talk to any machine learning expert and they’ll tell you the math and fundamentals haven’t really changed since the 90s, we’ve just gotten better at scaling. Transformers came onto the scene half a decade ago and we could scale them much better than CNNs, but like CNNs of today, we’ve hit the diminishing returns limit. Maybe look at actual data instead of being dismissive to different opinions.
  - olddustytrail 3y ago
    Ok, I checked. Alexnet 63%. Top rated 91%. That's a big difference. And what did you expect other than a log curve. The maximum is obviously 100%.
    - jackblemming 3y ago
      So interestingly, you can actually have linear or exponential curves on your way from 0 to 100. And you completely ignored how the basic building blocks and algorithms are more or less the same. I think I'm done discussing with non-experts.
jackblemming 3y ago
Vision models are still CNNs trained with backprop. Now they’ve begun to incorporate transformers into vision task, but the results aren’t x10 better. Please explain to me how vision models today are massively different than Alexnet and not just a bunch of slight optimizations and tricks to eek out some marginal accuracy improvements.
- versteegen 3y ago
  If you're talking about vision, fine. But you were replying to a comment about ChatGPT and then brought up AGI. No offence, but although some vision tasks are AI-hard, in general vision has minimal to do with AGI. That's why I quit the field. Transformers in vision may not be very interesting, but they certainly are a breakthrough in language.

deleted 3y ago

[deleted]