4 ms·

Are we finally past the AI winter? We seem to be seeing major advances at least once a year. I recall there was a bit of a lull after GPT3, but clearly the boun

Are we finally past the AI winter? We seem to be seeing major advances at least once a year. I recall there was a bit of a lull after GPT3, but clearly the boundaries of AI are expanding ridiculously fast.

SleekEagle 5y ago
GPT-3 was released 2 years ago, and in that time CLIP, GLIDE, and DALL-Es 1 and 2 have been released. All of this is just from OpenAI too! DL research is cranking along as quickly as ever imo!
- lurker619 5y ago
  Just need a music one please.
  - gwern 5y ago
    Jukebox. If you listen to Jukebox samples, recall that that was quite a while ago in dog/DL years, and imagine what the DALL-E 2 equivalent would be for a Jukebox 2...
    - p1esk 5y ago
      I'm surprised no one has tried to launch a music generation startup based on Jukebox. I'd be interested in collaboration if anyone wants to work on it (and has compute resources).
  - donkarma 5y ago
    You're going to be waiting a while, even now these models can't get the details right. Look at all the example pictures, everything is just wrong. You can obviously see what it is trying to get at, but it can't get there. Another example of the last 10% taking 90% of the time.
mellosouls 5y ago
In some ways (narrow AI), yes, its been a fantastic few years including tools like the one in context. In the important way that the AI winter originally referred to though, no, there doesn't seem to have been any progress towards AGI.
- SleekEagle 5y ago
  Was the original AI winter with reference to AGI? I thought it was in reference to the resulting lack of research and interest after the "bubble bust". If we're not close to AGI now I can't imagine researchers 40 years ago really thought AGI was around the corner, right? Just curious, I'm not an expert on the history of ML!
  - mellosouls 5y ago
    I think there have been several really, and they tend to follow hype periods, which over-promise. I do think the last few years have been more productive than previous periods in advancing narrow AI, and to be fair to those researchers who just get on with the work, it is not on them if the advances are over-sold by others.
  - visarga 5y ago
    > If we're not close to AGI now I bet we're closer than most people think. Instruct GPT-3 can do semantic tasks just as efficiently as DALL-E 2 can draw. NLP tasks that took whole teams multiple years can be simply described in a few words and they work right away. The entry barrier to implement new tasks will get very low. The large models will be the new operating system. This means more investments and data, leading to new improvements. I believe GPT-3 is already close to median human level on most semantic tasks that fit in a 4000 token window. I'm researching how to use it right now for a variety of tasks, it just works from plain text requirements with no training.
    - mellosouls 5y ago
      GPT-3 is brilliant, undeniably. But there's a quantum leap or two from (say) mindlessly producing comments that can occasionally fool readers on HN (as it has been used to do in the past) to it consciously joining in the conversation of its own volition and curiosity, then zoning out on Netflix while half-worrying about the future for GPT-4 jr and idly planning it's next server room refit.
jollybean 5y ago
At least the big public display of this tech seems to me that it's mostly merging photos in interesting ways. That the 'seed' comes from a word is not hugely interesting to me. I'm actually more curious if we could parse the underlying logic that ultimately it emulates to merge those images together. It 'looks like' something kind of sophisticated is being modelled with AI but there's some nice algorithms hidden in there.
- SleekEagle 5y ago
  Luckily the use of Transformer models makes what's going on under the hood a bit more interpretable, but I think the fundamental part at which ideas are merged is translating from CLIP text embeddings to CLIP image embeddings. The training principle of CLIP is very simple, but intuitively understanding how the diffusion prior maps between semantically similar textual and visual representations is a bit more unclear (if that's even a well-formulated question!)
- ma2rten 5y ago
  It doesn’t merge images. It generates them from scratch. Sure it’s trained on a corpus of existing images, but I don’t think it „merges“ them any more than human artists do with images they have seen in their lifetime.
  - jollybean 5y ago
    I don't believe 'creating them' is the write word. 'Merging' them is probably a bad choice of words on my part. More like 'averaging them' and finding variations from vast inputs. Which is more a long the lines of what I mean.
  - xsmasher 5y ago
    From one of the linked papers, it does both - generate from scratch (zero-shot), and edit existing template images. > While our model can render a wide variety of text prompts zero-shot, it can can have difficulty producing realistic im ages for complex prompts. Therefore, we provide our model with editing capabilities in addition to zero-shot generation, which allows humans to iteratively improve model samples until they match more complex prompts. Specifically, we fine-tune our model to perform image inpainting, finding that it is capable of making realistic edits to existing images using natural language prompts. Unless that only applies to GLIDE and not to DAL-E?
redredrobot 5y ago
There has not been an AI winter in at least a decade, arguably more.
- Polygator 5y ago
  I guess he's referring to the fact that the glut of investment around 2017-2018 was followed by disappointment due to startups overpromising. I agree that from the technical side (I mostly follow NLP, might be different in other subfields) there's been no hint of a winter.
- Tossrock 5y ago
  Indeed, and I called it 7 years ago: https://news.ycombinator.com/item?id=9882217 https://news.ycombinator.com/item?id=9882217
Der_Einzige 5y ago
I resent this notion that AI doesn't advance if we aren't making new larger and larger foundation models. Even during that lull between GPT3 and DALL-E/CLIP, there was tons of truly wonderful advances in AI...

SleekEagle 5y ago

GPT-3 was released 2 years ago, and in that time CLIP, GLIDE, and DALL-Es 1 and 2 have been released. All of this is just from OpenAI too! DL research is cranking along as quickly as ever imo!

lurker619 5y ago
Just need a music one please.
- gwern 5y ago
  Jukebox. If you listen to Jukebox samples, recall that that was quite a while ago in dog/DL years, and imagine what the DALL-E 2 equivalent would be for a Jukebox 2...
  - p1esk 5y ago
    I'm surprised no one has tried to launch a music generation startup based on Jukebox. I'd be interested in collaboration if anyone wants to work on it (and has compute resources).
- donkarma 5y ago
  You're going to be waiting a while, even now these models can't get the details right. Look at all the example pictures, everything is just wrong. You can obviously see what it is trying to get at, but it can't get there. Another example of the last 10% taking 90% of the time.

mellosouls 5y ago

In some ways (narrow AI), yes, its been a fantastic few years including tools like the one in context. In the important way that the AI winter originally referred to though, no, there doesn't seem to have been any progress towards AGI.

SleekEagle 5y ago
Was the original AI winter with reference to AGI? I thought it was in reference to the resulting lack of research and interest after the "bubble bust". If we're not close to AGI now I can't imagine researchers 40 years ago really thought AGI was around the corner, right? Just curious, I'm not an expert on the history of ML!
- mellosouls 5y ago
  I think there have been several really, and they tend to follow hype periods, which over-promise. I do think the last few years have been more productive than previous periods in advancing narrow AI, and to be fair to those researchers who just get on with the work, it is not on them if the advances are over-sold by others.
- visarga 5y ago
  > If we're not close to AGI now I bet we're closer than most people think. Instruct GPT-3 can do semantic tasks just as efficiently as DALL-E 2 can draw. NLP tasks that took whole teams multiple years can be simply described in a few words and they work right away. The entry barrier to implement new tasks will get very low. The large models will be the new operating system. This means more investments and data, leading to new improvements. I believe GPT-3 is already close to median human level on most semantic tasks that fit in a 4000 token window. I'm researching how to use it right now for a variety of tasks, it just works from plain text requirements with no training.
  - mellosouls 5y ago
    GPT-3 is brilliant, undeniably. But there's a quantum leap or two from (say) mindlessly producing comments that can occasionally fool readers on HN (as it has been used to do in the past) to it consciously joining in the conversation of its own volition and curiosity, then zoning out on Netflix while half-worrying about the future for GPT-4 jr and idly planning it's next server room refit.

jollybean 5y ago

At least the big public display of this tech seems to me that it's mostly merging photos in interesting ways. That the 'seed' comes from a word is not hugely interesting to me. I'm actually more curious if we could parse the underlying logic that ultimately it emulates to merge those images together. It 'looks like' something kind of sophisticated is being modelled with AI but there's some nice algorithms hidden in there.

SleekEagle 5y ago
Luckily the use of Transformer models makes what's going on under the hood a bit more interpretable, but I think the fundamental part at which ideas are merged is translating from CLIP text embeddings to CLIP image embeddings. The training principle of CLIP is very simple, but intuitively understanding how the diffusion prior maps between semantically similar textual and visual representations is a bit more unclear (if that's even a well-formulated question!)
ma2rten 5y ago
It doesn’t merge images. It generates them from scratch. Sure it’s trained on a corpus of existing images, but I don’t think it „merges“ them any more than human artists do with images they have seen in their lifetime.
- jollybean 5y ago
  I don't believe 'creating them' is the write word. 'Merging' them is probably a bad choice of words on my part. More like 'averaging them' and finding variations from vast inputs. Which is more a long the lines of what I mean.
- xsmasher 5y ago
  From one of the linked papers, it does both - generate from scratch (zero-shot), and edit existing template images. > While our model can render a wide variety of text prompts zero-shot, it can can have difficulty producing realistic im ages for complex prompts. Therefore, we provide our model with editing capabilities in addition to zero-shot generation, which allows humans to iteratively improve model samples until they match more complex prompts. Specifically, we fine-tune our model to perform image inpainting, finding that it is capable of making realistic edits to existing images using natural language prompts. Unless that only applies to GLIDE and not to DAL-E?

redredrobot 5y ago

There has not been an AI winter in at least a decade, arguably more.

Polygator 5y ago
I guess he's referring to the fact that the glut of investment around 2017-2018 was followed by disappointment due to startups overpromising. I agree that from the technical side (I mostly follow NLP, might be different in other subfields) there's been no hint of a winter.
Tossrock 5y ago
Indeed, and I called it 7 years ago: https://news.ycombinator.com/item?id=9882217 https://news.ycombinator.com/item?id=9882217

Der_Einzige 5y ago

I resent this notion that AI doesn't advance if we aren't making new larger and larger foundation models. Even during that lull between GPT3 and DALL-E/CLIP, there was tons of truly wonderful advances in AI...