4 ms·

At least the big public display of this tech seems to me that it's mostly merging photos in interesting ways. That the 'seed' comes from a word is not hugely in

At least the big public display of this tech seems to me that it's mostly merging photos in interesting ways. That the 'seed' comes from a word is not hugely interesting to me. I'm actually more curious if we could parse the underlying logic that ultimately it emulates to merge those images together. It 'looks like' something kind of sophisticated is being modelled with AI but there's some nice algorithms hidden in there.

SleekEagle 5y ago
Luckily the use of Transformer models makes what's going on under the hood a bit more interpretable, but I think the fundamental part at which ideas are merged is translating from CLIP text embeddings to CLIP image embeddings. The training principle of CLIP is very simple, but intuitively understanding how the diffusion prior maps between semantically similar textual and visual representations is a bit more unclear (if that's even a well-formulated question!)
ma2rten 5y ago
It doesn’t merge images. It generates them from scratch. Sure it’s trained on a corpus of existing images, but I don’t think it „merges“ them any more than human artists do with images they have seen in their lifetime.
- jollybean 5y ago
  I don't believe 'creating them' is the write word. 'Merging' them is probably a bad choice of words on my part. More like 'averaging them' and finding variations from vast inputs. Which is more a long the lines of what I mean.
- xsmasher 5y ago
  From one of the linked papers, it does both - generate from scratch (zero-shot), and edit existing template images. > While our model can render a wide variety of text prompts zero-shot, it can can have difficulty producing realistic im ages for complex prompts. Therefore, we provide our model with editing capabilities in addition to zero-shot generation, which allows humans to iteratively improve model samples until they match more complex prompts. Specifically, we fine-tune our model to perform image inpainting, finding that it is capable of making realistic edits to existing images using natural language prompts. Unless that only applies to GLIDE and not to DAL-E?
alar44 5y ago
Well that's a absolutely not what's happening. It seems like you haven't done any reading in this space, so I'm not even sure what to link for you.
- jollybean 5y ago
  'Merging' was a poor choice of words on my part, but I'm aware of what it does.

SleekEagle 5y ago

Luckily the use of Transformer models makes what's going on under the hood a bit more interpretable, but I think the fundamental part at which ideas are merged is translating from CLIP text embeddings to CLIP image embeddings. The training principle of CLIP is very simple, but intuitively understanding how the diffusion prior maps between semantically similar textual and visual representations is a bit more unclear (if that's even a well-formulated question!)

ma2rten 5y ago

It doesn’t merge images. It generates them from scratch. Sure it’s trained on a corpus of existing images, but I don’t think it „merges“ them any more than human artists do with images they have seen in their lifetime.

jollybean 5y ago
I don't believe 'creating them' is the write word. 'Merging' them is probably a bad choice of words on my part. More like 'averaging them' and finding variations from vast inputs. Which is more a long the lines of what I mean.
xsmasher 5y ago
From one of the linked papers, it does both - generate from scratch (zero-shot), and edit existing template images. > While our model can render a wide variety of text prompts zero-shot, it can can have difficulty producing realistic im ages for complex prompts. Therefore, we provide our model with editing capabilities in addition to zero-shot generation, which allows humans to iteratively improve model samples until they match more complex prompts. Specifically, we fine-tune our model to perform image inpainting, finding that it is capable of making realistic edits to existing images using natural language prompts. Unless that only applies to GLIDE and not to DAL-E?

alar44 5y ago

Well that's a absolutely not what's happening. It seems like you haven't done any reading in this space, so I'm not even sure what to link for you.

jollybean 5y ago
'Merging' was a poor choice of words on my part, but I'm aware of what it does.