2 ms·

First reactions before reading in detail: Does it give more lift than just using the training data from the diffusion model? You can search through LAION 5B in

First reactions before reading in detail: Does it give more lift than just using the training data from the diffusion model? You can search through LAION 5B in Clip embedding space and pull out lots of new training data without ever using a generative model. Are they able to use prompts to generate augmentations around whatever invariant features - different colors or positions or scenes- in order to train better?

minihat 3y ago
This is the right question. It is plausible to me that this is a way of condensing a vast training set into a reduced set of high quality examples.
- version_five 3y ago
  You could also consider some kinds of sampling of the dataset - basically not including images that are too close together in whatever space. There are also data distillation methods that are supposed to generate a smaller training set - the ones I've seen don't generate images that look like real training samples though. I think it's definitely an interesting thing to study - it would be worth exploring what specifically the generative model can add. I've actually done something like this with GANs, but the gains came largely from generating plausible training images that were augmentations around some invariance, as I mentioned above.
sharemywin 3y ago
is there an un-similarity search using clip. some reasonably efficient way to pull the X un-similar embeddings(to Y dataset) in Z dataset?
- version_five 3y ago
  Un-similarity would generally just mean further away in the latent space. I don't know if there are common heuristics used for this - taking the furthest away would give you outliers, and then probably lots of members of whatever class was furthest away. So you'd probably be best first clustering in the latent space and then sampling from each cluster.

2 ms·

First reactions before reading in detail: Does it give more lift than just using the training data from the diffusion model? You can search through LAION 5B in

by version_five 3y ago

minihat 3y ago
This is the right question. It is plausible to me that this is a way of condensing a vast training set into a reduced set of high quality examples.
- version_five 3y ago
  You could also consider some kinds of sampling of the dataset - basically not including images that are too close together in whatever space. There are also data distillation methods that are supposed to generate a smaller training set - the ones I've seen don't generate images that look like real training samples though. I think it's definitely an interesting thing to study - it would be worth exploring what specifically the generative model can add. I've actually done something like this with GANs, but the gains came largely from generating plausible training images that were augmentations around some invariance, as I mentioned above.
sharemywin 3y ago
is there an un-similarity search using clip. some reasonably efficient way to pull the X un-similar embeddings(to Y dataset) in Z dataset?
- version_five 3y ago
  Un-similarity would generally just mean further away in the latent space. I don't know if there are common heuristics used for this - taking the furthest away would give you outliers, and then probably lots of members of whatever class was furthest away. So you'd probably be best first clustering in the latent space and then sampling from each cluster.