3 ms·

> So to the extent that numbers somehow “exist” out there in reality apart from what we humans think or say about them, every digital file that you could put on

> So to the extent that numbers somehow “exist” out there in reality apart from what we humans think or say about them, every digital file that you could put on a computer — every iTunes music download, every movie, every picture, every podcast — already “exists” on the regular old integer number line we all know from grade school. I was once interested in the philosophy and history of numbers. The idea of coming up with counting symbols and concepts is deeper than what we might think. It's interesting to see what others think about the existence of numbers in the "real" world (pun not intended).

Natsu 4y ago
The weird part to me is that this makes me imagine a high-dimensional space where we can have some vector that lets you go from some starting position, like a picture of a muffin, and then take that in a "dog-like" direction. And yet I've seen the weird "dog or muffin" [1] images that make you realize there's not as clean a separation as one might imagine with some pictures. [1] See, e.g. https://www.freecodecamp.org/news/chihuahua-or-muffin-my-search-for-the-best-computer-vision-api-cbda4d6b425d/ https://www.freecodecamp.org/news/chihuahua-or-muffin-my-sea...
- nl 4y ago
  One of the issues with the current stable diffusion img2img implementation is that this is NOT what it does (but should!) Instead it takes the image then tries to denoise it using the conditioned latents of the prompt, and the "strength" of the prompt determines how many denoising loops it does.
  - Natsu 4y ago
    Could it be made to do that?
    - ShamelessC 4y ago
      Definitely! The model accepts CLIP embeds, so you could average the prompt embedding with the image embedding - and then do some sort of top-k selection over a few of these image embeddings to generate "variations" of an image. I think this is how DALLE-2 does it. Also, the model would benefit from being explicitly finetuned for the task of "inpainting". This is where you randomly erase parts of an image and force the model to predict what is missing based on the prompt. This allows one to edit existing images, filling in the gaps with what they need. This is technically already supported - but uses a similar "hack" as described above by sampling some noise into an existing image and starting a little bit later in the denoising schedule. Since the model wasn't finetuned for this task - a more generic method is used that doesn't work as well.
      - nl 4y ago
        I've made an attempt at image embedding averaging. That is insufficient in itself but it was a completely naive implementation. It really needs to do the denoising from one image to another and vice-versa over a few iterations. Or that's what I'm working on next anyway.
        Natsu 4y ago
        How does denoising work, exactly and why does it seem necessary to get good results?
        ShamelessC 4y ago
        tl;dr it allows the model to focus on different frequencies of pixel-space at different time steps. Time step zero being the low frequency/low-detail features and time step 1000 being a completely “filled out”, detailed image including all the high frequency features. Doing so in discretely many steps makes the problem tractable and differentiable. Gross oversimplification, but that’s the gist.
        Natsu 4y ago
        Different frequencies? Does that mean it's doing something like an FFT?

Natsu 4y ago

The weird part to me is that this makes me imagine a high-dimensional space where we can have some vector that lets you go from some starting position, like a picture of a muffin, and then take that in a "dog-like" direction. And yet I've seen the weird "dog or muffin" [1] images that make you realize there's not as clean a separation as one might imagine with some pictures. [1] See, e.g. https://www.freecodecamp.org/news/chihuahua-or-muffin-my-search-for-the-best-computer-vision-api-cbda4d6b425d/ https://www.freecodecamp.org/news/chihuahua-or-muffin-my-sea...

nl 4y ago
One of the issues with the current stable diffusion img2img implementation is that this is NOT what it does (but should!) Instead it takes the image then tries to denoise it using the conditioned latents of the prompt, and the "strength" of the prompt determines how many denoising loops it does.
- Natsu 4y ago
  Could it be made to do that?
  - ShamelessC 4y ago
    Definitely! The model accepts CLIP embeds, so you could average the prompt embedding with the image embedding - and then do some sort of top-k selection over a few of these image embeddings to generate "variations" of an image. I think this is how DALLE-2 does it. Also, the model would benefit from being explicitly finetuned for the task of "inpainting". This is where you randomly erase parts of an image and force the model to predict what is missing based on the prompt. This allows one to edit existing images, filling in the gaps with what they need. This is technically already supported - but uses a similar "hack" as described above by sampling some noise into an existing image and starting a little bit later in the denoising schedule. Since the model wasn't finetuned for this task - a more generic method is used that doesn't work as well.
    - nl 4y ago
      I've made an attempt at image embedding averaging. That is insufficient in itself but it was a completely naive implementation. It really needs to do the denoising from one image to another and vice-versa over a few iterations. Or that's what I'm working on next anyway.
      - Natsu 4y ago
        How does denoising work, exactly and why does it seem necessary to get good results?
        ShamelessC 4y ago
        tl;dr it allows the model to focus on different frequencies of pixel-space at different time steps. Time step zero being the low frequency/low-detail features and time step 1000 being a completely “filled out”, detailed image including all the high frequency features. Doing so in discretely many steps makes the problem tractable and differentiable. Gross oversimplification, but that’s the gist.
        Natsu 4y ago
        Different frequencies? Does that mean it's doing something like an FFT?