3 ms·
As someone who has a layman's understanding of neural networks, and who did some neural network programming ~20 years ago before the real explosion of the field
by hn_throwaway_99 4y ago
As someone who has a layman's understanding of neural networks, and who did some neural network programming ~20 years ago before the real explosion of the field, can someone point to some resources where I can get a better understanding about how this magic works?
I mean, from my perspective, the skill in these (and DALL-E's) image reproductions is truly astonishing. Just looking for more information about how the software actually works, even if there are big chunks of it that are "this is beyond your understanding without taking some in-depth courses".
- londons_explore 4y agoFigure A.4 in the linked paper is a good high level overview of this model. Shame it was hidden away on page 19 in the appendix! Each box you see there has a section in the paper explaining it in more detail.
- hn_throwaway_99 4y agoUhh, yeah, I'm going to need much more of an ELI5 than that! Looking at Figure A.4, I understand (again, at a very high-level) the first step of "Frozen Text Encoder", and I have a decent understanding of the upsampling techniques used in the last 2 diffusion model steps, but the middle "Text-to-Image Diffusion Model" step that magically outputs a 64x64 pixel image of an actual golden retriever wearing an actual blue checkered beret and red-dotted turtleneck is where I go "WTF??".
- f38zf5vdt 4y agoA good explanation is here. https://www.youtube.com/watch?v=344w5h24-h8 https://www.youtube.com/watch?v=344w5h24-h8
- sinenomine 4y ago> but the middle "Text-to-Image Diffusion Model" step that magically outputs a 64x64 pixel image of an actual golden retriever wearing an actual blue checkered beret and red-dotted turtleneck is where I go "WTF??". It doesn't output it outright, it basically forms it slowly, finding and strengthening more and more finer-grained features among the dwindling noise, combining the learned associations of memorized convolutional texture primitives vs encoded text embeddings. In the limit of enough data the associations and primitives turn out composable enough to suffice for out-of-distribution benchmark scenes. When you have a high-quality encoder of your modality into a compressed vector representation, the rest is optimization over a sufficiently high-dimensional, plastic computational substrate (model): https://moultano.wordpress.com/2020/10/18/why-deep-learning-works-even-though-it-shouldnt/ https://moultano.wordpress.com/2020/10/18/why-deep-learning-... It works because it should. The next question is: "What are the implications?". Can we meaningfully represent every available modality in a single latent space, and freely interconvert composable gestalts like this https://files.catbox.moe/rmy40q.jpg https://files.catbox.moe/rmy40q.jpg ?
- rvnx 4y agoCheck https://github.com/multimodalart/majesty-diffusion https://github.com/multimodalart/majesty-diffusion or https://github.com/lucidrains/DALLE2-pytorch https://github.com/lucidrains/DALLE2-pytorch There is a Google Colab workbook that you can try and run for free :) This is the image-text pairs behind: https://laion.ai/laion-400-open-dataset/ https://laion.ai/laion-400-open-dataset/
- astrange 4y ago> I mean, from my perspective, the skill in these (and DALL-E's) image reproductions is truly astonishing. A basic part of it is that neural networks combine learning and memorizing fluidly inside them, and these networks are really really big, so they can memorize stuff good. So when you see it reproduce a Shiba Inu well, don’t think of it as “the model understands Shiba Inus”. Think of it as making a collage out of some Shiba Inu clip art it found on the internet. You’d do the same if someone asked you to make this image. It’s certainly impressive that the lighting and blending are as good as they are though.
- hn_throwaway_99 4y agoTo be clear, I understand the general techniques about (a) how diffusion models can be used to upsample images and generate more photorealistic (or even "cartoon realistic") results and (b) I understand how they can do basic matching of "someone typed in Shiba Inu, look for images of Shiba Inus". What I don't understand is how they do the composition. E.g. for "A giant cobra snake on a farm. The snake is made out of corn." I think I could understand how it could reproduce the "A giant cobra snake on a farm" part. What I don't understand is how it accurately pictured "The snake is made out of corn." part, when I'm guessing it has never seen images of snakes made out of corn, and the way it combined "snake" with "made out of corn", in a way that is pretty much how I imagined it would look, is the part I'm baffled by.
- zone411 4y agoa) Diffusion is not just used to upsample images but also to create them. b) It has seen images with descriptions of "corn," "cobra," "farm," and it has seen images of "A made out of B" and "C on a D." To generate a high-scoring image, it has to make something that scores well on all of them put together.
- sinenomine 4y ago> What I don't understand is how they do the composition Convolutional filters lend themselves to rich combinatorics of compositions[1]: think of them as of context-dependent texture-atoms, repulsing and attracting over the variations of the local multi-dimensional context in the image. The composition is literally a convolutional transformation of local channels encoding related principal components of context. Astronomical amounts of computations spent via training allow the network to form a lego-set of these texture-atoms in a general distribution of contexts. At least this is my intuition for the nature of the convnets. 1. https://microscope.openai.com/models/contrastive_16x/image_block_4_0_Add_8_0?models.op.feature_vis.type=channel&models.op.technique=feature_vis https://microscope.openai.com/models/contrastive_16x/image_b...