16 ms·
How DALL-E 2 Works
- billconan 5y agoCan I train DALL-E2 on my personal computer with a fairly decent gpu? or it is out of the question?
- ShamelessC 5y agoNever gonna happen ha.
- oofbey 5y agoMaybe possible with a fabulous GPU, but still likely not, and if it did work it would take a horrendously long time. The real blocker is gonna be GPU memory. With an RTX 3090 you have 24 GB of GPU RAM and _might_ be able to try it, but I'm still not sure it would fit. The key model has 3.5 billion parameters, which at 16-bit requires 7GB of GPU-memory for each copy. Training requires 3 or 4 copies of the model, depending on the algorithm you use. And then you need memory for the data and activations, which you can reduce with a small batch size. But if it did fit, on a single GPU with a small batch size, you're probably looking at years of training time. Even an RTX 3080 is a complete non-starter.
- manquer 5y agoSomething like the Quadro RTX 8000 may theoretically work, it does have 48GB of RAM [1]. [1] https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/quadro-product-literature/quadro-rtx-8000-us-nvidia-946977-r1-web.pdf https://www.nvidia.com/content/dam/en-zz/Solutions/design-vi...
- oofbey 5y agoSure, with a $5k card like that it would be physically capable, but still unreasonably slow. FWIW RTX 8000 is previous-generation - the current gen is the A6000, which is also $5k, and similarly spec'd but faster. If you're looking for a deep learning card, they're actually a great value - I've got a bunch of them.
- jazzyjackson 5y agoI wonder whether it could be adapted to run on Apple M1 Ultra hardware, which can have 128GB of on-chip "unified" memory, can't find right away how much of that is available to the GPU or the neural cores, but if its even half of it, perhaps Apple will have the AI market cornered.
- simonw 5y agoI'm pretty confident that part of OpenAI's competitive edge is that they can train these models on GIANT clusters of machines. This article predicts that GPT-3 cost $10-$20m to train. I imagine DALL-E could cost even more: https://lastweekin.ai/p/gpt-3-is-no-longer-the-only-game?s=r https://lastweekin.ai/p/gpt-3-is-no-longer-the-only-game?s=r
- joshcryer 5y agoNope, and you'll still need a pretty beefy computer to run the trained data. Currently GPT-NeoX-20B, the "open source GPT3," requires 42 GB of VRAM, so you're looking at minimum a $5-6k graphics card (though a Quadro RTX 8000 is actually in stock so there's that). Or use a service like GooseAI. Eleuther.ai or some other open source / open research developers will likely try to reproduce DALL-E 2 but it'll take some time and a lot of donated hardware and cycles.
- SleekEagle 5y agoUnfortunately, it is out of the question. OpenAI trains on hundreds of thousands of dollars of GPUs and even then the trainings take two weeks. Also, as far as I know their training data (400 M image/caption pairs) is not available to the public!
- GaggiX 5y agofortunately there are even larger public datasets like LAION 5b
- manquer 5y agoIt is more like 10M+ for single run for the latest generation models[1]. This is a key reason why not lot of models are out there. Few groups have that kind of money to commit, also the viability is not yet very clear , i.e. how much the model with make if commercialized so they can recoup the investment. There is also cost of running the model on each API call, of course not factoring in any of the employee and other costs for sales marketing etc. [1] https://venturebeat.com/2020/06/01/ai-machine-learning-openai-gpt-3-size-isnt-everything/ https://venturebeat.com/2020/06/01/ai-machine-learning-opena...
- axg11 5y agoThis is a cute question. Not today! I hope someone comes back to read this question in 10-15 years time, when we will all have the ability to train Dall-E quality models on our AR glasses.
- pupppet 5y agoIf I have it generate a “bowl of soup” will I find an identical bowl in some clip art collection somewhere? How much does it deviate from the source images?
- SleekEagle 5y agoYou can try to reverse image search - from what I've seen of other people doing this, the renditions are quite distinct. The diffusion process is ultimately the root of the model's ability to not just copy images. Variational methods truly allow for the learning of a distribution, which is why VAEs can generate new data and AEs can't. Also, practically from a data point of view, the same object can be represented in numerous ways (different artistic styles, different filters, abstract paintings, etc.) and the model has to optimize across all of these samples. What this means is that the model truly is forced to learn the semantic meaning behind a concept and not just rely on specific features. Check out the dropdown under the "Significance of CLIP to DALL-E 2" section in the article
- corysama 5y agoI've played with tech like this for over a year now. You won't find the bowl in the source images. It doesn't evolve the noise into a source image. It slowly nudges the noise into feeling more and more bowl-like. Do that enough, and you get something that feels quite a bit like a bowl. Put it this way: The model file is absurdly smaller than the half billion source images files. If it actually contained the source images, it would be the greatest feat of image compression ever. Instead it only contains the impression left over by the images. A lot closer to a memory than a jpg.
- oofbey 5y agoOne thing I find really interesting about the DALL-E-2 is that the popular blog name ("DALL-E-2") never shows up in either of the research papers that describe it. The paper commonly referred to as DALL-E-2 calls its own algorithm "unCLIP". UnCLIP is _heavily_ based on a paper from a few months earlier called GLIDE - in fact you can't really understand the unCLIP paper by reading it without first reading the GLIDE paper. I suspect what's going on is that OpenAI has decoupled their PR activities from their science activities. They told the researchers to publish papers when they're ready, and then the PR apparatus decides when one is good enough to be crowned "DALL-E-2" and writes a blog post about it.
- KaoruAoiShiho 5y agoI don't get it, isn't this how literally every product launch works.
- radicaldreamer 5y agoI don't know why you're being voted down, internal/research names are often way weirder and decided on ad-hoc by the researchers themselves and then a good PM comes in when product-ionizing and part of this is deciding on a catchy name for public use.
- oofbey 5y agoThe phrasing "I don't get it" is fairly rude - it implies the post is obvious to the point of not being worth mentioning. However obvious this might seem to somebody, I would point out that turning AI research papers turn into products is hardly commonplace.
- SleekEagle 5y agoI noticed that as well! It confused me a bit at first. They say that their "image generation stack" is referred to as unCLIP, and I was trying to figure out how it's distinct from DALL-E 2 at first! My only guess would be that unCLIP is the end-to-end image generation model, but if the model is used for manipulation, interpolation, or variations, then it is referred to as DALL-E 2. So unCLIP is a subset of DALL-E 2.
- oofbey 5y agoDiffusion models seem like they're poised to completely replace GANs. They obviously work super well, and you don't have this super finicky minimax training problem.
- machinekob 5y agoBiggest problem for diffusion models were performance (as you need to iterate even at inference) But I'm not up to date with newest architectures maybe its already solved :P
- johndough 5y agoI was wondering it if would be possible to train a neural network to do multiple iterative steps at once. As it turns out, it has already been done and it requires about 4 to 8 distilled iterations for comparable quality. If this pace keeps up, we will probably see similar running time to GANs in the near future. https://arxiv.org/pdf/2202.00512.pdf https://arxiv.org/pdf/2202.00512.pdf
- SleekEagle 5y agoYeah, I haven't seen any big advancements in GANs in few years. Have I missed anything big or is the research volume trending down on them?
- astrange 5y agoThere's this but I don't know if it's been followed up on. https://www.microsoft.com/en-us/research/publication/manifold-matching-via-deep-metric-learning-for-generative-modeling/ https://www.microsoft.com/en-us/research/publication/manifol...
- SleekEagle 5y agoThank you for that link!
- mdda 5y ago
- mmastrac 5y agoAre we finally past the AI winter? We seem to be seeing major advances at least once a year. I recall there was a bit of a lull after GPT3, but clearly the boundaries of AI are expanding ridiculously fast.
- SleekEagle 5y agoGPT-3 was released 2 years ago, and in that time CLIP, GLIDE, and DALL-Es 1 and 2 have been released. All of this is just from OpenAI too! DL research is cranking along as quickly as ever imo!
- lurker619 5y agoJust need a music one please.
- gwern 5y agoJukebox. If you listen to Jukebox samples, recall that that was quite a while ago in dog/DL years, and imagine what the DALL-E 2 equivalent would be for a Jukebox 2...
- p1esk 5y agoI'm surprised no one has tried to launch a music generation startup based on Jukebox. I'd be interested in collaboration if anyone wants to work on it (and has compute resources).
- donkarma 5y agoYou're going to be waiting a while, even now these models can't get the details right. Look at all the example pictures, everything is just wrong. You can obviously see what it is trying to get at, but it can't get there. Another example of the last 10% taking 90% of the time.
- mellosouls 5y agoIn some ways (narrow AI), yes, its been a fantastic few years including tools like the one in context. In the important way that the AI winter originally referred to though, no, there doesn't seem to have been any progress towards AGI.
- madiator 5y agoI think DALL-E 3 will generate short clips. But I am curious to know what HN thinks will OpenAI will do with these technologies?
- SleekEagle 5y agoGPT-3 was sold with exclusive usage rights to MicroSoft, so maybe something along those lines with a different company (Meta?). As for what they will do with it, it's hard to say ...
- astrange 5y agoYou can use GPT-3 right now on OpenAI playground and there's commercial apps running on it that as far as I know aren't on Azure. It's not clear what they meant by exclusive.
- oofbey 5y agoOpenAI's ties to MSFT run very deep.[1] Meta's not going to get their hands on anything coming out of there anytime soon. [1] q.v. $1 billion investment https://techcrunch.com/2019/07/22/microsoft-invests-1-billion-in-openai-in-new-multiyear-partnership/ https://techcrunch.com/2019/07/22/microsoft-invests-1-billio...
- aabhay 5y agoTry to commercialize it, but fail to create much of a moat from it. Just like their past commercialization efforts.
- kingcharles 5y agoThe same thing they do every day, Pinky...
- breakfastduck 5y agoThis is a truly horrifying piece of technology, destined to destroy the livelihoods of countless artists. It's incredible in terms of the technology, but... scary in equal measure. I can't think of a single good reason for this to exist that doesn't have huge negative impacts on our world. Why pay an artist/graphic designer when this does what you need? "Now those damned creatives can go and find real jobs"
- sephlietz 5y agoHow is this any different than any other technological innovation which has made a job obsolete or otherwise allowed fewer people to do more work?
- wormer 5y agoI argue there is a difference because of the nature of the work. Machines aiding in farming is only a good thing, because it can maximize output and minimize input. People (largely) don't care about the process of how it was grown, but rather having the product to eat (Of course there's cruelty free agriculture, organic, etc. but stay with me here). But artistry is a personal thing, and maximizing the output of art pieces isn't something that most are interested in. Art is a uniquely unquantifiable subject, and we want it to have a personal and emotional connection to both the creator and the viewer, something that is lost when AI boil it down to it's essential components and rebuild them in it's image.
- WillPostForFood 5y agoMachines aiding in farming is only a good thing, because it can maximize output and minimize input. Machines aiding in art is only a good thing, because it can maximize output and minimize input? Makes art cheaper, more accessible, allows more people to create? It is like how digital filmmaking has cracked the Hollywood monopoly on content.
- wormer 5y agoI think that this doesn't really help artists as much as just do it for them. Art, the way I see it, requires a human to do because it is something that requires emotion, something a robot could replicate but not feel. For example, a gut wrenching image of innocents being beat by police is gut wrenching because it's something that exists in the real world, and the artist and the subjects are real and their emotion is real. But a computer generated image only has a likeness; it doesn't have actual emotion. I aldo don't think that it makes it "cheaper, more accessible, and allows more people to create". Digital art supplies being something readily available and relatively cheap to their classic counterparts is what makes things more accessible, and to make it more so would be to drive the cost down or something. Having the computer draw for you isn't exactly creating art. And art isn't a commodity and I argue it shouldn't be a commodity. It's something, again, personal and special. And this doesn't end at the visual arts, I think it applies too to writing. AI could write what's written in my journal word for word but my journal would have more value just by virtue of it being written by me.
- MengerSponge 5y agoHow does DALL-E 2 handle stereotypes? For example, what kind of output would you see for: > A person being shot by a police officer > A scientist emptying a dishwasher > A nurse driving a minivan AI training sets are famously biased, and I'm curious how egregious the outputs are...
- SleekEagle 5y agoGreat questions! I'd also be interested in this. I supposed the generations would mimic the general distribution of information that is on the internet, but what that would look like specifically is hard to say without OpenAI releasing more information.
- blamazon 5y agoOne way to dodge this and other issues related to depiction of human bodies is to trim the dataset such that humans are not generally recognizable as realistic humans in the output. It is also currently explicitly forbidden by OpenAI to share publicly realistic images of human faces generated by DE2. Via LessWrong.com: [1] >"One place where DE2 clearly falls down is in generating people. I generated an image for [four people playing poker in a dark room, with the table brightly lit by an ornate chandelier], and people didn't look human -- more like the typical GAN-style images where you can see the concept but the details are all wrong. >Update: image removed because the guidelines specifically call out not sharing realistic human faces. >Anything involving people, small defined objects, and so on, looks much more like the previous systems in this area. You can tell that it has all the concepts, but can't translate them into something realistic. >This could be deliberate, for safety reasons -- realistic images of people are much more open to abuse than other things. Porn, deep fakes, violence, and so on are much more worrisome with people. They also mentioned that they scrubbed out lots of bad stuff from the training data; possibly one way they did that was removing most images with people. >Things look much better with animals, and better again with an artistic style." [1]: https://www.lesswrong.com/posts/r99tazGiLgzqFX7ka/playing-with-dall-e-2 https://www.lesswrong.com/posts/r99tazGiLgzqFX7ka/playing-wi...
- deleted 5y ago[deleted]
- rglover 5y agoI've always been skeptical of AI stuff (for obvious reasons/long-term implications), but I have to say this application has me excited beyond belief. This is pure magic. Kudos to the OpenAI team.
- SleekEagle 5y agoA few years ago when photorealistic facial image generation models started getting really good I had my first "holy crap" moment. OpenAI expanding the domain from faces to essentially anything is absolutely mind blowing. An absolutely seminal step forward undoubtedly!
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- chrisco255 5y agoI find myself oscillating between excitement and sheer terror, sometimes several times a day.
- SleekEagle 5y agoSometimes at the same time!
- mokchira 5y agoFrom the article: "CLIP is trained on hundreds of millions of images and their associated captions..." Does anyone have any insight as to which images were trained on? Was it all open-domain stuff? And if not were the original authors of those images made aware their work was being use to train an AI that would likely put them out of work? Were they compensated appropriately?
- SleekEagle 5y agoAs far as I know, OpenAI has not made this dataset publicly available. IIRC the dataset is images scraped from instagram and their corresponding captions. Check out the CLIP paper for more details: https://arxiv.org/abs/2103.00020 https://arxiv.org/abs/2103.00020 Theoretically, you could build a web-scraping tool to do something like this, but even storing that data would take an absolutely insane amount of storage. I would assume OpenAI has some deal with Meta to make the creation of datasets like this easier.
- mokchira 5y agoThanks for the link. I hope they do make the data set publicly available at some point so that the artists whose work helped train this can know. I think, while it is absolutely impressive on a technical level what the OpenAI team has been able to do, it is also important to consider what damage it will do to artists and their livelihood. Many professional artists stake their career on one unique style of art that they have honed and developed over many years. It's this unique style that clients generally pay for, and that now faces a very real threat of being stolen from them by a technology that frankly no human can hope to compete with. Without artist compensation, this can only lead to artists terminating their careers early once the AI has co-opted all work from them. Or future artists never beginning their careers in the first place. This is a net loss for humanity, as it will deprive us of works and styles of art that have yet to be imagined. I'm not saying AI like this needs to go away. There is no putting that genie back in the bottle, of course. But it needs to be something that artists opt into. If someone's style is worth it for OpenAI to train on, then that style obviously should have a price tag. And it ought to be up to the artist whether they want to sell or not. Anything short of that is theft in my eyes.
- superasn 5y agoGaming is going to get so interesting with these emerging technologies. I played A.I. dungeon sometime ago and I was amazed at how good it was at making up believable stories on the fly. Now imagine joining this with dall-e and you truly have a game which has never existed until now, with it's own story and graphics that you are creating on the spot. Unlike the adventure games like King's quest where everything was pre-programmed, this is truly infinite never-ending game with a unique experience for every single player. Like the guy from the '2 min papers': what a time to be alive. I feel so happy and excited just thinking about the possibilities these techs are going to bring.
- mupuff1234 5y agoI doubt it will beat a curated experience any time soon, but I do see a future where it could assist in creating that curated experience.
- kromem 5y agoFor the format, AI dungeon already does. When I played with it, I started a quest as a wizard looking for a book. I was able to cast a tracking spell that led me to a giant library. I could have it read the titles of the books on a shelf in front of me. I could pick up any book and open to a random page and have it tell me what was in it. One was about a little half-elf that had a magic flute that broke. I cast a summoning spell to summon the half-elf and fix its flute, after which it happily played a song opening a door to another dimension filled with musical instruments. Give me that level of emergent gameplay in a VR open world, and then just take my money and all my free time, as I'm never leaving. We're simply very, very early on in what's arguably going to be the most transformative tech since the Internet. People predicted back then that the slow network which only offered basic things like email wasn't going to significantly disrupt things like retail. They were only right in that it didn't remain slow and ended up doing a lot more than email. This stuff is getting better way faster than any tech I've seen, and I used to consult for CEOs at Fortune 500s and sit on advisory boards on the topic of emerging tech. I wouldn't be so quick to bet against it. We really haven't even started to see what these models can do in application.
- 5y ago
- deleted 5y ago[deleted]
- password54321 5y agoTech bros are high-fiving their way to the top in every field with some neural nets. No one is safe.
- SleekEagle 5y agoThe rate of advancement over the past 10-15 years really has been incredible. Now the question is - is this growth curve logistic or exponential!
- ausbah 5y agothese are teams of PhD research scientists and research engineers. I wouldn't quantify them quite as just tech bros
- nsxwolf 5y agoI'll just never understand how any of this works. I know it is trained on millions of existing images, but when you say "... a bowl ..." in your prompt, how does it decide what the bowl should look like? Does it pick one of the bowls it's seen at random? It doesn't ever quite draw the same bowl twice, does it? Is it somehow "imagining" a bowl, the way a human would, and some all new image of a bowl pops into its "head"?
- simonw 5y agoThe trick is to start with random "gaussian noise" - something like https://opendatascience.com/wp-content/uploads/2017/03/noise3-560x300.jpg https://opendatascience.com/wp-content/uploads/2017/03/noise... - and then iteratively modify that image until it starts to look like the concept you want it to look like. I find the concept of a GAN - a Generative Adversarial Network - useful. My high-level attempt at explaining how those work is that you create two machine learning models, one that tries to create fake images and one that tries to see if an image is fake or not. The first one says "here's an image", the second one says "that's a fake", the first one learns from that and tries again, then keep going until an image scores highly on the test. The networks are adversarial because they are trying to outwit each other. (I'm sure a ML researcher could provide a better explanation than I can, but that's the way I think about it.)
- goodside 5y agoI don't believe Dall-E 2 incorporates GANs at all, but I haven't read the paper in detail. GANs were the best text-to-image models maybe a year ago but lately diffusion techniques are taking over.
- simonw 5y agoThanks for the keyword hint - this explanation looks good for diffusion models: https://ai.googleblog.com/2021/07/high-fidelity-image-generation-using.html https://ai.googleblog.com/2021/07/high-fidelity-image-genera...
- hackernewds 5y ago
- aantix 5y agoHow are objects differentiated from their background?
- ak391 5y agoopen source alternative to dalle: https://huggingface.co/spaces/multimodalart/latentdiffusion https://huggingface.co/spaces/multimodalart/latentdiffusion
- zimpenfish 5y agoI've had this running locally for a while and whilst it's an alternative you can actually use, it's not anywhere near the quality of DALL-E, let alone DALL-E-2. (Although since I mainly use it for abstracts and inspiration, that's fine.)
- achr2 5y agoOver the next decade, ML advancements will erode the monetary value of countless professions. Hopefully AI research will be turned towards solving the problems of society/economics/civilization before it is too late to avoid major disruptions in human wellbeing.
- seanwilson 5y ago> "DALL-E 2's works very simply: ... a model called the prior maps the text encoding to a corresponding image encoding that captures the semantic information of the prompt contained in the text encoding. Finally, an image decoding model stochastically generates an image which is a visual manifestation of this semantic information." > "The fundamental principles of training CLIP are quite simple: First, all images and their associated captions are passed through their respective encoders, mapping all objects into an m-dimensional space." Not scared to admit I don't find this simple at all and I'm probably not in the target audience. I'd love a description that doesn't assume machine learning basics. Is there one?
- pas 5y agohttps://ml.berkeley.edu/blog/posts/dalle2/ https://ml.berkeley.edu/blog/posts/dalle2/ it's "simple" because how it works is "just" brute-fucking-force. of course coming up with the architecture and making it fast (so it scales up well) is the challenge. and scaling works .. because .. well, no one knows why (but likely because it's just a nice architecture for learning, evolution also converged on it without knowing why) see also: https://www.gwern.net/Scaling-hypothesis https://www.gwern.net/Scaling-hypothesis
- exdsq 5y agoI've been generating little love stories between Putin and BoJo, things like going on romantic walks in the park and whatnot, and I cannot wait for DALL-E 2 access to add images to these :D
- hnuman 5y ago/fly /invisible /speed 100000 30 /jump_boost 1000 20 /spiderman /hammer