14 ms·
DALL-E 2 open source implementation
- imwillofficial 4y agoI’m not an AI person, when can I use my MacBook to make crazy cool images I’ve seen on various twitter threads via DALL-E 2?
- axg11 4y agoBallpark answer: ~5 years until this is running on local hardware owned by the average person. Hardware is always improving and the algorithms will get more efficient. Until then, you'll likely be able to use an API to generate images within the next 6 months.
- tekgnos 4y agoIf you have some command line experience you can rent out some compute and then run it on rented hardware. I am renting an A40 with 48G of vram for $0.40/hr on Vast.ai right now to make cool images. I am still using Big Dream which I think is still based on DALL-E, not DALL-E 2. I couldn't figure out how to run this latest iteration (yet!).
- natly 4y agoLucidrains is an incredible guy. Nice guy in his discord too.
- sputknick 4y agowhat is his discord? is it publicly accessible?
- cpill 4y agois there a pre trained middle for it? what's the smallest cluster of GPUs you'll need to run this?
- fxtentacle 4y agoThey are now starting to train the model. https://wandb.ai/laion/diffusion-prior?workspace=user- https://wandb.ai/laion/diffusion-prior?workspace=user- https://github.com/lucidrains/DALLE2-pytorch/issues/23 https://github.com/lucidrains/DALLE2-pytorch/issues/23
- petulla 4y agoHe really is.. how has he not been snatched up by OpenAi etc.? He embodies the commitment to open source.
- fuzzythinker 4y agoHe started quite a bit of other reimplementations, see https://github.com/lucidrains https://github.com/lucidrains . Before Deepmind open sourced AlpahFold2, his implementation was one of only two available.
- tmabraham 4y agoHe left Discord lol
- DitheringIdiot 4y agoThis is a wonderful thing. But does it have any positive applications outside of entertainment? I can think of lots of malicious things it can be used for.
- emilsedgh 4y agoThe guy drawing funny memes for John Oliver's Last Week Tonight (Like the following) can be replaced by this! https://i.ytimg.com/vi/WhMGcp9xIhY/maxresdefault.jpg https://i.ytimg.com/vi/WhMGcp9xIhY/maxresdefault.jpg
- skybrian 4y agoI expect all usages of this tool to border on fiction in some way. After all, the images don't depict anything real. They might accompany a nonfiction article as an "artist's impression," though, as so often happens in popular science articles about things we have no images of yet. It's become increasingly common to start off articles with an image, even when the article is describing an abstract concept, just to avoid having a "wall of text." If that's what you need to do, it seems like a more interesting alternative to clip art.
- lumost 4y agoThis honestly seems like just a fun activity. Why not start every doc with a DALL-E generated image?
- williamvds 4y agoI'd much prefer tech being open to everyone rather than kept to megacorporations that OpenAI deems worthy (and has received funding from). A neural network isn't a lethal weapon, the potential for malicious use is much more limited than its potential positive creative applications.
- mensetmanusman 4y ago“ A neural network isn't a lethal weapon “ If a click generates a deepfake that causes someone to commit suicide, it might be considered lethal in some jurisdiction.
- christoph 4y agoOne thing I’ve not seen mentioned/asked about DALL-E 2, which I’m really, really intrigued about is what hardware its typically running on and the average time it takes for image generation? Maybe I’ve somehow missed it, but I’m really intrigued by what their hardware set up is and how long it on average takes to generate the eight images from a users text input? I’m guessing it’s really quick, but I honestly have no idea… How does this system scale? Typical cost per image generated?
- axg11 4y agoNot a complete answer to your question but you may find this discussion interesting: https://github.com/lucidrains/DALLE2-pytorch/discussions/10 https://github.com/lucidrains/DALLE2-pytorch/discussions/10 Inference cost and scale seems to be much more favourable than large language models (for now).
- Sebb767 4y agoIn case anyone else is put off by the link referencing an answer that then links to something else with most likely higher hardware requirements that are not stated, the end of the rabbit hole seems to be here: https://github.com/openai/dalle-2-preview/issues/6#issuecomment-1100494083 https://github.com/openai/dalle-2-preview/issues/6#issuecomm... TL;DR: A single NVidia A100 is most likely sufficient; with a lot of optimization and stepwise execution a single 3090 Ti might also be within the realm of possibility.
- bitforger 4y agoAlso very interested in this. AFAIK, the best alternative to DALLE-type generation is CLIP-Guided generation (such as Disco Diffusion [1] and MidJourney[2]) which can take anywhere from 1 - 20 minutes on an RTX A5000. [1]: https://github.com/alembics/disco-diffusion https://github.com/alembics/disco-diffusion [2]: https://twitter.com/midjourney?t=-kKC5UE-gjIkMvAb709SyQ&s=09 https://twitter.com/midjourney?t=-kKC5UE-gjIkMvAb709SyQ&s=09
- nl 4y agoI think it would be relatively quick. The Latent_Diffusion_LAION_400M notebook generates six 512x512 images in about 45 seconds on a K80 on Collab. DALL-E2 is more complicated but presumably also better optimised.
- arsome 4y agoIs it just me or is OpenAI the exact opposite of open? They may as well just call it ClosedAI at this point, seems like every interesting thing they do is kept under wraps.
- gwern 4y agoHow much does it matter? Everyone's practically forgotten that compviz and Midjourney were just in the past few months, Sberbank has ruDALL-E which largest model is somewhere between DALL-E 1 and 2, FB has Make-A-Scene which is somewhat below GLIDE/DALL-E 2, and Tsinghua's Cogview2 paper was released just days ago and is claiming roughly DALL-E 2 parity. AI capabilities are advancing fast enough that whether or not OA releases something, there will be competitors soon. (Heck, you want a GPT-3 competitor? There's like a dozen of them of similar or better quality, several of which are public or API - BigScience, Aleph Alpha, & AI2 come to mind.)
- sva_ 4y agoI feel like their initial mission of "democratizing AI" was just a way to recruit people. They're everything but "open".
- moffkalast 4y agoThey're as open as microsoft is micro.
- a1371 4y agoSorry I'm being too lazy to run the code, but are there any examples of its output? Like, how comparable is it to the openAi's
- minimaxir 4y agoPer the discussions, a prototype trained version of the model was released a few hours ago. There probably won't be demos for a bit, but sooner-than-later.
- djleni 4y agoSoftware developer that knows nothing about ML: What would it take to mess around with this? From a skim of the README it seems like install the package, and then find some training dataset, and then ??? to use it.
- mgalgs 4y agoIt looks like big-sleep [1] is their CLI? I did: pip install big-sleep dream "a pyramid made of basketballs" and after a few hours got this: https://i.imgur.com/FxdfdmV.png https://i.imgur.com/FxdfdmV.png Not nearly as cool as the real DALL-e, but maybe I'm missing something. [1] https://github.com/lucidrains/big-sleep https://github.com/lucidrains/big-sleep
- moffkalast 4y agoWell you most likely need a CUDA compatible GPU with a non-negligible amount of vram to load the model. And a few hours of your time to mash your head at the keyboard in frustration because of the inevitable dependencies that won't want to resolve properly. Most "open source" models don't actully upload the final models either, just the code they used to train them (not sure if this is the case here), so the next step after that is downloading 100G of data and running your workstation for a few days to train the actual model first. I recall at one point I attempted to run BERT or GPT-J or one of those lighter language models locally, and it was going well until I realized I needed a 24G of vRAM to even load the model hah.
- hadlock 4y agoDo CUDA AI projects not containerize well? I understand that AI projects often have many, many dependencies, but this seems like something containers ("docker") would be ideally suited for. Something like `docker pull dall-e` and `docker run -ie -n dalle dall-e` probably volume mount in a local folder? I'm curious what the hurdles are here for packaging something like this.
- indiv0 4y agoThey do containerize but it’s not enough. You still need the right drivers and CUDA stuff installed on the docker host, which can be finicky to setup. Not to mention figuring out how to get docker to actually pass the GPU to the container.
- mensetmanusman 4y agoWe are at the beginning of a post verification society. Any image or video of anything will soon be faked by commoners. This will be a wild ride!
- danuker 4y agoIs manual image editing not already faking?
- langitbiru 4y agoWe're talking about the scale. This is "manual image editing" on steroid.
- benlivengood 4y agoOne solution is chain-of-custody attestation from trusted hardware. Have cameras sign the video or images (and audio) they record.
- ceejayoz 4y agoIn a world where "no, mRNA vaccines don't change your DNA" has to be explained (often, unsuccessfully), I'm not sure I wanna try explaining digital signatures with elections on the line.
- boredemployee 4y agomy peer at work didnt take the shot because he doesnt believe in "new technologies". hes an engineer, MSC, phd and worked as a researcher. I mean, I'm not talking about someone totally out of science
- mdp2021 4y agoPeople who deal with complexities understand that experience is the encounter with the disruptive unexpected. Similarly to the Socratic principle, ancient counterpart of the Dunning-Kruger effect. Trust is heavily "modulated" in the experienced, it's kept at a distance. It's normal. So, back to the topic, we may have a few ways to build new shields against the new weapons, but this will be not trivial in technical and social matters, and their interactions - having the involved parts adequately understand the new realities: "a document may not be just trusted", "there is no boolean trust value but a 0..1 fraction", "signatures may not be a total warranty"... To some people, "If you received an e-mail using that sender's name it does not mean he sent it; no, easily his account was not hacked (etc.)" is already a massive blow to their inner world, that makes them throw themselves in the chair finding it unbearable if taken seriously and incomprehensible if considered. And, some people will find it "less painful" to distrust /you/ than to break the laws in their inner world (it's a phenomenon you may also meet in this very forum) - some minds observe a regimen of strict conservative low expenditure. Complexity involves social dangers.
- lagrange77 4y agoThis technology always reminds me of the TNG episode Schisms, where the holodeck renders an alien site based on evolving text descriptions. https://www.youtube.com/watch?v=lPLiaYD1lUo https://www.youtube.com/watch?v=lPLiaYD1lUo
- cwillu 4y agoThe original clickbait: “Clicking noises. More. Faster. Louder.”
- webmaven 4y ago"Delete the wife": https://youtu.be/mNCybqmKugA https://youtu.be/mNCybqmKugA
- mvkel 4y agoYes! I have this exact recurring memory when I'm playing around in DALLE2. Specifically the part where they're trying to make the table.
- ridgered4 4y agoI seem to remember once they finally found a way to wake up in the real alien world it was a bit different than the group had imagined it, which was a nice touch. Certainly one of the creepier TNG episodes even if it was a one off meant to cash in on the popularity of alien abduction in media at the time.
- dkga 4y agoThere's been some discussion below on OpenAI not being open. Contrary to most arguments, I have to share that I like their current approach. We can all easily imagine the sort of negative things that could come out of the misuse of their models, and it is their responsibility to ensure it drips into public domain rather than just putting it out there. It also allows them to more carefully consider any implications that they might have missed during development. And even when you do have access to them, at least in my experience they make sure to properly caveat some uses even during runtime (eg, in their playground for the DaVinci model). So while I'm not associated with OpenAI in any way, I would like to go out and say I broadly support their "closedness" for now and hope we can all be in a place in the future well release of high-performance models does not need to be so carefully considered because people would be more reasonable overall.
- astrange 4y ago> We can all easily imagine the sort of negative things that could come out of the misuse of their models, and it is their responsibility to ensure it drips into public domain rather than just putting it out there. Can we? I’m not sure about that - one issue being that they’re reproducable, so other people can recreate them without constraints. It turns out to not be that hard. I think a better explanation is there’s a bunch of spare “AI ethicists” hanging around because they read too much material from effective altruism cultists and decided they’re going to stop the world conquering AGI. But as OpenAI work doesn’t actually produce AGI, the best they can do is make up random things that sound ethical like “don’t put faces in the training output” and do that. Btw, their GPT3 playground happily produces incredibly offensive/racist material in the same friendly voice InstructGPT uses for everything else, but there they decided to add a “this text looks kind of offensive, sorry” warning instead of not training it in at all. (And to be clear I think that's the right decision, but man some of the text is surprising when you get it.)
- maeil 4y agoDerailing a little here but I have to say I'm not a fan of implying this kind of 1:1 link between having an interest or participating in effective altruism and the AI ethics scene. The Venn diagram is not a circle, and the overlapping part may well be a loud minority. I get that "cultists" may be talking about just that minority but it doesn't much read like that.
- victor9000 4y agoIs there a reason why the pretrained models are not included as well? Or perhaps this is coming at some point in the future? There are many users who simply want to play around with the functionality, who lack the time and hardware required to replicate the training steps.
- nl 4y agoIt literally says: > To train CLIP, you can either use x-clip package, or join the LAION discord, where a lot of replication efforts are already underway.
- kvathupo 4y agoAs someone who has trained transformers with only ~5 layers on 1000 data point inputs for a decoder of length like 10 (so far fewer than the 12 billion parameters of DALL-E [1]), I learned two things: 1. Training is very finicky and time-consuming. 2. PyTorch model files are pretty large. For point 1, when I was training a different autoencoder for example, I would run it for a week on 4 GPUs, only to return and see that its results are subpar. For point 2, I would get 100 MB model files for a miniscule transformer (relative to DALL-E numbers). Combined with the strong dependence of the transformer on the training data, this can make it problematic when it comes to sharing pre-trained models for tasks that the user doesn't specify a priori. [1] - https://venturebeat.com/2021/01/05/openai-debuts-dall-e-for-generating-images-from-text/ https://venturebeat.com/2021/01/05/openai-debuts-dall-e-for-...
- Uehreka 4y agoI will gladly download a terabyte of model data if that’s what it would take, your numbers don’t scare me!
- webmaven 4y agoLarge models generally require beefier GPUs and more RAM. Wide (not just deep) models require more GPU cores, total model size requires more RAM. Some models require multiple GPUs to train and/or run. Some require non-consumer server GPUs. These are non-trivial hardware investments if purchased, and non-trivial cloud bills if rented, even at preemptible spot pricing.
- dannyw 4y agoI remember lucidrains from his site, Epicmafia. Good times! Made so many friends there, many of whom I still talk to today.
- londons_explore 4y agoThis needs distributed training... Years ago I made a shared tensor library[1] which should allow people to do training in a distributed fashion around the world. Even with relatively slow internet connections, training should still make good use of all the compute available because the whole lot runs asynchronously with highly compressed and approximate updates to shared weights. The end result is that every bit of computation added has some benefits. Obviously for a real large scale effort, anti-cheat and anti-spam mechanisms would be needed to ensure nodes aren't deliberately sending bad data to hurt the group effort. [1]: https://github.com/Hello1024/shared-tensor https://github.com/Hello1024/shared-tensor
- manquer 4y agoThe overlap between the hardware requirements for model training and web3 mining has me thinking could there be a coin where the proof of work could be training a model ?
- manigandham 4y agoPotentially, I was recently introduced to this: https://bws.xyz/ https://bws.xyz/ Not affiliated and haven't researched yet but looks interesting. There are a few other crypto mining companies offering general compute to manage changing power/price/hardware resources.
- londons_explore 4y agoUnfortunately ML model training is too easy to 'cheat' - ie. To look at the test data to fine-tune the model to overfit a lot, do really well at test time, but actually be useless for real world stuff. To work out, you'd need a way to keep some test data secret, which in turn kinda kills decentralisation.
- mejutoco 4y agoYou could do something like kaggle does, where you have some test data but the final score is determined by a more complete test data that you don't have access to.
- nonoesp 4y agoWouldn't you then be penalized when you train a model that yields bad results on the test data, even if you're doing what you're meant to be doing?
- mejutoco 4y agoAFAIK it works like this: you have a test data you develop against and some secret (bigger) test data that you only have access for the final score. While you are developing you can overfit if you want, but then probably you won't perform that well with the secret tests. What you are meant to be doing is to perform with the test data without overfiting. Even if it is not optimal, it mostly solves the overfiting problem. It might work for a cryptocurrency too.