5 ms·

One thing I’ve not seen mentioned/asked about DALL-E 2, which I’m really, really intrigued about is what hardware its typically running on and the average time

One thing I’ve not seen mentioned/asked about DALL-E 2, which I’m really, really intrigued about is what hardware its typically running on and the average time it takes for image generation? Maybe I’ve somehow missed it, but I’m really intrigued by what their hardware set up is and how long it on average takes to generate the eight images from a users text input? I’m guessing it’s really quick, but I honestly have no idea… How does this system scale? Typical cost per image generated?

axg11 4y ago
Not a complete answer to your question but you may find this discussion interesting: https://github.com/lucidrains/DALLE2-pytorch/discussions/10 https://github.com/lucidrains/DALLE2-pytorch/discussions/10 Inference cost and scale seems to be much more favourable than large language models (for now).
- Sebb767 4y ago
  In case anyone else is put off by the link referencing an answer that then links to something else with most likely higher hardware requirements that are not stated, the end of the rabbit hole seems to be here: https://github.com/openai/dalle-2-preview/issues/6#issuecomment-1100494083 https://github.com/openai/dalle-2-preview/issues/6#issuecomm... TL;DR: A single NVidia A100 is most likely sufficient; with a lot of optimization and stepwise execution a single 3090 Ti might also be within the realm of possibility.
bitforger 4y ago
Also very interested in this. AFAIK, the best alternative to DALLE-type generation is CLIP-Guided generation (such as Disco Diffusion [1] and MidJourney[2]) which can take anywhere from 1 - 20 minutes on an RTX A5000. [1]: https://github.com/alembics/disco-diffusion https://github.com/alembics/disco-diffusion [2]: https://twitter.com/midjourney?t=-kKC5UE-gjIkMvAb709SyQ&s=09 https://twitter.com/midjourney?t=-kKC5UE-gjIkMvAb709SyQ&s=09
nl 4y ago
I think it would be relatively quick. The Latent_Diffusion_LAION_400M notebook generates six 512x512 images in about 45 seconds on a K80 on Collab. DALL-E2 is more complicated but presumably also better optimised.
ggerganov 4y ago
Allegedly, based on info from a DALL-E 2 paper, it takes about 100k-200k GPU hours to train the model: https://github.com/lucidrains/DALLE2-pytorch/issues/22#issuecomment-1108171293 https://github.com/lucidrains/DALLE2-pytorch/issues/22#issue...
- p1esk 4y ago
  Assuming those are V100 GPUs at $3/hr, that’s at least $300k.
  - manquer 4y ago
    Relative to compensation for the developers and researchers working on projects like these this looks a small component. GPT was rumored to cost in the millions, perhaps the hours estimate is conservative?
    - agnosticmantis 4y ago
      And presumably one should also factor in the cost of experimentation before a good setting for the final training is found.
      - zdkl 4y ago
        How feasible is it to evaluate the quality of chosen parameters/settings without going through an extensive training? I mean <BIGNUM> GPU hours to discover your results are meh, jesus. (Small timer with absolutely no idea of practical ML here)
    - starnger 4y ago
      4.6 millions

axg11 4y ago

Not a complete answer to your question but you may find this discussion interesting: https://github.com/lucidrains/DALLE2-pytorch/discussions/10 https://github.com/lucidrains/DALLE2-pytorch/discussions/10 Inference cost and scale seems to be much more favourable than large language models (for now).

Sebb767 4y ago
In case anyone else is put off by the link referencing an answer that then links to something else with most likely higher hardware requirements that are not stated, the end of the rabbit hole seems to be here: https://github.com/openai/dalle-2-preview/issues/6#issuecomment-1100494083 https://github.com/openai/dalle-2-preview/issues/6#issuecomm... TL;DR: A single NVidia A100 is most likely sufficient; with a lot of optimization and stepwise execution a single 3090 Ti might also be within the realm of possibility.

bitforger 4y ago

Also very interested in this. AFAIK, the best alternative to DALLE-type generation is CLIP-Guided generation (such as Disco Diffusion [1] and MidJourney[2]) which can take anywhere from 1 - 20 minutes on an RTX A5000. [1]: https://github.com/alembics/disco-diffusion https://github.com/alembics/disco-diffusion [2]: https://twitter.com/midjourney?t=-kKC5UE-gjIkMvAb709SyQ&s=09 https://twitter.com/midjourney?t=-kKC5UE-gjIkMvAb709SyQ&s=09

nl 4y ago

I think it would be relatively quick. The Latent_Diffusion_LAION_400M notebook generates six 512x512 images in about 45 seconds on a K80 on Collab. DALL-E2 is more complicated but presumably also better optimised.

ggerganov 4y ago

Allegedly, based on info from a DALL-E 2 paper, it takes about 100k-200k GPU hours to train the model: https://github.com/lucidrains/DALLE2-pytorch/issues/22#issuecomment-1108171293 https://github.com/lucidrains/DALLE2-pytorch/issues/22#issue...

p1esk 4y ago
Assuming those are V100 GPUs at $3/hr, that’s at least $300k.
- manquer 4y ago
  Relative to compensation for the developers and researchers working on projects like these this looks a small component. GPT was rumored to cost in the millions, perhaps the hours estimate is conservative?
  - agnosticmantis 4y ago
    And presumably one should also factor in the cost of experimentation before a good setting for the final training is found.
    - zdkl 4y ago
      How feasible is it to evaluate the quality of chosen parameters/settings without going through an extensive training? I mean <BIGNUM> GPU hours to discover your results are meh, jesus. (Small timer with absolutely no idea of practical ML here)
  - starnger 4y ago
    4.6 millions