8 ms·

MIT Unveils Gen AI Tool That Generates High Res Images 30 Times Faster

188 points by mikhael 3y ago

cchance 3y ago
With all this tech i really wish we'd see it brought to Apples hardware, so much available unified memory, and yet generation times are shit in SD, even when converted to coreml
maytc 3y ago
Link to the paper: https://arxiv.org/abs/2311.18828 https://arxiv.org/abs/2311.18828
bl0b 3y ago
Am I missing it or is the source code not available?
- geor9e 3y ago
  I don't think they open sourced it. Thinking back, most AI breakthrough papers I've read don't include source code unfortunately. Researchers want their names out there explaining what they did and that they did it first, but MIT might want to license the implementation IP or Adobe (sounds like their interns discovered this during a summer) lawyers might be hanging onto it for a business edge.
  - Zambyte 3y ago
    Pretty ironic given this comment at the end of the page > While image generators like DALL-E and Meta AI's Imagine can produce extremely impressive results, these groups are highly protective of their technology and jealously guard it from curious public eyes. Meanwhile, you can go read about MIT's findings at this link.[0] [0] https://tianweiy.github.io/dmd/ https://tianweiy.github.io/dmd/
  - ilaksh 3y ago
    It's 100% Adobe.
FrustratedMonky 3y ago
Makes me wonder that on any given day someone could find way to improve something, and OpenAI value will evaporate quickly. Seems like these startups with high valuations, are more tenuous even than the startups of years past.
- atleastoptimal 3y ago
  There's more to startup value that whether or not they've implemented the latest nifty trick into their models. Their moat is architecture, research, distribution, design, and scale. Have all the local LLM hacks based off Llama 2 and the like done anything to diminish OpenAI's value?
  - aurareturn 3y ago
    I would say it has. If no one could demonstrate even GPT3.5 capabilities, then OpenAI could be worth a few hundred billion more in private valuations. But multiple vendors have already demonstrated viable alternatives.
    - atleastoptimal 3y ago
      Their valuation tripled to 80 billion in 10 months. I'm pretty sure they're not losing much to open source
      - aurareturn 3y ago
        It gives an idea in investors' minds that OpenAI's models can be matched or be "good enough" by open source models.
        p1esk 3y ago
        OpenAI has been at least one year ahead of everyone else in everything they have released do far. And there’s no sign they are stopping any time soon (e.g. GPT5 is expected this year).
fngjdflmdflg 3y ago
>Our method is similar to GANs in that a critic is jointly trained with the generator to minimize a divergence between the real and fake distributions, but differs in that our training does not play an adversarial game that may cause training instability, and our critic can fully leverage the weights of a pretrained diffusion model. Very glad to see GANs (or GAN-likes) coming back! Also I don't know if the examples were cherrypicked but a lot of the images look more realistic than SD. For example the dog in the snow[0] is severely oversaturated in the SD version giving it that distinct AI look (sometimes referred to as AIslop). Also the landscape scene[1] has a random lens flare in the SD version which makes it look oversaturated as well. The MIT images are much better in this regard. [0] https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_Image_0009.png https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_... vs https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_Image_0008.png https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_... [1] https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_Image_0001.png https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_... vs https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_Image_0005.png https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_...
- bosco_mcnasty 3y ago
  >Very glad to see GANs (or GAN-likes) coming back! Adobe has had GigaGAN out a while now though?
- amelius 3y ago
  Can GANs prevent the ">10 fingers" problem?
  - fzliu 3y ago
    The >10 fingers problem should be solved via higher quality data or conditioning rather than a better model architecture.
    - onion2k 3y ago
      Are you suggesting AI generates images of people with more than 10 fingers because there are too many pictures of people with more than 10 fingers in the input data? That seems unlikely..
ttul 3y ago
Diffusion models are exceptionally rich troves of statistical information. You just have to find it. The model will take any random noise and denoise of a bit. By guiding the denoising process, you can speed it up or improve the output tremendously.
robertclaus 3y ago
I'm glad to see another project tackling the challenge of speeding up image generation. After seeing the bang-for-buck jump from sdxl to sdxl-turbo, I assume there's tons more room for improvement on the speed side of things.
jasonjmcghee 3y ago
This is 30 times faster than SD (on its own, as opposed to LCM methods) right? Noting the "50" steps they referenced- whereas I've been using LCMs with 3-8 steps for months. Just trying to understand context of the paper / comparisons etc. 90ms is fast, but like 2-3x speedup (over what I've been seeing) which is still huge, but not 30x Can someone with more context explain?
- sirwhinesalot 3y ago
  It's 30x faster than 50 step SD with the same quality, unlike LCMs which can result in substantially lower quality. The actual page of the work shows the difference.
albert_e 3y ago
Is the text prompt input the only common thing between SD and MIT models when comparing outputs? If so I am surprised at how SIMILAR the outputs of both models look in the general layout / framing / composition of the image. How can for example both fox astronaut images have near identical backdrop of earth and earth alone on the same side of image at same apparent size. Virtually the same shade of deep blue for deep space. "Lightshow at the Dolomities" output is virtually identical. So similar that they almost look like iPhone versus Samsung Galaxy camera comparison shots of same scene (I am exaggerating of course but almost there). What results in such a close similarity in outputs? Same training data set? It is almost like they have the same DNA!
- mattnewton 3y ago
  I believe this is similar to the Latent Consistency Modeling approach, where it’s a replacement for the “diffusion” process, not the underlying weights. Basically, they have a more efficient process for pulling images out of the weights, not necessarily a set of new weights.
  - spi 3y ago
    The weights are different, because the model is different. As jzbontar below mentions, the crucial point is that the random noise mask is the same. The diffusion models are trained to turn random noise to an image, and they are deterministic at that - the same noise leads to the same image. What the authors did here was to find a smart way of training a new model able to "simulate" in a single step what diffusion achieves in many; to do so, they took many triplets of (prompt, noise, image) generated starting from random noise and a (fixed) pretrained stable diffusion checkpoint. The model is trained to replicate the results. So, it is surprising that this works at all at creating meaningful images, but it would be _really_ surprising (i.e. probably impossible) if it generated meaningful images which were seriously different from the ones it was pretrained with!
    - albert_e 3y ago
      Oh the images and prompts we see in article are from the training data? Pardon my ignorance ... Does MIT model then not work as a general text-to-image model to generate novel images based on arbitrary new text prompts that it has not seen before?
GaggiX 3y ago
There are so many similar diffusion model distillation techniques these days that it's become hard to tell the difference between them, this is probably the fourth example I've seen that uses adversarial loss to distill the model, the others being UFOgen, some other work by StabilityAI on SDXL-turbo and similar, SDXL-Lightning by ByteDance. I found this blog page that explains some of the differences: https://sander.ai/2024/02/28/paradox.html https://sander.ai/2024/02/28/paradox.html
- ilaksh 3y ago
  This is 10 times faster than almost all of the fast image generation models.
  - GaggiX 3y ago
    I mean, it's not the only one that can run SD-1.5 in one step, so no.
Linda231 3y ago
[dead]
smusamashah 3y ago
Article is only talking about 1 step generation. It doesn't anything if 2 steps improve the image at all. The images look good enough but not as good as 30-50 step generation by SD. The cat in water is much clear example of that.
th0ma5 3y ago
I think it is wild that "good" outputs are always some of the most uncanny imagery. Sure it is better than older versions, but "photorealistic" only makes sense if you are describing a photo to someone who can't see. I know it's "early" but they don't deserve this praise. It is cool they can assemble things but it really is a very low bar compared to reality. I know non creative people don't care but audiences care.
ChrisArchitect 3y ago
Official news release rather than whatever this site is: https://news.mit.edu/2024/ai-generates-high-quality-images-30-times-faster-single-step-0321 https://news.mit.edu/2024/ai-generates-high-quality-images-3...
- em1sar 3y ago
  [dead]
dragonwriter 3y ago
30 times faster than SD1.5. Which is in the same broad neighborhood as existing Stable Diffusion 1.5 LCM (and similar to the speedup, compared to SDXL, for the SDXL LCM, Turbo, and Lightning models.)
aussieguy1234 3y ago
Given the speed increase, how much work would be needed to create videos?
neya 3y ago
This is just an ad dumpster borderline malware link. Here is the link to the actual tool. https://tianweiy.github.io/dmd/ https://tianweiy.github.io/dmd/
holoduke 3y ago
I am waiting for the day i can hook up my webcam and do some live video to video :)
mosselman 3y ago
Is the tool available to use somewhere? I've been looking around, but I am not familiar with the research paper format of these types of tools so maybe I am overlooking some obvious link to a repository or something?
angiosperm 3y ago
The word "confabulate" is underused in AI image generation circles.
_l7dh 3y ago
So annoying to have articles not point to the original paper.

188 points by mikhael 3y ago

With all this tech i really wish we'd see it brought to Apples hardware, so much available unified memory, and yet generation times are shit in SD, even when converted to coreml

maytc 3y ago

Link to the paper: https://arxiv.org/abs/2311.18828 https://arxiv.org/abs/2311.18828

bl0b 3y ago

Am I missing it or is the source code not available?

geor9e 3y ago
I don't think they open sourced it. Thinking back, most AI breakthrough papers I've read don't include source code unfortunately. Researchers want their names out there explaining what they did and that they did it first, but MIT might want to license the implementation IP or Adobe (sounds like their interns discovered this during a summer) lawyers might be hanging onto it for a business edge.
- Zambyte 3y ago
  Pretty ironic given this comment at the end of the page > While image generators like DALL-E and Meta AI's Imagine can produce extremely impressive results, these groups are highly protective of their technology and jealously guard it from curious public eyes. Meanwhile, you can go read about MIT's findings at this link.[0] [0] https://tianweiy.github.io/dmd/ https://tianweiy.github.io/dmd/
- ilaksh 3y ago
  It's 100% Adobe.

FrustratedMonky 3y ago

Makes me wonder that on any given day someone could find way to improve something, and OpenAI value will evaporate quickly. Seems like these startups with high valuations, are more tenuous even than the startups of years past.

atleastoptimal 3y ago
There's more to startup value that whether or not they've implemented the latest nifty trick into their models. Their moat is architecture, research, distribution, design, and scale. Have all the local LLM hacks based off Llama 2 and the like done anything to diminish OpenAI's value?
- aurareturn 3y ago
  I would say it has. If no one could demonstrate even GPT3.5 capabilities, then OpenAI could be worth a few hundred billion more in private valuations. But multiple vendors have already demonstrated viable alternatives.
  - atleastoptimal 3y ago
    Their valuation tripled to 80 billion in 10 months. I'm pretty sure they're not losing much to open source
    - aurareturn 3y ago
      It gives an idea in investors' minds that OpenAI's models can be matched or be "good enough" by open source models.
      - p1esk 3y ago
        OpenAI has been at least one year ahead of everyone else in everything they have released do far. And there’s no sign they are stopping any time soon (e.g. GPT5 is expected this year).

fngjdflmdflg 3y ago

>Our method is similar to GANs in that a critic is jointly trained with the generator to minimize a divergence between the real and fake distributions, but differs in that our training does not play an adversarial game that may cause training instability, and our critic can fully leverage the weights of a pretrained diffusion model. Very glad to see GANs (or GAN-likes) coming back! Also I don't know if the examples were cherrypicked but a lot of the images look more realistic than SD. For example the dog in the snow[0] is severely oversaturated in the SD version giving it that distinct AI look (sometimes referred to as AIslop). Also the landscape scene[1] has a random lens flare in the SD version which makes it look oversaturated as well. The MIT images are much better in this regard. [0] https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_Image_0009.png https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_... vs https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_Image_0008.png https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_... [1] https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_Image_0001.png https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_... vs https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_Image_0005.png https://tianweiy.github.io/dmd/images/teaser/teaser2_Page_1_...

bosco_mcnasty 3y ago
>Very glad to see GANs (or GAN-likes) coming back! Adobe has had GigaGAN out a while now though?
amelius 3y ago
Can GANs prevent the ">10 fingers" problem?
- fzliu 3y ago
  The >10 fingers problem should be solved via higher quality data or conditioning rather than a better model architecture.
  - onion2k 3y ago
    Are you suggesting AI generates images of people with more than 10 fingers because there are too many pictures of people with more than 10 fingers in the input data? That seems unlikely..

ttul 3y ago

Diffusion models are exceptionally rich troves of statistical information. You just have to find it. The model will take any random noise and denoise of a bit. By guiding the denoising process, you can speed it up or improve the output tremendously.

robertclaus 3y ago

I'm glad to see another project tackling the challenge of speeding up image generation. After seeing the bang-for-buck jump from sdxl to sdxl-turbo, I assume there's tons more room for improvement on the speed side of things.

jasonjmcghee 3y ago

This is 30 times faster than SD (on its own, as opposed to LCM methods) right? Noting the "50" steps they referenced- whereas I've been using LCMs with 3-8 steps for months. Just trying to understand context of the paper / comparisons etc. 90ms is fast, but like 2-3x speedup (over what I've been seeing) which is still huge, but not 30x Can someone with more context explain?

sirwhinesalot 3y ago
It's 30x faster than 50 step SD with the same quality, unlike LCMs which can result in substantially lower quality. The actual page of the work shows the difference.

albert_e 3y ago

Is the text prompt input the only common thing between SD and MIT models when comparing outputs? If so I am surprised at how SIMILAR the outputs of both models look in the general layout / framing / composition of the image. How can for example both fox astronaut images have near identical backdrop of earth and earth alone on the same side of image at same apparent size. Virtually the same shade of deep blue for deep space. "Lightshow at the Dolomities" output is virtually identical. So similar that they almost look like iPhone versus Samsung Galaxy camera comparison shots of same scene (I am exaggerating of course but almost there). What results in such a close similarity in outputs? Same training data set? It is almost like they have the same DNA!

mattnewton 3y ago
I believe this is similar to the Latent Consistency Modeling approach, where it’s a replacement for the “diffusion” process, not the underlying weights. Basically, they have a more efficient process for pulling images out of the weights, not necessarily a set of new weights.
- spi 3y ago
  The weights are different, because the model is different. As jzbontar below mentions, the crucial point is that the random noise mask is the same. The diffusion models are trained to turn random noise to an image, and they are deterministic at that - the same noise leads to the same image. What the authors did here was to find a smart way of training a new model able to "simulate" in a single step what diffusion achieves in many; to do so, they took many triplets of (prompt, noise, image) generated starting from random noise and a (fixed) pretrained stable diffusion checkpoint. The model is trained to replicate the results. So, it is surprising that this works at all at creating meaningful images, but it would be _really_ surprising (i.e. probably impossible) if it generated meaningful images which were seriously different from the ones it was pretrained with!
  - albert_e 3y ago
    Oh the images and prompts we see in article are from the training data? Pardon my ignorance ... Does MIT model then not work as a general text-to-image model to generate novel images based on arbitrary new text prompts that it has not seen before?

GaggiX 3y ago

There are so many similar diffusion model distillation techniques these days that it's become hard to tell the difference between them, this is probably the fourth example I've seen that uses adversarial loss to distill the model, the others being UFOgen, some other work by StabilityAI on SDXL-turbo and similar, SDXL-Lightning by ByteDance. I found this blog page that explains some of the differences: https://sander.ai/2024/02/28/paradox.html https://sander.ai/2024/02/28/paradox.html

ilaksh 3y ago
This is 10 times faster than almost all of the fast image generation models.
- GaggiX 3y ago
  I mean, it's not the only one that can run SD-1.5 in one step, so no.

Linda231 3y ago

[dead]

smusamashah 3y ago

Article is only talking about 1 step generation. It doesn't anything if 2 steps improve the image at all. The images look good enough but not as good as 30-50 step generation by SD. The cat in water is much clear example of that.

th0ma5 3y ago

I think it is wild that "good" outputs are always some of the most uncanny imagery. Sure it is better than older versions, but "photorealistic" only makes sense if you are describing a photo to someone who can't see. I know it's "early" but they don't deserve this praise. It is cool they can assemble things but it really is a very low bar compared to reality. I know non creative people don't care but audiences care.

ChrisArchitect 3y ago

Official news release rather than whatever this site is: https://news.mit.edu/2024/ai-generates-high-quality-images-30-times-faster-single-step-0321 https://news.mit.edu/2024/ai-generates-high-quality-images-3...

em1sar 3y ago
[dead]

dragonwriter 3y ago

30 times faster than SD1.5. Which is in the same broad neighborhood as existing Stable Diffusion 1.5 LCM (and similar to the speedup, compared to SDXL, for the SDXL LCM, Turbo, and Lightning models.)

aussieguy1234 3y ago

Given the speed increase, how much work would be needed to create videos?

neya 3y ago

This is just an ad dumpster borderline malware link. Here is the link to the actual tool. https://tianweiy.github.io/dmd/ https://tianweiy.github.io/dmd/

holoduke 3y ago

I am waiting for the day i can hook up my webcam and do some live video to video :)

mosselman 3y ago

Is the tool available to use somewhere? I've been looking around, but I am not familiar with the research paper format of these types of tools so maybe I am overlooking some obvious link to a repository or something?

angiosperm 3y ago

The word "confabulate" is underused in AI image generation circles.

_l7dh 3y ago

So annoying to have articles not point to the original paper.