4 ms·

It's highly doubtful that Midjourney and OpenAI use Stable Diffusion or other Stability models.

by ipsum2 3y ago

jonplackett 3y ago
How do you know though?
- minimaxir 3y ago
  You can't use off-the-shelf models to get the results Midjourney and DALL-E generate, even with strong finetuning.
  - cthalupa 3y ago
    I pay for both MJ and DALL-E (though OpenAI mostly gets my money for GPT) and don't find them to produce significantly better images than popular checkpoints on CivitAI. What I do find is that they are significantly easier to work with. (Actually, my experience with hundreds of DALL-E generations is that it's actually quite poor in quality. I'm in several IRC channels where it's the image generator of choice for some IRC bots, and I'm never particularly impressed with the visual quality.) For MJ in particular, knowing that they at least used to use Stable Diffusion under the hood, it would not surprise me if the majority of the secret sauce is actually a middle layer that processes the prompt and converts it to one that is better for working with SD. Prompting SD to get output at the MJ quality level takes significantly more tokens, lots of refinement, heavy tweaking of negative prompting, etc. Also a stack of embeddings and LoRAs, though I would place those more in the category of finetuning like you had mentioned.
    - emadm 3y ago
      If you try diffusionGPT with regional prompting added and a GAN corrector you can get a good idea of what is possible https://diffusiongpt.github.io https://diffusiongpt.github.io
      - euazOn 3y ago
        That looks very impressive unless the demo is cherrypicked, would be great if this could be implemented into a frontend like Fooocus https://github.com/lllyasviel/Fooocus https://github.com/lllyasviel/Fooocus
    - millgrove 3y ago
      What do you use it for? I haven't found a great use for it myself (outside of generating assets for landing pages / apps, where it's really really good). But I have seen endless subreddits / instagram pages dedicated to various forms of AI content, so it seems lots of people are using it for fun?
      - cthalupa 3y ago
        Nothing professional. I run a variety of tabletop RPGs for friends, so I mostly use it for making visual aids there. I've also got a large format printer that I was no longer using for it's original purpose, so I bought a few front-loading art frames that I generate art for and rotate through periodically. I've also used it to generate art for deskmats I got printed at https://specterlabs.co/ https://specterlabs.co/ For commercial stuff I still pay human artists.
        throwanem 3y ago
        Whose frames do you use? Do you like them? I print my photos to frame and hang, and wouldn't at all mind being able to rotate them more conveniently and inexpensively than dedicating a frame to each allows.
        cthalupa 3y ago
        https://www.spotlightdisplays.com/ https://www.spotlightdisplays.com/ I like them quite a bit, and you can get basically any size cut to fit your needs even if they don't directly offer it on the site.
        throwanem 3y ago
        Perfectly suited to go alongside the style of frame I already have lots of, and very reasonably priced off the shelf for the 13x19 my printer tops out at. Thanks so much! It'll be easier to fill that one blank wall now.
    - soultrees 3y ago
      What IRC Channels do you frequent?
      - cthalupa 3y ago
        Largely some old channels from the 90s/00s that really only exist as vestiges of their former selves - not really related to their original purpose, just rooms for hanging out with friends made there back when they had a point besides being a group chat.
    - xionplean 3y ago
      I use comfyUI/SD and MJ and I have never seen anything on the level of what I get out of MJ. Nothing at CivitAI is impressive to me next to what I get from MJ. Of course, art is so subjective none of this has any real meaning. MJ routinely blows my mind though and it is very rare something from SD does. The secret MJ sauce is obviously all the human feedback that has gone into the model at this point. I think AI video will be a different story though. I think that is when comfyUI/SD will destroy MJ because MJ is simply not going to be able to have an economic model with the amount of compute needed.
  - yreg 3y ago
    That's not really true, MJ and DALL-E are just more beginner friendly.
  - orbital-decay 3y ago
    Midjourney has absolutely nothing to offer compared to proper finetunes. DALL-E has: it generalizes well (can make objects interact properly for example) and has great prompt adherence. But it can also be unpredictable as hell because it rewrites the prompts. DALL-E's quality is meh - it has terrible artifacts on all pixel-sized details, hallucinations on small details, and limited resolution. Controlnets, finetuning/zero-shot reference transfer, and open tooling would have made a beast of a model of it, but they aren't available.
    - xionplean 3y ago
      You obviously have no idea what you are talking about or just make stupid anime porn. For fine art MJ destroys everything and it is even close. I say this with using comfyUI/SD all the time. But you do you and your anime porn.
      - orbital-decay 3y ago
        I'm actually a person making technical decisions (art decisions in the past) in a VFX/art studio, and I'm talking about production use. No generative AI currently passes any reasonable production quality bar, but is being tried by everyone for doing the work that can't be done or is cost-prohibitive otherwise, for example animation, long series with style transfer, filler assets creation etc. Anything that only has a text prompt can be discarded instantly. You have to be able to finetune it on your own material for consistency (of course I'm not talking about dubious 3rd party models), you need higher order guidance (e.g. controlnets, especially custom ones) and many other things. In the hands of a skilled person, a trivial Krita/Photoshop plugin (Firefly, SD, SD realtime) blows anything MJ can offer out of the water, simply because it has all that and you can't do much with text, it doesn't have enough semantic capacity to express artistic intent. I'm not even starting on animation. In fact, anything that involves non-explicitly guided one-shot generation of anything with light/shadow/colors/perspective is entirely out of the question with the current crop, because all models are hallucinating hard and aren't controllable within a single generation. There are attempts at fixing the perspective without explicit guidance, but it's going to be a long way and it's not super relevant to how things are done anyway. And for fine art, nothing beats a human painter, doing it by throwing prompts at AI mostly misses the point. I'm not even sure what you mean by fine art in this context, actually - surely not generating artsy-looking images from a prompt for fun?
cthalupa 3y ago
Midjourney 100% at least used to use Stable Diffusion: https://twitter.com/EMostaque/status/1561917541743841280 https://twitter.com/EMostaque/status/1561917541743841280 I am not sure if that is still the case.
- refulgentis 3y ago
  It trialled it as an explicitly optional model for a moment a couple years ago. (or only a year? time moves so fast. somewhere in v2/v3 timeframe and around when SD came out). I am sure it is no longer the case.
liuliu 3y ago
DALL-E shares the same autoencoders as SD v1.x. It is probably similar to how Meta's Emu-class models work though. They tweaked the architecture quite a bit, trained on their own dataset, reused some components (or in Emu case, trained all the components from scratch but reused the same arch).

It's highly doubtful that Midjourney and OpenAI use Stable Diffusion or other Stability models.

by ipsum2 3y ago

It's highly doubtful that Midjourney and OpenAI use Stable Diffusion or other Stability models.

jonplackett 3y ago

How do you know though?

minimaxir 3y ago
You can't use off-the-shelf models to get the results Midjourney and DALL-E generate, even with strong finetuning.
- cthalupa 3y ago
  I pay for both MJ and DALL-E (though OpenAI mostly gets my money for GPT) and don't find them to produce significantly better images than popular checkpoints on CivitAI. What I do find is that they are significantly easier to work with. (Actually, my experience with hundreds of DALL-E generations is that it's actually quite poor in quality. I'm in several IRC channels where it's the image generator of choice for some IRC bots, and I'm never particularly impressed with the visual quality.) For MJ in particular, knowing that they at least used to use Stable Diffusion under the hood, it would not surprise me if the majority of the secret sauce is actually a middle layer that processes the prompt and converts it to one that is better for working with SD. Prompting SD to get output at the MJ quality level takes significantly more tokens, lots of refinement, heavy tweaking of negative prompting, etc. Also a stack of embeddings and LoRAs, though I would place those more in the category of finetuning like you had mentioned.
  - emadm 3y ago
    If you try diffusionGPT with regional prompting added and a GAN corrector you can get a good idea of what is possible https://diffusiongpt.github.io https://diffusiongpt.github.io
    - euazOn 3y ago
      That looks very impressive unless the demo is cherrypicked, would be great if this could be implemented into a frontend like Fooocus https://github.com/lllyasviel/Fooocus https://github.com/lllyasviel/Fooocus
  - millgrove 3y ago
    What do you use it for? I haven't found a great use for it myself (outside of generating assets for landing pages / apps, where it's really really good). But I have seen endless subreddits / instagram pages dedicated to various forms of AI content, so it seems lots of people are using it for fun?
    - cthalupa 3y ago
      Nothing professional. I run a variety of tabletop RPGs for friends, so I mostly use it for making visual aids there. I've also got a large format printer that I was no longer using for it's original purpose, so I bought a few front-loading art frames that I generate art for and rotate through periodically. I've also used it to generate art for deskmats I got printed at https://specterlabs.co/ https://specterlabs.co/ For commercial stuff I still pay human artists.
      - throwanem 3y ago
        Whose frames do you use? Do you like them? I print my photos to frame and hang, and wouldn't at all mind being able to rotate them more conveniently and inexpensively than dedicating a frame to each allows.
        cthalupa 3y ago
        https://www.spotlightdisplays.com/ https://www.spotlightdisplays.com/ I like them quite a bit, and you can get basically any size cut to fit your needs even if they don't directly offer it on the site.
        throwanem 3y ago
        Perfectly suited to go alongside the style of frame I already have lots of, and very reasonably priced off the shelf for the 13x19 my printer tops out at. Thanks so much! It'll be easier to fill that one blank wall now.
  - soultrees 3y ago
    What IRC Channels do you frequent?
    - cthalupa 3y ago
      Largely some old channels from the 90s/00s that really only exist as vestiges of their former selves - not really related to their original purpose, just rooms for hanging out with friends made there back when they had a point besides being a group chat.
  - xionplean 3y ago
    I use comfyUI/SD and MJ and I have never seen anything on the level of what I get out of MJ. Nothing at CivitAI is impressive to me next to what I get from MJ. Of course, art is so subjective none of this has any real meaning. MJ routinely blows my mind though and it is very rare something from SD does. The secret MJ sauce is obviously all the human feedback that has gone into the model at this point. I think AI video will be a different story though. I think that is when comfyUI/SD will destroy MJ because MJ is simply not going to be able to have an economic model with the amount of compute needed.
- yreg 3y ago
  That's not really true, MJ and DALL-E are just more beginner friendly.
- orbital-decay 3y ago
  Midjourney has absolutely nothing to offer compared to proper finetunes. DALL-E has: it generalizes well (can make objects interact properly for example) and has great prompt adherence. But it can also be unpredictable as hell because it rewrites the prompts. DALL-E's quality is meh - it has terrible artifacts on all pixel-sized details, hallucinations on small details, and limited resolution. Controlnets, finetuning/zero-shot reference transfer, and open tooling would have made a beast of a model of it, but they aren't available.
  - xionplean 3y ago
    You obviously have no idea what you are talking about or just make stupid anime porn. For fine art MJ destroys everything and it is even close. I say this with using comfyUI/SD all the time. But you do you and your anime porn.
    - orbital-decay 3y ago
      I'm actually a person making technical decisions (art decisions in the past) in a VFX/art studio, and I'm talking about production use. No generative AI currently passes any reasonable production quality bar, but is being tried by everyone for doing the work that can't be done or is cost-prohibitive otherwise, for example animation, long series with style transfer, filler assets creation etc. Anything that only has a text prompt can be discarded instantly. You have to be able to finetune it on your own material for consistency (of course I'm not talking about dubious 3rd party models), you need higher order guidance (e.g. controlnets, especially custom ones) and many other things. In the hands of a skilled person, a trivial Krita/Photoshop plugin (Firefly, SD, SD realtime) blows anything MJ can offer out of the water, simply because it has all that and you can't do much with text, it doesn't have enough semantic capacity to express artistic intent. I'm not even starting on animation. In fact, anything that involves non-explicitly guided one-shot generation of anything with light/shadow/colors/perspective is entirely out of the question with the current crop, because all models are hallucinating hard and aren't controllable within a single generation. There are attempts at fixing the perspective without explicit guidance, but it's going to be a long way and it's not super relevant to how things are done anyway. And for fine art, nothing beats a human painter, doing it by throwing prompts at AI mostly misses the point. I'm not even sure what you mean by fine art in this context, actually - surely not generating artsy-looking images from a prompt for fun?

cthalupa 3y ago

Midjourney 100% at least used to use Stable Diffusion: https://twitter.com/EMostaque/status/1561917541743841280 https://twitter.com/EMostaque/status/1561917541743841280 I am not sure if that is still the case.

refulgentis 3y ago
It trialled it as an explicitly optional model for a moment a couple years ago. (or only a year? time moves so fast. somewhere in v2/v3 timeframe and around when SD came out). I am sure it is no longer the case.

liuliu 3y ago

DALL-E shares the same autoencoders as SD v1.x. It is probably similar to how Meta's Emu-class models work though. They tweaked the architecture quite a bit, trained on their own dataset, reused some components (or in Emu case, trained all the components from scratch but reused the same arch).