4 ms·

What's behind MidJourney, and how has it not been matched/defeated? There's a lot of interest in the space and money to be made, and MidJourney proves that qual

by prawn 3y ago

What's behind MidJourney, and how has it not been matched/defeated? There's a lot of interest in the space and money to be made, and MidJourney proves that quality results are possible. Is it because of the sheer costs or a secret sauce?

storkhm 3y ago
Well you can gain a free GPU hour by rating images. I am pretty sure midjourney is doing some kind of reinforcement learning from human feedback. The 5.1 raw model is simply unbelievable for photo realistic images. There is 4a,4b,4c, 5, 5.1, 5.1 raw, the anime model I never use. All are interesting but how often do you read anyone mention what midjourney model they are talking about? Basically never. There are certain circumstances that I have seen stable diffusion match the output but overall I don't even think it is close. Personally, it wouldn't even occur to me to look for stock photos now as opposed to just making variations in midjourney. There is a bit of a learning curve with so many options in midjourney so I take anything I read about it with a grain of salt. Most people are using these tools to make absolute shit.
aenvoker 3y ago
They combine multiple custom algorithms that are designed to run on high-RAM (>24GB) GPUs. And, they incorporate a lot of user-provided feedback into the training of new models. If you have more time on your hands than cash, you can get some free generation credit by rating images.
orbital-decay 3y ago
That's mostly because they make a finished end-user product, which nobody else does (except OpenAI and DALL-E, but they are mostly focused on language processing right now). - It runs in the cloud with more VRAM available than on consumer machines, so it's able to use more parameters and custom layers. This makes it more coherent. - It's finetuned on a large RLHF dataset, which they have from the users voting for bot-generated images in their massively popular Discord server. - It includes a decent share of custom processing and manual tuning (system prompts, custom layers trained on god knows what etc) Their closest contender, Stability AI, is currently busy training SDXL, their new model, on a RLHF dataset they are collecting on their much smaller Discord channel. So Midjourney is SOTA for text-to-image. There's DeepFloyd IF which is large enough and includes a 11B LLM for natural language understanding, so it has much less issues understanding complex prompts such as "koi fish doing a handstand on a longboard rolling down the slope of a moon crater". But it's not a finished product (no complex pipelines, no RLHF), and the license is restrictive. That said, Midjourney's strengths are also its weaknesses. Midjourney is good for the text-to-image, but not much else, and image generation is much more than just text-to-image. MJ gives great results off the bat, but struggles with unusual requests, complex compositions with multiple objects, and can't use various elaborate trickery available in the much smaller Stable Diffusion (which is mediocre by default but has powerful higher-order guidance and training, and is also much better at complex things). Midjourney is also censored - you can't have Joe Biden eating a slice of porcupine.

by prawn 3y ago

storkhm 3y ago

Well you can gain a free GPU hour by rating images. I am pretty sure midjourney is doing some kind of reinforcement learning from human feedback. The 5.1 raw model is simply unbelievable for photo realistic images. There is 4a,4b,4c, 5, 5.1, 5.1 raw, the anime model I never use. All are interesting but how often do you read anyone mention what midjourney model they are talking about? Basically never. There are certain circumstances that I have seen stable diffusion match the output but overall I don't even think it is close. Personally, it wouldn't even occur to me to look for stock photos now as opposed to just making variations in midjourney. There is a bit of a learning curve with so many options in midjourney so I take anything I read about it with a grain of salt. Most people are using these tools to make absolute shit.

aenvoker 3y ago

They combine multiple custom algorithms that are designed to run on high-RAM (>24GB) GPUs. And, they incorporate a lot of user-provided feedback into the training of new models. If you have more time on your hands than cash, you can get some free generation credit by rating images.

orbital-decay 3y ago

That's mostly because they make a finished end-user product, which nobody else does (except OpenAI and DALL-E, but they are mostly focused on language processing right now). - It runs in the cloud with more VRAM available than on consumer machines, so it's able to use more parameters and custom layers. This makes it more coherent. - It's finetuned on a large RLHF dataset, which they have from the users voting for bot-generated images in their massively popular Discord server. - It includes a decent share of custom processing and manual tuning (system prompts, custom layers trained on god knows what etc) Their closest contender, Stability AI, is currently busy training SDXL, their new model, on a RLHF dataset they are collecting on their much smaller Discord channel. So Midjourney is SOTA for text-to-image. There's DeepFloyd IF which is large enough and includes a 11B LLM for natural language understanding, so it has much less issues understanding complex prompts such as "koi fish doing a handstand on a longboard rolling down the slope of a moon crater". But it's not a finished product (no complex pipelines, no RLHF), and the license is restrictive. That said, Midjourney's strengths are also its weaknesses. Midjourney is good for the text-to-image, but not much else, and image generation is much more than just text-to-image. MJ gives great results off the bat, but struggles with unusual requests, complex compositions with multiple objects, and can't use various elaborate trickery available in the much smaller Stable Diffusion (which is mediocre by default but has powerful higher-order guidance and training, and is also much better at complex things). Midjourney is also censored - you can't have Joe Biden eating a slice of porcupine.