5 ms·

Flux is so frustrating to me. Really good prompt adherence, strong ability to keep track of multiple parts of a scene, it's technically very impressive. However

Flux is so frustrating to me. Really good prompt adherence, strong ability to keep track of multiple parts of a scene, it's technically very impressive. However it seems to have had no training on art-art. I can't get it to generate even something that looks like Degas, for instance. And, I can't even fine tune a painterly art style of any sort into Flux dev. I get that there was working, living artist backlash at SD and I can therefore imagine that the BFL team has decided not to train on art, but, it's a real loss. Both in terms of human knowledge of, say composition, emotion, and so on, but also for style diversity. For goodness sake, the MET in New York has a massive trove of open CC0 type licensed art. Dear BFL, please ease up a bit on this, and add some art-art to your models, they will be better as a result.

weebull 2y ago
I wonder if part of the reason it's good is because it's been trained for a more specific task. I can only imagine that if your concept of a "house" includes range from a stately home to "a pineapple under the sea" you're going to end up with a very generalised concept. It's then takes specific prompting to remove the influences you're not interested in. I suspect the same goes for art styles. There's such huge variety that really they'd be better surveys by separate models.
throwup238 2y ago
I’ve had the same problem with photography styles, even though the photographer I’m going for is Prokudin-Gorskii who used emulsion plates in the 1910s and the entire Library of Congress collection is in the public domain. I’m curious how they even managed to remove them from the training data since the entire LoC is such an easy dataset to access.
- throwaway314155 2y ago
  i'm fairly confident they did a broad FirstName LastName removal.
- vessenes 2y ago
  Yes, exactly. I think they purposely did not train on stuff like this. I'd bet that you could do a LoRa of Prokudin-Gorskii though; there's a lot of photographic content in flux's training set.
gs17 2y ago
And I can't imagine there's a real copyright (or ethical) issue with including artwork in the public domain because the artist died over a century ago.
pdntspa 2y ago
I wonder if you can use Flux to generate the base image then img2img on SD1.4 to impart artistic style?
- vunderba 2y ago
  That's what a refiner is for in auto1111. Taking an image the last 10% and touching it up with an alternative model. I actually use flux to generate image for purposes of adherence, then pull it in as a canny/depth controlnet with more established models like realvis, unstableXL, etc.
  - andersa 2y ago
    That is an interesting idea, I somehow hadn't thought of using flux in a chain like that, thanks!
  - vessenes 2y ago
    Yes, that is my current workflow as well.
crystal_revenge 2y ago
I've had a similar experience, incredible at generating a very specific style of image, but not great at generating anything with a specific style. I suspect we'll see the answer to this is LoRAs. Two examples that stick out are: - Flux Tarot v1 [0] - Flux Amateur Photography [1] Both of these do a great job of combining all the benefits of Flux with custom styles that seem to work quite well. [0] https://huggingface.co/multimodalart/flux-tarot-v1 https://huggingface.co/multimodalart/flux-tarot-v1 [1] https://civitai.com/models/652699?modelVersionId=756149 https://civitai.com/models/652699?modelVersionId=756149
- vessenes 2y ago
  I like those, and there's an electroshock lora that's just awesome out there. That said, Tarot and others like it are "illustrator" type styles with extra juice. I have not successfully trained a LoRa for any painting style, Flux does not seem to know about painting.
  - davidbarker 2y ago
    I'm curious to give this a go. I've been training a lot of LoRAs for FLUX dev recently (purely for fun). I'm sure there must be a way to get this working. Here are a few I've recently trained: https://civitai.com/user/dvyio https://civitai.com/user/dvyio
    - spython 2y ago
      This looks really good! What is your process to get this kind of high quality LoRAs?
      - davidbarker 2y ago
        Thank you! A reasonable amount of training images (50 or so), and then I train for 2,000-ish steps for a new style. Many of them work well with Flux, particularly if they're illustration-based. Some don't seem to work at all, so I didn't upload those!
thomastjeffery 2y ago
I think that's part of what makes FLUX.1 so good: the content it's trained on is very similar. Diversity is a double-edged sword. It's a desirable feature where you want it, and an undesirable feature everywhere else. If you want an impressionist painting, then it's good to have Monet and Degas in the training corpus. On the other hand, if you want a photograph of water lilies, then it's good to keep Monet out of the training data.
- doctorpangloss 2y ago
  DALL-E3 doesn't struggle with this. It's just opinions. There's no technical limitation. They chose to weaken the model in this regard.
  - thomastjeffery 2y ago
    Nonsense. FLUX.1-dev is famous for its consistency, prompt adherence, etc.; and it fits on a consumer GPU. That has to come with compromises. You can call any optimization weakness: that's the nature of compromise.
whywhywhywhy 2y ago
>However it seems to have had no training on art-art. I can't get it to generate even something that looks like Degas, for instance It feels like they just removed names from the datasets to make it worse at recreating famous people and artists.
- vessenes 2y ago
  No, they absolutely did not just do that in this case, although that was the SD plan. If you prompt for "painterly, oil painting, thick brush strokes, impressionistic oil painting style" to flux, you will get ... anime-ish renderings.
  - whywhywhywhy 2y ago
    That's not what I'm talking about, SDXL you can literally prompt a famous artists entire style and mix and match them, even conceptual artists and sculptors.
skort 2y ago
>but, it's a real loss. Both in terms of human knowledge of, say composition, emotion, and so on, but also for style diversity But that real art still exists, and can still be found, so what exactly is the loss here?
- vessenes 2y ago
  We may differ on our take about the usefulness of diffusion models, but I'd say it's a loss in that many of the visuals humans will see in the next ten years are going to be generated by these models, and I for one wish they weren't just trained on weeb shit.
  - dagaci 2y ago
    Just think that before 1995 (and in reality, decades later than that) most of the world would never have access to 99% of the worlds art. And between 1995 and 2022 the amount of Art produced surpasses the cumulative output of all other periods of human history.
    - vessenes 2y ago
      ... And between 2022 and 2025 the amount of imagery generated will drive the percent of Art created to roughly 0% of all imagery.
  - skort 2y ago
    You'll still be able to ask a person to create art in a specific style if you'd like.
    - vessenes 2y ago
      Unfortunately we will have a generation of young artists who learn to draw based on models like flux, unless they get classical training..
DeathArrow 2y ago
There are people who undistilled Flux so it can be further finetuned, so adding art training won't be an issue. https://huggingface.co/nyanko7/flux-dev-de-distill https://huggingface.co/nyanko7/flux-dev-de-distill

weebull 2y ago

I wonder if part of the reason it's good is because it's been trained for a more specific task. I can only imagine that if your concept of a "house" includes range from a stately home to "a pineapple under the sea" you're going to end up with a very generalised concept. It's then takes specific prompting to remove the influences you're not interested in. I suspect the same goes for art styles. There's such huge variety that really they'd be better surveys by separate models.

throwup238 2y ago

I’ve had the same problem with photography styles, even though the photographer I’m going for is Prokudin-Gorskii who used emulsion plates in the 1910s and the entire Library of Congress collection is in the public domain. I’m curious how they even managed to remove them from the training data since the entire LoC is such an easy dataset to access.

throwaway314155 2y ago
i'm fairly confident they did a broad FirstName LastName removal.
vessenes 2y ago
Yes, exactly. I think they purposely did not train on stuff like this. I'd bet that you could do a LoRa of Prokudin-Gorskii though; there's a lot of photographic content in flux's training set.

gs17 2y ago

And I can't imagine there's a real copyright (or ethical) issue with including artwork in the public domain because the artist died over a century ago.

pdntspa 2y ago

I wonder if you can use Flux to generate the base image then img2img on SD1.4 to impart artistic style?

vunderba 2y ago
That's what a refiner is for in auto1111. Taking an image the last 10% and touching it up with an alternative model. I actually use flux to generate image for purposes of adherence, then pull it in as a canny/depth controlnet with more established models like realvis, unstableXL, etc.
- andersa 2y ago
  That is an interesting idea, I somehow hadn't thought of using flux in a chain like that, thanks!
- vessenes 2y ago
  Yes, that is my current workflow as well.

crystal_revenge 2y ago

I've had a similar experience, incredible at generating a very specific style of image, but not great at generating anything with a specific style. I suspect we'll see the answer to this is LoRAs. Two examples that stick out are: - Flux Tarot v1 [0] - Flux Amateur Photography [1] Both of these do a great job of combining all the benefits of Flux with custom styles that seem to work quite well. [0] https://huggingface.co/multimodalart/flux-tarot-v1 https://huggingface.co/multimodalart/flux-tarot-v1 [1] https://civitai.com/models/652699?modelVersionId=756149 https://civitai.com/models/652699?modelVersionId=756149

vessenes 2y ago
I like those, and there's an electroshock lora that's just awesome out there. That said, Tarot and others like it are "illustrator" type styles with extra juice. I have not successfully trained a LoRa for any painting style, Flux does not seem to know about painting.
- davidbarker 2y ago
  I'm curious to give this a go. I've been training a lot of LoRAs for FLUX dev recently (purely for fun). I'm sure there must be a way to get this working. Here are a few I've recently trained: https://civitai.com/user/dvyio https://civitai.com/user/dvyio
  - spython 2y ago
    This looks really good! What is your process to get this kind of high quality LoRAs?
    - davidbarker 2y ago
      Thank you! A reasonable amount of training images (50 or so), and then I train for 2,000-ish steps for a new style. Many of them work well with Flux, particularly if they're illustration-based. Some don't seem to work at all, so I didn't upload those!

thomastjeffery 2y ago

I think that's part of what makes FLUX.1 so good: the content it's trained on is very similar. Diversity is a double-edged sword. It's a desirable feature where you want it, and an undesirable feature everywhere else. If you want an impressionist painting, then it's good to have Monet and Degas in the training corpus. On the other hand, if you want a photograph of water lilies, then it's good to keep Monet out of the training data.

doctorpangloss 2y ago
DALL-E3 doesn't struggle with this. It's just opinions. There's no technical limitation. They chose to weaken the model in this regard.
- thomastjeffery 2y ago
  Nonsense. FLUX.1-dev is famous for its consistency, prompt adherence, etc.; and it fits on a consumer GPU. That has to come with compromises. You can call any optimization weakness: that's the nature of compromise.

whywhywhywhy 2y ago

>However it seems to have had no training on art-art. I can't get it to generate even something that looks like Degas, for instance It feels like they just removed names from the datasets to make it worse at recreating famous people and artists.

vessenes 2y ago
No, they absolutely did not just do that in this case, although that was the SD plan. If you prompt for "painterly, oil painting, thick brush strokes, impressionistic oil painting style" to flux, you will get ... anime-ish renderings.
- whywhywhywhy 2y ago
  That's not what I'm talking about, SDXL you can literally prompt a famous artists entire style and mix and match them, even conceptual artists and sculptors.

skort 2y ago

>but, it's a real loss. Both in terms of human knowledge of, say composition, emotion, and so on, but also for style diversity But that real art still exists, and can still be found, so what exactly is the loss here?

vessenes 2y ago
We may differ on our take about the usefulness of diffusion models, but I'd say it's a loss in that many of the visuals humans will see in the next ten years are going to be generated by these models, and I for one wish they weren't just trained on weeb shit.
- dagaci 2y ago
  Just think that before 1995 (and in reality, decades later than that) most of the world would never have access to 99% of the worlds art. And between 1995 and 2022 the amount of Art produced surpasses the cumulative output of all other periods of human history.
  - vessenes 2y ago
    ... And between 2022 and 2025 the amount of imagery generated will drive the percent of Art created to roughly 0% of all imagery.
- skort 2y ago
  You'll still be able to ask a person to create art in a specific style if you'd like.
  - vessenes 2y ago
    Unfortunately we will have a generation of young artists who learn to draw based on models like flux, unless they get classical training..

DeathArrow 2y ago

There are people who undistilled Flux so it can be further finetuned, so adding art training won't be an issue. https://huggingface.co/nyanko7/flux-dev-de-distill https://huggingface.co/nyanko7/flux-dev-de-distill