5 ms·

There are some fundamental pieces missing in AI art generation that when solved will completely change the game. Forever. I think most fundamentally is the capa

There are some fundamental pieces missing in AI art generation that when solved will completely change the game. Forever. I think most fundamentally is the capability for AI to have some kind of memory or maybe more technically a way for the AI to be capable of do style "character style transfer" more effectively. I think this is probably possible, it's akin to making deep fakes but for drawn/photography art. With this tool suddenly the effort of making a series of compositions that are coherent will dramatically change the game, specially in the videogame industry. I think there are a lot of programmers out there capable of making great games but might be lacking the resources to fully complete their visions due to having to needing assets for their games. It seems to me like the approach stable diffusion has taken has dramatically increased interest and utility of these tools. So I'm hoping they follow similar lines for other types of AI. Every week I'm reading of a new novel use for these generators that I hadn't really considered before.

prox 4y ago
I wonder where it is going to. I was thinking of my favorite concept artists and what it means for them. Probably they can churn out 10x the work, and the role becomes more of curation, as in, scanning and judging the work put out by SD. They probably still going to adapt the work to fit their needs, and so companies probably still want actual artists for entertainment and artistic purposes. Small companies (and individuals) probably can use it to circumvent costly stock images all together, so the lower tier photographers/artists there gonna have a problem.
- gpderetta 4y ago
  I think most actual work will be done via img2img, outpainting and inpainting, and an real artist can actually give an important contribution.
  - prox 4y ago
    At the end of the day you still want something that sticks out, and artists are trained at that.
potatoman22 4y ago
This style transfer is possible through both textual inversion and dreambooth models, though they take a while to train. https://textual-inversion.github.io/ https://textual-inversion.github.io/ https://dreambooth.github.io/ https://dreambooth.github.io/
- danielbln 4y ago
  Training can be done in under an hour[1], really not that long. And yes, what OP is saying is already possible, which seems to be par for the course for this "new" AI space, as it's moving so fast. [1] https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast-DreamBooth.ipynb https://colab.research.google.com/github/TheLastBen/fast-sta...
  - DrSiemer 4y ago
    It is possible, but right now it still takes quite a bit of time and effort to get it right. The main challenge is finding the right balance between "make something that looks exactly like this" and "put it in a completely different context". Better similarity equals less flexibility. For now, the most effective combination will be artist + AI, although it does feel a bit like those that incorporate it in their workflow are helping to dig their own grave.
    - sillysaurusx 4y ago
      Do you happen to have any screenshots of what you mean? I’m really curious to see dreambooth’s capabilities in the field, and it sounds like you’ve had experience with some of its pitfalls.
      - danielbln 4y ago
        Basically OP says that overfitting is a common pitfall, you don't want to overtrain the model, because then everything will look like your training data, and vice versa with not enough training steps. So it's a bit of a balance. If you search for "dreambooth" on the SD subreddit, you will see a lot of examples of dreambooth results and also some that show overfitted and underfitted results. https://www.reddit.com/r/StableDiffusion/search?q=dreambooth&restrict_sr=on&sort=relevance&t=all https://www.reddit.com/r/StableDiffusion/search?q=dreambooth...
MrPatan 4y ago
Aren't the fundamental pieces already there? 1 Generate a bunch of characters or objects based on a prompt. 2 Pick the one you like 3 Tell the AI to extract the character's traits from that one picture 4 Miracle happens (I don't know. What does a "character definition file" look like?) 5 Make a new prompt, but add the character definition from step 4, so you get the same traits, only in a different setting or position, etc
magic_hamster 4y ago
It sounds like you're literally describing Dreambooth. You can easily and quickly train models for certain styles like Disney characters, certain illustrators, specific artists and even tailor the model to a specific person so you can produce countless pictures of them in different poses, locations and clothes. This is already available.
- sillysaurusx 4y ago
  Indeed: https://dreambooth.github.io https://dreambooth.github.io Note that training is extremely expensive, and is beyond the capabilities of most end users. Here are the details of their training method: > Given ~3-5 images of a subject we fine tune a text-to-image diffusion in two steps: (a) fine tuning the low-resolution text-to-image model with the input images paired with a text prompt containing a unique identifier and the name of the class the subject belongs to (e.g., "A photo of a [T] dog”), in parallel, we apply a class-specific prior preservation loss, which leverages the semantic prior that the model has on the class and encourages it to generate diverse instances belong to the subject's class by injecting the class name in the text prompt (e.g., "A photo of a dog”). (b) fine-tuning the super resolution components with pairs of low-resolution and high-resolution images taken from our input images set, which enables us to maintain high-fidelity to small details of the subject. Each fine-tuned model is a copy of the original model. So if the model is 10GB, the fine tuned version will be a separate 10GB file. That might not sound like a lot, but it quickly adds up. In this case, end users are artists. One could imagine a cloud-based art program which will fine tune on demand. That certainly seems like a good startup idea.
  - gpderetta 4y ago
    > Note that training is extremely expensive, and is beyond the capabilities of most end users. You can't run it on consumer hardware, but you can just rent a GPU (or use a free collab book) for a few hours to generate the model. Then you download and reuse it locally at will. Yes, you need storage and if you train often it can get expensive, but it is by no mean out of the end user, at least professional end users, capabilities. And of course there are growing libraries of freely available pretrained models. > In this case, end users are artists. One could imagine a cloud-based art program which will fine tune on demand. That certainly seems like a good startup idea. Very much agree about this. At least for a while, I strongly believe that AI will just be another tool for artists willing to embrace it, far from replacing them.
otabdeveloper4 4y ago
"AI" is only copy-pasting elements from a database of existing images. It's effectively just a fancy image search engine. You can use it to make procedurally-generated art, but it's still obvious what the source material was.
- sillysaurusx 4y ago
  This isn’t true, but it’s 4am, and I regret that I can’t type out a full rebuttal on my phone. One obvious counterexample is stylegan interpolations. If you interpolate between two images, the midpoint is usually unique — it’s often not obvious what the source material was. (E.g. Gwern’s anime interpolations; sure, they’re anime faces, but from where? “All of danbooru” might as well be “all styles of anime ever created.”) Maybe someone else can argue the point further.
- deleted 4y ago
  [deleted]
- grumbel 4y ago
  Go to your favorite image search engine and try to replicate anything AI has generated. It's impossible. The results you get from image search are nowhere near as specific as what the AI produces. It's not even close. The only time image search can compete is when you are highly specific, e.g. "painting of the Mono Lisa", both AI and image search will produce very similar results. But for a generic prompt, image search will come up with nothing that gets even close to the query, while AI can produce a highly specific image. DreamBooth really should have destroyed all doubt about this point point, as the AI can generated highly specific images of subjects that aren't even in the original training set.
  - otabdeveloper4 4y ago
    > as the AI can generated highly specific images of subjects that aren't even in the original training set I haven't seen any concrete examples of this yet.
- cthalupa 4y ago
  It's pretty impressive that Stable Diffusion can compress over 200TB of already previously compressed images from LAION 5B down to a few gigabytes! And that it can search that database so quickly to copy and paste things from the right images! /s There's a lot of good discussion to be had around the ethics of AI art, training on copyrighted materials, etc. But it is equivocally not just copy pasting from a database of images.

prox 4y ago

I wonder where it is going to. I was thinking of my favorite concept artists and what it means for them. Probably they can churn out 10x the work, and the role becomes more of curation, as in, scanning and judging the work put out by SD. They probably still going to adapt the work to fit their needs, and so companies probably still want actual artists for entertainment and artistic purposes. Small companies (and individuals) probably can use it to circumvent costly stock images all together, so the lower tier photographers/artists there gonna have a problem.

gpderetta 4y ago
I think most actual work will be done via img2img, outpainting and inpainting, and an real artist can actually give an important contribution.
- prox 4y ago
  At the end of the day you still want something that sticks out, and artists are trained at that.

potatoman22 4y ago

This style transfer is possible through both textual inversion and dreambooth models, though they take a while to train. https://textual-inversion.github.io/ https://textual-inversion.github.io/ https://dreambooth.github.io/ https://dreambooth.github.io/

danielbln 4y ago
Training can be done in under an hour[1], really not that long. And yes, what OP is saying is already possible, which seems to be par for the course for this "new" AI space, as it's moving so fast. [1] https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast-DreamBooth.ipynb https://colab.research.google.com/github/TheLastBen/fast-sta...
- DrSiemer 4y ago
  It is possible, but right now it still takes quite a bit of time and effort to get it right. The main challenge is finding the right balance between "make something that looks exactly like this" and "put it in a completely different context". Better similarity equals less flexibility. For now, the most effective combination will be artist + AI, although it does feel a bit like those that incorporate it in their workflow are helping to dig their own grave.
  - sillysaurusx 4y ago
    Do you happen to have any screenshots of what you mean? I’m really curious to see dreambooth’s capabilities in the field, and it sounds like you’ve had experience with some of its pitfalls.
    - danielbln 4y ago
      Basically OP says that overfitting is a common pitfall, you don't want to overtrain the model, because then everything will look like your training data, and vice versa with not enough training steps. So it's a bit of a balance. If you search for "dreambooth" on the SD subreddit, you will see a lot of examples of dreambooth results and also some that show overfitted and underfitted results. https://www.reddit.com/r/StableDiffusion/search?q=dreambooth&restrict_sr=on&sort=relevance&t=all https://www.reddit.com/r/StableDiffusion/search?q=dreambooth...

MrPatan 4y ago

Aren't the fundamental pieces already there? 1 Generate a bunch of characters or objects based on a prompt. 2 Pick the one you like 3 Tell the AI to extract the character's traits from that one picture 4 Miracle happens (I don't know. What does a "character definition file" look like?) 5 Make a new prompt, but add the character definition from step 4, so you get the same traits, only in a different setting or position, etc

magic_hamster 4y ago

It sounds like you're literally describing Dreambooth. You can easily and quickly train models for certain styles like Disney characters, certain illustrators, specific artists and even tailor the model to a specific person so you can produce countless pictures of them in different poses, locations and clothes. This is already available.

sillysaurusx 4y ago
Indeed: https://dreambooth.github.io https://dreambooth.github.io Note that training is extremely expensive, and is beyond the capabilities of most end users. Here are the details of their training method: > Given ~3-5 images of a subject we fine tune a text-to-image diffusion in two steps: (a) fine tuning the low-resolution text-to-image model with the input images paired with a text prompt containing a unique identifier and the name of the class the subject belongs to (e.g., "A photo of a [T] dog”), in parallel, we apply a class-specific prior preservation loss, which leverages the semantic prior that the model has on the class and encourages it to generate diverse instances belong to the subject's class by injecting the class name in the text prompt (e.g., "A photo of a dog”). (b) fine-tuning the super resolution components with pairs of low-resolution and high-resolution images taken from our input images set, which enables us to maintain high-fidelity to small details of the subject. Each fine-tuned model is a copy of the original model. So if the model is 10GB, the fine tuned version will be a separate 10GB file. That might not sound like a lot, but it quickly adds up. In this case, end users are artists. One could imagine a cloud-based art program which will fine tune on demand. That certainly seems like a good startup idea.
- gpderetta 4y ago
  > Note that training is extremely expensive, and is beyond the capabilities of most end users. You can't run it on consumer hardware, but you can just rent a GPU (or use a free collab book) for a few hours to generate the model. Then you download and reuse it locally at will. Yes, you need storage and if you train often it can get expensive, but it is by no mean out of the end user, at least professional end users, capabilities. And of course there are growing libraries of freely available pretrained models. > In this case, end users are artists. One could imagine a cloud-based art program which will fine tune on demand. That certainly seems like a good startup idea. Very much agree about this. At least for a while, I strongly believe that AI will just be another tool for artists willing to embrace it, far from replacing them.

otabdeveloper4 4y ago

"AI" is only copy-pasting elements from a database of existing images. It's effectively just a fancy image search engine. You can use it to make procedurally-generated art, but it's still obvious what the source material was.

sillysaurusx 4y ago
This isn’t true, but it’s 4am, and I regret that I can’t type out a full rebuttal on my phone. One obvious counterexample is stylegan interpolations. If you interpolate between two images, the midpoint is usually unique — it’s often not obvious what the source material was. (E.g. Gwern’s anime interpolations; sure, they’re anime faces, but from where? “All of danbooru” might as well be “all styles of anime ever created.”) Maybe someone else can argue the point further.
deleted 4y ago
[deleted]
grumbel 4y ago
Go to your favorite image search engine and try to replicate anything AI has generated. It's impossible. The results you get from image search are nowhere near as specific as what the AI produces. It's not even close. The only time image search can compete is when you are highly specific, e.g. "painting of the Mono Lisa", both AI and image search will produce very similar results. But for a generic prompt, image search will come up with nothing that gets even close to the query, while AI can produce a highly specific image. DreamBooth really should have destroyed all doubt about this point point, as the AI can generated highly specific images of subjects that aren't even in the original training set.
- otabdeveloper4 4y ago
  > as the AI can generated highly specific images of subjects that aren't even in the original training set I haven't seen any concrete examples of this yet.
cthalupa 4y ago
It's pretty impressive that Stable Diffusion can compress over 200TB of already previously compressed images from LAION 5B down to a few gigabytes! And that it can search that database so quickly to copy and paste things from the right images! /s There's a lot of good discussion to be had around the ethics of AI art, training on copyrighted materials, etc. But it is equivocally not just copy pasting from a database of images.