23 ms·

AI and the Future of Pixel Art

173 points by aleyan 4y ago

sawzaw 4y ago
My wife is an artist and has been using some of these systems to inspire her own work with clients - she's said it's both given her creative new ideas, and improved her efficiency by about 30%.
jerojero 4y ago
There are some fundamental pieces missing in AI art generation that when solved will completely change the game. Forever. I think most fundamentally is the capability for AI to have some kind of memory or maybe more technically a way for the AI to be capable of do style "character style transfer" more effectively. I think this is probably possible, it's akin to making deep fakes but for drawn/photography art. With this tool suddenly the effort of making a series of compositions that are coherent will dramatically change the game, specially in the videogame industry. I think there are a lot of programmers out there capable of making great games but might be lacking the resources to fully complete their visions due to having to needing assets for their games. It seems to me like the approach stable diffusion has taken has dramatically increased interest and utility of these tools. So I'm hoping they follow similar lines for other types of AI. Every week I'm reading of a new novel use for these generators that I hadn't really considered before.
- prox 4y ago
  I wonder where it is going to. I was thinking of my favorite concept artists and what it means for them. Probably they can churn out 10x the work, and the role becomes more of curation, as in, scanning and judging the work put out by SD. They probably still going to adapt the work to fit their needs, and so companies probably still want actual artists for entertainment and artistic purposes. Small companies (and individuals) probably can use it to circumvent costly stock images all together, so the lower tier photographers/artists there gonna have a problem.
  - gpderetta 4y ago
    I think most actual work will be done via img2img, outpainting and inpainting, and an real artist can actually give an important contribution.
    - prox 4y ago
      At the end of the day you still want something that sticks out, and artists are trained at that.
- potatoman22 4y ago
  This style transfer is possible through both textual inversion and dreambooth models, though they take a while to train. https://textual-inversion.github.io/ https://textual-inversion.github.io/ https://dreambooth.github.io/ https://dreambooth.github.io/
  - danielbln 4y ago
    Training can be done in under an hour[1], really not that long. And yes, what OP is saying is already possible, which seems to be par for the course for this "new" AI space, as it's moving so fast. [1] https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast-DreamBooth.ipynb https://colab.research.google.com/github/TheLastBen/fast-sta...
    - DrSiemer 4y ago
      It is possible, but right now it still takes quite a bit of time and effort to get it right. The main challenge is finding the right balance between "make something that looks exactly like this" and "put it in a completely different context". Better similarity equals less flexibility. For now, the most effective combination will be artist + AI, although it does feel a bit like those that incorporate it in their workflow are helping to dig their own grave.
      - sillysaurusx 4y ago
        Do you happen to have any screenshots of what you mean? I’m really curious to see dreambooth’s capabilities in the field, and it sounds like you’ve had experience with some of its pitfalls.
        danielbln 4y ago
        Basically OP says that overfitting is a common pitfall, you don't want to overtrain the model, because then everything will look like your training data, and vice versa with not enough training steps. So it's a bit of a balance. If you search for "dreambooth" on the SD subreddit, you will see a lot of examples of dreambooth results and also some that show overfitted and underfitted results. https://www.reddit.com/r/StableDiffusion/search?q=dreambooth&restrict_sr=on&sort=relevance&t=all https://www.reddit.com/r/StableDiffusion/search?q=dreambooth...
- MrPatan 4y ago
  Aren't the fundamental pieces already there? 1 Generate a bunch of characters or objects based on a prompt. 2 Pick the one you like 3 Tell the AI to extract the character's traits from that one picture 4 Miracle happens (I don't know. What does a "character definition file" look like?) 5 Make a new prompt, but add the character definition from step 4, so you get the same traits, only in a different setting or position, etc
- magic_hamster 4y ago
  It sounds like you're literally describing Dreambooth. You can easily and quickly train models for certain styles like Disney characters, certain illustrators, specific artists and even tailor the model to a specific person so you can produce countless pictures of them in different poses, locations and clothes. This is already available.
  - sillysaurusx 4y ago
    Indeed: https://dreambooth.github.io https://dreambooth.github.io Note that training is extremely expensive, and is beyond the capabilities of most end users. Here are the details of their training method: > Given ~3-5 images of a subject we fine tune a text-to-image diffusion in two steps: (a) fine tuning the low-resolution text-to-image model with the input images paired with a text prompt containing a unique identifier and the name of the class the subject belongs to (e.g., "A photo of a [T] dog”), in parallel, we apply a class-specific prior preservation loss, which leverages the semantic prior that the model has on the class and encourages it to generate diverse instances belong to the subject's class by injecting the class name in the text prompt (e.g., "A photo of a dog”). (b) fine-tuning the super resolution components with pairs of low-resolution and high-resolution images taken from our input images set, which enables us to maintain high-fidelity to small details of the subject. Each fine-tuned model is a copy of the original model. So if the model is 10GB, the fine tuned version will be a separate 10GB file. That might not sound like a lot, but it quickly adds up. In this case, end users are artists. One could imagine a cloud-based art program which will fine tune on demand. That certainly seems like a good startup idea.
    - gpderetta 4y ago
      > Note that training is extremely expensive, and is beyond the capabilities of most end users. You can't run it on consumer hardware, but you can just rent a GPU (or use a free collab book) for a few hours to generate the model. Then you download and reuse it locally at will. Yes, you need storage and if you train often it can get expensive, but it is by no mean out of the end user, at least professional end users, capabilities. And of course there are growing libraries of freely available pretrained models. > In this case, end users are artists. One could imagine a cloud-based art program which will fine tune on demand. That certainly seems like a good startup idea. Very much agree about this. At least for a while, I strongly believe that AI will just be another tool for artists willing to embrace it, far from replacing them.
- otabdeveloper4 4y ago
  "AI" is only copy-pasting elements from a database of existing images. It's effectively just a fancy image search engine. You can use it to make procedurally-generated art, but it's still obvious what the source material was.
  - sillysaurusx 4y ago
    This isn’t true, but it’s 4am, and I regret that I can’t type out a full rebuttal on my phone. One obvious counterexample is stylegan interpolations. If you interpolate between two images, the midpoint is usually unique — it’s often not obvious what the source material was. (E.g. Gwern’s anime interpolations; sure, they’re anime faces, but from where? “All of danbooru” might as well be “all styles of anime ever created.”) Maybe someone else can argue the point further.
  - deleted 4y ago
    [deleted]
  - grumbel 4y ago
    Go to your favorite image search engine and try to replicate anything AI has generated. It's impossible. The results you get from image search are nowhere near as specific as what the AI produces. It's not even close. The only time image search can compete is when you are highly specific, e.g. "painting of the Mono Lisa", both AI and image search will produce very similar results. But for a generic prompt, image search will come up with nothing that gets even close to the query, while AI can produce a highly specific image. DreamBooth really should have destroyed all doubt about this point point, as the AI can generated highly specific images of subjects that aren't even in the original training set.
    - otabdeveloper4 4y ago
      > as the AI can generated highly specific images of subjects that aren't even in the original training set I haven't seen any concrete examples of this yet.
  - cthalupa 4y ago
    It's pretty impressive that Stable Diffusion can compress over 200TB of already previously compressed images from LAION 5B down to a few gigabytes! And that it can search that database so quickly to copy and paste things from the right images! /s There's a lot of good discussion to be had around the ethics of AI art, training on copyrighted materials, etc. But it is equivocally not just copy pasting from a database of images.
onion2k 4y ago
I honestly thought pixel art style games were made by creating 3D assets and then rendering them in low resolutions as sprite sheets these days. Or using the 3D models in the game with a 'pixelate' shader. The idea that people still hand draw sprites slightly blows my mind.
- mikkom 4y ago
  Why? It's much easier to draw a pixel sprite instead of creating a 3d model and then doing tricks to render it as 2d.. And hand-drawn images look better if the artist is skilled.
  - CodeArtisan 4y ago
    King of Fighter 12 sprites, which are regarded as being high quality, were made from 3d models. They first animated 3D models, selected specifics frames, then traced sprite over those. https://kofaniv.snk-corp.co.jp/english/info/15th_anniv/2d_dot/creation/index.php https://kofaniv.snk-corp.co.jp/english/info/15th_anniv/2d_do...
    - josefx 4y ago
      > then traced sprite over those. In the past this was called roto scoping and used to capture outlines and movement of real objects for animation. The end result is still a hand drawn and shaded object instead of just a screenshot of a posed 3D model.
      - CodeArtisan 4y ago
        ah yes. now that you say it, i recall they did that for Terminator 2. https://www.youtube.com/watch?v=D0xp74uIZO4 https://www.youtube.com/watch?v=D0xp74uIZO4
  - onion2k 4y ago
    Why? It's much easier to draw a pixel sprite instead of creating a 3d model and then doing tricks to render it as 2d.. It's never a sprite though. Even back on the Super Nintendo game sprites had 9 angles * lots of actions * lots of frames for every animation. Multiply that by different lighting in modern games, and using the power of a 3D engine starts looking like an obvious choice. I know lots of games do this for environments. I assumed they did it for everything.
seibelj 4y ago
hollowturtle 4y ago
As a hobby game programmer stable diffusion looks exciting, the idea of quickly generating some assets for a game jam is very appealing. But as a hobby musician it strikes me hard. I want to write, compose and play my own music, so I would understand someone feeling the same with regards to hand drawing arts. This whole stable diffusion is exciting on one side, but on the other makes me wonder what we will be left with once this technology reaches higher capabilities?
- yieldcrv 4y ago
  Do the extended creative process for yourself Just like a hipster/enthusiast with a more manual photography and development process some people might appreciate it, the market likely wont but thats already the same before AI
- mkmk3 4y ago
  It looks bad for career artists. Maybe it's an important step in terms of human expression and communication though, that visual and musical art won't require years of skill to convey your ideas or emotions effectively. I don't have much of a solarpunk outlook on this, I don't think it's going to be that beneficial. It'd be nice to be wrong though.
  - hollowturtle 4y ago
    That's actually my point, without the requirement of years skill building, what's left for us to do? Also, imo, what makes art so human is also failure, it's part of the game. e.g. a successful musician releasing a bad record, or a not more relevant musician that after years of experimentation reinvent himself and publish a great record. I don't have a clear outlook on this too, only questions about the future
  - Applejinx 4y ago
    I don't know. I've put out an album largely driven by modular synth, meaning that I've got many percussion elements driven literally by a machine, defeating the need for me as a human to play drums in perfect time (which I'm not good at) But in doing that, I've spent years learning how the micro-timing of musical grooves work (some of it's very obvious, like how a heavy snare backbeat will lag slightly behind what the perfect time is, almost to the point of being a 'flam') As a result, I was able to make an album that conveyed certain kinds of groove not immediately accessible to novice musicians (or drummers)… but given the same information, any schmoe could push a button and have a preset in his DAW produce the same effect. I think to some extent if the person doesn't really understand the purpose or need for such an effect, their grasp of how to implement and craft it will be pretty loose. If you don't know why you're doing the effortless thing you're not going to guide it very well and your results will be kind of generic. Rather than thinking of years of skill, maybe call it years of focus, or years of purpose? To some extent, we collectively respond to creations with a profound sense of purpose. If that purpose comes out of AI it will have to be something from the AI, and not simply a blind reflection of us and our crudest drives.
noobermin 4y ago
For the last time, the issue with "AI" is not that it exists as a tool, all of the issue that anyone should have is in how the data used to fit is obtained. No one would have any issue if you took the time to draw thousands of images then trained on them and lived off of that, just don't steal data from others, hand-wave as "free use" and carry off into the sunset on data you didn't generate. The tools behind AI are fine and have honestly existed for decades. If anyone is up against AI because it isn't authentic or something, that's a fools errand really because people, artists themselves and developers, will find them useful. The problem is and always will be how the data you fit on and how you obtained it.
- fleddr 4y ago
  Even the issue you mention, training without consent, won't stop this. Chances are really very low that this will become illegal and enforceable. It would require some very draconian laws, whilst copyright legislation is low priority in government circles, even more so in these times. Even if somehow this would be outlawed in the US, nobody cares internationally. Right now, on Amazon you can buy knockoffs of millions of products from China that violate IP/copyright. Nobody cares. Do you think they will care about something as worthless as a digital image? A digital image that can't even be reliably detected as being AI generated? And there's yet another work-around. Scrape images that don't require consent or make consent part of terms and conditions. Google made Google Photos free for about a decade, and trained it for free on all your stuff.
spencerf 4y ago
With the image generators coming out I’ve been scrambling to understanding their place, I’ve been looking to the chess world as a model of the future. In chess, people would still rather see people play chess than a robot. The top chess players in the world started as a brute force obsession about the game. Those would go on to teach the next generation. They advent of computers allowed for historically statically advantaged moves. ML came along and disrupted even further. Now many of the top chess players consult the ML chess oracle. I see the same thing happening in a lot of areas: grammar, image generation, text replies. I see a world where humans are celebrated for their humanness while machines assist.
- magic_hamster 4y ago
  This is vastly different. Chess is a battle of minds between people while you don't necessarily need context to enjoy art, and that's where AI art is likely to take over. Yes, considering the story behind art pieces and the artist does significantly impact the way we interact with art, but you can still just like a painting or a piece of pixel art without knowing anything else about it. While Chess is about the players, art has products that exist on their own.
  - _bkyr 4y ago
    > That's where AI art is likely to take over I guess that depends on what the definition of art is. If it's a digitally rendered illustration then maybe. If it's a physical object crafted to imperfect perfections then no. AI can probably produce some alternative version of Guernica but that's just a fascimile of something that's already been made in a digital space. I can see an artist using these tools as a way to produce ground-breaking work in the future but I would wager the artist that does that could make good art without any of these tools. You still need a craftsman to master the tools and without knowing the basics you're left with images of Elon Musk as a Disney princess on repeat.
- qikInNdOutReply 4y ago
  Sort of like the paralympics?
- 4y ago
kleiba 4y ago
I've already lost track of all the different apps that let you play with Stable Diffusion et al. Do you have any recommendations for (web)apps that allow you to generate images from prompts in good resolutions? How about ones for img2img?
- asicsp 4y ago
  Check out https://stablehorde.net/ https://stablehorde.net/
asicsp 4y ago
The article linked to a twitter search: https://twitter.com/search?q=KaliYuga%20pixel%20diffusion&src=typed_query https://twitter.com/search?q=KaliYuga%20pixel%20diffusion&sr... Here's another option: https://old.reddit.com/r/StableDiffusion/search?q=pixel+model&restrict_sr=on https://old.reddit.com/r/StableDiffusion/search?q=pixel+mode... For example: * Pixel art sprite sheets: https://old.reddit.com/r/StableDiffusion/comments/y54isd/couldnt_generate_pixel_art_with_sd_so_i_trained_a/ https://old.reddit.com/r/StableDiffusion/comments/y54isd/cou... * pixel-art-v1: https://old.reddit.com/r/StableDiffusion/comments/yj1kbi/ive_trained_a_new_model_to_output_pixel_art/ https://old.reddit.com/r/StableDiffusion/comments/yj1kbi/ive...
aww_dang 4y ago
Looking forward to the day when I can generate 4 direction sprite sheets, walk cycle, attacks and equipment.
woolion 4y ago
As someone who draws, there are some obvious aspects where AI generation just works perfectly: - generate details and texture, which photobashing was already used for; but that mostly solve the licensing problem for it - generate random inspiration boards, for which image search was used (again, mostly solve the licensing problem) - generate derivative stuff, e.g. typical game portraits or props In practice, only the third point is generally discussed, because it lowers tremendously the entry barrier to generate images for people without any skills. It's like if you could pick up screenshots from other content, clean them up, and you're free to use it. Whereas it essentially does not work for: - cartoony generation. It relies too much on line consistency, visual clarity and abstraction - concept design --not the flashy 10 minutes speedpaint type, but where you have to combine ideas in meaningful ways. In particular hard-surface design which required good 3D thinking and consistency of the whole. These fundamental flaws are omnipresent, but can be hidden by certain styles where things are implied by color blobs, hidden by stylized brushstrokes, or simply an overflow of details (something Midjourney is very good at). All in all, it feels like AI is a danger for people at the bottom of the profession hierarchy, but will elevate people at the top, whose work cannot be replaced. In other words, people who are more akin to be considered "artisans" rather than artists, who will take a prompt and simply clean it. In particular, drawing has something like the 20/80 rule, where all the creative input is in the first 20% and the rest is 'rendering', a very mechanical task which you can mostly do with your brain turned off. As Yumenoley put it, "it was a mistake to let the AI do the interesting part".
- sillysaurusx 4y ago
  Cartoony generation is probably a matter of training on the appropriate source material, for what it’s worth. https://dreambooth.github.io https://dreambooth.github.io shows a glimpse of the future. You’ll be able to upload a few drawings that you want to emulate (e.g. Mickey Mouse), and then you can give it a specific prompt (e.g. Mickey Mouse doing a handstand). You’re probably right about the consistency of 3D concepts, though. On the other hand, I was going to say “If you need a specific table, AI might not be able to help” — but again, dreambooth shows that we might be able to upload a few photos of a certain table, and it’ll take care of the details. Give it a few years. :) I think you’re spot on that AI will be an incredible tool for artisans. I used it to make some video game music: https://soundcloud.com/theshawwn/sets/ai-generated-videogame-music https://soundcloud.com/theshawwn/sets/ai-generated-videogame... Even though I can’t play any instruments too well, I was able to craft each piece uniquely. (My favorite is “Crossing the Channel”, which has a strange rhythm because I’m pretty sure the AI made a mistake at the beginning, and then extrapolated the next “actually, this isn’t a mistake” song that it thought of, which turned out to sound cool. A bit like a guitarist doing improv.)
hownottowrite 4y ago
Creative director is one of the many hats I wear these days. I am using components of AI in nearly all of my visual assets. The future is already here.
TaupeRanger 4y ago
Using these generators as tools to find inspiration seems like the best case scenario. I think people are assigning too little probability to the potential future scenario that current systems are about as good as we're going to get with current ML methods, though some refinements will marginally improve them. Without a major breakthrough, I don't see any AI systems replacing professional artists on video games, e.g., unless it's a very low effort, low quality game.
- kuratkull 4y ago
  In 2014 we considered it almost impossible for computers to understand the contents of images. It took a few years for that to become possible. Now ML models are creating amazing images themselves. I don't see what your skepticism is based on. https://xkcd.com/1425/ https://xkcd.com/1425/
- grumbel 4y ago
  There are still major breakthrough and improvements every few weeks, so I really wouldn't worry about already having hit rock bottom. But even ignoring that, we have barely even stared exploring what we can do with the technology as is. A lot of it is still just experiments living in a git repository or need more GPU than the average person has. Give it a few more months or years, and you'll have it integrated into every major photo and video editor software and optimized to run on normal consumer hardware. That simple improvement in accessibility will have very wide reaching consequences just by itself, even without improving the underlying AI drastically. And no, you won't replace the professional artists anytime soon, after all somebody still need to have the final say into what goes into the game, but it will drastically transform how that artist will work and the amount of content they'll be able to produce. > e.g., unless it's a very low effort, low quality game. The output of Midjourney and Co. already looks spectacular, easily better than a lot of games out there. I could easily see that replacing or enhancing a lot of art in 2D RPGs, point&click adventures or visual novels. Everything that needs animation or 3D meshes will take a while longer, but for 2D games it's already more than good enough. It's really more an issue with artists and game developers still needing to catch up on all the rapid new developments that happened over the last few months.
a_shovel 4y ago
> 200x200 is relatively large for pixel art, but if a single pixel makes this much of a difference it should probably be larger. Huh? Half the point of pixel art is that single pixels make a difference. That art is way above the threshold where one pixel can make too much of a difference.

173 points by aleyan 4y ago

My wife is an artist and has been using some of these systems to inspire her own work with clients - she's said it's both given her creative new ideas, and improved her efficiency by about 30%.

jerojero 4y ago

There are some fundamental pieces missing in AI art generation that when solved will completely change the game. Forever. I think most fundamentally is the capability for AI to have some kind of memory or maybe more technically a way for the AI to be capable of do style "character style transfer" more effectively. I think this is probably possible, it's akin to making deep fakes but for drawn/photography art. With this tool suddenly the effort of making a series of compositions that are coherent will dramatically change the game, specially in the videogame industry. I think there are a lot of programmers out there capable of making great games but might be lacking the resources to fully complete their visions due to having to needing assets for their games. It seems to me like the approach stable diffusion has taken has dramatically increased interest and utility of these tools. So I'm hoping they follow similar lines for other types of AI. Every week I'm reading of a new novel use for these generators that I hadn't really considered before.

prox 4y ago
I wonder where it is going to. I was thinking of my favorite concept artists and what it means for them. Probably they can churn out 10x the work, and the role becomes more of curation, as in, scanning and judging the work put out by SD. They probably still going to adapt the work to fit their needs, and so companies probably still want actual artists for entertainment and artistic purposes. Small companies (and individuals) probably can use it to circumvent costly stock images all together, so the lower tier photographers/artists there gonna have a problem.
- gpderetta 4y ago
  I think most actual work will be done via img2img, outpainting and inpainting, and an real artist can actually give an important contribution.
  - prox 4y ago
    At the end of the day you still want something that sticks out, and artists are trained at that.
potatoman22 4y ago
This style transfer is possible through both textual inversion and dreambooth models, though they take a while to train. https://textual-inversion.github.io/ https://textual-inversion.github.io/ https://dreambooth.github.io/ https://dreambooth.github.io/
- danielbln 4y ago
  Training can be done in under an hour[1], really not that long. And yes, what OP is saying is already possible, which seems to be par for the course for this "new" AI space, as it's moving so fast. [1] https://colab.research.google.com/github/TheLastBen/fast-stable-diffusion/blob/main/fast-DreamBooth.ipynb https://colab.research.google.com/github/TheLastBen/fast-sta...
  - DrSiemer 4y ago
    It is possible, but right now it still takes quite a bit of time and effort to get it right. The main challenge is finding the right balance between "make something that looks exactly like this" and "put it in a completely different context". Better similarity equals less flexibility. For now, the most effective combination will be artist + AI, although it does feel a bit like those that incorporate it in their workflow are helping to dig their own grave.
    - sillysaurusx 4y ago
      Do you happen to have any screenshots of what you mean? I’m really curious to see dreambooth’s capabilities in the field, and it sounds like you’ve had experience with some of its pitfalls.
      - danielbln 4y ago
        Basically OP says that overfitting is a common pitfall, you don't want to overtrain the model, because then everything will look like your training data, and vice versa with not enough training steps. So it's a bit of a balance. If you search for "dreambooth" on the SD subreddit, you will see a lot of examples of dreambooth results and also some that show overfitted and underfitted results. https://www.reddit.com/r/StableDiffusion/search?q=dreambooth&restrict_sr=on&sort=relevance&t=all https://www.reddit.com/r/StableDiffusion/search?q=dreambooth...
MrPatan 4y ago
Aren't the fundamental pieces already there? 1 Generate a bunch of characters or objects based on a prompt. 2 Pick the one you like 3 Tell the AI to extract the character's traits from that one picture 4 Miracle happens (I don't know. What does a "character definition file" look like?) 5 Make a new prompt, but add the character definition from step 4, so you get the same traits, only in a different setting or position, etc
magic_hamster 4y ago
It sounds like you're literally describing Dreambooth. You can easily and quickly train models for certain styles like Disney characters, certain illustrators, specific artists and even tailor the model to a specific person so you can produce countless pictures of them in different poses, locations and clothes. This is already available.
- sillysaurusx 4y ago
  Indeed: https://dreambooth.github.io https://dreambooth.github.io Note that training is extremely expensive, and is beyond the capabilities of most end users. Here are the details of their training method: > Given ~3-5 images of a subject we fine tune a text-to-image diffusion in two steps: (a) fine tuning the low-resolution text-to-image model with the input images paired with a text prompt containing a unique identifier and the name of the class the subject belongs to (e.g., "A photo of a [T] dog”), in parallel, we apply a class-specific prior preservation loss, which leverages the semantic prior that the model has on the class and encourages it to generate diverse instances belong to the subject's class by injecting the class name in the text prompt (e.g., "A photo of a dog”). (b) fine-tuning the super resolution components with pairs of low-resolution and high-resolution images taken from our input images set, which enables us to maintain high-fidelity to small details of the subject. Each fine-tuned model is a copy of the original model. So if the model is 10GB, the fine tuned version will be a separate 10GB file. That might not sound like a lot, but it quickly adds up. In this case, end users are artists. One could imagine a cloud-based art program which will fine tune on demand. That certainly seems like a good startup idea.
  - gpderetta 4y ago
    > Note that training is extremely expensive, and is beyond the capabilities of most end users. You can't run it on consumer hardware, but you can just rent a GPU (or use a free collab book) for a few hours to generate the model. Then you download and reuse it locally at will. Yes, you need storage and if you train often it can get expensive, but it is by no mean out of the end user, at least professional end users, capabilities. And of course there are growing libraries of freely available pretrained models. > In this case, end users are artists. One could imagine a cloud-based art program which will fine tune on demand. That certainly seems like a good startup idea. Very much agree about this. At least for a while, I strongly believe that AI will just be another tool for artists willing to embrace it, far from replacing them.
otabdeveloper4 4y ago
"AI" is only copy-pasting elements from a database of existing images. It's effectively just a fancy image search engine. You can use it to make procedurally-generated art, but it's still obvious what the source material was.
- sillysaurusx 4y ago
  This isn’t true, but it’s 4am, and I regret that I can’t type out a full rebuttal on my phone. One obvious counterexample is stylegan interpolations. If you interpolate between two images, the midpoint is usually unique — it’s often not obvious what the source material was. (E.g. Gwern’s anime interpolations; sure, they’re anime faces, but from where? “All of danbooru” might as well be “all styles of anime ever created.”) Maybe someone else can argue the point further.
- deleted 4y ago
  [deleted]
- grumbel 4y ago
  Go to your favorite image search engine and try to replicate anything AI has generated. It's impossible. The results you get from image search are nowhere near as specific as what the AI produces. It's not even close. The only time image search can compete is when you are highly specific, e.g. "painting of the Mono Lisa", both AI and image search will produce very similar results. But for a generic prompt, image search will come up with nothing that gets even close to the query, while AI can produce a highly specific image. DreamBooth really should have destroyed all doubt about this point point, as the AI can generated highly specific images of subjects that aren't even in the original training set.
  - otabdeveloper4 4y ago
    > as the AI can generated highly specific images of subjects that aren't even in the original training set I haven't seen any concrete examples of this yet.
- cthalupa 4y ago
  It's pretty impressive that Stable Diffusion can compress over 200TB of already previously compressed images from LAION 5B down to a few gigabytes! And that it can search that database so quickly to copy and paste things from the right images! /s There's a lot of good discussion to be had around the ethics of AI art, training on copyrighted materials, etc. But it is equivocally not just copy pasting from a database of images.

onion2k 4y ago

I honestly thought pixel art style games were made by creating 3D assets and then rendering them in low resolutions as sprite sheets these days. Or using the 3D models in the game with a 'pixelate' shader. The idea that people still hand draw sprites slightly blows my mind.

mikkom 4y ago
Why? It's much easier to draw a pixel sprite instead of creating a 3d model and then doing tricks to render it as 2d.. And hand-drawn images look better if the artist is skilled.
- CodeArtisan 4y ago
  King of Fighter 12 sprites, which are regarded as being high quality, were made from 3d models. They first animated 3D models, selected specifics frames, then traced sprite over those. https://kofaniv.snk-corp.co.jp/english/info/15th_anniv/2d_dot/creation/index.php https://kofaniv.snk-corp.co.jp/english/info/15th_anniv/2d_do...
  - josefx 4y ago
    > then traced sprite over those. In the past this was called roto scoping and used to capture outlines and movement of real objects for animation. The end result is still a hand drawn and shaded object instead of just a screenshot of a posed 3D model.
    - CodeArtisan 4y ago
      ah yes. now that you say it, i recall they did that for Terminator 2. https://www.youtube.com/watch?v=D0xp74uIZO4 https://www.youtube.com/watch?v=D0xp74uIZO4
- onion2k 4y ago
  Why? It's much easier to draw a pixel sprite instead of creating a 3d model and then doing tricks to render it as 2d.. It's never a sprite though. Even back on the Super Nintendo game sprites had 9 angles * lots of actions * lots of frames for every animation. Multiply that by different lighting in modern games, and using the power of a 3D engine starts looking like an obvious choice. I know lots of games do this for environments. I assumed they did it for everything.

seibelj 4y ago

hollowturtle 4y ago

As a hobby game programmer stable diffusion looks exciting, the idea of quickly generating some assets for a game jam is very appealing. But as a hobby musician it strikes me hard. I want to write, compose and play my own music, so I would understand someone feeling the same with regards to hand drawing arts. This whole stable diffusion is exciting on one side, but on the other makes me wonder what we will be left with once this technology reaches higher capabilities?

yieldcrv 4y ago
Do the extended creative process for yourself Just like a hipster/enthusiast with a more manual photography and development process some people might appreciate it, the market likely wont but thats already the same before AI
mkmk3 4y ago
It looks bad for career artists. Maybe it's an important step in terms of human expression and communication though, that visual and musical art won't require years of skill to convey your ideas or emotions effectively. I don't have much of a solarpunk outlook on this, I don't think it's going to be that beneficial. It'd be nice to be wrong though.
- hollowturtle 4y ago
  That's actually my point, without the requirement of years skill building, what's left for us to do? Also, imo, what makes art so human is also failure, it's part of the game. e.g. a successful musician releasing a bad record, or a not more relevant musician that after years of experimentation reinvent himself and publish a great record. I don't have a clear outlook on this too, only questions about the future
- Applejinx 4y ago
  I don't know. I've put out an album largely driven by modular synth, meaning that I've got many percussion elements driven literally by a machine, defeating the need for me as a human to play drums in perfect time (which I'm not good at) But in doing that, I've spent years learning how the micro-timing of musical grooves work (some of it's very obvious, like how a heavy snare backbeat will lag slightly behind what the perfect time is, almost to the point of being a 'flam') As a result, I was able to make an album that conveyed certain kinds of groove not immediately accessible to novice musicians (or drummers)… but given the same information, any schmoe could push a button and have a preset in his DAW produce the same effect. I think to some extent if the person doesn't really understand the purpose or need for such an effect, their grasp of how to implement and craft it will be pretty loose. If you don't know why you're doing the effortless thing you're not going to guide it very well and your results will be kind of generic. Rather than thinking of years of skill, maybe call it years of focus, or years of purpose? To some extent, we collectively respond to creations with a profound sense of purpose. If that purpose comes out of AI it will have to be something from the AI, and not simply a blind reflection of us and our crudest drives.

noobermin 4y ago

For the last time, the issue with "AI" is not that it exists as a tool, all of the issue that anyone should have is in how the data used to fit is obtained. No one would have any issue if you took the time to draw thousands of images then trained on them and lived off of that, just don't steal data from others, hand-wave as "free use" and carry off into the sunset on data you didn't generate. The tools behind AI are fine and have honestly existed for decades. If anyone is up against AI because it isn't authentic or something, that's a fools errand really because people, artists themselves and developers, will find them useful. The problem is and always will be how the data you fit on and how you obtained it.

fleddr 4y ago
Even the issue you mention, training without consent, won't stop this. Chances are really very low that this will become illegal and enforceable. It would require some very draconian laws, whilst copyright legislation is low priority in government circles, even more so in these times. Even if somehow this would be outlawed in the US, nobody cares internationally. Right now, on Amazon you can buy knockoffs of millions of products from China that violate IP/copyright. Nobody cares. Do you think they will care about something as worthless as a digital image? A digital image that can't even be reliably detected as being AI generated? And there's yet another work-around. Scrape images that don't require consent or make consent part of terms and conditions. Google made Google Photos free for about a decade, and trained it for free on all your stuff.

spencerf 4y ago

With the image generators coming out I’ve been scrambling to understanding their place, I’ve been looking to the chess world as a model of the future. In chess, people would still rather see people play chess than a robot. The top chess players in the world started as a brute force obsession about the game. Those would go on to teach the next generation. They advent of computers allowed for historically statically advantaged moves. ML came along and disrupted even further. Now many of the top chess players consult the ML chess oracle. I see the same thing happening in a lot of areas: grammar, image generation, text replies. I see a world where humans are celebrated for their humanness while machines assist.

magic_hamster 4y ago
This is vastly different. Chess is a battle of minds between people while you don't necessarily need context to enjoy art, and that's where AI art is likely to take over. Yes, considering the story behind art pieces and the artist does significantly impact the way we interact with art, but you can still just like a painting or a piece of pixel art without knowing anything else about it. While Chess is about the players, art has products that exist on their own.
- _bkyr 4y ago
  > That's where AI art is likely to take over I guess that depends on what the definition of art is. If it's a digitally rendered illustration then maybe. If it's a physical object crafted to imperfect perfections then no. AI can probably produce some alternative version of Guernica but that's just a fascimile of something that's already been made in a digital space. I can see an artist using these tools as a way to produce ground-breaking work in the future but I would wager the artist that does that could make good art without any of these tools. You still need a craftsman to master the tools and without knowing the basics you're left with images of Elon Musk as a Disney princess on repeat.
qikInNdOutReply 4y ago
Sort of like the paralympics?
4y ago

kleiba 4y ago

I've already lost track of all the different apps that let you play with Stable Diffusion et al. Do you have any recommendations for (web)apps that allow you to generate images from prompts in good resolutions? How about ones for img2img?

asicsp 4y ago
Check out https://stablehorde.net/ https://stablehorde.net/

asicsp 4y ago

The article linked to a twitter search: https://twitter.com/search?q=KaliYuga%20pixel%20diffusion&src=typed_query https://twitter.com/search?q=KaliYuga%20pixel%20diffusion&sr... Here's another option: https://old.reddit.com/r/StableDiffusion/search?q=pixel+model&restrict_sr=on https://old.reddit.com/r/StableDiffusion/search?q=pixel+mode... For example: * Pixel art sprite sheets: https://old.reddit.com/r/StableDiffusion/comments/y54isd/couldnt_generate_pixel_art_with_sd_so_i_trained_a/ https://old.reddit.com/r/StableDiffusion/comments/y54isd/cou... * pixel-art-v1: https://old.reddit.com/r/StableDiffusion/comments/yj1kbi/ive_trained_a_new_model_to_output_pixel_art/ https://old.reddit.com/r/StableDiffusion/comments/yj1kbi/ive...

aww_dang 4y ago

Looking forward to the day when I can generate 4 direction sprite sheets, walk cycle, attacks and equipment.

woolion 4y ago

As someone who draws, there are some obvious aspects where AI generation just works perfectly: - generate details and texture, which photobashing was already used for; but that mostly solve the licensing problem for it - generate random inspiration boards, for which image search was used (again, mostly solve the licensing problem) - generate derivative stuff, e.g. typical game portraits or props In practice, only the third point is generally discussed, because it lowers tremendously the entry barrier to generate images for people without any skills. It's like if you could pick up screenshots from other content, clean them up, and you're free to use it. Whereas it essentially does not work for: - cartoony generation. It relies too much on line consistency, visual clarity and abstraction - concept design --not the flashy 10 minutes speedpaint type, but where you have to combine ideas in meaningful ways. In particular hard-surface design which required good 3D thinking and consistency of the whole. These fundamental flaws are omnipresent, but can be hidden by certain styles where things are implied by color blobs, hidden by stylized brushstrokes, or simply an overflow of details (something Midjourney is very good at). All in all, it feels like AI is a danger for people at the bottom of the profession hierarchy, but will elevate people at the top, whose work cannot be replaced. In other words, people who are more akin to be considered "artisans" rather than artists, who will take a prompt and simply clean it. In particular, drawing has something like the 20/80 rule, where all the creative input is in the first 20% and the rest is 'rendering', a very mechanical task which you can mostly do with your brain turned off. As Yumenoley put it, "it was a mistake to let the AI do the interesting part".

sillysaurusx 4y ago
Cartoony generation is probably a matter of training on the appropriate source material, for what it’s worth. https://dreambooth.github.io https://dreambooth.github.io shows a glimpse of the future. You’ll be able to upload a few drawings that you want to emulate (e.g. Mickey Mouse), and then you can give it a specific prompt (e.g. Mickey Mouse doing a handstand). You’re probably right about the consistency of 3D concepts, though. On the other hand, I was going to say “If you need a specific table, AI might not be able to help” — but again, dreambooth shows that we might be able to upload a few photos of a certain table, and it’ll take care of the details. Give it a few years. :) I think you’re spot on that AI will be an incredible tool for artisans. I used it to make some video game music: https://soundcloud.com/theshawwn/sets/ai-generated-videogame-music https://soundcloud.com/theshawwn/sets/ai-generated-videogame... Even though I can’t play any instruments too well, I was able to craft each piece uniquely. (My favorite is “Crossing the Channel”, which has a strange rhythm because I’m pretty sure the AI made a mistake at the beginning, and then extrapolated the next “actually, this isn’t a mistake” song that it thought of, which turned out to sound cool. A bit like a guitarist doing improv.)

hownottowrite 4y ago

Creative director is one of the many hats I wear these days. I am using components of AI in nearly all of my visual assets. The future is already here.

TaupeRanger 4y ago

Using these generators as tools to find inspiration seems like the best case scenario. I think people are assigning too little probability to the potential future scenario that current systems are about as good as we're going to get with current ML methods, though some refinements will marginally improve them. Without a major breakthrough, I don't see any AI systems replacing professional artists on video games, e.g., unless it's a very low effort, low quality game.

kuratkull 4y ago
In 2014 we considered it almost impossible for computers to understand the contents of images. It took a few years for that to become possible. Now ML models are creating amazing images themselves. I don't see what your skepticism is based on. https://xkcd.com/1425/ https://xkcd.com/1425/
grumbel 4y ago
There are still major breakthrough and improvements every few weeks, so I really wouldn't worry about already having hit rock bottom. But even ignoring that, we have barely even stared exploring what we can do with the technology as is. A lot of it is still just experiments living in a git repository or need more GPU than the average person has. Give it a few more months or years, and you'll have it integrated into every major photo and video editor software and optimized to run on normal consumer hardware. That simple improvement in accessibility will have very wide reaching consequences just by itself, even without improving the underlying AI drastically. And no, you won't replace the professional artists anytime soon, after all somebody still need to have the final say into what goes into the game, but it will drastically transform how that artist will work and the amount of content they'll be able to produce. > e.g., unless it's a very low effort, low quality game. The output of Midjourney and Co. already looks spectacular, easily better than a lot of games out there. I could easily see that replacing or enhancing a lot of art in 2D RPGs, point&click adventures or visual novels. Everything that needs animation or 3D meshes will take a while longer, but for 2D games it's already more than good enough. It's really more an issue with artists and game developers still needing to catch up on all the rapid new developments that happened over the last few months.

a_shovel 4y ago

> 200x200 is relatively large for pixel art, but if a single pixel makes this much of a difference it should probably be larger. Huh? Half the point of pixel art is that single pixels make a difference. That art is way above the threshold where one pixel can make too much of a difference.