12 ms·

I am frustrated with Stable Diffusion

172 points by luu 4y ago

jarbus 4y ago
Online, stable diffusion looks incredible. But it's been so, so bad for me, and when you look closely at lots of other "good" pictures, they have lots of flaws too, we just aren't good at picking them out. I wonder if there will be a way to make SD align with what humans want more, maybe with some GAN approach, without needed actual "understanding" of the real world.
- dwringer 4y ago
  Using img2img and making subtle alterations/substitutions to the prompts and/or images as you go can allow one to serve the adversarial role oneself as the user. Ultimately it's sort of like sitting across the table from an artist who can't speak but you can pass drawings back and forth iterating on them until they are complete. Of course it can be drawings, 3D animation style rendering, painting, photorealistic renders, etc.
- kecupochren 4y ago
  Try MidJourney instead. Very little is needed to get amazing results. You just set the theme and let it run wild. Or you can also be very specific and it will handle it.
skybrian 4y ago
I tried this using Stable Diffusion v2.1 at dreamstudio.ai. I'm not getting good results yet. It seems like an odd place to build a dock, if you can avoid it. Is there a real-world place like that?
- paulcole 4y ago
  This place in Malta is pretty close: https://www.alamy.com/stock-photo-boats-at-dock-near-a-set-of-steps-leading-up-cliffs-in-malta-51572197.html https://www.alamy.com/stock-photo-boats-at-dock-near-a-set-o... I actually thought the first version in the blog post was pretty good. It looked like a floating dock to me (that goes up and down with the water level) rather than one secured by pilings.
- robocat 4y ago
  Cap de Formentor in Majorca is an example, and I'm sure you could find better examples for other lighthouses. I remember clambering down the epic walkway to where all the lighthouse building materials were brought in - and there must have been a jetty or something because it's fucking steep. The best description and image I could find: https://www.majorcadailybulletin.com/news/local/2021/03/16/80955/mallorca-culture-cap-formentor-lighthouse.html https://www.majorcadailybulletin.com/news/local/2021/03/16/8... I think that the "cove" and walkway is shown on the left of the third photo in the article. That said, any jetty would have had to be cantilevered from the rockface. The guy in the article drawing piles next to a shear cliff of rock is hardly thinking the engineering though!
humanistbot 4y ago
I had the exact same experience. I foolishly paid for a pro account without even trying out the free version, because I was so excited by it. Only to learn that it took so much trial and error --- sorry, "prompt engineering" --- to get anything remotely usable. This is the same with all kinds of generative models. Humans are a critical part of the system. If this is "artificial intelligence" it only appears intelligent because there is a human gatekeeper.
- yunwal 4y ago
  All AI models so far are pretty much only useful as a middle visualization step in a creative process. You have to come up with a vague idea of what to draw, SD can give you a number of compositions to choose from and will get close enough that you can tell the final product would be good. You will still have to recreate or heavily edit whatever it creates though. It’s still useful, but it’s not good enough to be a revolutionary product yet or declare photoshop obsolete. Similarly, ChatGPT is a great replacement for a search engine. I can ask it to guide me through writing a metric to Prometheus from a batch job and it’ll dig out the relevant pieces from the documentation and present them to me in an easy-to-understand format. But I’m not giving it write access to main yet, or letting it make business decisions.
- Slow_Hand 4y ago
  This sounds like any other tool or instrument that requires practice to render good results. SD is magic, but it's not a miracle. It still requires work and understanding to get good results. It's just a different kind of work than we're used to doing. Don't expect it read your mind. It's still a computer. As a client, if I hire an illustrator to commission a specific image I'm not just hiring them for their technical ability to render images. The best illustrators are going to have a broad and deep awareness of all styles, modes of presentation, and compositional techniques. They're going to use all of their experience and knowledge to interpret what my needs are and deliver on that. That's not a trivial thing to do. Even the best contractors have to deal with clients who don't know how to ask for what they want. The "I'll know it when I see it" type of client. As the "client" in this situation, you need to know how to ask for what you want in a way that your illustrator can interpret.
fy20 4y ago
What nobody tells you is that to get the results you see people post online often requires hours of work for a single image. First you need to refine the prompt so it is ultra specific (which OP has not done), then you need to generate a hundred or so images and pick the best ones. From there you can use img2img to refine it more, and once that's done you might want to go to Photoshop to add some finishing touches. To get good results it is an art form at the moment, but of course as the tools get better eventually it will just be a click of a button.
- Fricken 4y ago
  So long as you're content with some random crazy image, you can get fancy stuff pretty easily with Midjourney V4. Getting specific details correct, however...
- mch82 4y ago
  Correct. It’s a new medium for a new school of artist, not a replacement for artists. Very exciting!
  - prpl 4y ago
    it’s the new sketch
- vhold 4y ago
  Another trick you can do is when you find a composition you like, but it's off in whatever way, is lock down that seed, and then apply variation to that initial noise state, and produce a bunch more images from there. The popular AUTOMATIC1111 webui implements it: https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#variations https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki...
- godelski 4y ago
  As someone researching generative models, this is one of my pet peeves. There's been a rat race to show off high quality cherry-picked samples. Very few works actually show random samples. It is rather frustrating because when pointed out it still goes ignored. I'm not sure why we look the other way, aren't these issues we want to solve? But if you tried to publish a paper with random results you won't get accepted. I am also not worried about AI taking over art, exactly for the reasons you say. Even if alignment was a whole lot better you're never going to be able to perfectly describe an image in your head. Language is too limited. So it will always be a tool. Of course, it can enable fraud. But not many people are putting digital arts on their walls. The other pet peeve is that people think generative models are only image or text. They so so much more.
sirwhinesalot 4y ago
What (barely) works for me is: - Make a prompt with a similar structure to the ones that generated good images online. - Generate a bunch of low quality images. - Pick the low quality image that kinda sort of has the shape and colors I'm looking for. - Iterate on that image with various other prompts and img2img, increasing how much the original image contributes as it gets better.
theCrowing 4y ago
Mhm...looks like he put more effort into writing this blog post than learning how to work with SD.
InvisGhost 4y ago
I've had a lot of luck using Invoke.AI's solution. It allows for a lot more control over what the AI model is allowed to modify and lets you tailor each prompt for each individual edit.
- xrd 4y ago
  This is my favorite as well. Are you running on your own hardware? I've found the latest commits to be less stable and it's hard to know what to run.
hackyhacky 4y ago
This should really be an easy fix, since Stable Diffusion is open source. Just find and remove the code that reads: if (drawing_bottom_of_cliff) fuckup(dock, RIGHT_SIDE) The team is waiting for your pull request!
Aeolun 4y ago
In my experience, Midjourney has been so much better than Stable Diffusion at anything game related that it’s absurd. It might be worth trying that.
- anentropic 4y ago
  I've had exactly the same issues with midjourney, can't really tell any difference between then
kyleyeats 4y ago
It's all a composition and image-bashing game with Stable Diffusion. A better strategy here would be keeping the same seed and then try adding 'dock' phrases to the prompt. Then image-bashing what it gives you over the first one. Image-bashing here basically means putting it in a new layer in GIMP over the old one and erasing the parts you don't want. It's really great with seed alternates because the two images will be very similar. Inpainting is good for stuff you don't want the viewer to look at. If there's an errant shrub in your desert picture[0], you can inpaint it out. It almost never works for faces, heads, characters or any other focal object. The new depth stuff might fix it, but inpainting generally just doesn't have a good sense of how to orient things in the scene. [0] I generated a bunch of SD images to spice up my CSS library's homepage: https://casscss.github.io/cass/ https://casscss.github.io/cass/
WheelsAtLarge 4y ago
I love SD for its unexpected results. You really don't know what you are going to get. Trying to get it to create the exact image you have in your mind is next to impossible. That's where this guy went wrong he thought he was getting an artist that would do as he asked but SD is not that. There's a lot of randomness associated with it.
elliottkember 4y ago
The gap between something totally new existing, and someone taking it for granted and saying it totally sucks, is still so short. I'm reminded of Louis CK's bit about wifi on planes: "And it's fast, and I'm watching YouTube clips. It's amaz--I'm on an airplane! And then it breaks down. And they apologize, the Internet's not working. And the guy next to me goes, 'This is bullshit.' I mean, how quickly does the world owe him something that he knew existed only 10 seconds ago?"
- esprehn 4y ago
  To be fair in the case of airplane wifi this is compounded by the exorbitant pricing. If you pay a premium price you expect premium service. There's no competition for airplane wifi though, you either use it or you don't. It's not like you choose between two different vendors... So they're competing against no wifi at all which allows for pretty mediocre and unreliable service as that's "better than nothing". These free ML models exist in a slightly different world.
  - guiambros 4y ago
    > To be fair in the case of airplane wifi this is compounded by the exorbitant pricing. Is this still true, though? I regularly pay $8 for wifi on a 6h flight coast-to-coast with United. And also paid $15 for a 10h transatlantic flight last week, which seems pretty fair to me. Oh, and messaging and chat apps are free, so you don't really need to pay if you only want to keep in touch with folks on the ground. Sure, quality is sometimes spotty, but seems to have improved over the last couple of years.
    - agolio 4y ago
      I don't know what it is, but somehow paying even a penny for wi-fi hotspots is incomprehensible by my brain. It could be a 16-hour flight to Australia, and still my brain would be saying no. Then I would order a $10 coffee and tell myself I deserve to treat myself.
      - 4y ago
krisoft 4y ago
One super obvious trick, which the author seems to be not aware of: If you have two pictures and you like one part from one and an other part from an other just photoshop/gimp them together. If a particular manipulation is easier manualy there is no reason you need to try to convince the AI to do it. Just do it yourself. In this case you drag the two images to two layers and use the erase tool to remove the blemishes which are present in your “top” image but not present in the “bottom” one.
- joeld42 4y ago
  You can only get so far with this, the lighting and perspective won't match and you'll end up with a "collage" look, especially as you add more images. You can sometimes feed the result back through img2img and get it to fix it. Even in an illustrative style like this the color tone is going to look different.
  - krisoft 4y ago
    Sure thing. But I mean it in a much more restrained way. They have one picture where they like the left side of the dock, but say that the AI screwed up the right side. And they have a previous one where the right side is good but the left side is bad. And they are frustrated that the AI can’t or won’t make a single generation where both things are good. And the two pictures have the same structure, same illumination and look barely distinguishable otherwise. That is the case where editing is trivial. You might wonder, how often does this really happen in real life, and the answer is: all the time. You get a picture you like overall, and it looks good from 10 feet distance but on closer inspection you notice some blemishes with it. One way to fix this is to “inpaint” the blemishes. Basically you tell the model to keep all the pixels unchanged, but generate new ones in a small region masking out the blemishes. And of course you run many such inpainting generations and pick the best. And when you have multiple blemishes, which is often the case, then very often one gets filled in good in one generation while the other is better in an other. And you can keep rolling the dice hoping that some day the AI will get it right everywhere, or you can get them in photoshop and do the selection manually.
  - pjgalbraith 4y ago
krisoft 4y ago
> The original, 1990s Myst had 2,500 images, which would cost me more than I expect to make this year. Also, it would take an absurd amount of time. Myst was computer rendered. If that is the feel the author is going for why don’t they use the same technique? The number of individual images would certainly not be a limit then.
- agildehaus 4y ago
  Indeed. It's also quite a bit easier, faster, and cheaper to make INSANELY better quality images than it was in 1992 when Myst was being developed. The software and hardware differences between 1992 and today are immense. What hasn't changed though is the artist. Robyn Miller and Chuck Carter squeezed every ounce out of what was available to them. This guy could do the same with Stable Diffusion, but will take work it doesn't seem like he wants to do. He wants Stable Diffusion to be his perfect human-quality artist, and it isn't that.
pie_flavor 4y ago
My first attempt with SD2 was my favorite prompt in SD1 - 'nebulaic sky above a golden sea, 4k desktop wallpaper'. SD1 routinely made the nebula golden instead of the sea, but what the heck, still makes for a great Zoom background. But SD2 just gave me a golden nebula. No sea. I tried everything I could think of to vary the prompt, and nothing would produce an image with the sea in it. Then I ran a sanity check, and asked it 'Sea'. I got some interesting abstract patterns, a comic strip about an airplane, and a bird that looked rather diseased. No seas. Maybe one of the abstract patterns kind of looked like the sea does in anime?. Just to double-sanity-check that prompts don't need to be more refined than that, I searched 'forest' and got some lovely photorealistic pine trees in the mist. SD2 literally doesn't know what the sea is. I just checked and SD2.1 gives you more sea-ish abstract art than SD2.0 did, but still no dice. 'photorealistic sea' gives me two motorcycles and an elephant. edit: Seems v2.1 understands 'ocean' some of the time (not all!). v2.0 did not understand either one.
- Nition 4y ago
  > two motorcycles and an elephant Maybe SEA = Southeast Asia.
- ralfd 4y ago
  What happens with the prompt "nebulaic sky above a golden ocean, 4k desktop wallpaper'"?
- jemmyw 4y ago
  maybe most images are labelled "ocean"?
- TillE 4y ago
  I'd been using SD1 for fantasy game character portraits, with fairly decent results. SD2 refuses to give me a simple headshot no matter what I try, it's just generating pictures which include most of the body as well.
  - astrange 4y ago
    Try using negative prompting, especially negative textual-inversion prompts.
dcj4 4y ago
I'm not.
w4ffl35 4y ago
SD works best when you combine it with a paint program. for example, I generated superman but part of his cape was missing. I brought the image into GIMP and manually painted his cape on (sloppy, with pure red, no attempt at making it nice). Then I ran the image through "img2img" and the cape looked perfect.
- sophrocyne 4y ago
  This is one of the the main reasons for InvokeAI building the Unified Canvas https://youtu.be/hIYBfDtKaus https://youtu.be/hIYBfDtKaus
dorkwood 4y ago
I find this attempt to create something of value without putting in any effort to be frustrating. > Yes, I could pay an artist to make art for the game. But I’m not yet good enough at game marketing for this to make sense. There's a third option: learn to make the art yourself! Yes, it's time consuming, but so is anything worth doing. > The original, 1990s Myst had 2,500 images, which would cost me more than I expect to make this year. Start smaller. Or use a simpler, more achievable art style.
- oneoff786 4y ago
  > There's a third option: learn to make the art yourself! Yes, it's time consuming, but so is anything worth doing. This is terrible logic. If you don’t want to make art, don’t waste time learning to make art.
  - dorkwood 4y ago
    > If you don’t want to make art, don’t waste time learning to make art. Oh, I agree. I wouldn't suggest someone who wasn't interested in art spend time learning to make art. That would be ludicrous. But that's not what I'm seeing in this article. Most of it chronicles the author's attempt to create a single illustration in a particular style. What are they doing, if not attempting to make art?
    - oneoff786 4y ago
      Trying to get art
wokwokwok 4y ago
? Is this a re-post? I swear this is from like 2 days ago, and it turns up in the HN search from two days ago: > I am frustrated with Stable Diffusion(https://novalis.org/blog/2022-12-05-i-am-frustrated-with-stable-diffusion.html https://novalis.org/blog/2022-12-05-i-am-frustrated-with-sta...) > 43 points|luu|2 days ago|40 comments <--- ..but it's been resurrected now? > 42 points by luu 5 hours ago | hide | past | favorite | 38 comments <--- ?? I mean, whatever, but the timestamps seem like they've been edited or something, which is very strange.
- fragmede 4y ago
  Second chance pool https://news.ycombinator.com/item?id=26998309 https://news.ycombinator.com/item?id=26998309
elil17 4y ago
I'd argue the images he generated are good enough to use in an indie game (based on the expectation that good gameplay would make up for the jank).
nineteen999 4y ago
Non artist struggles to make game art with an AI image generator. News at 11.
noduerme 4y ago
So ... this is me as a 42 year old artist, a coder in many languages, an art director for some decent sized companies (probably overpaid while my illustrators are definitely underpaid), and Stable Diffusion enthusiast: HIRE SOME ARTISTS. [edit] if you feel like a long discussion of the cognitive failure of AI over human artists, bring it on; if you're too cheap to hire artists, I totally get it!!! Make your own art! Otherwise, take your brilliant idea and pay other people to execute the parts you're not capable of executing yourself! Don't expect someone else to hand you a robot/AI replacement for your lack of skill for free ;)
- badsectoracula 4y ago
  He brings that up at the end of the article: > Yes, I could pay an artist to make art for the game. But I’m not yet good enough at game marketing for this to make sense: The original, 1990s Myst had 2,500 images, which would cost me more than I expect to make this year. Also, it would take an absurd amount of time.
  - noduerme 4y ago
    I spent 8 years developing all the art for my own game platform and another 4 years on the next... so .. "also, it would take an absurd amount of time" ? Yeah? Well while you search for shortcuts or hope for a free ride, you're not making the art you need or want. What constitutes an "absurd" amount of time is pretty relative. If you hope to make a sizeable amount off your game, maybe it will take what seems right now to be an absurd amount of effort. It only seems absurd because you haven't connected with the reality of spending your time 24/7 making things. i.e. putting out the effort to earn the acclaim you hope for. Slightly a side story, so no one complains: In 1994 when I was 14, "Heart of China" came out and VideoSpigot came out and all I wanted for Xmas was a video camera and an interface to record short QuickTime movies in locations around my house, so I could string them together into a dark futuristic CD-ROM game. (in SuperCard, if you're wondering). That would have solved all my problems creating art! It wouldn't have. Make your game as a sketch with whatever sketching tools you have...but don't expect technology to rescue you from needing other humans to make it good. AND DON'T UNDER-RATE OR UNDERPAY THOSE PEOPLE. Because trust me, if you're disappointed that AI isn't giving you good enough art, you're no visionary. You haven't even tried people.
jasfi 4y ago
I'm tackling this problem with InventAI: https://inventai.xyz https://inventai.xyz. Doing everything I can to launch the MVP this year! Prompt engineering isn't what it should be. It will use DALL-E 2 to start with, but Stable Diffusion 2 is the next back-end I'll add.
ChaitanyaSai 4y ago
We've been experimenting with SD and results are quite underwhelming. DALLE2 seems a lot more usable. Any reason why most are sticking with SD? (If that assumption is true)
- trifurcate 4y ago
  Openness for me. Stable diffusion runs on a machine that sits 5 feet away from me and delivers good quality results in under 3 seconds per image. I don't have to shy away from politically (or otherwise) questionable prompts, or in the case of business applications, potentially leaking sensitive data, because I know that they aren't being recorded forever on a remote server somewhere. I can also tinker with the code, which has helped me add a bunch of features that really synergize with my workflow. As others have noted, getting good results from SD is very much a numbers and parameters game. I've witnessed first hand how some people get frustrated easily and quit tinkering because their initial approaches don't work right away. Prompt and parameter engineering really is a skill that you have to grow, and it makes a world of difference. img2img is also much finickier than text2img.
- speedgoose 4y ago
  StableDiffusion produces better pictures once you get what you want. Dall.e2 is very good at understanding your prompt but the images are not very detailed when you look closely. They are full of artefacts, and the faces are zombi like unless they fill the whole frame. But you can use stable diffusion and dall.e together. Or use MidJourney v4 which is simply the best right now.
  - ChaitanyaSai 4y ago
    How does MidJourney v4 do illustrations. The early versions we saw seemed a little too artistic and also had this distinct "looks like midjourney" signature. Any links you can point me to? Thanks!
    - speedgoose 4y ago
      I haven’t found a good explanation of how mid journey works in detail. Many people guess that they are not collecting the data for no reasons, and that the model is fine tuned using data and user ratings from the previous generations.
nitwit005 4y ago
Points out the original Myst had 2500 images. Even if an AI tool was so good it only took you 10 minutes per picture, that'd be around 416 hours of work, or 52 eight hour days. Presumably some things will need to be reworked, so probably quite a bit more. It'd probably be easier to create an ordinary 3D game world. I'm not going to claim 3D art is easy, but you only need to make your Myst island one time, and then you can just move the camera around.
prakhar897 4y ago
I'm with the author on this one. I'm working on a Stable Diffusion based word guessing game (https://diffudle.com/ https://diffudle.com/). If you look at the prompts and the actual images, they are miles apart. This is after I reject multiple prompts and multiple images from the selected prompt. The worst part is Midjourney is able to create much better images from the same prompt. At this point, I'm seriously considering renaming the product and just move with another image generator.
thot_experiment 4y ago
I dunno man, I put it into img2img and it worked on like the third try. https://imgur.com/a/tbqz59G https://imgur.com/a/tbqz59G a simple rickety dock at the base of an ocean cliff, approach from the sea, picture from a boat, an illustration by moebius, complexla style, an outdoor scene, simple, expansive, ocean, cliff, stairs, dock, realistic colors Negative prompt: isometric, photograph, boat, multiple frames, abstract, complex, borders Steps: 20, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 1136556607, Size: 1024x1024, Model hash: 5e107da3, Denoising strength: 0.51, Eta: 0.4, Mask blur: 4
- jerpint 4y ago
  Wow impressive! I need to learn more about negative prompting
kecupochren 4y ago
Try Midjourney instead. It's miles ahead. Very little is needed to get amazing results. You just set the theme and let it run wild.
- anentropic 4y ago
  Are there different versions of midjourney or something? I tried it last week and the results were equally underwhelming
  - kecupochren 4y ago
    Yes, you need to use --v 4 parameter to use the latest version
_ph_ 4y ago
What I always find so disappointing is, that you might get an almost perfect image out of it, but then there is something decisive very wrong. Like a fifth leg on an anmial oder persons with 3 arms. Or the face completely missing in an otherwise good rendering. So while the results often are astonishing - I am still faszinated that it is possible to create realistic images just from a short text input - this shows that there is no true intelligence behind this. It will be interesting when this gap can be breached. Till then, I can hope that there will be good tooling around SD, which make image creation easier by offering a good UI to make iterations easier and allow the user to guide the creation better.
avereveard 4y ago
eh, it's all anectdotes unless he shows us his prompts
boredhedgehog 4y ago
I don't know much about SD, but the author might be confused about the technical terms. He wants a pier but keeps asking for merely a dock.
Fricken 4y ago
Here is my first attempt at "wooden pier beneath sea cliffs by Moebius --v 4", in Midjourney: https://imgur.com/KnNCFJQ https://imgur.com/KnNCFJQ
cloogshicer 4y ago
When I see things like this, I always have to think of this Scott Aaronson quote: "If P=NP, then the world would be a profoundly different place than we usually assume it to be. There would be no special value in “creative leaps,” no fundamental gap between solving a problem and recognizing the solution once it’s found. Everyone who could appreciate a symphony would be Mozart; everyone who could follow a step-by-step argument would be Gauss; everyone who could recognize a good investment strategy would be Warren Buffett." Maybe the difference between recognizing art and making it yourself is not as big as we assumed it to be.
smcl 4y ago
> Don’t you know how solid objects work? No
sophrocyne 4y ago
There is a significant learning curve to using SD - Not just in developing the craft of prompting, but recognizing that fine-tuned model choice and tooling choices play a significant role in output quality, and the ability to control the generation process. The more you abstract control from end users and “do it for them”, the easier generations can be made, at the loss of control. Midjourney is an excellent example of this. If you want fine-tuned control, SD has an ecosystem that is rapidly accelerating in capabilities. I’m part of a team building open source toolkit on top of SD, and the power users of our tool have shown themselves to be exceptional in their ability to pull high quality works out of the aether.
woolion 4y ago
I made a similar post [0], hoping to 'replace myself', but being really underwhelmed with the results. And this is after a few full days of prompt hacking, Dreambooth training on different base models, etc. It's clearly not at a level where you can easily replace even low-tier artists in even semi-pro environments. It's not the case however for things like one-off illustrations (books, music albums) where it's easy to see that indies have totally embraced it over commissions already. It doesn't require general consistency, nor to be very specific, nor to have a very unique design. http://woolion.art/2022/11/16/SDDB.html http://woolion.art/2022/11/16/SDDB.html

172 points by luu 4y ago

Online, stable diffusion looks incredible. But it's been so, so bad for me, and when you look closely at lots of other "good" pictures, they have lots of flaws too, we just aren't good at picking them out. I wonder if there will be a way to make SD align with what humans want more, maybe with some GAN approach, without needed actual "understanding" of the real world.

dwringer 4y ago
Using img2img and making subtle alterations/substitutions to the prompts and/or images as you go can allow one to serve the adversarial role oneself as the user. Ultimately it's sort of like sitting across the table from an artist who can't speak but you can pass drawings back and forth iterating on them until they are complete. Of course it can be drawings, 3D animation style rendering, painting, photorealistic renders, etc.
kecupochren 4y ago
Try MidJourney instead. Very little is needed to get amazing results. You just set the theme and let it run wild. Or you can also be very specific and it will handle it.

skybrian 4y ago

I tried this using Stable Diffusion v2.1 at dreamstudio.ai. I'm not getting good results yet. It seems like an odd place to build a dock, if you can avoid it. Is there a real-world place like that?

paulcole 4y ago
This place in Malta is pretty close: https://www.alamy.com/stock-photo-boats-at-dock-near-a-set-of-steps-leading-up-cliffs-in-malta-51572197.html https://www.alamy.com/stock-photo-boats-at-dock-near-a-set-o... I actually thought the first version in the blog post was pretty good. It looked like a floating dock to me (that goes up and down with the water level) rather than one secured by pilings.
robocat 4y ago
Cap de Formentor in Majorca is an example, and I'm sure you could find better examples for other lighthouses. I remember clambering down the epic walkway to where all the lighthouse building materials were brought in - and there must have been a jetty or something because it's fucking steep. The best description and image I could find: https://www.majorcadailybulletin.com/news/local/2021/03/16/80955/mallorca-culture-cap-formentor-lighthouse.html https://www.majorcadailybulletin.com/news/local/2021/03/16/8... I think that the "cove" and walkway is shown on the left of the third photo in the article. That said, any jetty would have had to be cantilevered from the rockface. The guy in the article drawing piles next to a shear cliff of rock is hardly thinking the engineering though!

humanistbot 4y ago

I had the exact same experience. I foolishly paid for a pro account without even trying out the free version, because I was so excited by it. Only to learn that it took so much trial and error --- sorry, "prompt engineering" --- to get anything remotely usable. This is the same with all kinds of generative models. Humans are a critical part of the system. If this is "artificial intelligence" it only appears intelligent because there is a human gatekeeper.

yunwal 4y ago
All AI models so far are pretty much only useful as a middle visualization step in a creative process. You have to come up with a vague idea of what to draw, SD can give you a number of compositions to choose from and will get close enough that you can tell the final product would be good. You will still have to recreate or heavily edit whatever it creates though. It’s still useful, but it’s not good enough to be a revolutionary product yet or declare photoshop obsolete. Similarly, ChatGPT is a great replacement for a search engine. I can ask it to guide me through writing a metric to Prometheus from a batch job and it’ll dig out the relevant pieces from the documentation and present them to me in an easy-to-understand format. But I’m not giving it write access to main yet, or letting it make business decisions.
Slow_Hand 4y ago
This sounds like any other tool or instrument that requires practice to render good results. SD is magic, but it's not a miracle. It still requires work and understanding to get good results. It's just a different kind of work than we're used to doing. Don't expect it read your mind. It's still a computer. As a client, if I hire an illustrator to commission a specific image I'm not just hiring them for their technical ability to render images. The best illustrators are going to have a broad and deep awareness of all styles, modes of presentation, and compositional techniques. They're going to use all of their experience and knowledge to interpret what my needs are and deliver on that. That's not a trivial thing to do. Even the best contractors have to deal with clients who don't know how to ask for what they want. The "I'll know it when I see it" type of client. As the "client" in this situation, you need to know how to ask for what you want in a way that your illustrator can interpret.

fy20 4y ago

What nobody tells you is that to get the results you see people post online often requires hours of work for a single image. First you need to refine the prompt so it is ultra specific (which OP has not done), then you need to generate a hundred or so images and pick the best ones. From there you can use img2img to refine it more, and once that's done you might want to go to Photoshop to add some finishing touches. To get good results it is an art form at the moment, but of course as the tools get better eventually it will just be a click of a button.

Fricken 4y ago
So long as you're content with some random crazy image, you can get fancy stuff pretty easily with Midjourney V4. Getting specific details correct, however...
mch82 4y ago
Correct. It’s a new medium for a new school of artist, not a replacement for artists. Very exciting!
- prpl 4y ago
  it’s the new sketch
vhold 4y ago
Another trick you can do is when you find a composition you like, but it's off in whatever way, is lock down that seed, and then apply variation to that initial noise state, and produce a bunch more images from there. The popular AUTOMATIC1111 webui implements it: https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#variations https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki...
godelski 4y ago
As someone researching generative models, this is one of my pet peeves. There's been a rat race to show off high quality cherry-picked samples. Very few works actually show random samples. It is rather frustrating because when pointed out it still goes ignored. I'm not sure why we look the other way, aren't these issues we want to solve? But if you tried to publish a paper with random results you won't get accepted. I am also not worried about AI taking over art, exactly for the reasons you say. Even if alignment was a whole lot better you're never going to be able to perfectly describe an image in your head. Language is too limited. So it will always be a tool. Of course, it can enable fraud. But not many people are putting digital arts on their walls. The other pet peeve is that people think generative models are only image or text. They so so much more.

sirwhinesalot 4y ago

What (barely) works for me is: - Make a prompt with a similar structure to the ones that generated good images online. - Generate a bunch of low quality images. - Pick the low quality image that kinda sort of has the shape and colors I'm looking for. - Iterate on that image with various other prompts and img2img, increasing how much the original image contributes as it gets better.

theCrowing 4y ago

Mhm...looks like he put more effort into writing this blog post than learning how to work with SD.

InvisGhost 4y ago

I've had a lot of luck using Invoke.AI's solution. It allows for a lot more control over what the AI model is allowed to modify and lets you tailor each prompt for each individual edit.

xrd 4y ago
This is my favorite as well. Are you running on your own hardware? I've found the latest commits to be less stable and it's hard to know what to run.

hackyhacky 4y ago

This should really be an easy fix, since Stable Diffusion is open source. Just find and remove the code that reads: if (drawing_bottom_of_cliff) fuckup(dock, RIGHT_SIDE) The team is waiting for your pull request!

Aeolun 4y ago

In my experience, Midjourney has been so much better than Stable Diffusion at anything game related that it’s absurd. It might be worth trying that.

anentropic 4y ago
I've had exactly the same issues with midjourney, can't really tell any difference between then

kyleyeats 4y ago

It's all a composition and image-bashing game with Stable Diffusion. A better strategy here would be keeping the same seed and then try adding 'dock' phrases to the prompt. Then image-bashing what it gives you over the first one. Image-bashing here basically means putting it in a new layer in GIMP over the old one and erasing the parts you don't want. It's really great with seed alternates because the two images will be very similar. Inpainting is good for stuff you don't want the viewer to look at. If there's an errant shrub in your desert picture[0], you can inpaint it out. It almost never works for faces, heads, characters or any other focal object. The new depth stuff might fix it, but inpainting generally just doesn't have a good sense of how to orient things in the scene. [0] I generated a bunch of SD images to spice up my CSS library's homepage: https://casscss.github.io/cass/ https://casscss.github.io/cass/

WheelsAtLarge 4y ago

I love SD for its unexpected results. You really don't know what you are going to get. Trying to get it to create the exact image you have in your mind is next to impossible. That's where this guy went wrong he thought he was getting an artist that would do as he asked but SD is not that. There's a lot of randomness associated with it.

elliottkember 4y ago

The gap between something totally new existing, and someone taking it for granted and saying it totally sucks, is still so short. I'm reminded of Louis CK's bit about wifi on planes: "And it's fast, and I'm watching YouTube clips. It's amaz--I'm on an airplane! And then it breaks down. And they apologize, the Internet's not working. And the guy next to me goes, 'This is bullshit.' I mean, how quickly does the world owe him something that he knew existed only 10 seconds ago?"

esprehn 4y ago
To be fair in the case of airplane wifi this is compounded by the exorbitant pricing. If you pay a premium price you expect premium service. There's no competition for airplane wifi though, you either use it or you don't. It's not like you choose between two different vendors... So they're competing against no wifi at all which allows for pretty mediocre and unreliable service as that's "better than nothing". These free ML models exist in a slightly different world.
- guiambros 4y ago
  > To be fair in the case of airplane wifi this is compounded by the exorbitant pricing. Is this still true, though? I regularly pay $8 for wifi on a 6h flight coast-to-coast with United. And also paid $15 for a 10h transatlantic flight last week, which seems pretty fair to me. Oh, and messaging and chat apps are free, so you don't really need to pay if you only want to keep in touch with folks on the ground. Sure, quality is sometimes spotty, but seems to have improved over the last couple of years.
  - agolio 4y ago
    I don't know what it is, but somehow paying even a penny for wi-fi hotspots is incomprehensible by my brain. It could be a 16-hour flight to Australia, and still my brain would be saying no. Then I would order a $10 coffee and tell myself I deserve to treat myself.
    - 4y ago

krisoft 4y ago

One super obvious trick, which the author seems to be not aware of: If you have two pictures and you like one part from one and an other part from an other just photoshop/gimp them together. If a particular manipulation is easier manualy there is no reason you need to try to convince the AI to do it. Just do it yourself. In this case you drag the two images to two layers and use the erase tool to remove the blemishes which are present in your “top” image but not present in the “bottom” one.

joeld42 4y ago
You can only get so far with this, the lighting and perspective won't match and you'll end up with a "collage" look, especially as you add more images. You can sometimes feed the result back through img2img and get it to fix it. Even in an illustrative style like this the color tone is going to look different.
- krisoft 4y ago
  Sure thing. But I mean it in a much more restrained way. They have one picture where they like the left side of the dock, but say that the AI screwed up the right side. And they have a previous one where the right side is good but the left side is bad. And they are frustrated that the AI can’t or won’t make a single generation where both things are good. And the two pictures have the same structure, same illumination and look barely distinguishable otherwise. That is the case where editing is trivial. You might wonder, how often does this really happen in real life, and the answer is: all the time. You get a picture you like overall, and it looks good from 10 feet distance but on closer inspection you notice some blemishes with it. One way to fix this is to “inpaint” the blemishes. Basically you tell the model to keep all the pixels unchanged, but generate new ones in a small region masking out the blemishes. And of course you run many such inpainting generations and pick the best. And when you have multiple blemishes, which is often the case, then very often one gets filled in good in one generation while the other is better in an other. And you can keep rolling the dice hoping that some day the AI will get it right everywhere, or you can get them in photoshop and do the selection manually.
- pjgalbraith 4y ago

krisoft 4y ago

> The original, 1990s Myst had 2,500 images, which would cost me more than I expect to make this year. Also, it would take an absurd amount of time. Myst was computer rendered. If that is the feel the author is going for why don’t they use the same technique? The number of individual images would certainly not be a limit then.

agildehaus 4y ago
Indeed. It's also quite a bit easier, faster, and cheaper to make INSANELY better quality images than it was in 1992 when Myst was being developed. The software and hardware differences between 1992 and today are immense. What hasn't changed though is the artist. Robyn Miller and Chuck Carter squeezed every ounce out of what was available to them. This guy could do the same with Stable Diffusion, but will take work it doesn't seem like he wants to do. He wants Stable Diffusion to be his perfect human-quality artist, and it isn't that.

pie_flavor 4y ago

My first attempt with SD2 was my favorite prompt in SD1 - 'nebulaic sky above a golden sea, 4k desktop wallpaper'. SD1 routinely made the nebula golden instead of the sea, but what the heck, still makes for a great Zoom background. But SD2 just gave me a golden nebula. No sea. I tried everything I could think of to vary the prompt, and nothing would produce an image with the sea in it. Then I ran a sanity check, and asked it 'Sea'. I got some interesting abstract patterns, a comic strip about an airplane, and a bird that looked rather diseased. No seas. Maybe one of the abstract patterns kind of looked like the sea does in anime?. Just to double-sanity-check that prompts don't need to be more refined than that, I searched 'forest' and got some lovely photorealistic pine trees in the mist. SD2 literally doesn't know what the sea is. I just checked and SD2.1 gives you more sea-ish abstract art than SD2.0 did, but still no dice. 'photorealistic sea' gives me two motorcycles and an elephant. edit: Seems v2.1 understands 'ocean' some of the time (not all!). v2.0 did not understand either one.

Nition 4y ago
> two motorcycles and an elephant Maybe SEA = Southeast Asia.
ralfd 4y ago
What happens with the prompt "nebulaic sky above a golden ocean, 4k desktop wallpaper'"?
jemmyw 4y ago
maybe most images are labelled "ocean"?
TillE 4y ago
I'd been using SD1 for fantasy game character portraits, with fairly decent results. SD2 refuses to give me a simple headshot no matter what I try, it's just generating pictures which include most of the body as well.
- astrange 4y ago
  Try using negative prompting, especially negative textual-inversion prompts.

dcj4 4y ago

I'm not.

w4ffl35 4y ago

SD works best when you combine it with a paint program. for example, I generated superman but part of his cape was missing. I brought the image into GIMP and manually painted his cape on (sloppy, with pure red, no attempt at making it nice). Then I ran the image through "img2img" and the cape looked perfect.

sophrocyne 4y ago
This is one of the the main reasons for InvokeAI building the Unified Canvas https://youtu.be/hIYBfDtKaus https://youtu.be/hIYBfDtKaus

dorkwood 4y ago

I find this attempt to create something of value without putting in any effort to be frustrating. > Yes, I could pay an artist to make art for the game. But I’m not yet good enough at game marketing for this to make sense. There's a third option: learn to make the art yourself! Yes, it's time consuming, but so is anything worth doing. > The original, 1990s Myst had 2,500 images, which would cost me more than I expect to make this year. Start smaller. Or use a simpler, more achievable art style.

oneoff786 4y ago
> There's a third option: learn to make the art yourself! Yes, it's time consuming, but so is anything worth doing. This is terrible logic. If you don’t want to make art, don’t waste time learning to make art.
- dorkwood 4y ago
  > If you don’t want to make art, don’t waste time learning to make art. Oh, I agree. I wouldn't suggest someone who wasn't interested in art spend time learning to make art. That would be ludicrous. But that's not what I'm seeing in this article. Most of it chronicles the author's attempt to create a single illustration in a particular style. What are they doing, if not attempting to make art?
  - oneoff786 4y ago
    Trying to get art

wokwokwok 4y ago

? Is this a re-post? I swear this is from like 2 days ago, and it turns up in the HN search from two days ago: > I am frustrated with Stable Diffusion(https://novalis.org/blog/2022-12-05-i-am-frustrated-with-stable-diffusion.html https://novalis.org/blog/2022-12-05-i-am-frustrated-with-sta...) > 43 points|luu|2 days ago|40 comments <--- ..but it's been resurrected now? > 42 points by luu 5 hours ago | hide | past | favorite | 38 comments <--- ?? I mean, whatever, but the timestamps seem like they've been edited or something, which is very strange.

fragmede 4y ago
Second chance pool https://news.ycombinator.com/item?id=26998309 https://news.ycombinator.com/item?id=26998309

elil17 4y ago

I'd argue the images he generated are good enough to use in an indie game (based on the expectation that good gameplay would make up for the jank).

nineteen999 4y ago

Non artist struggles to make game art with an AI image generator. News at 11.

noduerme 4y ago

So ... this is me as a 42 year old artist, a coder in many languages, an art director for some decent sized companies (probably overpaid while my illustrators are definitely underpaid), and Stable Diffusion enthusiast: HIRE SOME ARTISTS. [edit] if you feel like a long discussion of the cognitive failure of AI over human artists, bring it on; if you're too cheap to hire artists, I totally get it!!! Make your own art! Otherwise, take your brilliant idea and pay other people to execute the parts you're not capable of executing yourself! Don't expect someone else to hand you a robot/AI replacement for your lack of skill for free ;)

badsectoracula 4y ago
He brings that up at the end of the article: > Yes, I could pay an artist to make art for the game. But I’m not yet good enough at game marketing for this to make sense: The original, 1990s Myst had 2,500 images, which would cost me more than I expect to make this year. Also, it would take an absurd amount of time.
- noduerme 4y ago
  I spent 8 years developing all the art for my own game platform and another 4 years on the next... so .. "also, it would take an absurd amount of time" ? Yeah? Well while you search for shortcuts or hope for a free ride, you're not making the art you need or want. What constitutes an "absurd" amount of time is pretty relative. If you hope to make a sizeable amount off your game, maybe it will take what seems right now to be an absurd amount of effort. It only seems absurd because you haven't connected with the reality of spending your time 24/7 making things. i.e. putting out the effort to earn the acclaim you hope for. Slightly a side story, so no one complains: In 1994 when I was 14, "Heart of China" came out and VideoSpigot came out and all I wanted for Xmas was a video camera and an interface to record short QuickTime movies in locations around my house, so I could string them together into a dark futuristic CD-ROM game. (in SuperCard, if you're wondering). That would have solved all my problems creating art! It wouldn't have. Make your game as a sketch with whatever sketching tools you have...but don't expect technology to rescue you from needing other humans to make it good. AND DON'T UNDER-RATE OR UNDERPAY THOSE PEOPLE. Because trust me, if you're disappointed that AI isn't giving you good enough art, you're no visionary. You haven't even tried people.

jasfi 4y ago

I'm tackling this problem with InventAI: https://inventai.xyz https://inventai.xyz. Doing everything I can to launch the MVP this year! Prompt engineering isn't what it should be. It will use DALL-E 2 to start with, but Stable Diffusion 2 is the next back-end I'll add.

ChaitanyaSai 4y ago

We've been experimenting with SD and results are quite underwhelming. DALLE2 seems a lot more usable. Any reason why most are sticking with SD? (If that assumption is true)

trifurcate 4y ago
Openness for me. Stable diffusion runs on a machine that sits 5 feet away from me and delivers good quality results in under 3 seconds per image. I don't have to shy away from politically (or otherwise) questionable prompts, or in the case of business applications, potentially leaking sensitive data, because I know that they aren't being recorded forever on a remote server somewhere. I can also tinker with the code, which has helped me add a bunch of features that really synergize with my workflow. As others have noted, getting good results from SD is very much a numbers and parameters game. I've witnessed first hand how some people get frustrated easily and quit tinkering because their initial approaches don't work right away. Prompt and parameter engineering really is a skill that you have to grow, and it makes a world of difference. img2img is also much finickier than text2img.
speedgoose 4y ago
StableDiffusion produces better pictures once you get what you want. Dall.e2 is very good at understanding your prompt but the images are not very detailed when you look closely. They are full of artefacts, and the faces are zombi like unless they fill the whole frame. But you can use stable diffusion and dall.e together. Or use MidJourney v4 which is simply the best right now.
- ChaitanyaSai 4y ago
  How does MidJourney v4 do illustrations. The early versions we saw seemed a little too artistic and also had this distinct "looks like midjourney" signature. Any links you can point me to? Thanks!
  - speedgoose 4y ago
    I haven’t found a good explanation of how mid journey works in detail. Many people guess that they are not collecting the data for no reasons, and that the model is fine tuned using data and user ratings from the previous generations.

nitwit005 4y ago

Points out the original Myst had 2500 images. Even if an AI tool was so good it only took you 10 minutes per picture, that'd be around 416 hours of work, or 52 eight hour days. Presumably some things will need to be reworked, so probably quite a bit more. It'd probably be easier to create an ordinary 3D game world. I'm not going to claim 3D art is easy, but you only need to make your Myst island one time, and then you can just move the camera around.

prakhar897 4y ago

I'm with the author on this one. I'm working on a Stable Diffusion based word guessing game (https://diffudle.com/ https://diffudle.com/). If you look at the prompts and the actual images, they are miles apart. This is after I reject multiple prompts and multiple images from the selected prompt. The worst part is Midjourney is able to create much better images from the same prompt. At this point, I'm seriously considering renaming the product and just move with another image generator.

thot_experiment 4y ago

I dunno man, I put it into img2img and it worked on like the third try. https://imgur.com/a/tbqz59G https://imgur.com/a/tbqz59G a simple rickety dock at the base of an ocean cliff, approach from the sea, picture from a boat, an illustration by moebius, complexla style, an outdoor scene, simple, expansive, ocean, cliff, stairs, dock, realistic colors Negative prompt: isometric, photograph, boat, multiple frames, abstract, complex, borders Steps: 20, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 1136556607, Size: 1024x1024, Model hash: 5e107da3, Denoising strength: 0.51, Eta: 0.4, Mask blur: 4

jerpint 4y ago
Wow impressive! I need to learn more about negative prompting

kecupochren 4y ago

Try Midjourney instead. It's miles ahead. Very little is needed to get amazing results. You just set the theme and let it run wild.

anentropic 4y ago
Are there different versions of midjourney or something? I tried it last week and the results were equally underwhelming
- kecupochren 4y ago
  Yes, you need to use --v 4 parameter to use the latest version

_ph_ 4y ago

What I always find so disappointing is, that you might get an almost perfect image out of it, but then there is something decisive very wrong. Like a fifth leg on an anmial oder persons with 3 arms. Or the face completely missing in an otherwise good rendering. So while the results often are astonishing - I am still faszinated that it is possible to create realistic images just from a short text input - this shows that there is no true intelligence behind this. It will be interesting when this gap can be breached. Till then, I can hope that there will be good tooling around SD, which make image creation easier by offering a good UI to make iterations easier and allow the user to guide the creation better.

avereveard 4y ago

eh, it's all anectdotes unless he shows us his prompts

boredhedgehog 4y ago

I don't know much about SD, but the author might be confused about the technical terms. He wants a pier but keeps asking for merely a dock.

Fricken 4y ago

Here is my first attempt at "wooden pier beneath sea cliffs by Moebius --v 4", in Midjourney: https://imgur.com/KnNCFJQ https://imgur.com/KnNCFJQ

cloogshicer 4y ago

When I see things like this, I always have to think of this Scott Aaronson quote: "If P=NP, then the world would be a profoundly different place than we usually assume it to be. There would be no special value in “creative leaps,” no fundamental gap between solving a problem and recognizing the solution once it’s found. Everyone who could appreciate a symphony would be Mozart; everyone who could follow a step-by-step argument would be Gauss; everyone who could recognize a good investment strategy would be Warren Buffett." Maybe the difference between recognizing art and making it yourself is not as big as we assumed it to be.

smcl 4y ago

> Don’t you know how solid objects work? No

sophrocyne 4y ago

There is a significant learning curve to using SD - Not just in developing the craft of prompting, but recognizing that fine-tuned model choice and tooling choices play a significant role in output quality, and the ability to control the generation process. The more you abstract control from end users and “do it for them”, the easier generations can be made, at the loss of control. Midjourney is an excellent example of this. If you want fine-tuned control, SD has an ecosystem that is rapidly accelerating in capabilities. I’m part of a team building open source toolkit on top of SD, and the power users of our tool have shown themselves to be exceptional in their ability to pull high quality works out of the aether.

woolion 4y ago

I made a similar post [0], hoping to 'replace myself', but being really underwhelmed with the results. And this is after a few full days of prompt hacking, Dreambooth training on different base models, etc. It's clearly not at a level where you can easily replace even low-tier artists in even semi-pro environments. It's not the case however for things like one-off illustrations (books, music albums) where it's easy to see that indies have totally embraced it over commissions already. It doesn't require general consistency, nor to be very specific, nor to have a very unique design. http://woolion.art/2022/11/16/SDDB.html http://woolion.art/2022/11/16/SDDB.html