8 ms·
We ran over 600 image generations to compare AI image models
- alienbaby 1y agoInteresting experiment, though I'm not certain quite how the models are usefully compared.
- Fnoord 1y agoWell, that is a good point. That is for everyone themselves to decide, I suppose. To me, I like to think in times the model failed versus success. So what I did, is I looked every time at the worst result. To me, the one which stood out (negatively) is Gemini. OpenAI had some very good results but also some missing the mark. SeeDream (which I never heard of previously) missed the mark less often than Gemini, and at times where OpenAI failed, SeeDream came out clearly on top. So, if I were to use the effects of the mentioned models, I wouldn't bother with Gemini; only OpenAI and SeeDream.
- Dwedit 1y agoYou can always identify the OpenAI result because it's yellow.
- Bombthecat 1y agoAnd mid journey because it's cell shading:)
- Hoasi 1y agoAlso because it’s mid :)
- jstummbillig 1y ago> If you made it all the way down here you probably don’t need a summary Love the optimism
- LogicFailsMe 1y agoI skipped to the end to see if they did any local models. spoilers: they didn't.
- ojr 1y agothey are building a product and said the unit economics must make sense, local models have slower latency, unless you run a gpu on for hours which gets expensive fast
- kalleboo 1y agoLocal models will make a lot more sense once we have the scale for it, but when your user count is still small paying cents per image is a much better deal than paying for a GPU either in a data center or physically. Local models are definitely something I want to dive into more, if only out of personal interest.
- CWuestefeld 1y agoHonestly, I think it was misfounded. As an photographer and artist myself, I find the OpenAI results head-and-shoulders above the others. It's not perfect, and in a few cases one or the other alternative did better, but if I had to pick one, it would be OpenAI for sure. The gap between their aesthetics and mine makes me question ever using their other products (which is purely academic since I'm not an Apple person).
- Retric 1y agoHow many of those result did you actually look at? I thought it did ok with the cats, but check the other images and OpenAI strait up failed to do the prompt a large fraction of the time.
- CWuestefeld 1y agoI'm wondering if I read the same article. Yeah, I looked at every single one the results. And although it's been a couple of hours since, I don't recall ANY examples where it completely failed to do anything. Can you point me to an example?
- kevin009 1y agoEveryday I generate more than 600 image and also compare them, it takes me 5 hours
- sema4hacker 1y agoAre artists and illustrators going the way of the horse and buggy?
- LogicFailsMe 1y agoNo, but this is the beginning of a new generation of tools to accelerate productivity. What surprises me is that the AI companies are not market savvy enough to build those tools yet. Adobe seems to have gotten the memo though.
- somenameforme 1y agoIn testing some local image gen software, it takes about 10 seconds to generate a high quality image on my relatively old computer. I have no idea the latency on a current high end computer, but I expect it's probably near instantaneous. Right now though the software for local generation is horrible. It's a mish-mash of open source stuff with varying compatibility loaded with casually excessive use of vernacular and acronyms. To say nothing of the awkwardness of it mostly being done in python scripts. But once it gets inevitably cleaned up, I expect people in the future are going to take being able to generate unlimited, near instantaneous images, locally, for free, for granted.
- dingnuts 1y ago[dead]
- pkroll 1y agoDid you test some local image gen software in that you installed the Python code on the github page for a local model, which is clearly a LOT for a normal user... or did you look at ComfyUI, which is how most people are running local video and image models? There are "just install this" versions, which eases the path for users (but it's still, admittedly, chaos beneath the surface).
- somenameforme 1y ago
- th0ma5 1y agoThis seems to imply that the capabilities being tested are like the descriptive words used in the prompts, but, as a category using random words would be just as valid for exercising the extents of the underlying math. And when I think of that reality I wonder why a list of tests like this should be interesting and to what ends. The repeated nature of the iteration implies that some control or better quality is being sought but the mechanism of exploration is just trial and error and not informative of what would be repeatable success for anyone else in any other circumstance given these discoveries.
- XYZ12334 1y ago[flagged]
- turbonigger 1y ago[dead]
- deleted 1y ago[deleted]
- fsniper 1y agoIs it me or ChatGPT change subtle or sometimes more prominent things? Like ball holding position of the hand, face features like for head, background trees and alike?
- qayxc 1y agoIt's not you. The model seems to refuse to accurately reproduce details. It changes things and leaves stuff out every time.
- yapyap 1y agoUsing gen. ai for filters is stupid, a filter guarantees the same object but filtered, a gen. AI version of this guarantees nothing and an expensive AI bill. It’s like using gen. ai to do math instead of extracting the numbers from a story and just doing the math with +, -, / and *
- whoaoweird 1y agoIt was interesting to see how often the OpenAI model changed the face of the child. Often the other two models wouldn't, but OpenAI would alter the structure of their head (making it rounder), eyes (making them rounder), or altering the position and facing of the children in the background. It's like OpenAI is reducing to some sort of median face a little on all of these, whereas the other two models seemed to reproduce the face. For some things, exactly reproducing the face is a problem -- for example in making them a glass etching, Gemini seemed unwilling to give up the specific details of the child's face, even though that would make sense in that context.
- sasper 1y agoI've noticed that OpenAI modifies faces on a regular basis. I was using it to try and create examples of different haircuts and the face would randomly turn into a different face -- similar but noticeably changed. Even when I prompted to not modify the face, it would do it regardless. Perhaps part of their "safety" for modifying pictures of people?
- SweetSoftPillow 1y agohttps://www.reddit.com/r/ChatGPT/comments/1n8dung/chatgpt_prompted_to_create_the_exact_replica_of/ https://www.reddit.com/r/ChatGPT/comments/1n8dung/chatgpt_pr...
- deleted 1y ago[deleted]
- fsniper 1y agoIt's also changing scene features too. Like removed background trees.
- captainclam 1y agoIt looks to me like OpenAI's image pipeline takes an image as input, derives the semantic details, and then essentially regenerates an entirely new image based on the "description" obtained from the input image. Even Sam Altman's "Ghiblified" twitter avatar looks nothing like him (at least to me). Other models seem much more able to operate directly on the input image.
- frotaur 1y agoIt's crazy that the 'piss filter' of openAI image generation hasn't been fixed yet. I wonder if it's on purpose for some reason ?
- CamperBob2 1y agoThey don't want you creating images that mimic either works of other artists to an extent that's likely to confuse viewers (or courts), or that mimic realistic photographs to an extent that allows people to generate low-effort fake news. So they impose an intentionally-crappy orange-cyan palette on everything the model generates. Peak quality in terms of realistic color rendering was probably the initial release of DALL-E 3. Once they saw what was going to happen, they fixed that bug fast.
- asadotzler 1y agoLOL. If you believe that, let me tell you about this bridge I've got.
- CamperBob2 1y ago(Shrug) It fits the facts. Do a GIS for images from DALL-E 3 and provide an alternative explanation for what you see. It absolutely did not do that on day 1.
- RestartKernel 1y agoOpenAI's new image generation model is autoregressive, while DALL-E was diffusion. The yellowish tone is an artefact of their autoregressive pipeline, if I recall correctly.
- CamperBob2 1y agoCould be. My point is that if the pipeline itself didn't impart an unmistakable character to the generated images, OpenAI would feel compelled to make it do so on purpose. Most DALL-E 3 images have a orange-blue cast, which is absolutely not an unintended artifact. You'd literally have to be blind to miss it, or at least color-blind. That wasn't true at first -- check the original paper, and try the same prompts! It was something they started doing not long after release, and it's hardly a stretch to imagine why. They will be doing the same thing for the same reasons today, assuming it doesn't just happen as a side effect.
- gs17 1y agoIt's interesting to me that the models often have their "quirks". GPT has the orange tint, but it also is much worse at being consistent with details. Gemini has a problem where it often returns the image unchanged or almost unchanged, to the point where I gave up on using it for editing anything. Not sure if Seedream has a similar defining "feature". They noted the Gemini issue too: > Especially with photos of people, Gemini seems to refuse to apply any edits at all
- minimaxir 1y agoNano Banana in general cannot do style transfer effectively unless the source image/subject is a similar style as the target style, which is an interesting and unexpected model quirk. Even the documentation examples unintentionally demonstrates this. Seedream will always alter the global color balance with edits.
- basch 1y agoSomething like a style transfer works better in Whisk. Still quirky and hit and miss.
- dwringer 1y agoI've definitely noticed Gemini's tendency to return the image basically unchanged, but not noticed it being worse or better for images of people. When I tested by having it change aspects of a photo of me, I found it was far more likely to cooperate when I'd specify, for instance, "change the hair from long to short" rather than "Make the hair short" (the latter routinely failed completely). It also helped to specify which other parts should not be changed, otherwise it was rather unpredictable about whether it would randomly change other aspects.
- basch 1y agoNot only does it return the image unchanged, but if you are using the Gemini interface, it confidently tells you it made the changes.
- 1y ago
- jrflowers 1y agoI like that they call openai’s image generator ground breaking and then explain that it’s prone to taking eight times longer to generate an image before showing it add a third cat over and over and over again
- kalleboo 1y agoI meant to say it was ground-breaking when it was released, the other models came later.
- treesciencebot 1y agowe build our sandbox just for this use case, fal.ai/sandbox. take the same image/prompt, and compare across tens of models.
- beezle 1y agoFound OpenAI too often heavy handed. On balance, I'd probably pick Gemini narrowly over Seedream and just learn that sometimes Gemini needs a more specific prompt.
- Scrapemist 1y agoSeedream is the only one that outputs 4k. Last time I checked that is..
- stevage 1y agoI wish they'd used a better image than the low contrast mountain, which rarely transformed into anything much.
- justhw 1y agoI came to the same conclusion as the authors after generating 1000s of thumbnails[1]. OpenAI alters faces too much and smoothes out details by default. NanoBanana is the best but lacks high fidelity option. SeeDream is catching up to NanoBanana and sometimes is better. It's been too long since OpenAI's gpt-img-1 came out, hope they launch a better model soon. [1] = https://thumbnail.ai/ https://thumbnail.ai/
- Libidinalecon 1y agoI am probably at 50k-60k image generations from various models. It is just very hard to make any generalizations because any single prompt will lead to so many different types of images. The only thing I would really say to generalize is every model has strengths and weaknesses depending on what you are going for. It is also generally very hard to explore all the possibilities of a model. So many times I thought I seen what the model could do just to be completely blown away by a particular generation.
- jncfhnb 1y agoWhy so many?
- pram 1y agoFWIW when I do txt2img or img2img locally I have the batch set to 8-12 (so 12 variation images are generated from the same seed in the same gen) so it’s fairly easy to numerically end up with tens of thousands, which are usually 99% not good.
- georgefrowny 1y agoWhat do you even do with 50k images? Even at just 10 seconds attention each, that's an solid entire week of waking time.
- weq 1y agoYoutube is full of AI slop right now, doesnt take much imagine to regconise how scammers (listed on an exhcange or not) are utilising it this... Take for instance a political influence organisation, generating avatars for vast bot networks that are implanted into social media to influence.
- chanw 1y agoHey. We'd love to fund thr generations for free for you to try Riverflow 2 out if you're up for it. Riverflow 1 ranks above them all and 2 is now in preview this week.
- angry_albatross 1y agoThe shortcut to flip between models in an expanded view is nice, but the original image should also be included as one of the things to flip between, and should be included in the side by side view.
- specproc 1y agoI dunno about you lot, but I actually really like Stable Diffusion 1.5. I like giving it weird, non-prompts, like lines from songs or novels. I then run it for a few hundred generations locally and doing stuff with the malformed shit it comes out with. I have a few art projects like this. Aphex Twin vibes.
- alvinveroy 1y ago[dead]
- deaux 1y agoLLM slop comment. 5 of them within 5 minutes.
- amanverasia 1y agoTimings were measured on a consumer internet connection in Japan (Fiber connection, 10 Gbps nominal bandwidth) during a limited test run in a short time period. "consumer internet connection in Japan", "10 Gbps nominal bandwidth" Coming from a third world country, that surprises me.
- kalleboo 1y agoThe 10gbit connection costs me ¥5,000/mo (around USD 30/mo), which was actually slightly cheaper than I was paying for 1 Gbit... The main issue is latency and bandwidth across the oceans since Asia far away from the US where a lot of servers live, and even for services that are distributed, I live in a rural prefectural capitol of Japan 1000 km away from Tokyo where all the "Japan" data centers are, so my ping is always unimpressive despite the bandwidth.
- MrBuddyCasino 1y agoWould have loved to see Grok (xAI) in there, by my (limited) experience it is often better than OpenAI or Gemini.
- shakir_amarri 1y agoA thing worth noting is that Seedream 4.0 is uncensored and was seemingly trained on a lot of uncensored stuff.
- krige 1y agoHighly amusing that OAI model is still ghibli-zing its outputs.
- JCM9 1y agoFor all the AI slop and studies saying AI is more hype than substance I will say that this use case is one that seems very legit. The stock photo industry was always pretty bad and silly expensive. Being able to custom generate visuals and photos to replace that is a good use case of AI IMHO. Yes sometimes it does goofy things, but it’s getting quite good. If AI blows up the stock photo industry few will shed a tear.
- red-iron-pine 1y agoyeah fuck stock photos
- cwoolfe 1y agoChatGPT is the only one I've found that can transform an image into a specified size. i.e. "resize this image to be 1280x1024 pixels"
- emsign 1y agoIt's disturbing how the models sometimes alter the objects in the images when they're only supposed to add an effect. That's not just a complete failure of the task, it also means manual work since a human has to double check evry detail in every image.
- the_real_cher 1y agoA tiny, high-signal TL;DR: • OpenAI (gpt-image-1): The wild artist. Best for creative, transformative, style-heavy edits—Ghibli, watercolor, fantasy additions, portals, sci-fi stuff, etc. But it hallucinates a lot and often distorts fine details (especially faces). Slowest. • Gemini (flash-image / nanoBanana): The cautious realist. Best for subtle, photorealistic edits—fog, lighting tweaks, gentle filters, lens effects. Almost never ruins details, but sometimes refuses to do artsy transformations, especially on human photos. • Seedream: The adventurous middle child. Faster, cheaper, and often surprisingly good at aesthetic effects—bokeh, low-poly, ukiyo-e, metallic sheen, etc. Not as creative as OpenAI, not as conservative as Gemini. Can hallucinate, but in fun ways. Bottom line: • Creative prompts → OpenAI • Realistic photo edits → Gemini • Budget-friendly, balanced option → Seedream If you’re planning an automated pipeline, routing “artistic” prompts to OpenAI and “photorealistic” ones to Gemini (with Seedream as a wildcard) matches their own conclusion.
- lunias 1y agoAre people doing image generation really using these models much? I've generated a lot of images, but I always use ComfyUI with local models and custom workflows. I only have 8GB of VRAM and I can easily do 1000s of images per day if I want to.