37 ms·

No elephants: Breakthroughs in image generation

445 points by Kerrick 2y ago

roenxi 2y ago
That proper "no elephants" first image is hilarious. Another key point here is the generative AI's meme game is getting rather strong. Which isn't a small thing, humour is an advanced soft skill.
- YurgenJurgensen 2y ago
  Having thousands of copies of that image in your training set isn’t a skill at all.
  - hnbad 2y ago
    AI's "meme game is going strong" by the same metric I would use to try to argue that Elon Musk's is. I wouldn't call it a good metric, though.
- loudmax 2y ago
  The "Game Design Otter" action figure seems to come with a pair of flashlights. I bet that's a residue from the previous prompt about illuminating the tablet with a flashlight.
- aenvoker 2y ago
  Having used lots of different image generators, so far the only one with a sense of humor has been Dall-e3. https://www.reddit.com/r/dalle2/s/khb5XuNFdl https://www.reddit.com/r/dalle2/s/khb5XuNFdl There’s probably some sort of connection to ChatGPT in there. But, I don’t know enough about how it works.
x187463 2y ago
This is a before/after moment for image generation. A simple example is the background images on a ton of (mediocre) music youtube channels. They almost all use AI generated images that are full of nonsense the closer you look. Jazz channels will feature coffee shops with garbled text on the menu and furniture blending together. I bet all of that disappears over the next few months. On another note, and perhaps others are feeling similarly, but I am finding myself surprised at how little use I have for this stuff, LLMs included. If, ten years ago, you told me I would have access to tools like this, I'm sure I would have responded with a never ending stream of ideas and excitement. But now that they're here, I just sort of poke at it for a minute and carry on with my day. Maybe it's the unreliability on all fronts, I don't know. I ask a lot of programming questions and appreciate some of the autocomplete in vscode, but I know I'm not anywhere close to taking full advantage of what these systems can do.
- Retr0id 2y ago
  I've never used a stock photo site before, so I suppose it's no surprise I have no real use for "generate any image on demand".
  - YurgenJurgensen 2y ago
    Their main application appears to be taking blog posts and internal memos and making them three times longer and use ten times the bandwidth to convey no more information. So exactly the application AI is ‘good’ at.
    - wongarsu 2y ago
      If anything stock image websites are even worse at this then AI. With AI you come up with an image idea, then try to make the AI produce something close to it. With stock images you come up with an image idea, then hope some photographer had a similar idea and uploaded it to a stock website.
  - esperent 2y ago
    I've used stock photo sites occasionally but I use vector art and icon sites multiple times a week. Even today, I used an few different sites while designing some stuff on Canva. The reason I don't use AI is because it gives me far less reliable and impossible to specify results than just searching through the limited lists of human made art. Today, for undisclosed reasons, I needed vector art of peanuts. I found imperfect but usable human made art within seconds from a search engine. I then spent around 15 - 25 minutes trying to get something closer to my vision using ChatGPT, and using the imperfect art I'd found as a style guide. I got lots of "huh that's cool what AI can do" but nothing useful. Nothing closer to my vision than what I started with. By coincidence it's the first time I'vr tried making art with AI in about a year, but back then I bought a Midjourney account and spent a month making loads of art, then installed SD on my laptop and spent another couple of weeks playing around with that. So it's not like I'm lacking experience. What I've found so far is that AI art generators are great for generating articles like this one. And they do make some genuinely cool pictures, it blows my mind that computers can do this now. It's just when I sit down with a real world task that has specific, concrete requirements... I find them useless.
Retr0id 2y ago
I had a reasonable intuition for how the "old" method works, but I still don't grok this new approach. "in multimodal image generation, images are created in the same way that LLMs create text, a token at a time" Is there some way to visualise these "image tokens", in the same way I can view tokenized text?
- fxtentacle 2y ago
  Imagine you cut the image into 32x32 pixel blocks. And then for each block, you can chose 1 out of 128,000 variations. And then a post-processing step smoothes out the borders between blocks and adjusts small details. That's basically how a transformer image generation model works. As such, the process is remarkably similar to old fixed-font ASCII art. It's just that modern AIs have a larger alphabet and, thus, more character shapes to choose from.
  - rwmj 2y ago
    I don't get how this would produce consistent images. In the article, the text could be on a grid, but the window and doorway and sofa don't seem to be grid-aligned. (Or maybe the text is overlaid?)
    - danielbln 2y ago
      The model looks ahead, just like LLMs look ahead. An LLM outputs token by token but can still output a fully coherent and consistent story for example. This new crop of auto-regressive image models does the same.
- WhiteNoiz3 2y ago
  I haven't see any details on how OpenAI's model works, but the tokens it generates aren't directly translated into pixels - those tokens are probably fed into a diffusion process which generates the actual image.. The tokens are the latent space or conditioning for the actual image generation process.
  - 2y ago
TacticalCoder 2y ago
[dead]
card_zero 2y ago
Looking at the example where the coffee table is swapped, I notice every time the image is reprocessed it mutates, based on the previous iteration, and objects become more bizarre each time, like chinese whispers. * The weird-ass basket decoration on the table originally has some big chain links (maybe anchor chain, to keep the theme with the beach painting). By the third version, they're leathery and are merging with the basket. * The candelabra light on the wall, with branch decorations, turns into a sort of skinny minimalist gold stag head, and then just a branch. * The small table in the background gradually loses one of its three legs, and ends up defying gravity. * The freaky green lamps in the window become at first more regular, then turn into topiary. * Making the carpet less faded turns up the saturation on everything else, too, including the wood the table is made from.
- bla3 2y ago
  The pictures on the wall change too.
  - card_zero 2y ago
    Yes, first a still life and something impressionist, then a blob and a blob, then a smear and a smear. And what about the reflections and transparency of the glass table top? It gets very indistinct. Keep working at the same image and it looks like you'll end up with some Deep Dream weirdness. I think the fireplace might be turning into some tiny stairs leading down. :)
    - YurgenJurgensen 2y ago
      Only sailors know how to leave.
  - rob74 2y ago
    Actually, almost everything changes slightly - the number, shape and pattern of the chairs, the number and pattern of the pillows, the pattern of the curtains, the scene outside the window, the wooden part of the table, the pattern of the carpet... The blue couch stays largely the same, it just loses some detail...
NitpickLawyer 2y ago
> The results are not as good as a professional designer could create but are an impressive first prototype. I like to look at how far we've come since the early days of Stable Diffusion. It was fascinating to play with it back then, but it quickly became apparent that it was "generic" and not suited for "real work" because it lacked consistency, text capabilities, fingers! and so on... Looking at these results now, I'm amazed at the quality, consistency and ease of use. Gone are the days of doing alchemy on words and adding a bunch of "in the style of Rutkovsky, golden hour, hd, 4k, pretty please ..." at the end of prompts.
probably_wrong 2y ago
> Is it okay to reproduce the hard-won style of other artists using AI? Who owns the resulting art? Who profits from it? Which artists are in the training data for AI, and what is the legal and ethical status of using copyrighted work for training? These were important questions before multimodal AI, but now developing answers to them is increasingly urgent. I have to disagree with the conclusion. This was an important discussion to have two to three years ago, then we had it online, and then we more or less agreed that it's unfair for artists to have their works sucked up with no recourse. What the post should say is "we know that this is unfair to artists, but the tech companies are making too much money from them and we have no way to force them to change".
- shkkmo 2y ago
  > This was an important discussion to have two to three years ago, then we had it online, and then we more or less agreed that it's unfair for artists to have their works sucked up with no recourse. Speak for yourself, there was no consensus online. There are plenty of us that think that dramatically expanding the power of copyright would be a huge mistake that would primarily benefit larger companies and do little to protect or fund small artists.
  - OtherShrezzing 2y ago
    >There are plenty of us that think that dramatically expanding the power of copyright would be a huge mistake that would primarily benefit larger companies and do little to protect or fund small artists. The status quo also primarily benefits larger companies, and does little (exactly nothing, if we're being earnest) to protect or fund small artists. It's reasonable to hold both opinions that: 1) artists aren't being compensated, even though their work is being used by these tools, and 2) massive expansion of copyright isn't the appropriate response to 1).
    - deleted 2y ago
      [deleted]
- eadmund 2y ago
rkharsan64 2y ago
Are there any local models that use this new approach to generating images?
- GaggiX 2y ago
  GPT-4o is the only model that seems to work well in the text-image joint space to this degree, even Gemini Flash 2.0 with native image support is not nearly as good so it will probably be a while for a good open source alternative to pop up (a while in the context of AI development).
  - gerash 2y ago
    depends on the use case. I used GPT-4o for some image editing (adding or removing things) to an image of a person and they distort the look of the people after each edit but (Gemini Flash + image out) did much better. The main problem is there is little control. For example I asked to add a helicopter to an image in a ski resort but then it seems cumbersome for me to have to write a full paragraph to describe where exactly I want this helicopter to be rather than if I could just do it by dragging things with a mouse.
- DeathArrow 2y ago
  Yes, there's HiDream which yields even better results than 01. https://github.com/HiDream-ai/HiDream-I1 https://github.com/HiDream-ai/HiDream-I1
  - GaggiX 2y ago
    This is just a diffusion text-to-image model like many others, completely different than a LLM with a native image support.
orbital-decay 2y ago
4o still exhibits the "pink elephant effect", it's just... subtler, and tends to reveal itself on a complex or confusing prompt. Negations are also still not handled properly, they tend to slightly confuse the model and decrease the accuracy of the answer or the generated picture. The same is true for any other LLM. Moreover, the author is asking the model to rationalize the decision he already made ("tell me why there can't be any elephants"), which could work as an equivalent to a CoT step. It's "just" a much bigger and much better trained model. Which is a quality on its own, absolutely no doubt about that. Fundamentally the issue is still there though, just less prominent. Which kind of makes sense - imagine the prompt "not green", what even is that? It's likely slightly out of distribution and requires representing a more complex abstraction, so the accuracy will necessarily be worse than stating the range of colors directly. The result might be accurate, until the model is confused/misdirected by something else, and suddenly it's not. I think in the end none of the architectural differences will matter beyond the scaling. What will matter a lot more is data diversity and training quality.
- danielbln 2y ago
  But it's literally a different architecture (auto-regressive, presumably sequence based vs diffusion). In my experiments it is significantly, overwhelmingly better at consistency, coherence and prompt adherence. Things I needed control nets before it just... does it. and even zooming into fine details, they make sense. Here is an example with a bunch of negations: https://i.imgur.com/P8G5ICs.png https://i.imgur.com/P8G5ICs.png
  - wat10000 2y ago
    It’s interesting that out of all the aquatic animals it could have used, it chose one that perhaps looks the most like an elephant.
    - danielbln 2y ago
      That's just luck of the draw, in a new session with the same prompt it outputted a bird: https://i.imgur.com/SLT8cYe.png https://i.imgur.com/SLT8cYe.png
d4rkp4ttern 2y ago
Diagrams are still a big unsolved problem. Making diagrams for a talk or paper is an extremely tedious process and I am still waiting for a good multimodal LLM solution for this. It should take a sketch and/or text description of what you want and in a few iterations you should get what you want. GPT4o tries hard but results are still bad.
- chthonicdaemon 2y ago
  I've had the best luck at getting it to produce diagrams-as-code like mermaid or plantuml.
  - d4rkp4ttern 2y ago
    I know but those diagrams often don’t adequately capture what I want. Think of diagrams in nice technical talks or papers. I’ve even tried having the LLM (Claude) generate SVGs. They all fall short.
shubhamjain 2y ago
The Ghibli trend completely missed the real breakthrough — and it’s this. The ability to closely follow text, understand the input image, and maintain context of what’s already there is a massive leap in image generation. While Midjourney delivered visually stunning results, I constantly struggled to get anything specific out of it, making it pretty much useless for actual workflows. 4o is the first image generation model that feels genuinely useful not just for pretty things. It can produce comics, app designs, UI mockups, storyboards, marketing assets, and so on. I saw someone make a multi-panel comic with it with consistent characters. Obviously, it's not perfect. But just getting there 90% is a game changer.
- empath75 2y ago
  I had chatgpt generate a flow chart with mermaid js for something at work and then write a scott mccloud style comic book explaining it in detail and it looked so convincing, even though it got some of the details a bit wrong. It's _so close_ to making completely usable graphics out of the box.
freeamz 2y ago
Hmmm isn't stable diffusion already doing that?
- aenvoker 2y ago
  SD has a very primitive conceptual model. Basically “bag of words nudging pixels around for a while”. Words near each other influence each other. But, there’s nearly no understanding of grammar. Midjourney is similar with text prompts. But, with image prompts it is able to understand content separately from style. You can give it a photo of two people and it can return many images of recognizable approximations of those people in different poses. SD can only start from pixels, blur and deblur those pixels in place. MJ image prompts probably works via image-to-tokens added on to your text-to-tokens-to-image.
thrance 2y ago
Each generation follows the prompt a little bit better than the last, but I don't see any revolutions. Fingers are still messed up, eyes are wonky and legs sometimes still fork into two. Fundamentally it's still the same diffusion technique, with the same limitations.
- xigency 2y ago
  Not only that, but the people working in this space still aren't reading the room, which is not encouraging.
gcanyon 2y ago
It's interesting to hear people side with the artists when in previous discussions on this forum I've gotten significant approval/agreement arguing that copyright is far too long. As I've argued in the past, I think copyright should last maybe five years: in this modern era, monetizing your work doesn't (usually) have to take more than a short time. I'd happily concede to some sort of renewal process to extend that period, especially if some monetization method is in process. Or some sort of mechanical rights process to replace the "public domain" phase early on. Or something -- I haven't thought about it that deeply. So thinking about that in this process: everyone is "ghiblifying" things. Studio Ghibli has been around for very nearly 40 years, and their "style" was well established over 35 years ago. To me, that (should) make(s) it fair game. The underlying assumption, I think, is that all the "starving" artists are being ripped off, but are they? Let's consider the numbers -- there are a handful of large-scale artists whose work is obviously replicable: Ghibli, the Simpsons, Pixar, etc. None of them is going hungry because a machine model can render a prom pic in their style. Then you get the other 99.999% of artists, all of whose work went into the model. They will be hurt, but not specifically because their style has been ingested and people want to replicate their style. Rather, they will be hurt because no one knows their style, nor cares about it; people just want to be able to say e.g. "Make a charcoal illustration of me in this photo, but make me sitting on a horse in the mountains." It's very much like the arguments about piracy in the past: 99.99% of people were never going to pay an artist to create that charcoal sketch. The 0.01% who might are arguably causing harm to the artist(s) by not using them to create that thing, but the rest were never going to pay for it in the first place. All to say it's complicated, and obviously things are changing dramatically, but it's difficult to make the argument that "artists need to be compensated for their work being used to train the model" without both a reasonable plan for how that might be done, and a better-supported argument for why.
- Taek 2y ago
  I find it unlikely that someone who was willing to pay an artist for a charcoal sketch would be satisfied with an AI alternative. You don't just buy art for the aethstetic, you buy it for a lot of reasons and AI doesn't give any of the same satisfaction.
qiqitori 2y ago
Wha- wha- what? I tried to generate an image in ChatGPT after the announcement a while back and the image wasn't bad, but the text on it (numbers) was nonsense. (Analog gauge with nonsense numbers instead of e.g. 10, 20, 30, 40, etc.) Gave it another chance now, explicitly calling out the numbers. Well, they are improved but not sure how useful this result is (the spacing between numbers is a little off and there's still some curious counting going on. Maybe it kind of looks like the numbers are pasted in after the fact? https://chatgpt.com/share/67f4fa33-70dc-8012-8e1e-2dea563d3def https://chatgpt.com/share/67f4fa33-70dc-8012-8e1e-2dea563d3d...
- nvalis 2y ago
  These images are still created with the old model. The share link states "Made with the old version of image generation. New images coming soon." below the first image.
haswell 2y ago
> The question isn't whether these tools will change visual media, but whether we'll be thoughtful enough to shape that change intentionally. Unfortunately I think the answer to this question is a resounding “no”. The time for thoughtful shaping was a few years ago. It feels like we’re hurtling toward a future where instead we’ll be left picking up the pieces and assessing the damage. These tools are impressive and will undoubtedly unlock new possibilities for existing artists and for people who are otherwise unable to create art. But I think it’s going to be a rough ride, and whatever new equilibrium we reach will be the result of much turmoil. Employment for artists won’t disappear, but certain segments of the market will just use AI because it’s faster, cheaper, and doesn’t require time consuming iterations and communication of vision. The results will be “good enough” for many. I say this as someone who has found these tools incredibly helpful for thinking. I have aphantasia, and my ability to visualize via AI is pretty remarkable. But I can’t bring myself to actually publish these visualizations. A growing number of blogs and YouTube channels don’t share these qualms and every time I encounter them in the wild I feel an “ick”. It’ll be interesting to see if more people develop this feeling.
- pixl97 2y ago
  >But I think it’s going to be a rough ride, and whatever new equilibrium we reach will be the result of much turmoil. Honestly visual media just seems to be the start. In the past two years we've seen about as much robotics progress as the last 20. If this momentum keeps up then we're not just talking about artists that are going to have issues.
  - TheGrognardling 2y ago
    Honestly, I'm pretty encouraged by all of the projects and efforts within legislation and organizations regarding clear lines being drawn - i.e., through watermarking to clearly label whether something is AI-generated or not - as well as the efforts by industries for livelihoods to be protected, specifically in the creative space, where human intentionality and feeling are still of the utmost essentiality. We've seen, are seeing, and will see cultural and societal acceptance and backlash against one thing or another, but I'm confident that we will adapt. Ultimately, pushback, thanks to the Web itself, is already pretty monumental among artists and even other AI researchers in many respects - regulations for the internet, largely due to lack of the Web, were far slower to materialize, on an exponential scale. I remain optimistic that we will find the niches where AI is needed, where it isn't, and where it is detrimental.
AIPedant 2y ago
[dead]
eapriv 2y ago
It’s always fun to read posts like that: they say “look at this amazing thing it drew”, and the image is utter garbage.
NiloCK 2y ago
The 'before' image passes the test this time in a "Treachery of Images" sort of way.
morkalork 2y ago
Huh, the coffee table reminds me of all those cheap e-retailers who very clearly (and badly) photoshop their clothes on to same 2 or 3 stock model images. If you thought shopping online sucked before, it's just going to get even worse now.
vunderba 2y ago
4o, despite OpenAI's practically draconian content policies, is a pretty big leap forward. I put together a comparison of some of the most competitive generative models (Imagen, 4o, Flux, and MJ7) where I prioritized increasingly difficult prompt adherence. If Imagen 3 had 4o's multimodal capabilities (being able to make constant adjustments against a generated image by prompting) I would say its nearly on-par with 4o. https://genai-showdown.specr.net https://genai-showdown.specr.net
deleted 2y ago
[deleted]
ge96 2y ago
> we guac you covered
xnorswap 2y ago
The "How to build a boardgame" infographic looks like half my linkedin "feed" now, but a boardgame instead of random basic programming / recruimentment / sales topic. Feed is in quotes because my feed seems to be 90% suggested posts.
- tantalor 2y ago
  This could help: https://www.linkedin.com/help/linkedin/answer/a1379064 https://www.linkedin.com/help/linkedin/answer/a1379064
hansmayer 2y ago
> " Image generation is likely to be very disruptive in ways we don’t understand right now. " Is anyone getting tired of these formulations ? When a tech is disruptive, we know it immediately. Uber was disruptive. AirBnB, Gmail, Amazon, even Facebook at one point. You just knew it, nobody was writing long essays trying to justify those products. Robots generating statistically median images is impressive, but not disruptive at all. If something is "likely" to be "disruptive", but in ways "we don't understand yet", how can the claim even be made? What is it based on? If we do not understand it yet, how can we understand if it is "likely to be disruptive".
Der_Einzige 2y ago
The #1 reason that this technology won't proliferate more quickly is that humans are a bunch of COOMERS! We get Stable Diffusion V1.5 and SDXL and what does the community go do with it? Lmao see civit.ai and it's literal hundreds of thousands of NSFW loras. The most popular model today on that website is the NSFW anime version of SDXL, called "Pony Diffusion" (I'm literally not making this up. A bunch of Bronies made this model!) Imagine that an open source image generator which does tokens autoregressively like this at this quality is released. The world is simply not ready for the amount of horny stuff that is going to be produced (especially without consent). It appears that the male libido really is the reason for most bad things in the world. We are truly the "villains of history".
- sfn42 2y ago
  I don't see the problem. Most people don't know this stuff exists. You pretty much have to look for it to find it. So what if someone wants to create rule 34 stuff? Let them, who cares? It doesn't hurt anyone. There was already a large market for artists drawing weird fetish stuff, AI doesn't really change anything.
smusamashah 2y ago
I am waiting for when I could provide these a scene snippet from "Hitchhiker's Guide To Galaxy" (or any book) and it could draw that for me. The gold planets, the waking up on the beach, total perspective vortex etc. I like the book, but there are quite a few scenes which are quite hard to visualize and make sense. An image generator that can follow that language and detail will be amazing. Even more awesome will be if it remains consistent in follow ups.
- wrboyce 2y ago
  I love the idea, but I feel like I have to say that I’ve got a pretty solid idea of what the total perspective vortex would look like for someone being subjected to it. When I first read the books I immediately had a visual and that has never changed when I’ve read them again (and again…). I’m not sure what that says about either of us, but I would say that your definitive “quite hard to visualise” statement is very much subjective.
  - smusamashah 2y ago
    Vortex may be not so much but there are other hard to visualize things. I am on the third book, and I have no idea what Beeblebrox's two heads look like. Second head is often mentioned in passing. Sometimes its mentioned as its always there, other times it feels like its just pops out of somewhere, otherwise, it's like it doesn't exist. There is the scene when they see themselves on the beach on first rescue by the ship. That was hard to grasp. Or the insides of the ship itself, the bridge, the panels etc. Also that black ship they stole. But may be its just me having a hard time with these concepts. It's not just about scene being difficult to visualize, even if I can see them in my head, I want to see them on paper too because those thing excite me.
    - thwarted 2y ago
      We have alt text for images, you want alt images for text. You can see other people's interpretation of Zaphod's two heads by watching the BBC HHGTTG show (Mark Wing-Davey) or the movie (Sam Rockwell), among other renditions, which offer completely different interpretations, none of them canonical (not the least of which is because there was no canonical version of HHGTTG according to DA). I'm sure there are multitudes of fan art for HHGTTG on deviantart. Having AI generate an image doesn't offer any more "official" visualization. Zaphod's second head is mentioned just as much is warranted. If a character has a limp or a crazy haircut it is not mentioned every time, because it has nothing to do with what is going on. And the book mentions that one head is often distracted/asleep, so it sounds like you do have a good visual of what his two heads are like. While I understand that people think differently and some people are more visual thinkers, a good portion of the concepts expressed through writing are meant to be mindfucks that are difficult to express visually. A picture may be worth a thousand words, but the meat of writing is usually not the visual representation of its concepts. That's a great thing about writing: you can fill in the visuals yourself and it's fodder for fans to discuss. (BTW, Hotblack Desiato's ship would just be black. Your eyes couldn't focus on it. Even the controls were black labels on a black background. There is nothing here to visualize other than, well, blackness).
nowittyusername 2y ago
There is circumstantial evidence out there that 4o image manipulation isn't done within the 4o image generator in one shot but is a workflow done by an agentic system. Meaning this, user inputs prompt "create an image with no elephants in the room" > prompt goes to an llm which preprocesses the human prompt > outputs a a prompt that it knows works withing this image generator well > create an image of a room > and that llm processed prompt is sent to the image generator. Same happens with edits but a lot more complicated, meaning function calling tools are involved with many layers of edits being done behind the scenes. Try it yourself, take an image, send it it, and have the 4o edit it for you in some way, then ask it to edit again, and again, and so on. you will notice noticeable sepia filter being applied every edit, and the image ends up more and more sepia toned with more edits. This is because in the workflow that is one of the steps that is naively applied without consideration of multi edit possibility. If this was a one shot solution where editing is done within 4o image model by itself, the sepia problem wouldn't be there.
- renewiltord 2y ago
  The prompt enrichment thing is pretty standard. Everyone does that bit, though some make it user-visible. On Grok it used to populate to the frontend via the download name on the image. The image editing is interesting.
  - genewitch 2y ago
    All the stable diffusion software I've used names the files after some form of the prompt, and probably because SD weights the first tokens higher than the last tokens, probably as a side effect of the way the CLIP/BLIP works. I doubt any of these companies have rolled their own interface to stable diffusion / transformers. It's copy and paste from huggingface all the way down. I'm still waiting for a confirmed Diffusion Language Model to be released as gguf that works with llama.cpp
    - danielbln 2y ago
      Auto1111 and co are using the prompt in the filename because it's convenient, not due to some inherent CLIP mechanism. If you think that companies like OpenAI (for all the criticisms they deserve) don't use their own inference harness and image models I have a bridge to sell to you.
justinator 2y ago
But the annotations are still wrong, https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3bca939e-53c3-40cb-a7cc-6177d2765233_2544x985.png https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_... (nice URL btw) The room, the door, the ceiling are all of a scale to fit many sizes of elephants.
- sfn42 2y ago
  The lines don't really make sense either, like the one above the sofa should probably go along the corner between the floor and wall.
  - justinator 2y ago
    Just like text results from AI, when images like this get better, the more and more subtle yet absolute wrongness is going to be a nightmare to deal with. Imagine a ask AI to show me a sewer cap that's less than a foot wide (or whatever, I dunno, watch TMNT right now). And it does, just by showing a sewer cap that looks photorealistic and a ruler, that has markings from one end to the other that only go up to 8 inches. That doesn't mean sewer caps come in that size, it just means you can produce a rendered image to fit what you asked for.
lupusreal 2y ago
The image annotated to explain why no elephants are possible is very amusing. To me, this kind of image generation isn't very interesting for creating final products, but is extremely useful for communicating design intent to other people when collaborating on large creative projects. Previously I used crude "ms paint" sketches for this, which was much more tedious and less effective.
ziofill 2y ago
I usually agree with most of Gary Marcus' points, but I'd really like to hear his take on this. One of his examples is that "the system can't generate a horse riding an astronaut" and in fact I tried a lot in the past but it would always draw the astronaut on top of the horse. Well, here is the result now: https://postimg.cc/QFtRjbHM https://postimg.cc/QFtRjbHM
- jsheard 2y ago
  Whenever one of these well known gotcha prompts gets "solved" the question is always whether they actually solved the underlying reason it used to fail, or did they just have a bunch of third-world workers tag pictures of horses and astronauts until the model started handling that specific example more reliably. As the saying goes, every measure which becomes a target becomes a bad measure.
  - pixl97 2y ago
    Well, what you do is try examples close to the problem space around it. For example a fiddler crab riding a chameleon https://sora.com/g/gen_01jrbq91wtefjtpb8ceajdh9mt https://sora.com/g/gen_01jrbq91wtefjtpb8ceajdh9mt and then iterate around other combinations to see if it's generalized or not.
DonHopkins 2y ago
Q: How do you know if there's an elephant hiding under your bed? A: Your face is pressed up against the ceiling!
lou1306 2y ago
Putting my Wittgenstein hat on: How can I ever be sure that the machine is not generating an incredibly tiny elephant, maybe hidden under the sofa?
cadamsdotcom 2y ago
Anyone stuck claiming AI isn’t useful - there are so many useful things it can now do. With text that makes sense you can generate invitations for your next picnic. That wasn’t possible mere weeks ago. Wonderful to be alive for these step changes in human capability.
klik99 2y ago
I’ve seen a few YouTube thumbnail generation examples on Reddit (I’m on vacation so not gonna search for a link) that show multimodal with inline text giving specific instructions. It’s impressed me in a way that I haven’t been with LLMs for 2 years, IE it’s not just getting better at what it already does, but a totally new and intuitive way of working with generative AI. My understanding is it’s a meta-LLM approach, using multiple models and having them interact. I feel like it’s also evidence that OpenAI is not seriously pursuing AGI (just my opinion, I know there’s some on here who would aggressively disagree), but rather market use cases. It feels like an acceptance that any given model, at least now, has its own limitations but can get more useful in combination.
1970-01-01 2y ago
For the first 9 years of an elephant's life, it can easily walk into that room. I don't find this to be a breakthrough. I find it to be clickbait.
m4thfr34k 2y ago
I am very impressed with the current image generators out there, 4o / Leonardo / etc., but I cannot wait until they include some step to actually "check their work". Ask it to produce a watch with the time of 6:37. It fails every time, because almost all watch photos out there are set to a specific time, and seems like something an initial "did I do this right" check could confirm. The time example is trivial but a general "does this output actually make sense considering what the user asked" checked would be tremendously valuable.
stfupniggger 2y ago
[flagged]
swframe2 2y ago
I'm hoping this alternative image prompt preprocessing technique gets more attention: https://art-msra.github.io/ https://art-msra.github.io/ Basically, the user's image prompt is converted to a several prompts to generate parts of the final image in layers which are combined. The layers are still available so that edits can cleanly update one section without affecting the others.
mrconter11 2y ago
Isn't it ironic that it ended up being harder to get a computer to explicitly not create a photorealistic image of an elephant than to have create one?
Zr01 2y ago
I'm more interested in the technical details than the publicity. Pretty much anyone these days can learn what a diffusion model is, how they're implemented, what the control flow is. What about this new multimodal LLM? They have no problems with text, they generate images using tokens, but how exactly? There's no open-source implementations that I know of, and I'm struggling to find details.
- famouswaffles 2y ago
  This video is very good. https://youtu.be/EzDsrEvdgNQ?si=EWp3U1GMkwg1bMQQ https://youtu.be/EzDsrEvdgNQ?si=EWp3U1GMkwg1bMQQ One thing i'd add is that generating the tokens at the target resolution from the start is no longer the only approach to autoregressive image generation. Rather than predicting each patch at the target resolution right away, it starts with the image (as patches) at a very small resolution and increasingly scales up. Paper here - https://arxiv.org/abs/2404.02905 https://arxiv.org/abs/2404.02905
globnomulous 2y ago
> The past couple years have been spent trying to figure out what text AI models are good for, and new use cases are being developed continuously. In other words, people who care about money and only money are pushing for these tools because they're convinced they'll reduce labor costs and somehow also improve the resulting product, while engineers and creative professionals who have these tools foisted upon them by unimaginative business people continue to insist that the tools are a solution in search of a problem, that they're stochastic parrots and plagiarism automata that bypass all of the important parts of engineering and creativity and make the absolutely, breathtakingly idiotic mistake of supposing it's possible to leap to a finished product without all the work and problem solving involved in getting there. > The line between human and AI creation will continue to blur This is utter nonsense, and hype-man prognosticators in the tech world like the author of the article turn out pretty much 100% of the time to be either grifters or saps who have fallen for the grifters' nonsense.

445 points by Kerrick 2y ago

That proper "no elephants" first image is hilarious. Another key point here is the generative AI's meme game is getting rather strong. Which isn't a small thing, humour is an advanced soft skill.

YurgenJurgensen 2y ago
Having thousands of copies of that image in your training set isn’t a skill at all.
- hnbad 2y ago
  AI's "meme game is going strong" by the same metric I would use to try to argue that Elon Musk's is. I wouldn't call it a good metric, though.
loudmax 2y ago
The "Game Design Otter" action figure seems to come with a pair of flashlights. I bet that's a residue from the previous prompt about illuminating the tablet with a flashlight.
aenvoker 2y ago
Having used lots of different image generators, so far the only one with a sense of humor has been Dall-e3. https://www.reddit.com/r/dalle2/s/khb5XuNFdl https://www.reddit.com/r/dalle2/s/khb5XuNFdl There’s probably some sort of connection to ChatGPT in there. But, I don’t know enough about how it works.

x187463 2y ago

This is a before/after moment for image generation. A simple example is the background images on a ton of (mediocre) music youtube channels. They almost all use AI generated images that are full of nonsense the closer you look. Jazz channels will feature coffee shops with garbled text on the menu and furniture blending together. I bet all of that disappears over the next few months. On another note, and perhaps others are feeling similarly, but I am finding myself surprised at how little use I have for this stuff, LLMs included. If, ten years ago, you told me I would have access to tools like this, I'm sure I would have responded with a never ending stream of ideas and excitement. But now that they're here, I just sort of poke at it for a minute and carry on with my day. Maybe it's the unreliability on all fronts, I don't know. I ask a lot of programming questions and appreciate some of the autocomplete in vscode, but I know I'm not anywhere close to taking full advantage of what these systems can do.

Retr0id 2y ago
I've never used a stock photo site before, so I suppose it's no surprise I have no real use for "generate any image on demand".
- YurgenJurgensen 2y ago
  Their main application appears to be taking blog posts and internal memos and making them three times longer and use ten times the bandwidth to convey no more information. So exactly the application AI is ‘good’ at.
  - wongarsu 2y ago
    If anything stock image websites are even worse at this then AI. With AI you come up with an image idea, then try to make the AI produce something close to it. With stock images you come up with an image idea, then hope some photographer had a similar idea and uploaded it to a stock website.
- esperent 2y ago
  I've used stock photo sites occasionally but I use vector art and icon sites multiple times a week. Even today, I used an few different sites while designing some stuff on Canva. The reason I don't use AI is because it gives me far less reliable and impossible to specify results than just searching through the limited lists of human made art. Today, for undisclosed reasons, I needed vector art of peanuts. I found imperfect but usable human made art within seconds from a search engine. I then spent around 15 - 25 minutes trying to get something closer to my vision using ChatGPT, and using the imperfect art I'd found as a style guide. I got lots of "huh that's cool what AI can do" but nothing useful. Nothing closer to my vision than what I started with. By coincidence it's the first time I'vr tried making art with AI in about a year, but back then I bought a Midjourney account and spent a month making loads of art, then installed SD on my laptop and spent another couple of weeks playing around with that. So it's not like I'm lacking experience. What I've found so far is that AI art generators are great for generating articles like this one. And they do make some genuinely cool pictures, it blows my mind that computers can do this now. It's just when I sit down with a real world task that has specific, concrete requirements... I find them useless.

Retr0id 2y ago

I had a reasonable intuition for how the "old" method works, but I still don't grok this new approach. "in multimodal image generation, images are created in the same way that LLMs create text, a token at a time" Is there some way to visualise these "image tokens", in the same way I can view tokenized text?

fxtentacle 2y ago
Imagine you cut the image into 32x32 pixel blocks. And then for each block, you can chose 1 out of 128,000 variations. And then a post-processing step smoothes out the borders between blocks and adjusts small details. That's basically how a transformer image generation model works. As such, the process is remarkably similar to old fixed-font ASCII art. It's just that modern AIs have a larger alphabet and, thus, more character shapes to choose from.
- rwmj 2y ago
  I don't get how this would produce consistent images. In the article, the text could be on a grid, but the window and doorway and sofa don't seem to be grid-aligned. (Or maybe the text is overlaid?)
  - danielbln 2y ago
    The model looks ahead, just like LLMs look ahead. An LLM outputs token by token but can still output a fully coherent and consistent story for example. This new crop of auto-regressive image models does the same.
WhiteNoiz3 2y ago
I haven't see any details on how OpenAI's model works, but the tokens it generates aren't directly translated into pixels - those tokens are probably fed into a diffusion process which generates the actual image.. The tokens are the latent space or conditioning for the actual image generation process.
- 2y ago

TacticalCoder 2y ago

[dead]

card_zero 2y ago

Looking at the example where the coffee table is swapped, I notice every time the image is reprocessed it mutates, based on the previous iteration, and objects become more bizarre each time, like chinese whispers. * The weird-ass basket decoration on the table originally has some big chain links (maybe anchor chain, to keep the theme with the beach painting). By the third version, they're leathery and are merging with the basket. * The candelabra light on the wall, with branch decorations, turns into a sort of skinny minimalist gold stag head, and then just a branch. * The small table in the background gradually loses one of its three legs, and ends up defying gravity. * The freaky green lamps in the window become at first more regular, then turn into topiary. * Making the carpet less faded turns up the saturation on everything else, too, including the wood the table is made from.

bla3 2y ago
The pictures on the wall change too.
- card_zero 2y ago
  Yes, first a still life and something impressionist, then a blob and a blob, then a smear and a smear. And what about the reflections and transparency of the glass table top? It gets very indistinct. Keep working at the same image and it looks like you'll end up with some Deep Dream weirdness. I think the fireplace might be turning into some tiny stairs leading down. :)
  - YurgenJurgensen 2y ago
    Only sailors know how to leave.
- rob74 2y ago
  Actually, almost everything changes slightly - the number, shape and pattern of the chairs, the number and pattern of the pillows, the pattern of the curtains, the scene outside the window, the wooden part of the table, the pattern of the carpet... The blue couch stays largely the same, it just loses some detail...

NitpickLawyer 2y ago

> The results are not as good as a professional designer could create but are an impressive first prototype. I like to look at how far we've come since the early days of Stable Diffusion. It was fascinating to play with it back then, but it quickly became apparent that it was "generic" and not suited for "real work" because it lacked consistency, text capabilities, fingers! and so on... Looking at these results now, I'm amazed at the quality, consistency and ease of use. Gone are the days of doing alchemy on words and adding a bunch of "in the style of Rutkovsky, golden hour, hd, 4k, pretty please ..." at the end of prompts.

probably_wrong 2y ago

> Is it okay to reproduce the hard-won style of other artists using AI? Who owns the resulting art? Who profits from it? Which artists are in the training data for AI, and what is the legal and ethical status of using copyrighted work for training? These were important questions before multimodal AI, but now developing answers to them is increasingly urgent. I have to disagree with the conclusion. This was an important discussion to have two to three years ago, then we had it online, and then we more or less agreed that it's unfair for artists to have their works sucked up with no recourse. What the post should say is "we know that this is unfair to artists, but the tech companies are making too much money from them and we have no way to force them to change".

shkkmo 2y ago
> This was an important discussion to have two to three years ago, then we had it online, and then we more or less agreed that it's unfair for artists to have their works sucked up with no recourse. Speak for yourself, there was no consensus online. There are plenty of us that think that dramatically expanding the power of copyright would be a huge mistake that would primarily benefit larger companies and do little to protect or fund small artists.
- OtherShrezzing 2y ago
  >There are plenty of us that think that dramatically expanding the power of copyright would be a huge mistake that would primarily benefit larger companies and do little to protect or fund small artists. The status quo also primarily benefits larger companies, and does little (exactly nothing, if we're being earnest) to protect or fund small artists. It's reasonable to hold both opinions that: 1) artists aren't being compensated, even though their work is being used by these tools, and 2) massive expansion of copyright isn't the appropriate response to 1).
  - deleted 2y ago
    [deleted]
eadmund 2y ago

rkharsan64 2y ago

Are there any local models that use this new approach to generating images?

GaggiX 2y ago
GPT-4o is the only model that seems to work well in the text-image joint space to this degree, even Gemini Flash 2.0 with native image support is not nearly as good so it will probably be a while for a good open source alternative to pop up (a while in the context of AI development).
- gerash 2y ago
  depends on the use case. I used GPT-4o for some image editing (adding or removing things) to an image of a person and they distort the look of the people after each edit but (Gemini Flash + image out) did much better. The main problem is there is little control. For example I asked to add a helicopter to an image in a ski resort but then it seems cumbersome for me to have to write a full paragraph to describe where exactly I want this helicopter to be rather than if I could just do it by dragging things with a mouse.
DeathArrow 2y ago
Yes, there's HiDream which yields even better results than 01. https://github.com/HiDream-ai/HiDream-I1 https://github.com/HiDream-ai/HiDream-I1
- GaggiX 2y ago
  This is just a diffusion text-to-image model like many others, completely different than a LLM with a native image support.

orbital-decay 2y ago

4o still exhibits the "pink elephant effect", it's just... subtler, and tends to reveal itself on a complex or confusing prompt. Negations are also still not handled properly, they tend to slightly confuse the model and decrease the accuracy of the answer or the generated picture. The same is true for any other LLM. Moreover, the author is asking the model to rationalize the decision he already made ("tell me why there can't be any elephants"), which could work as an equivalent to a CoT step. It's "just" a much bigger and much better trained model. Which is a quality on its own, absolutely no doubt about that. Fundamentally the issue is still there though, just less prominent. Which kind of makes sense - imagine the prompt "not green", what even is that? It's likely slightly out of distribution and requires representing a more complex abstraction, so the accuracy will necessarily be worse than stating the range of colors directly. The result might be accurate, until the model is confused/misdirected by something else, and suddenly it's not. I think in the end none of the architectural differences will matter beyond the scaling. What will matter a lot more is data diversity and training quality.

danielbln 2y ago
But it's literally a different architecture (auto-regressive, presumably sequence based vs diffusion). In my experiments it is significantly, overwhelmingly better at consistency, coherence and prompt adherence. Things I needed control nets before it just... does it. and even zooming into fine details, they make sense. Here is an example with a bunch of negations: https://i.imgur.com/P8G5ICs.png https://i.imgur.com/P8G5ICs.png
- wat10000 2y ago
  It’s interesting that out of all the aquatic animals it could have used, it chose one that perhaps looks the most like an elephant.
  - danielbln 2y ago
    That's just luck of the draw, in a new session with the same prompt it outputted a bird: https://i.imgur.com/SLT8cYe.png https://i.imgur.com/SLT8cYe.png

d4rkp4ttern 2y ago

Diagrams are still a big unsolved problem. Making diagrams for a talk or paper is an extremely tedious process and I am still waiting for a good multimodal LLM solution for this. It should take a sketch and/or text description of what you want and in a few iterations you should get what you want. GPT4o tries hard but results are still bad.

chthonicdaemon 2y ago
I've had the best luck at getting it to produce diagrams-as-code like mermaid or plantuml.
- d4rkp4ttern 2y ago
  I know but those diagrams often don’t adequately capture what I want. Think of diagrams in nice technical talks or papers. I’ve even tried having the LLM (Claude) generate SVGs. They all fall short.

shubhamjain 2y ago

The Ghibli trend completely missed the real breakthrough — and it’s this. The ability to closely follow text, understand the input image, and maintain context of what’s already there is a massive leap in image generation. While Midjourney delivered visually stunning results, I constantly struggled to get anything specific out of it, making it pretty much useless for actual workflows. 4o is the first image generation model that feels genuinely useful not just for pretty things. It can produce comics, app designs, UI mockups, storyboards, marketing assets, and so on. I saw someone make a multi-panel comic with it with consistent characters. Obviously, it's not perfect. But just getting there 90% is a game changer.

empath75 2y ago
I had chatgpt generate a flow chart with mermaid js for something at work and then write a scott mccloud style comic book explaining it in detail and it looked so convincing, even though it got some of the details a bit wrong. It's _so close_ to making completely usable graphics out of the box.

freeamz 2y ago

Hmmm isn't stable diffusion already doing that?

aenvoker 2y ago
SD has a very primitive conceptual model. Basically “bag of words nudging pixels around for a while”. Words near each other influence each other. But, there’s nearly no understanding of grammar. Midjourney is similar with text prompts. But, with image prompts it is able to understand content separately from style. You can give it a photo of two people and it can return many images of recognizable approximations of those people in different poses. SD can only start from pixels, blur and deblur those pixels in place. MJ image prompts probably works via image-to-tokens added on to your text-to-tokens-to-image.

thrance 2y ago

Each generation follows the prompt a little bit better than the last, but I don't see any revolutions. Fingers are still messed up, eyes are wonky and legs sometimes still fork into two. Fundamentally it's still the same diffusion technique, with the same limitations.

xigency 2y ago
Not only that, but the people working in this space still aren't reading the room, which is not encouraging.

gcanyon 2y ago

It's interesting to hear people side with the artists when in previous discussions on this forum I've gotten significant approval/agreement arguing that copyright is far too long. As I've argued in the past, I think copyright should last maybe five years: in this modern era, monetizing your work doesn't (usually) have to take more than a short time. I'd happily concede to some sort of renewal process to extend that period, especially if some monetization method is in process. Or some sort of mechanical rights process to replace the "public domain" phase early on. Or something -- I haven't thought about it that deeply. So thinking about that in this process: everyone is "ghiblifying" things. Studio Ghibli has been around for very nearly 40 years, and their "style" was well established over 35 years ago. To me, that (should) make(s) it fair game. The underlying assumption, I think, is that all the "starving" artists are being ripped off, but are they? Let's consider the numbers -- there are a handful of large-scale artists whose work is obviously replicable: Ghibli, the Simpsons, Pixar, etc. None of them is going hungry because a machine model can render a prom pic in their style. Then you get the other 99.999% of artists, all of whose work went into the model. They will be hurt, but not specifically because their style has been ingested and people want to replicate their style. Rather, they will be hurt because no one knows their style, nor cares about it; people just want to be able to say e.g. "Make a charcoal illustration of me in this photo, but make me sitting on a horse in the mountains." It's very much like the arguments about piracy in the past: 99.99% of people were never going to pay an artist to create that charcoal sketch. The 0.01% who might are arguably causing harm to the artist(s) by not using them to create that thing, but the rest were never going to pay for it in the first place. All to say it's complicated, and obviously things are changing dramatically, but it's difficult to make the argument that "artists need to be compensated for their work being used to train the model" without both a reasonable plan for how that might be done, and a better-supported argument for why.

Taek 2y ago
I find it unlikely that someone who was willing to pay an artist for a charcoal sketch would be satisfied with an AI alternative. You don't just buy art for the aethstetic, you buy it for a lot of reasons and AI doesn't give any of the same satisfaction.

qiqitori 2y ago

Wha- wha- what? I tried to generate an image in ChatGPT after the announcement a while back and the image wasn't bad, but the text on it (numbers) was nonsense. (Analog gauge with nonsense numbers instead of e.g. 10, 20, 30, 40, etc.) Gave it another chance now, explicitly calling out the numbers. Well, they are improved but not sure how useful this result is (the spacing between numbers is a little off and there's still some curious counting going on. Maybe it kind of looks like the numbers are pasted in after the fact? https://chatgpt.com/share/67f4fa33-70dc-8012-8e1e-2dea563d3def https://chatgpt.com/share/67f4fa33-70dc-8012-8e1e-2dea563d3d...

nvalis 2y ago
These images are still created with the old model. The share link states "Made with the old version of image generation. New images coming soon." below the first image.

haswell 2y ago

> The question isn't whether these tools will change visual media, but whether we'll be thoughtful enough to shape that change intentionally. Unfortunately I think the answer to this question is a resounding “no”. The time for thoughtful shaping was a few years ago. It feels like we’re hurtling toward a future where instead we’ll be left picking up the pieces and assessing the damage. These tools are impressive and will undoubtedly unlock new possibilities for existing artists and for people who are otherwise unable to create art. But I think it’s going to be a rough ride, and whatever new equilibrium we reach will be the result of much turmoil. Employment for artists won’t disappear, but certain segments of the market will just use AI because it’s faster, cheaper, and doesn’t require time consuming iterations and communication of vision. The results will be “good enough” for many. I say this as someone who has found these tools incredibly helpful for thinking. I have aphantasia, and my ability to visualize via AI is pretty remarkable. But I can’t bring myself to actually publish these visualizations. A growing number of blogs and YouTube channels don’t share these qualms and every time I encounter them in the wild I feel an “ick”. It’ll be interesting to see if more people develop this feeling.

pixl97 2y ago
>But I think it’s going to be a rough ride, and whatever new equilibrium we reach will be the result of much turmoil. Honestly visual media just seems to be the start. In the past two years we've seen about as much robotics progress as the last 20. If this momentum keeps up then we're not just talking about artists that are going to have issues.
- TheGrognardling 2y ago
  Honestly, I'm pretty encouraged by all of the projects and efforts within legislation and organizations regarding clear lines being drawn - i.e., through watermarking to clearly label whether something is AI-generated or not - as well as the efforts by industries for livelihoods to be protected, specifically in the creative space, where human intentionality and feeling are still of the utmost essentiality. We've seen, are seeing, and will see cultural and societal acceptance and backlash against one thing or another, but I'm confident that we will adapt. Ultimately, pushback, thanks to the Web itself, is already pretty monumental among artists and even other AI researchers in many respects - regulations for the internet, largely due to lack of the Web, were far slower to materialize, on an exponential scale. I remain optimistic that we will find the niches where AI is needed, where it isn't, and where it is detrimental.

AIPedant 2y ago

[dead]

eapriv 2y ago

It’s always fun to read posts like that: they say “look at this amazing thing it drew”, and the image is utter garbage.

NiloCK 2y ago

The 'before' image passes the test this time in a "Treachery of Images" sort of way.

morkalork 2y ago

Huh, the coffee table reminds me of all those cheap e-retailers who very clearly (and badly) photoshop their clothes on to same 2 or 3 stock model images. If you thought shopping online sucked before, it's just going to get even worse now.

vunderba 2y ago

4o, despite OpenAI's practically draconian content policies, is a pretty big leap forward. I put together a comparison of some of the most competitive generative models (Imagen, 4o, Flux, and MJ7) where I prioritized increasingly difficult prompt adherence. If Imagen 3 had 4o's multimodal capabilities (being able to make constant adjustments against a generated image by prompting) I would say its nearly on-par with 4o. https://genai-showdown.specr.net https://genai-showdown.specr.net

deleted 2y ago

[deleted]

ge96 2y ago

> we guac you covered

xnorswap 2y ago

The "How to build a boardgame" infographic looks like half my linkedin "feed" now, but a boardgame instead of random basic programming / recruimentment / sales topic. Feed is in quotes because my feed seems to be 90% suggested posts.

tantalor 2y ago
This could help: https://www.linkedin.com/help/linkedin/answer/a1379064 https://www.linkedin.com/help/linkedin/answer/a1379064

hansmayer 2y ago

> " Image generation is likely to be very disruptive in ways we don’t understand right now. " Is anyone getting tired of these formulations ? When a tech is disruptive, we know it immediately. Uber was disruptive. AirBnB, Gmail, Amazon, even Facebook at one point. You just knew it, nobody was writing long essays trying to justify those products. Robots generating statistically median images is impressive, but not disruptive at all. If something is "likely" to be "disruptive", but in ways "we don't understand yet", how can the claim even be made? What is it based on? If we do not understand it yet, how can we understand if it is "likely to be disruptive".

Der_Einzige 2y ago

The #1 reason that this technology won't proliferate more quickly is that humans are a bunch of COOMERS! We get Stable Diffusion V1.5 and SDXL and what does the community go do with it? Lmao see civit.ai and it's literal hundreds of thousands of NSFW loras. The most popular model today on that website is the NSFW anime version of SDXL, called "Pony Diffusion" (I'm literally not making this up. A bunch of Bronies made this model!) Imagine that an open source image generator which does tokens autoregressively like this at this quality is released. The world is simply not ready for the amount of horny stuff that is going to be produced (especially without consent). It appears that the male libido really is the reason for most bad things in the world. We are truly the "villains of history".

sfn42 2y ago
I don't see the problem. Most people don't know this stuff exists. You pretty much have to look for it to find it. So what if someone wants to create rule 34 stuff? Let them, who cares? It doesn't hurt anyone. There was already a large market for artists drawing weird fetish stuff, AI doesn't really change anything.

smusamashah 2y ago

I am waiting for when I could provide these a scene snippet from "Hitchhiker's Guide To Galaxy" (or any book) and it could draw that for me. The gold planets, the waking up on the beach, total perspective vortex etc. I like the book, but there are quite a few scenes which are quite hard to visualize and make sense. An image generator that can follow that language and detail will be amazing. Even more awesome will be if it remains consistent in follow ups.

wrboyce 2y ago
I love the idea, but I feel like I have to say that I’ve got a pretty solid idea of what the total perspective vortex would look like for someone being subjected to it. When I first read the books I immediately had a visual and that has never changed when I’ve read them again (and again…). I’m not sure what that says about either of us, but I would say that your definitive “quite hard to visualise” statement is very much subjective.
- smusamashah 2y ago
  Vortex may be not so much but there are other hard to visualize things. I am on the third book, and I have no idea what Beeblebrox's two heads look like. Second head is often mentioned in passing. Sometimes its mentioned as its always there, other times it feels like its just pops out of somewhere, otherwise, it's like it doesn't exist. There is the scene when they see themselves on the beach on first rescue by the ship. That was hard to grasp. Or the insides of the ship itself, the bridge, the panels etc. Also that black ship they stole. But may be its just me having a hard time with these concepts. It's not just about scene being difficult to visualize, even if I can see them in my head, I want to see them on paper too because those thing excite me.
  - thwarted 2y ago
    We have alt text for images, you want alt images for text. You can see other people's interpretation of Zaphod's two heads by watching the BBC HHGTTG show (Mark Wing-Davey) or the movie (Sam Rockwell), among other renditions, which offer completely different interpretations, none of them canonical (not the least of which is because there was no canonical version of HHGTTG according to DA). I'm sure there are multitudes of fan art for HHGTTG on deviantart. Having AI generate an image doesn't offer any more "official" visualization. Zaphod's second head is mentioned just as much is warranted. If a character has a limp or a crazy haircut it is not mentioned every time, because it has nothing to do with what is going on. And the book mentions that one head is often distracted/asleep, so it sounds like you do have a good visual of what his two heads are like. While I understand that people think differently and some people are more visual thinkers, a good portion of the concepts expressed through writing are meant to be mindfucks that are difficult to express visually. A picture may be worth a thousand words, but the meat of writing is usually not the visual representation of its concepts. That's a great thing about writing: you can fill in the visuals yourself and it's fodder for fans to discuss. (BTW, Hotblack Desiato's ship would just be black. Your eyes couldn't focus on it. Even the controls were black labels on a black background. There is nothing here to visualize other than, well, blackness).

nowittyusername 2y ago

There is circumstantial evidence out there that 4o image manipulation isn't done within the 4o image generator in one shot but is a workflow done by an agentic system. Meaning this, user inputs prompt "create an image with no elephants in the room" > prompt goes to an llm which preprocesses the human prompt > outputs a a prompt that it knows works withing this image generator well > create an image of a room > and that llm processed prompt is sent to the image generator. Same happens with edits but a lot more complicated, meaning function calling tools are involved with many layers of edits being done behind the scenes. Try it yourself, take an image, send it it, and have the 4o edit it for you in some way, then ask it to edit again, and again, and so on. you will notice noticeable sepia filter being applied every edit, and the image ends up more and more sepia toned with more edits. This is because in the workflow that is one of the steps that is naively applied without consideration of multi edit possibility. If this was a one shot solution where editing is done within 4o image model by itself, the sepia problem wouldn't be there.

renewiltord 2y ago
The prompt enrichment thing is pretty standard. Everyone does that bit, though some make it user-visible. On Grok it used to populate to the frontend via the download name on the image. The image editing is interesting.
- genewitch 2y ago
  All the stable diffusion software I've used names the files after some form of the prompt, and probably because SD weights the first tokens higher than the last tokens, probably as a side effect of the way the CLIP/BLIP works. I doubt any of these companies have rolled their own interface to stable diffusion / transformers. It's copy and paste from huggingface all the way down. I'm still waiting for a confirmed Diffusion Language Model to be released as gguf that works with llama.cpp
  - danielbln 2y ago
    Auto1111 and co are using the prompt in the filename because it's convenient, not due to some inherent CLIP mechanism. If you think that companies like OpenAI (for all the criticisms they deserve) don't use their own inference harness and image models I have a bridge to sell to you.

justinator 2y ago

But the annotations are still wrong, https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3bca939e-53c3-40cb-a7cc-6177d2765233_2544x985.png https://substackcdn.com/image/fetch/w_1456,c_limit,f_webp,q_... (nice URL btw) The room, the door, the ceiling are all of a scale to fit many sizes of elephants.

sfn42 2y ago
The lines don't really make sense either, like the one above the sofa should probably go along the corner between the floor and wall.
- justinator 2y ago
  Just like text results from AI, when images like this get better, the more and more subtle yet absolute wrongness is going to be a nightmare to deal with. Imagine a ask AI to show me a sewer cap that's less than a foot wide (or whatever, I dunno, watch TMNT right now). And it does, just by showing a sewer cap that looks photorealistic and a ruler, that has markings from one end to the other that only go up to 8 inches. That doesn't mean sewer caps come in that size, it just means you can produce a rendered image to fit what you asked for.

lupusreal 2y ago

The image annotated to explain why no elephants are possible is very amusing. To me, this kind of image generation isn't very interesting for creating final products, but is extremely useful for communicating design intent to other people when collaborating on large creative projects. Previously I used crude "ms paint" sketches for this, which was much more tedious and less effective.

ziofill 2y ago

I usually agree with most of Gary Marcus' points, but I'd really like to hear his take on this. One of his examples is that "the system can't generate a horse riding an astronaut" and in fact I tried a lot in the past but it would always draw the astronaut on top of the horse. Well, here is the result now: https://postimg.cc/QFtRjbHM https://postimg.cc/QFtRjbHM

jsheard 2y ago
Whenever one of these well known gotcha prompts gets "solved" the question is always whether they actually solved the underlying reason it used to fail, or did they just have a bunch of third-world workers tag pictures of horses and astronauts until the model started handling that specific example more reliably. As the saying goes, every measure which becomes a target becomes a bad measure.
- pixl97 2y ago
  Well, what you do is try examples close to the problem space around it. For example a fiddler crab riding a chameleon https://sora.com/g/gen_01jrbq91wtefjtpb8ceajdh9mt https://sora.com/g/gen_01jrbq91wtefjtpb8ceajdh9mt and then iterate around other combinations to see if it's generalized or not.

DonHopkins 2y ago

Q: How do you know if there's an elephant hiding under your bed? A: Your face is pressed up against the ceiling!

lou1306 2y ago

Putting my Wittgenstein hat on: How can I ever be sure that the machine is not generating an incredibly tiny elephant, maybe hidden under the sofa?

cadamsdotcom 2y ago

Anyone stuck claiming AI isn’t useful - there are so many useful things it can now do. With text that makes sense you can generate invitations for your next picnic. That wasn’t possible mere weeks ago. Wonderful to be alive for these step changes in human capability.

klik99 2y ago

I’ve seen a few YouTube thumbnail generation examples on Reddit (I’m on vacation so not gonna search for a link) that show multimodal with inline text giving specific instructions. It’s impressed me in a way that I haven’t been with LLMs for 2 years, IE it’s not just getting better at what it already does, but a totally new and intuitive way of working with generative AI. My understanding is it’s a meta-LLM approach, using multiple models and having them interact. I feel like it’s also evidence that OpenAI is not seriously pursuing AGI (just my opinion, I know there’s some on here who would aggressively disagree), but rather market use cases. It feels like an acceptance that any given model, at least now, has its own limitations but can get more useful in combination.

1970-01-01 2y ago

For the first 9 years of an elephant's life, it can easily walk into that room. I don't find this to be a breakthrough. I find it to be clickbait.

m4thfr34k 2y ago

I am very impressed with the current image generators out there, 4o / Leonardo / etc., but I cannot wait until they include some step to actually "check their work". Ask it to produce a watch with the time of 6:37. It fails every time, because almost all watch photos out there are set to a specific time, and seems like something an initial "did I do this right" check could confirm. The time example is trivial but a general "does this output actually make sense considering what the user asked" checked would be tremendously valuable.

stfupniggger 2y ago

[flagged]

swframe2 2y ago

I'm hoping this alternative image prompt preprocessing technique gets more attention: https://art-msra.github.io/ https://art-msra.github.io/ Basically, the user's image prompt is converted to a several prompts to generate parts of the final image in layers which are combined. The layers are still available so that edits can cleanly update one section without affecting the others.

mrconter11 2y ago

Isn't it ironic that it ended up being harder to get a computer to explicitly not create a photorealistic image of an elephant than to have create one?

Zr01 2y ago

I'm more interested in the technical details than the publicity. Pretty much anyone these days can learn what a diffusion model is, how they're implemented, what the control flow is. What about this new multimodal LLM? They have no problems with text, they generate images using tokens, but how exactly? There's no open-source implementations that I know of, and I'm struggling to find details.

famouswaffles 2y ago
This video is very good. https://youtu.be/EzDsrEvdgNQ?si=EWp3U1GMkwg1bMQQ https://youtu.be/EzDsrEvdgNQ?si=EWp3U1GMkwg1bMQQ One thing i'd add is that generating the tokens at the target resolution from the start is no longer the only approach to autoregressive image generation. Rather than predicting each patch at the target resolution right away, it starts with the image (as patches) at a very small resolution and increasingly scales up. Paper here - https://arxiv.org/abs/2404.02905 https://arxiv.org/abs/2404.02905

globnomulous 2y ago

> The past couple years have been spent trying to figure out what text AI models are good for, and new use cases are being developed continuously. In other words, people who care about money and only money are pushing for these tools because they're convinced they'll reduce labor costs and somehow also improve the resulting product, while engineers and creative professionals who have these tools foisted upon them by unimaginative business people continue to insist that the tools are a solution in search of a problem, that they're stochastic parrots and plagiarism automata that bypass all of the important parts of engineering and creativity and make the absolutely, breathtakingly idiotic mistake of supposing it's possible to leap to a finished product without all the work and problem solving involved in getting there. > The line between human and AI creation will continue to blur This is utter nonsense, and hype-man prognosticators in the tech world like the author of the article turn out pretty much 100% of the time to be either grifters or saps who have fallen for the grifters' nonsense.