3 ms·

Excellent site! OpenAI 4o is more than mildly frighting in it's capabilities to understand the prompt. Seems mostly what's holding it back is a tendency away fr

Excellent site! OpenAI 4o is more than mildly frighting in it's capabilities to understand the prompt. Seems mostly what's holding it back is a tendency away from photo-realism (or even typical digital art styles) and it's own safeguards.

echelon 1y ago
Multimodal is the only image generation modality that matters going forward. Flux, HiDream, Stable Diffusion, and the like are going to be relegated to the past once multimodal becomes more common. Text-to-image sucks, and image-to-image with all the ControlNets and Comfy nodes is cumbersome in comparison to true multimodal instructiveness. I hope that we get an open weights multimodal image gen model. I'm slightly concerned that if these things take tens to hundreds of millions of dollars to train, that only Google and OpenAI will provide them. That said, the one weakness in multimodal models is that they don't let you structure the outputs yet. Multimodal + ControlNets would fix that, and that would be like literally painting with the mind. The future, when these models are deeply refined and perfected, is going to be wild.
- zaptrem 1y ago
  Good chance a future llama will output image tokens
  - echelon 1y ago
    That's my hope. That Llama or Qwen bring multimodal image generation capabilities to open source so we're not left in the dark. If that happens, then I'm sure we'll see slimmer multimodal models over the course of the next year or so. And that teams like Black Forest Labs will make more focused and performant multimodal variants. We need the incredible instructivity of multimodality. That's without question. But we also need to be able to fine tune, use ControlNets to guide diffusion, and to compose these into workflows.
avereveard 1y ago
It's a bit expensive/slow but for styled request I let it do the base image and when happy with the composition I ask to remake it as a picture or in whatever style needed.
troupo 1y ago
I also find it weird how it defaults/devolves into this overall brown-ish style. Once you see it, you see it everywhere
- flir 1y ago
  I've played around with "create an image based on this image" chains quite a lot, and yep, everything goes brown with 4o. You append the images to each other as a filmstrip and it's almost like a gradient. They also simplify over the generations (eg a basket full of stuff slowly loses the stuff), but I guess that's to be expected.
- jlarcombe 1y ago
  yes. it's absolutely horrible looking.

echelon 1y ago

Multimodal is the only image generation modality that matters going forward. Flux, HiDream, Stable Diffusion, and the like are going to be relegated to the past once multimodal becomes more common. Text-to-image sucks, and image-to-image with all the ControlNets and Comfy nodes is cumbersome in comparison to true multimodal instructiveness. I hope that we get an open weights multimodal image gen model. I'm slightly concerned that if these things take tens to hundreds of millions of dollars to train, that only Google and OpenAI will provide them. That said, the one weakness in multimodal models is that they don't let you structure the outputs yet. Multimodal + ControlNets would fix that, and that would be like literally painting with the mind. The future, when these models are deeply refined and perfected, is going to be wild.

zaptrem 1y ago
Good chance a future llama will output image tokens
- echelon 1y ago
  That's my hope. That Llama or Qwen bring multimodal image generation capabilities to open source so we're not left in the dark. If that happens, then I'm sure we'll see slimmer multimodal models over the course of the next year or so. And that teams like Black Forest Labs will make more focused and performant multimodal variants. We need the incredible instructivity of multimodality. That's without question. But we also need to be able to fine tune, use ControlNets to guide diffusion, and to compose these into workflows.

avereveard 1y ago

It's a bit expensive/slow but for styled request I let it do the base image and when happy with the composition I ask to remake it as a picture or in whatever style needed.

troupo 1y ago

I also find it weird how it defaults/devolves into this overall brown-ish style. Once you see it, you see it everywhere

flir 1y ago
I've played around with "create an image based on this image" chains quite a lot, and yep, everything goes brown with 4o. You append the images to each other as a filmstrip and it's almost like a gradient. They also simplify over the generations (eg a basket full of stuff slowly loses the stuff), but I guess that's to be expected.
jlarcombe 1y ago
yes. it's absolutely horrible looking.