3 ms·

What do you mean LLMs are blind? All frontier models are multimodal, which means they literally consume images as tokens. They can “see” exactly as well as they

What do you mean LLMs are blind? All frontier models are multimodal, which means they literally consume images as tokens. They can “see” exactly as well as they can “read”. Also, GPT-Image-2 is not a diffusion model, it is based on Transformers, like other LLMs are.

slashdave 5mo ago
Tokens are not a substitute for a numerical measurement. Ask a LLM how much time has passed. Watch it hallucinate wildly. Has anyone noticed that Opus has trouble building ascii diagrams (often leaves out spaces so lines are misaligned)?
- semiquaver 5mo ago
  Ok? Your comment is in no way responsive to anything I said.
- arjie 5mo ago
  LLMs are just one mechanical component. One might as well say "Ask your println how much time has passed". That is not a question that makes sense. As an example, I did not construct my agent specifically to answer your question and when I saw your question I queried the agent. And it is correct. https://imgur.com/a/j8j7hL9 https://imgur.com/a/j8j7hL9 As semiquaver said, modern LLMs are multi-modal, they can reason in image-space and audio-space as well as in text-space. It is not a translate then operate kind of situation. Claude Design is not a raw LLM, nor an instruction-tuned LLM. It is an agent harness around an LLM that allows it to do certain things.
embedding-shape 5mo ago
I guess they do "see" but more like "see an explanation of the image", not "see" as in experience visually. They're really bad at details and perfection when it comes to images, and doesn't understand things like visual hierarchy, affordances and other fundamental design concepts. Most of them are able to describe those things with letters, but doesn't seem to actually fundamentally grasp it when asking it to do UIs even when mentioning these things. Try doing 100% vibe-coding with an agent and loosely specify what kind of application you want, and observe how the resulting UI and UX is a complete mess, unless you specify exactly how the UI and UX should work in practice. If they actually had spatial understanding, together with being able to visually experience images, then they'd probably be able to build proper UI/UX from the get go, but since they only could describe what those things are, you end up with the messes even the current SOTAs produce.
- marcus_holmes 5mo ago
  This is my experience too, but with all other aspects of the application. If you only loosely describe it, it comes out as a mess. You have to know what you're building to get the LLM to actually build something decent. I don't think this is purely a visual or design constraint.
  - embedding-shape 5mo ago
    When I'm using agents for programming, I can have a AGENTS.md outlining exactly what requirements, guidelines and constraints all the code need to follow, and the agent (codex in my case) will pretty much nail that. I've tried doing the same for design work, just really outlining exactly how the UI and UX needs to look and work, but for some reason it struggles a whole bunch with it, regardless of how clear I am. Maybe it's I'm just worse at explaining and describing what UI and UX I'm actually after though, I suppose.
    - marcus_holmes 5mo ago
      I once worked at a startup where the CEO was originally a designer. He once spent two days huddled with the main designer for the product, trying to pick exactly the right font for the product. I have no idea how you'd have that kind of discussion with an LLM. But then, I would not spend more than five minutes on this decision, so I'm probably the wrong audience for this ;)
      - embedding-shape 5mo ago
        Used to work in a designer-heavy company doing frontend work, one of the founders could spot by naked eye if you got the alignment of something wrong by 2-3 pixels during the reviews. The UI and UX of the product was amazing, and took some time to get used to actually delivering pixel-perfect designs across three different browsers, but fun times regardless :) Probably takes a certain individual to enjoy that sort of experience though.
- spongebobstoes 5mo ago
bombcar 5mo ago
Claude has been kicking ass at code, but I asked it to “sketch” a second floor with a stairway and bedrooms with large closets and it made … something that resembles something akin to not at all what I asked.
embedding-shape 5mo ago
> Also, GPT-Image-2 is not a diffusion model, it is based on Transformers, like other LLMs are. Where are you getting this from btw? AFAIK, OpenAI hasn't openly talked about what exactly is powering the Images 2.0 stuff, unless I missed something? I think they've said it's not a diffusion model, but I'm not sure they've said what they're doing instead, have they?
- simonw 5mo ago
  I believe it's an evolution of the technique used in GPT-Image-1 (or whatever they called that), which was derived from their work on making GPT-4o an "omni" model that can directly output images and audio in addition to text. The 2024 GPT-4o launch post https://openai.com/index/hello-gpt-4o/ https://openai.com/index/hello-gpt-4o/ hints about how that works: "With GPT‑4o, we trained a single new model end-to-end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network."
  - embedding-shape 5mo ago
    Yeah, that's my belief as well, but haven't seen any concrete explanations about how it works, just the marketing/press releases sadly.

slashdave 5mo ago

Tokens are not a substitute for a numerical measurement. Ask a LLM how much time has passed. Watch it hallucinate wildly. Has anyone noticed that Opus has trouble building ascii diagrams (often leaves out spaces so lines are misaligned)?

semiquaver 5mo ago
Ok? Your comment is in no way responsive to anything I said.
arjie 5mo ago
LLMs are just one mechanical component. One might as well say "Ask your println how much time has passed". That is not a question that makes sense. As an example, I did not construct my agent specifically to answer your question and when I saw your question I queried the agent. And it is correct. https://imgur.com/a/j8j7hL9 https://imgur.com/a/j8j7hL9 As semiquaver said, modern LLMs are multi-modal, they can reason in image-space and audio-space as well as in text-space. It is not a translate then operate kind of situation. Claude Design is not a raw LLM, nor an instruction-tuned LLM. It is an agent harness around an LLM that allows it to do certain things.

embedding-shape 5mo ago

I guess they do "see" but more like "see an explanation of the image", not "see" as in experience visually. They're really bad at details and perfection when it comes to images, and doesn't understand things like visual hierarchy, affordances and other fundamental design concepts. Most of them are able to describe those things with letters, but doesn't seem to actually fundamentally grasp it when asking it to do UIs even when mentioning these things. Try doing 100% vibe-coding with an agent and loosely specify what kind of application you want, and observe how the resulting UI and UX is a complete mess, unless you specify exactly how the UI and UX should work in practice. If they actually had spatial understanding, together with being able to visually experience images, then they'd probably be able to build proper UI/UX from the get go, but since they only could describe what those things are, you end up with the messes even the current SOTAs produce.

marcus_holmes 5mo ago
This is my experience too, but with all other aspects of the application. If you only loosely describe it, it comes out as a mess. You have to know what you're building to get the LLM to actually build something decent. I don't think this is purely a visual or design constraint.
- embedding-shape 5mo ago
  When I'm using agents for programming, I can have a AGENTS.md outlining exactly what requirements, guidelines and constraints all the code need to follow, and the agent (codex in my case) will pretty much nail that. I've tried doing the same for design work, just really outlining exactly how the UI and UX needs to look and work, but for some reason it struggles a whole bunch with it, regardless of how clear I am. Maybe it's I'm just worse at explaining and describing what UI and UX I'm actually after though, I suppose.
  - marcus_holmes 5mo ago
    I once worked at a startup where the CEO was originally a designer. He once spent two days huddled with the main designer for the product, trying to pick exactly the right font for the product. I have no idea how you'd have that kind of discussion with an LLM. But then, I would not spend more than five minutes on this decision, so I'm probably the wrong audience for this ;)
    - embedding-shape 5mo ago
      Used to work in a designer-heavy company doing frontend work, one of the founders could spot by naked eye if you got the alignment of something wrong by 2-3 pixels during the reviews. The UI and UX of the product was amazing, and took some time to get used to actually delivering pixel-perfect designs across three different browsers, but fun times regardless :) Probably takes a certain individual to enjoy that sort of experience though.
spongebobstoes 5mo ago

bombcar 5mo ago

Claude has been kicking ass at code, but I asked it to “sketch” a second floor with a stairway and bedrooms with large closets and it made … something that resembles something akin to not at all what I asked.

embedding-shape 5mo ago

> Also, GPT-Image-2 is not a diffusion model, it is based on Transformers, like other LLMs are. Where are you getting this from btw? AFAIK, OpenAI hasn't openly talked about what exactly is powering the Images 2.0 stuff, unless I missed something? I think they've said it's not a diffusion model, but I'm not sure they've said what they're doing instead, have they?

simonw 5mo ago
I believe it's an evolution of the technique used in GPT-Image-1 (or whatever they called that), which was derived from their work on making GPT-4o an "omni" model that can directly output images and audio in addition to text. The 2024 GPT-4o launch post https://openai.com/index/hello-gpt-4o/ https://openai.com/index/hello-gpt-4o/ hints about how that works: "With GPT‑4o, we trained a single new model end-to-end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network."
- embedding-shape 5mo ago
  Yeah, that's my belief as well, but haven't seen any concrete explanations about how it works, just the marketing/press releases sadly.