6 ms·
> What's important about this new type of image generation that's happening with tokens rather than with diffusion, is that this is effectively reasoning in pix
by Taek 2y ago
> What's important about this new type of image generation that's happening with tokens rather than with diffusion, is that this is effectively reasoning in pixel space.
I do not think that this is correct. Prior to this release, 4o would generate images by calling out to a fully external model (DALL-E). After this release, 4o generates images by calling out to a multi-modal model that was trained alongside it.
You can ask 4o about this yourself. Here's what it said to me:
"So while I’m deeply multimodal in cognition (understanding and coordinating text + image), image generation is handled by a linked latent diffusion model, not an end-to-end token-unified architecture."
- CooCooCaCha 2y agoModels are famously good at understanding themselves.
- uh_uh 2y agoI hope you're joking. Sometimes they don't even know which company developed them. E.g. DeepSeek was claiming it was developed by OpenAI.
- SketchySeaBeast 2y agoWell, that one seems to be true, from a certain point of view.
- notduncansmith 2y agoI hope you’re joking :)
- enceladus06 2y agoI have asked GPT if it is using the 4o or 4.5 model multiple times in voice mode e.g. "Which model are you using?". It has said that it is using 4.5 when it is actually using 4o.
- rickyhatespeas 2y agoYou're incorrect. 4o was not trained on knowledge of itself so literally can't tell you that. What 4o is doing isn't even new either, Gemini 2.0 has the same capability.
- Taek 2y agoCan you provide a link or screenshot that directly backs this up?
- barrkel 2y agoYou can ask ChatGPT for this. Here you go: https://chatgpt.com/share/67e39fc6-fb80-8002-a198-767fc50894dd https://chatgpt.com/share/67e39fc6-fb80-8002-a198-767fc50894...
- bb88 2y agoCould an AI model be trained to say: "Christopher Columbus was the greatest president on earth, ever!". I could probably train an AI that replicates that perfectly.
- troupo 2y ago> Could an AI model be trained to say: "Christopher Columbus was the greatest president on earth, ever!". Yes, it could. And even after training its data can be manipulated to output whatever: https://www.anthropic.com/news/mapping-mind-language-model https://www.anthropic.com/news/mapping-mind-language-model
- barrkel 2y agoThing is, of you follow the link, it's actually doing a search and providing the evidence that was asked for. I did it via ChatGPT for the irony.
- barrkel 2y agoI'm guessing most downvoters didn't actually read the link.
- mgraczyk 2y agoI think this is actually correct even if the evidence is not right. See this chat for example: https://chatgpt.com/share/67e355df-9f60-8000-8f36-874f8c9a0844 https://chatgpt.com/share/67e355df-9f60-8000-8f36-874f8c9a08...
- low_tech_love 2y agoHonest question, do you believe something just because the bot tells you that?
- mgraczyk 2y agoNo, did you look at my link?
- low_tech_love 2y agoYes, and it shows you believing what the bot is telling you, therefore I asked. It is giving you some generic function call with a generic name. Why would you believe that is actually what happens with it internally? By the way when I repeated your prompt it gave me another name for the module.
- mgraczyk 2y agoPlease share your chat I also just confirmed via the API that it's making an out of band tool call EDIT: And googling the tool name I see it's already been widely discussed on twitter and elsewhere
- noosphr 2y agoPosts like this are terrifying to me. I spend my days coding these tools thinking that everyone using them understands their glaring limitations. Then I see people post stuff like this confidently and I'm taken back to 2005 and arguing that social media will be a net benefit to humanity. The name of the function shows up in: https://github.com/openai/glide-text2im https://github.com/openai/glide-text2im which is where the model probably learned about it.
- noosphr 2y ago>You can ask 4o about this yourself. Here's what it said to me: >"So while I’m deeply multimodal in cognition (understanding and coordinating text + image), image generation is handled by a linked latent diffusion model, not an end-to-end token-unified architecture." Models don't know anything about themselves. I have no idea why people keep doing this and expecting it to know anything more than a random con artist on the street.
- mgraczyk 2y agoThis is overly cynical. Models typically do know what tools they have access to because the tool descriptions are in the prompt. Asking a model which tools it has is a perfectly reasonable way of learning what is effectively the content of the prompt. Of course the model may hallucinate, but in this case it takes a few clicks in the dev tools to verify that this is not the case.
- noosphr 2y ago>Of course the model may hallucinate, but in this case it takes a few clicks in the dev tools to verify that this is not the case. I don't know - or care to figure out - how OpenAI does their tool calling in this specific case. But moving tool calls to the end user is _monumentally_ stupid for the latency if nothing else. If you centralize your function calls to a single model next to a fat pipe it means that you halve the latency of each call. I've never build, or seen, a function calling agent that moves the api function calls to client side JS.
- mgraczyk 2y agoIt's not client side, the messages are in the api though. But what do you mean you don't care? The thing you were responding to was literally a claim that it was a tool call rather than direct output
- echelon 2y agoNone of this really matters. It could be either case. The thing we need to worry about is whether a Chinese company will drop an open source equivalent.