3 ms·

I had a reasonable intuition for how the "old" method works, but I still don't grok this new approach. "in multimodal image generation, images are created in t

I had a reasonable intuition for how the "old" method works, but I still don't grok this new approach. "in multimodal image generation, images are created in the same way that LLMs create text, a token at a time" Is there some way to visualise these "image tokens", in the same way I can view tokenized text?

fxtentacle 2y ago
Imagine you cut the image into 32x32 pixel blocks. And then for each block, you can chose 1 out of 128,000 variations. And then a post-processing step smoothes out the borders between blocks and adjusts small details. That's basically how a transformer image generation model works. As such, the process is remarkably similar to old fixed-font ASCII art. It's just that modern AIs have a larger alphabet and, thus, more character shapes to choose from.
- rwmj 2y ago
  I don't get how this would produce consistent images. In the article, the text could be on a grid, but the window and doorway and sofa don't seem to be grid-aligned. (Or maybe the text is overlaid?)
  - danielbln 2y ago
    The model looks ahead, just like LLMs look ahead. An LLM outputs token by token but can still output a fully coherent and consistent story for example. This new crop of auto-regressive image models does the same.
WhiteNoiz3 2y ago
I haven't see any details on how OpenAI's model works, but the tokens it generates aren't directly translated into pixels - those tokens are probably fed into a diffusion process which generates the actual image.. The tokens are the latent space or conditioning for the actual image generation process.
- bonoboTP 2y ago
  > I haven't see any details on how OpenAI's model works Exactly. People just confidently make things up. There are many possible ways, and without details, "native generation" is just a marketing buzzword without clear definition. It's a proprietary system, there is no code release, there is no publication. We simply don't know how exactly it's done.
  - famouswaffles 2y ago
    Open AI have both said it's native image generation and autoregressive. It has the signs of it too. It's probably an implementation of VAR (https://arxiv.org/abs/2404.02905 https://arxiv.org/abs/2404.02905) - autoregressive image generation with a small twist. Rather than predict every token at the target resolution directly, start with predicting it at a small resolution, cranking it higher and higher until the desired resolution.

fxtentacle 2y ago

Imagine you cut the image into 32x32 pixel blocks. And then for each block, you can chose 1 out of 128,000 variations. And then a post-processing step smoothes out the borders between blocks and adjusts small details. That's basically how a transformer image generation model works. As such, the process is remarkably similar to old fixed-font ASCII art. It's just that modern AIs have a larger alphabet and, thus, more character shapes to choose from.

rwmj 2y ago
I don't get how this would produce consistent images. In the article, the text could be on a grid, but the window and doorway and sofa don't seem to be grid-aligned. (Or maybe the text is overlaid?)
- danielbln 2y ago
  The model looks ahead, just like LLMs look ahead. An LLM outputs token by token but can still output a fully coherent and consistent story for example. This new crop of auto-regressive image models does the same.

WhiteNoiz3 2y ago

I haven't see any details on how OpenAI's model works, but the tokens it generates aren't directly translated into pixels - those tokens are probably fed into a diffusion process which generates the actual image.. The tokens are the latent space or conditioning for the actual image generation process.

bonoboTP 2y ago
> I haven't see any details on how OpenAI's model works Exactly. People just confidently make things up. There are many possible ways, and without details, "native generation" is just a marketing buzzword without clear definition. It's a proprietary system, there is no code release, there is no publication. We simply don't know how exactly it's done.
- famouswaffles 2y ago
  Open AI have both said it's native image generation and autoregressive. It has the signs of it too. It's probably an implementation of VAR (https://arxiv.org/abs/2404.02905 https://arxiv.org/abs/2404.02905) - autoregressive image generation with a small twist. Rather than predict every token at the target resolution directly, start with predicting it at a small resolution, cranking it higher and higher until the desired resolution.