3 ms·
can someone give an intuition for the unified tokenizer that they mentioned? I understand the idea of using a GAN or an autoencoder to make a image tokenizer, b
by chaos_emergent 2y ago
can someone give an intuition for the unified tokenizer that they mentioned? I understand the idea of using a GAN or an autoencoder to make a image tokenizer, but it's unclear how you combine that with text to have a single tokenizer over both modalities (if I'm understanding the paper correctly)
- chaos_emergent 2y agoactually I just went into the paper and skimmed it. Their image tokenizer creates image encodings in a discrete space of dimension 8192; each image consists of 1024 discrete tokens. The 8192 vector size matches their language tokenizer (discrete and also size 8192). Then they just mash those tokens together, which leads to downstream problems in training: because the two token spaces have different entropies, stable training in later steps is difficult, and they have to do a bunch of regularizations to make sure that gradients don't start exploding. Interesting stuff!