The ViT paper has the details: project 16x16 groups of pixels through a learned embedding, combine with positional encoding, and feed to an attention layer. It

The ViT paper has the details: project 16x16 groups of pixels through a learned embedding, combine with positional encoding, and feed to an attention layer. It's delightful that this is practically identical to the NLP architecture with only the tiniest adaptive tweak!