The reason it's not discussed much is that what goes on downstream of tokenization is extremely opaque. It's lots of layers of the transformer network so the ov

The reason it's not discussed much is that what goes on downstream of tokenization is extremely opaque. It's lots of layers of the transformer network so the overall structure is documented but what exactly those numbers mean is hard to figure out. There's an article here where the structure of an image generation network is explored a bit: https://openai.com/research/sparse-transformer https://openai.com/research/sparse-transformer They have a visualization of what the different layers are paying attention to. There are also some good explanations of transformers elsewhere online. This one is old but I found it helpful: http://jalammar.github.io/illustrated-transformer/ http://jalammar.github.io/illustrated-transformer/