More like 256 of them, just an image is actually 3 layers with RGB. If you are going into combinations, I could easily say that triplets of tokens are actually

More like 256 of them, just an image is actually 3 layers with RGB. If you are going into combinations, I could easily say that triplets of tokens are actually what's important, and so 250k^3 is the real number of whatever. Actually now that I think about it, neural networks work with real numbers, so a pixel is just 3 numbers. Typical input for an image model would then be around 300x300x3 values. While an input for a language model is around 2000 tokens, but while each token is inputted as an integer into the model, mathematically it represents a 250k length vector, so mathematically the input is 250k x 2000 values. So 90k vs 500M. Also pixels next to each other in an image are related, so you can reduce model size by taking advantage of that (CNNs).