4 ms·
The pallette for LLM is tokens not characters.
by hgsgm 4y ago
The pallette for LLM is tokens not characters.
- Der_Einzige 4y agoDepends on the LLM. Character based LLMs exist, and even have advantages vs regular LLMs...
- dragonwriter 4y agoAnd if it is characters (as it is for some models), its more than 26 of them for English. Between space, case, punctuation, and digits, its basically 7-bit ASCII without most of the control characters (newline is semantically important, the rest not), almost 100 characters.
- kelipso 4y agoThe number of tokens is more in the range of 40k to 250k tokens.
- jameshart 4y agoRight. Whereas for pixels there are over 16 million. How many more than 250k is that? Over 16 million.
- kelipso 4y agoMore like 256 of them, just an image is actually 3 layers with RGB. If you are going into combinations, I could easily say that triplets of tokens are actually what's important, and so 250k^3 is the real number of whatever. Actually now that I think about it, neural networks work with real numbers, so a pixel is just 3 numbers. Typical input for an image model would then be around 300x300x3 values. While an input for a language model is around 2000 tokens, but while each token is inputted as an integer into the model, mathematically it represents a 250k length vector, so mathematically the input is 250k x 2000 values. So 90k vs 500M. Also pixels next to each other in an image are related, so you can reduce model size by taking advantage of that (CNNs).