4 ms·
Would an LLM predicting the next bit rather than the next token have any advantages over GPT? In some ways it seems more general - for example it can predict an
by fancyfredbot 4y ago
Would an LLM predicting the next bit rather than the next token have any advantages over GPT? In some ways it seems more general - for example it can predict an image or even executable machine code. I'm guessing that it's more difficult to train however?
- deleted 4y ago[deleted]
- thethimble 4y agoGPT’s vocab size is north of tens of thousands. Included in this vocab are the 256 unique byte values (2^8). This means GPT can already generate/interpret all byte sequences. The vocab size is larger than 256 (or in your example 2) because some sequences of bytes occur much more frequently than others. This allows the model to generalize more efficiently. By analogy consider that we think of “words” as atomic concepts much more than we think of the individual lines that make up the letters in a word - even though the former can be thought of being composed of the latter.