3 ms·

Has anyone ever tried a GPT trained on, say, 256 tokens representing bytes in a byte stream or even more simply binary digits? I imagine there are efficiency t

by api 3y ago

Has anyone ever tried a GPT trained on, say, 256 tokens representing bytes in a byte stream or even more simply binary digits? I imagine there are efficiency trade-offs but I just wonder if it works at all.

sandinmyjoints 3y ago
Not a GPT, but I think Megabyte does that.
sebzim4500 3y ago
I'm sure it would work, but there are obvious downsides (slower and less history) with few upsides (simpler, no glitch tokens)
PeterisP 3y ago
Sure, the concept has been explored; for example see the classic 2015 Karpathy's http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/ as a cool description of a character-level model. IIRC the early papers on subword tokenization also sometimes included explicit comparisons with character-level models, but people don't do it nowadays because there's a clear consensus on the expected outcome - yes, it works, but it's simply worse. Technically it's the exact outcome that you get if you put in a vocabulary size of 256 (and do tokenization on byte-level, not unicode), so it's just an extreme case of vocabulary size choice, and there's enough research on how vocabulary size affects stuff to assume that 256 is not an optimal size. You can do it for exploring capabilities though - see "Bytes is all you need" https://news.ycombinator.com/item?id=36176756 https://news.ycombinator.com/item?id=36176756 discussion on trying to abstract away complex file formats by just passing the bytes of the file to the neural network directly - again, it obviously works worse, but it kind of works.
belladoreai 3y ago
Tangentially related to what you ask: LLaMA tokenizer has fallback to byte-level tokens.
sanxiyn 3y ago
Yes, "ByT5: Towards a token-free future with pre-trained byte-to-byte models" for example. https://arxiv.org/abs/2105.13626 https://arxiv.org/abs/2105.13626

by api 3y ago

sandinmyjoints 3y ago

Not a GPT, but I think Megabyte does that.

sebzim4500 3y ago

I'm sure it would work, but there are obvious downsides (slower and less history) with few upsides (simpler, no glitch tokens)

PeterisP 3y ago

Sure, the concept has been explored; for example see the classic 2015 Karpathy's http://karpathy.github.io/2015/05/21/rnn-effectiveness/ http://karpathy.github.io/2015/05/21/rnn-effectiveness/ as a cool description of a character-level model. IIRC the early papers on subword tokenization also sometimes included explicit comparisons with character-level models, but people don't do it nowadays because there's a clear consensus on the expected outcome - yes, it works, but it's simply worse. Technically it's the exact outcome that you get if you put in a vocabulary size of 256 (and do tokenization on byte-level, not unicode), so it's just an extreme case of vocabulary size choice, and there's enough research on how vocabulary size affects stuff to assume that 256 is not an optimal size. You can do it for exploring capabilities though - see "Bytes is all you need" https://news.ycombinator.com/item?id=36176756 https://news.ycombinator.com/item?id=36176756 discussion on trying to abstract away complex file formats by just passing the bytes of the file to the neural network directly - again, it obviously works worse, but it kind of works.

belladoreai 3y ago

Tangentially related to what you ask: LLaMA tokenizer has fallback to byte-level tokens.

sanxiyn 3y ago

Yes, "ByT5: Towards a token-free future with pre-trained byte-to-byte models" for example. https://arxiv.org/abs/2105.13626 https://arxiv.org/abs/2105.13626