3 ms·
What is a token? One character? Or is it a currency?
by strombofulous 6y ago
What is a token? One character? Or is it a currency?
- lt 6y agoA character, both input and output.
- throwaway287391 6y agoNot exactly, GPT-3 uses a variant of BPE [1], so one token can correspond to a character, an entire word or more, or anything in between. The paper [2] says a token corresponds to 0.7 words on average. [1] https://en.wikipedia.org/wiki/Byte_pair_encoding https://en.wikipedia.org/wiki/Byte_pair_encoding [2] https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165, page 24