4 ms·
Could someone clarify what exactly are those tokens ?
by dgellow 6y ago
Could someone clarify what exactly are those tokens ?
- winchester6788 6y agoLength of given text after encoding it with their BPE. In general, you can expect it to be 1.2x-2x the len(words) in your text.
- minimaxir 6y agoThe Pricing FAQ notes that it's, on average, 4 characters, which IMO is fair. Super common punctuation is its own token, however.