3 ms·
How come the token to word ratio is smaller than 1 if tokens are either words or part of words? Shouldn't you expect more tokens than words?
by contravariant 3y ago
How come the token to word ratio is smaller than 1 if tokens are either words or part of words? Shouldn't you expect more tokens than words?
- yonixw 3y agoThat is how I understood it, a token is on average a 3/4 of a word. "Token to word". So if you want to buy 1000 tokens you would get effectively 750 words.
- deleted 3y ago[deleted]
- renewiltord 3y agoIt's the token to word multiplier, yeah. i.e. x tokens = 0.75x words.
- furyofantares 3y agoI think all the ratios given are x:1 and they tell you x.
- contravariant 3y agoThat would make it 0.75 tokens to 1 word right?
- furyofantares 3y agolol, yes, I'm glad they clarified because I understood it correctly then made the mistake GP did when I replied to them.
- qeternity 3y agoIt’s the other way around. 1 GPT4 token is equivalent to 50 GPT3.5 tokens. 1 token is equivalent to 0.75 words.
- waleedk 3y ago[Author] Fair point -- I clarified the language and gave a concrete example. Hope that helps!