2 ms·
How consistent is the tokenization across different model families? It always served as a mental hangup for me when comparing LLM inference pricing.
by nisegami 1y ago
How consistent is the tokenization across different model families? It always served as a mental hangup for me when comparing LLM inference pricing.
- alexellman 1y agoThey all tokenize a little differently so they are not exactly 1-1. However I plan on addressing this by having each model complete a test task and getting the actual price from each api + token count to make a real 1-1 comparison.