2 ms·

I really appreciate your engagement here and think it has great value on a personal level, but the length and claims tend to hide two very obvious, straightforw

I really appreciate your engagement here and think it has great value on a personal level, but the length and claims tend to hide two very obvious, straightforward things that are hilariously bad to the point its unbelievable: 1. They only support GPT3.5 and GPT4.0. Note here: [1], and that gpt-4o would get swallowed into gpt-4-0613. 2. This will lead to massive, significant, embarrassingly large error in calculations. Tokenizers are not mostly the same, within 10% error. # Explicating #1, Responsive to ex. "It's not just C100K though. It is for a few models [0]". The link is to Tiktoken, OpenAI's tokenization library. There are literally more than GPT3.5 and GPT4.0 there, but they're just OpenAI's models, no one else's, none of the others in the long list in their documentation, and certainly not 400. Most damning? There's only 2 other tokenizers, long deprecated, used only for deprecated models not served anymore, thus you're not calculating costs with them. The only live ones are c100k and o200k. As described above, and shown in [1], their own code kneecaps the o200k and will use c100k # Explicating #2 Let me know what you'd want to see if you're curious about the 30%+ error thing. I don't want to guess at a test suite that would make you confident you need to revise a prior that there's only +/- 10% difference between arbitrary tokenizers. For context, I run about 20 unit tests, for each of the big 5 providers, with the same prompts, to capture their input and output token counts to make sure I'm billing accurately. # Conclusion Just to save you time, I think the best way I can provide some value is token count experiments demonstrating error. You won't be able to talk me down to "eh, lets just say its +/- 10%, thats good enough for most people!" --- It matters, if it didn't, they'd explicate at least some of this. Instead, its "tokenization for 400 LLMs!" [1] https://github.com/AgentOps-AI/tokencost/blob/e1d52dbaa3ada232aa68dabf5b58662da4bc2363/tokencost/costs.py#L46\\\\\\ https://github.com/AgentOps-AI/tokencost/blob/e1d52dbaa3ada2...*

hansvm 2y ago
Oh I see (tiktoken). That's my mistake. I naively assumed the only good reason to pull in a 3rd party lib like that is if it actually did a reasonable amount of work. > curious about the 30%+ error thing I'm mildly curious. I have no doubt that small strings will often have high relative errors. I'd be surprised though if sum(estimated)/sum(actual) were very large if you copied in either a large piece of text or many small pieces of text, outside of specialized domains out of the normal scope of that tokenizer (e.g., throwing latex code into something trained just on wikipedia). That's more for entropic reasons than anything else. The only way that's true is if (1) some of these tokenizers are much less naive than the normal LLM literature and actually approach entropic bounds, or (2) the baseline implementations are especially bad so that there's a lot of headroom for improvements. What happens when you throw in something like a medium-sized plain-text wikipedia article (say, the first half as input and the second as output)? > messaging -- tokenization for 400 LLMs Alright, I'm sold. I'm still partial to Hanlon's razor for these sort of things, but that ought to be patched.

hansvm 2y ago

Oh I see (tiktoken). That's my mistake. I naively assumed the only good reason to pull in a 3rd party lib like that is if it actually did a reasonable amount of work. > curious about the 30%+ error thing I'm mildly curious. I have no doubt that small strings will often have high relative errors. I'd be surprised though if sum(estimated)/sum(actual) were very large if you copied in either a large piece of text or many small pieces of text, outside of specialized domains out of the normal scope of that tokenizer (e.g., throwing latex code into something trained just on wikipedia). That's more for entropic reasons than anything else. The only way that's true is if (1) some of these tokenizers are much less naive than the normal LLM literature and actually approach entropic bounds, or (2) the baseline implementations are especially bad so that there's a lot of headroom for improvements. What happens when you throw in something like a medium-sized plain-text wikipedia article (say, the first half as input and the second as output)? > messaging -- tokenization for 400 LLMs Alright, I'm sold. I'm still partial to Hanlon's razor for these sort of things, but that ought to be patched.