4 ms·

This is unnecessarily harsh. Not every model has a publicly available tokenizer, and using a fallback like cl100k is usually a decent enough estimator from my e

This is unnecessarily harsh. Not every model has a publicly available tokenizer, and using a fallback like cl100k is usually a decent enough estimator from my experience. Besides, there's a warning message for when you specify a model without a known tokenizer. If you're upset with the implementation, you can always raise an issue or fix it yourself

refulgentis 2y ago
> This is unnecessarily harsh. Which part? All I can tease out from your comment are "the lies are impossible" (agreed!) and "close enough afaik". (it's not, the closest in the Big 5 has percent error of 32%, see end of comment. ex. GPT4o has a tokenizer with 2x the vocab so you'd expect ~1/2 the tokens) > Not every model has a publicly available tokenizer, Right. Ex. Claude 3s and Geminis. So why are Claude 3s and Geminis listed as supported models? > using a fallback CL100K isn't a fallback, its the only tokenizer. > like cl100k is usually a decent enough estimator from my experience. I'm very surprised to hear this, per stats demonstrating minimum error of 32%. > If you're upset with the implementation, you can always raise an issue I'm not "upset with the implementation", I'm sharing that the claims about being able to make financial calculations for 400 different LLMs is lying. > or fix it yourself How? As you pointed out, its unfixable for at least some subset of the ones they're claiming, ex. Gemini and Claude 3s. Let's pretend it was possible. Why? If someone puts out a library making wildly false claims, is th right thing to do to stay quiet and fix the library making false claims until its claims are true? > usually a decent enough estimator No, not for financial things certainly, which is the stated core purpose of the library. As promised, data: I picked the simplest example from my unit tests because you won't believe the divergence on larger ones. OpenAI (CL100K) - 18 in/1 out = 19. Gemini 1.5 - 41 in/14 out = 55. (65% error) Claude 3 - 21 in/4 out = 25. (24% error) Llama 3 - 23 in/5 out = 28. (32% error) Mistral - 10 in/3 out = 13. (46% error)
- weird-eye-issue 2y ago
  I can tell you've never actually built anything worthwhile
  - refulgentis 2y ago
    Lol. Drive by insult that's A) obviously wrong, and funnily enough, it's the attention to detail that got me there B) in service of caping for "33% error in financial calculations is actually fine"
    - weird-eye-issue 2y ago
      That's fine, different people have different definitions of worthwhile
      - refulgentis 2y ago
        I hope your day gets better!
- hansvm 2y ago
  This thread looks spicy, so I won't address most of it. On the "2x vocab == 1/2 the tokens" idea though: It usually doesn't pan out that way. Tokens aren't uniformly likely in normal text. They tend to follow some kind of a power law (like pdf(x) ~ x^0.4), and those 100k extra tokens, even assuming they're all available for use in purely textual inputs/outputs, will only move you from something like 11.4 bits of entropy per token to 12.1 (a 6% improvement). With that base idea in mind, how do we square that with your observations of large errors? It's a bit hard to know for certain since you didn't tell us which thing you're encoding, but: 1. Using an estimator, even if fairly precise and unbiased across all text, will have high variance for small inputs. If you actually need to estimate costs accurately for _each_ query (not just have roughly accurate costs summed across many queries or an accurate cost for a large query), this project is, as you pointed out, definitely not going to work. 2. Assuming your query distribution matches the tokenizer's training data in some sense, you would expect those errors to balance out over many queries (comparing total predicted costs to total actual costs) or over a single large query. That's still useful for a lot of people (e.g., to estimate the cost of running models across a large internal corpus). 3. Out-of-distribution queries are another interesting use-case where this project falls flat. IIRC somebody here on HN comments frequently about using LLMs for Hebrew text (specifically noting good performance with no tokenization, which is another fun avenue of research), and if any of the models included Hebrew-specific tokenization (I think code-specific tokenization is probably more likely in the big models, not that the specific example matters), you'll likely find that the model in question is much cheaper than the rest for those kinds of queries. There's no free lunch, and you'll necessarily also find pockets of other query types where that model's tokenizer is more expensive than the other tokenizers. This project doesn't have the ability to divine that sort of discrepancy. 4. Even being within 2x on costs (especially when we're talking about 3-4 orders of magnitude of discrepancy in the costs for the different models) is useful. It lets you accomplish things like figuring out roughly the best model you can afford for a kind of task. Separately: > This is unnecessarily harsh, I'm not "upset with the implementation", ... I think the problem people were noting was your tone. Saying you're uncomfortable with the methodology, that you won't use it, highlighting the error bars, pointing out pathological inputs, ..., are all potentially "interesting" to someone. Telling a person they should feel ashamed and are clueless is maybe appropriate somewhere sometimes, but it's strongly frowned upon at HN, usually shouldn't be done publicly in any setting, and is also a bit extreme for a project which is useful even in its current state. Cluelessness is an especially hard claim to justify when the project actually addresses your concern (just via a warning instead of a hard failure or something, which isn't your preferred behavior).

refulgentis 2y ago

> This is unnecessarily harsh. Which part? All I can tease out from your comment are "the lies are impossible" (agreed!) and "close enough afaik". (it's not, the closest in the Big 5 has percent error of 32%, see end of comment. ex. GPT4o has a tokenizer with 2x the vocab so you'd expect ~1/2 the tokens) > Not every model has a publicly available tokenizer, Right. Ex. Claude 3s and Geminis. So why are Claude 3s and Geminis listed as supported models? > using a fallback CL100K isn't a fallback, its the only tokenizer. > like cl100k is usually a decent enough estimator from my experience. I'm very surprised to hear this, per stats demonstrating minimum error of 32%. > If you're upset with the implementation, you can always raise an issue I'm not "upset with the implementation", I'm sharing that the claims about being able to make financial calculations for 400 different LLMs is lying. > or fix it yourself How? As you pointed out, its unfixable for at least some subset of the ones they're claiming, ex. Gemini and Claude 3s. Let's pretend it was possible. Why? If someone puts out a library making wildly false claims, is th right thing to do to stay quiet and fix the library making false claims until its claims are true? > usually a decent enough estimator No, not for financial things certainly, which is the stated core purpose of the library. As promised, data: I picked the simplest example from my unit tests because you won't believe the divergence on larger ones. OpenAI (CL100K) - 18 in/1 out = 19. Gemini 1.5 - 41 in/14 out = 55. (65% error) Claude 3 - 21 in/4 out = 25. (24% error) Llama 3 - 23 in/5 out = 28. (32% error) Mistral - 10 in/3 out = 13. (46% error)

weird-eye-issue 2y ago
I can tell you've never actually built anything worthwhile
- refulgentis 2y ago
  Lol. Drive by insult that's A) obviously wrong, and funnily enough, it's the attention to detail that got me there B) in service of caping for "33% error in financial calculations is actually fine"
  - weird-eye-issue 2y ago
    That's fine, different people have different definitions of worthwhile
    - refulgentis 2y ago
      I hope your day gets better!
hansvm 2y ago
This thread looks spicy, so I won't address most of it. On the "2x vocab == 1/2 the tokens" idea though: It usually doesn't pan out that way. Tokens aren't uniformly likely in normal text. They tend to follow some kind of a power law (like pdf(x) ~ x^0.4), and those 100k extra tokens, even assuming they're all available for use in purely textual inputs/outputs, will only move you from something like 11.4 bits of entropy per token to 12.1 (a 6% improvement). With that base idea in mind, how do we square that with your observations of large errors? It's a bit hard to know for certain since you didn't tell us which thing you're encoding, but: 1. Using an estimator, even if fairly precise and unbiased across all text, will have high variance for small inputs. If you actually need to estimate costs accurately for _each_ query (not just have roughly accurate costs summed across many queries or an accurate cost for a large query), this project is, as you pointed out, definitely not going to work. 2. Assuming your query distribution matches the tokenizer's training data in some sense, you would expect those errors to balance out over many queries (comparing total predicted costs to total actual costs) or over a single large query. That's still useful for a lot of people (e.g., to estimate the cost of running models across a large internal corpus). 3. Out-of-distribution queries are another interesting use-case where this project falls flat. IIRC somebody here on HN comments frequently about using LLMs for Hebrew text (specifically noting good performance with no tokenization, which is another fun avenue of research), and if any of the models included Hebrew-specific tokenization (I think code-specific tokenization is probably more likely in the big models, not that the specific example matters), you'll likely find that the model in question is much cheaper than the rest for those kinds of queries. There's no free lunch, and you'll necessarily also find pockets of other query types where that model's tokenizer is more expensive than the other tokenizers. This project doesn't have the ability to divine that sort of discrepancy. 4. Even being within 2x on costs (especially when we're talking about 3-4 orders of magnitude of discrepancy in the costs for the different models) is useful. It lets you accomplish things like figuring out roughly the best model you can afford for a kind of task. Separately: > This is unnecessarily harsh, I'm not "upset with the implementation", ... I think the problem people were noting was your tone. Saying you're uncomfortable with the methodology, that you won't use it, highlighting the error bars, pointing out pathological inputs, ..., are all potentially "interesting" to someone. Telling a person they should feel ashamed and are clueless is maybe appropriate somewhere sometimes, but it's strongly frowned upon at HN, usually shouldn't be done publicly in any setting, and is also a bit extreme for a project which is useful even in its current state. Cluelessness is an especially hard claim to justify when the project actually addresses your concern (just via a warning instead of a hard failure or something, which isn't your preferred behavior).