3 ms·
you use the wrong word the Anthropic tokenizer is not worse, its more expensive/verbose
by not-a-llm 3mo ago
you use the wrong word
the Anthropic tokenizer is not worse, its more expensive/verbose
- nullsanity 3mo agoSo, worse? Because we benchmark off token use when talking about token use, and everyone else understood that.
- otikik 3mo agoIt’s better for them
- jascha_eng 3mo agoI mean it might lead to better performance on the model side. So the tokenizer is better but more expensive.
- marcosdumay 3mo agoThe most important feature of a tokenizer is dividing the inputs into independent values that the neural network can work with. It's not the size.
- nullsanity 3mo ago[dead]