3 ms·
So, worse? Because we benchmark off token use when talking about token use, and everyone else understood that.
by nullsanity 3mo ago
So, worse? Because we benchmark off token use when talking about token use, and everyone else understood that.
- otikik 3mo agoIt’s better for them
- jascha_eng 3mo agoI mean it might lead to better performance on the model side. So the tokenizer is better but more expensive.
- marcosdumay 2mo agoThe most important feature of a tokenizer is dividing the inputs into independent values that the neural network can work with. It's not the size.
- nullsanity 2mo ago[dead]