3 ms·
So, worse? Because we benchmark off token use when talking about token use, and everyone else understood that.
by nullsanity 4mo ago
So, worse? Because we benchmark off token use when talking about token use, and everyone else understood that.
- otikik 4mo agoIt’s better for them
- jascha_eng 4mo agoI mean it might lead to better performance on the model side. So the tokenizer is better but more expensive.
- marcosdumay 4mo agoThe most important feature of a tokenizer is dividing the inputs into independent values that the neural network can work with. It's not the size.
- nullsanity 4mo ago[dead]