yuvalpinter

3 ms·

1. 1.▲
   by yuvalpinter 1y ago

   We have a paper under review that's gonna be up on arXiv soon, where we test this for ~10,000 words and find consistent decline in counting ability based on how many characters are in the tokens where the target character appears . It
2. 2.▲
   by yuvalpinter 1y ago

   A few years ago we released SaGe, which is a contextual tokenizer, meaning that it builds a vocab that's fit for an LM use case because tokens are selected to appear within as clear a context set as possible in a corpus. https:&#x2F
3. 3.▲
   by yuvalpinter 1y ago

   This can also be done within the tokenization framework, see our work here: https://arxiv.org/abs/2504.00178
4. 4.▲
   by yuvalpinter 1y ago

   We've been working on this problem for quite some time in my lab. We released a benchmark piling together several "intrinsic evaluations" that don't require model training. We're currently investigating correlations