4 ms·
A token already has multiple meanings because words (and part-of-words) can have multiple meanings.
by kkzz99 2y ago
A token already has multiple meanings because words (and part-of-words) can have multiple meanings.
- KTibow 2y agoSure, there is some of the same problem with current tokenizers. However, I think this would increase it from "some tokens aren't words" to "(almost) all tokens aren't words". Correct me if I'm missing something though.