2 ms·
Isn’t this just exploiting the loss of letter-level resolution from tokenization?
by brookst 3y ago
Isn’t this just exploiting the loss of letter-level resolution from tokenization?
- HarHarVeryFunny 3y agoNo - makes no difference if you give the input string, and replacement rules, as sequences of space separated letters. FWIW, despite tokenized input, these models all seem capable of giving you the sequence of letters in a word if you ask.
- geor9e 3y agoHe addresses this in the tweet comments (it's the most common comment) and says it holds true even with byte (letter) level tokenization.