4 ms·
I've noticed that it correctly splits warm|est, cold|est, bleak|est, but darkest is a single token. I've also seen it group `?"`, `."`, `!"`, and `.--` into si
by rhdunn 4y ago
I've noticed that it correctly splits warm|est, cold|est, bleak|est, but darkest is a single token.
I've also seen it group `?"`, `."`, `!"`, and `.--` into single tokens.
It also splits some words like "Elton" as El|ton. Presumably in that case it has mis-idetified a -ton prefix.