3 ms·
I am quite sure GPT can undo tokenization if needed. It has no problem handling misspellings, plus you can just ask it to "spell things" L-I-K-E S-O.
by H8crilA 3y ago
I am quite sure GPT can undo tokenization if needed. It has no problem handling misspellings, plus you can just ask it to "spell things" L-I-K-E S-O.
- RugnirViking 3y agoI'm pretty sure there is a spellchecker ran before the tokenisation also it is often wrong with spelling stuff out like that, especially with rarer or more interesting tokens