3 ms·

Thanks, that's really interesting. Do they correct for spelling mistakes or internationalised spellings? For example, does `colour` and `color` end up in the sa

Thanks, that's really interesting. Do they correct for spelling mistakes or internationalised spellings? For example, does `colour` and `color` end up in the same token stream?

lyu07282 11mo ago
No it just looks at exact character sequences, try it out yourself here: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer