3 ms·
Do we know exactly what the GPT3 tokenizer is? It could be combining multiple words together, for example common bigrams. Never mind, it looks like they have a
by janalsncm 4y ago
Do we know exactly what the GPT3 tokenizer is? It could be combining multiple words together, for example common bigrams.
Never mind, it looks like they have a tokenizer tool online and every bigram I’ve given it BPEs to multiple tokens:
https://beta.openai.com/tokenizer https://beta.openai.com/tokenizer