3 ms·
I believe GPT-3 uses byte pair encoding, which allows it to do tokenization in a language-neutral manner: https://en.wikipedia.org/wiki/Byte_pair_encoding http
by andrewmutz 4y ago
I believe GPT-3 uses byte pair encoding, which allows it to do tokenization in a language-neutral manner:
https://en.wikipedia.org/wiki/Byte_pair_encoding https://en.wikipedia.org/wiki/Byte_pair_encoding
- mrfusion 4y agoI thought I read it uses word2vec?
- axiom92 4y agoYeah it's BPE. OpenAI has a nice tool that allows you to play with the tokenizer https://beta.openai.com/tokenizer https://beta.openai.com/tokenizer.
- minimaxir 4y agoAdditionally, the tokenizer vocabulary is unchanged from GPT-2. You can use HuggingFace's GPT-2 tokenizer as well. (some of OpenAI's GPT-3 notebooks do just that).