4 ms·
Can you explain what tokens are in this context? Edit: and better yet, is there a good resource for learning the vernacular in general? Should I just read some
by triyambakam 4y ago
Can you explain what tokens are in this context?
Edit: and better yet, is there a good resource for learning the vernacular in general? Should I just read something like "Dive into Deep Learning"?
- deleted 4y ago[deleted]
- aitball 4y agono the language model decides what a token is
- jncraton 4y agoIf an example would be helpful, OpenAI's tokenizer is publicly usable on their website: https://platform.openai.com/tokenizer https://platform.openai.com/tokenizer You can drop sample text in there and visually see how it is split into tokens. The GPT2/3 tokenizer uses about 50k unique tokens that were learned to be an efficient representation of the training data.