4 ms·
> It’s not a mapping of words. It’s a mapping of character sequences. This is plainly wrong if you read the word2vec paper.
by dvt 3y ago
> It’s not a mapping of words. It’s a mapping of character sequences.
This is plainly wrong if you read the word2vec paper.
- jstanley 3y agoIt depends which tokenisation you use. Here's the token list for GPT-4: https://incoherency.co.uk/interest/gpt4-token-list.txt https://incoherency.co.uk/interest/gpt4-token-list.txt Some of them are words, some of them are not. It uses the same list even if you have read the word2vec paper. It probably doesn't even know whether you've read it.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]