2 ms·
Is this tokenization basically the same process as what was done in word2vec?
by bondarchuk 3y ago
Is this tokenization basically the same process as what was done in word2vec?
- ftxbro 3y agoNo. This tokenization is huffman encoding and word2vec is embedding in cartesian space. They are very different.
- bondarchuk 3y agoAhh. Thanks for the clarification!