2 ms·
> For instance, why not use whole words as tokens? Word-only tokenizers what people did in the RNN/LSTM days. There's no functional improvement over tokenizati
by minimaxir 2y ago
> For instance, why not use whole words as tokens?
Word-only tokenizers what people did in the RNN/LSTM days. There's no functional improvement over tokenization schemes like BPE or even WordPiece/SentencePiece, and it results in worse quality since you can't use meaningful semantic hints such as punctuation.
- neuroelectron 2y agoYou can encode semantic hints in the layers instead. Admittedly, this is more expensive which is kind to counter of the word-as-tokens idea.