3 ms·
I find subword tokenisers like SentencePiece to be pretty great. It makes the sequence length smaller and you can get away with very, very small vocabularies.
by jszymborski 4y ago
I find subword tokenisers like SentencePiece to be pretty great. It makes the sequence length smaller and you can get away with very, very small vocabularies.