2 ms·
SentencePiece is not a different algorithm to WordPiece or BPE, despite its naming. One of the main pulls of the SentencePiece library was the pre-tokenization
by zwaps 2y ago
SentencePiece is not a different algorithm to WordPiece or BPE, despite its naming.
One of the main pulls of the SentencePiece library was the pre-tokenization being less reliant on white space and therefore more adaptable to non Western languages.