4 ms·
Huggingface tokenizers (e.g. BertTokenizerFast which can load the BERT model vocabularies) also can provide the offsets into the original text.
by PeterisP 5y ago
Huggingface tokenizers (e.g. BertTokenizerFast which can load the BERT model vocabularies) also can provide the offsets into the original text.