This is exactly why tokenizers should provide offsets into the original text! The Spacy tokenizer provides this, though the original wordpiece tokenizer provid

This is exactly why tokenizers should provide offsets into the original text! The Spacy tokenizer provides this, though the original wordpiece tokenizer provided for BERT does not. It is relatively easy to add offset information to that tokenizer, though. After you have start and end offsets for each token, it's not hard to just align tokens by offset overlap.