4 ms·
How to calculate the alignment between BERT and spaCy tokens
- cochne 5y agoThis is exactly why tokenizers should provide offsets into the original text! The Spacy tokenizer provides this, though the original wordpiece tokenizer provided for BERT does not. It is relatively easy to add offset information to that tokenizer, though. After you have start and end offsets for each token, it's not hard to just align tokens by offset overlap.
- PeterisP 5y agoHuggingface tokenizers (e.g. BertTokenizerFast which can load the BERT model vocabularies) also can provide the offsets into the original text.
- occupy_paul_st 5y agoClicked on this thinking it was some kind of crazy decentralized crypto algorithm... yeesh, I gotta spend less time on the blockchain!!
- tracyhenry 5y agoCan someone explain how BERT and spaCy are typically combined in practice for a custom NER task?
- tastroder 5y agoThese days, at least in non-research contexts, people seem to mostly use spacy-transformers for that, already contains all the necessary glue code between huggingface transformers (including BERT) and spacy. https://explosion.ai/blog/spacy-transformers https://explosion.ai/blog/spacy-transformers / https://spacy.io/universe/project/spacy-transformers https://spacy.io/universe/project/spacy-transformers
- fouc 5y agoDoes anyone know how those beautiful "terminal" images were generated? Or recommend some resources for how to make similar images? Or how to actually configure iterm2/vim/etc to that point?
- polm23 5y agohttps://carbon.now.sh/ https://carbon.now.sh/