4 ms·
The tokenizer appears to be the original GPT-2 tokenizer, with some curious added tokens: https://huggingface.co/Salesforce/xgen-7b-8k-base/blob/5e1ad70389cda91
by minimaxir 3y ago
The tokenizer appears to be the original GPT-2 tokenizer, with some curious added tokens: https://huggingface.co/Salesforce/xgen-7b-8k-base/blob/5e1ad70389cda911e995522368165a7cc4b3da21/tokenization_xgen.py#L40 https://huggingface.co/Salesforce/xgen-7b-8k-base/blob/5e1ad...
- imrehg 3y agoThose look all programming related tokens, and I think they should relate to their focus of improving this model's code generation capabilities, and adding quite a bit of data related to that (BigCode Starcoder in the Second stage in pre-training https://blog.salesforceairesearch.com/xgen/#pre-training-data https://blog.salesforceairesearch.com/xgen/#pre-training-dat... )