3 ms·
It seems to use a different tokenizer than LLaMA, though the neural network architecture is the same.
by jxy 3y ago
It seems to use a different tokenizer than LLaMA, though the neural network architecture is the same.
- minimaxir 3y agoThe tokenizer appears to be the original GPT-2 tokenizer, with some curious added tokens: https://huggingface.co/Salesforce/xgen-7b-8k-base/blob/5e1ad70389cda911e995522368165a7cc4b3da21/tokenization_xgen.py#L40 https://huggingface.co/Salesforce/xgen-7b-8k-base/blob/5e1ad...
- imrehg 3y agoThose look all programming related tokens, and I think they should relate to their focus of improving this model's code generation capabilities, and adding quite a bit of data related to that (BigCode Starcoder in the Second stage in pre-training https://blog.salesforceairesearch.com/xgen/#pre-training-data https://blog.salesforceairesearch.com/xgen/#pre-training-dat... )
- imrehg 3y agoLooks like this is the limitation why it cannot just be plugged into llama.cpp (doesn't have a saved tokeniser model, and I'm not sure how would one go about creating one). Otherwise it would be cool to try it out, the metrics in the article are promising to have something running locally on an M1 Mac...