3 ms·

> The training recipe and model architecture follow LLaMA This is huge. MPT and Falcon are cool, but the inference runtimes and various tooling is mostly opti

> The training recipe and model architecture follow LLaMA This is huge. MPT and Falcon are cool, but the inference runtimes and various tooling is mostly optimized for LLaMA. If this is a drop-in replacement for 7B, it's going to catch on much faster than any other small model.

jxy 3y ago
It seems to use a different tokenizer than LLaMA, though the neural network architecture is the same.
- minimaxir 3y ago
  The tokenizer appears to be the original GPT-2 tokenizer, with some curious added tokens: https://huggingface.co/Salesforce/xgen-7b-8k-base/blob/5e1ad70389cda911e995522368165a7cc4b3da21/tokenization_xgen.py#L40 https://huggingface.co/Salesforce/xgen-7b-8k-base/blob/5e1ad...
  - imrehg 3y ago
    Those look all programming related tokens, and I think they should relate to their focus of improving this model's code generation capabilities, and adding quite a bit of data related to that (BigCode Starcoder in the Second stage in pre-training https://blog.salesforceairesearch.com/xgen/#pre-training-data https://blog.salesforceairesearch.com/xgen/#pre-training-dat... )
- imrehg 3y ago
  Looks like this is the limitation why it cannot just be plugged into llama.cpp (doesn't have a saved tokeniser model, and I'm not sure how would one go about creating one). Otherwise it would be cool to try it out, the metrics in the article are promising to have something running locally on an M1 Mac...
lhl 3y ago
There's also OpenLLaMA, which has a 13B version as well and is a straight drop-in (except for code generation due to multiple space tokenization: https://github.com/openlm-research/open_llama#update-06152023 https://github.com/openlm-research/open_llama#update-0615202...). XGen-7B is probably the superior 7B model, it's trained on more tokens and a longer default sequence length (although both presumably can adopt SuperHOT (Position Interpolation) to extend context), but larger models still probably perform better on an absolute basis.

jxy 3y ago

It seems to use a different tokenizer than LLaMA, though the neural network architecture is the same.

minimaxir 3y ago
The tokenizer appears to be the original GPT-2 tokenizer, with some curious added tokens: https://huggingface.co/Salesforce/xgen-7b-8k-base/blob/5e1ad70389cda911e995522368165a7cc4b3da21/tokenization_xgen.py#L40 https://huggingface.co/Salesforce/xgen-7b-8k-base/blob/5e1ad...
- imrehg 3y ago
  Those look all programming related tokens, and I think they should relate to their focus of improving this model's code generation capabilities, and adding quite a bit of data related to that (BigCode Starcoder in the Second stage in pre-training https://blog.salesforceairesearch.com/xgen/#pre-training-data https://blog.salesforceairesearch.com/xgen/#pre-training-dat... )
imrehg 3y ago
Looks like this is the limitation why it cannot just be plugged into llama.cpp (doesn't have a saved tokeniser model, and I'm not sure how would one go about creating one). Otherwise it would be cool to try it out, the metrics in the article are promising to have something running locally on an M1 Mac...

lhl 3y ago

There's also OpenLLaMA, which has a 13B version as well and is a straight drop-in (except for code generation due to multiple space tokenization: https://github.com/openlm-research/open_llama#update-06152023 https://github.com/openlm-research/open_llama#update-0615202...). XGen-7B is probably the superior 7B model, it's trained on more tokens and a longer default sequence length (although both presumably can adopt SuperHOT (Position Interpolation) to extend context), but larger models still probably perform better on an absolute basis.