10 ms·
That's what they say but I just spent 10 minutes searching the git repo, reading the relavent .py files and looking at their homepage and the vicuna-7b-delta an
by superkuh 4y ago
That's what they say but I just spent 10 minutes searching the git repo, reading the relavent .py files and looking at their homepage and the vicuna-7b-delta and vicuna-13b-delta-v0 files are no where to be found. Am I blind or did they announce a release without actually releasing?
- zhwu 4y agoIf you follow this command in their instruction, the delta will be automatically downloaded and applied to the base model. https://github.com/lm-sys/FastChat#vicuna-13b https://github.com/lm-sys/FastChat#vicuna-13b: `python3 -m fastchat.model.apply_delta --base /path/to/llama-13b --target /output/path/to/vicuna-13b --delta lmsys/vicuna-13b-delta-v0`
- deleted 4y ago[deleted]
- viraptor 4y agoThis can be then quantized to the llama.cpp/gpt4all format, right? Specifically, this only tweaks the existing weights slightly, without changing the structure?
- hotpathdev 4y agoI may have missed the detail, but it also expects the pytorch conversion rather than original LLaMa model.
- zhwu 4y agoYes, you need to convert the original LLaMA model to the huggingface format, according to https://github.com/lm-sys/FastChat#vicuna-weights https://github.com/lm-sys/FastChat#vicuna-weights and https://huggingface.co/docs/transformers/main/model_doc/llama https://huggingface.co/docs/transformers/main/model_doc/llam...
- MMMercy2 4y agoYou can use this command to apply the delta weights. (https://github.com/lm-sys/FastChat#vicuna-13b https://github.com/lm-sys/FastChat#vicuna-13b) The delta weights are hosted on huggingface and will be automatically downloaded.
- superkuh 4y agoThanks! https://huggingface.co/lmsys/vicuna-13b-delta-v0 https://huggingface.co/lmsys/vicuna-13b-delta-v0 Edit, later: I found some instructive pages on how to use the vicuna weights with llama.cpp (https://lmsysvicuna.miraheze.org/wiki/How_to_use_Vicuna#Use_with_llama.cpp%3A https://lmsysvicuna.miraheze.org/wiki/How_to_use_Vicuna#Use_...) and pre-made ggml format compatible 4-bit quantized vicuna weights, https://huggingface.co/eachadea/ggml-vicuna-13b-4bit/tree/main https://huggingface.co/eachadea/ggml-vicuna-13b-4bit/tree/ma... (8GB ready to go, no 60+GB RAM steps needed)
- eurekin 4y agoI did try, but got: ``` ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported. ```
- superkuh 4y ago> Unfortunately there's a mismatch between the model generated by the delta patcher and the tokenizer (32001 vs 32000 tokens). There's a tool to fix this at llama-tools (https://github.com/Ronsor/llama-tools https://github.com/Ronsor/llama-tools). Add 1 token like (C controltoken), and then run the conversion script.
- DrSiemer 4y agoJust rename it in the tokenconfig.json
- eurekin 4y agoThanks, that indeed worked! This and using conda in wsl2, instead on bare windows