3 ms·
How so? how is this: npx dalai llama 7B npx dalai serve harder than: # build this repo git clone https://github.com/ggerganov/llama.cpp
by mastazi 4y ago
How so?
how is this:
npx dalai llama 7B
npx dalai serve
harder than:
# build this repo
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make
#For Windows and CMake, use the following command instead:
cd <path_to_llama_folder>
mkdir build
cd build
cmake ..
cmake --build . --config Release
# obtain the original LLaMA model weights and place them in ./models
ls ./models
65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model
# install Python dependencies
python3 -m pip install torch numpy sentencepiece
# convert the 7B model to ggml FP16 format
python3 convert-pth-to-ggml.py models/7B/ 1
# quantize the model to 4-bits (using method 2 = q4_0)
./quantize ./models/7B/ggml-model-f16.bin ./models/7B/ggml-model-q4_0.bin 2
# run the inference
./main -m ./models/7B/ggml-model-q4_0.bin -n 128
- hrpnk 4y agodalai does a lot of magic in the background. I had to run the package from source to debug the error messages. When run in on a Mac with zsh, it still runs bash under the hood to setup the python venv. Took me an hour to debug that as bash did not have poetry and pyenv configured to use <3.11 python since I never use bash. Yet, sentencepiece does not have a 3.11 wheel available, thus running dalai fails. Today, just 2d after running dalai successfully, it refuses to start and just hangs in there. Unclear why.
- MintPaw 4y agollama.cpp and alpaca.cpp both ship binary releases. Dalai does not seem to.
- vorticalbox 4y agofor one the script after cloning llama.cpp repo then calls `exit()` which then drops me out of my fish shell.
- msm_ 4y agoThe first version won't work on my linux distribution (nixos). The second one also won't work, but I know how to make it work (C++ and Python without any additional abstraction layers are easy).