4 ms·

llama.cpp is simpler to get working than this is.

• mastazi 4y ago
  How so? how is this: npx dalai llama 7B npx dalai serve harder than: # build this repo git clone https://github.com/ggerganov/llama.cpp cd llama.cpp make #For Windows and CMake, use the following command instead: cd <path_to_llama_folder> mkdir build cd build cmake .. cmake --build . --config Release # obtain the original LLaMA model weights and place them in ./models ls ./models 65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model # install Python dependencies python3 -m pip install torch numpy sentencepiece # convert the 7B model to ggml FP16 format python3 convert-pth-to-ggml.py models/7B/ 1 # quantize the model to 4-bits (using method 2 = q4_0) ./quantize ./models/7B/ggml-model-f16.bin ./models/7B/ggml-model-q4_0.bin 2 # run the inference ./main -m ./models/7B/ggml-model-q4_0.bin -n 128
  • hrpnk 4y ago
    dalai does a lot of magic in the background. I had to run the package from source to debug the error messages. When run in on a Mac with zsh, it still runs bash under the hood to setup the python venv. Took me an hour to debug that as bash did not have poetry and pyenv configured to use <3.11 python since I never use bash. Yet, sentencepiece does not have a 3.11 wheel available, thus running dalai fails. Today, just 2d after running dalai successfully, it refuses to start and just hangs in there. Unclear why.
  • MintPaw 4y ago
    llama.cpp and alpaca.cpp both ship binary releases. Dalai does not seem to.
  • vorticalbox 4y ago
    for one the script after cloning llama.cpp repo then calls `exit()` which then drops me out of my fish shell.
  • msm_ 4y ago
    The first version won't work on my linux distribution (nixos). The second one also won't work, but I know how to make it work (C++ and Python without any additional abstraction layers are easy).