3 ms·
Not hard with text-generation-ui! https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui I personally use llama.
by sodality2 3y ago
Not hard with text-generation-ui! https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui
I personally use llama.cpp as the driver since I run CPU-only but another may be better suited for GPU usage. But then it's as simple as downloading the model and placing it in the directory.
- pmarreck 3y agoMy Threadripper has 64 cores/128 threads and I'm wondering if any models can take advantage of CPU concurrency to at least mitigate some of the loss from not using a GPU (should one not be using a GPU)
- sodality2 3y agoFrom my experience llama.cpp doesn’t take full advantage of parallelism as it could. Tested this on an HPC cluster - increasing thread count certainly did increase CPU usage but did not meaningfully improve tok/s past 6-8 cores. Same behavior with whisper.cpp. :( I wonder if there’s another backend that scales better.
- moonchrome 3y agoI'm guessing the problem is that you're constrained by memory bandwidth and not computation power and this is inherent to the algorithm, not an artifact of any one implementation.
- pmarreck 3y agoFailed to build llama-cpp-python ERROR: Could not build wheels for llama-cpp-python, which is required to install pyproject.toml-based projects Command '. "/Users/pmarreck/Downloads/oobabooga_macos/installer_files/conda/etc/profile.d/conda.sh" && conda activate "/Users/pmarreck/Downloads/oobabooga_macos/installer_files/env" && python -m pip install -r requirements.txt --upgrade' failed with exit status code '1'. Exiting... I'm admittedly running Sonoma beta, that's probably why I will try it on my Linux machine later tonight (currently AFK with laptop)