4 ms·

I went self hosted. It was about time to build a new desktop anyways (roughly 4 to 6 years before the old one goes to frolic at the server farm in the basement

I went self hosted. It was about time to build a new desktop anyways (roughly 4 to 6 years before the old one goes to frolic at the server farm in the basement) and $2,000 will easily buy a machine that can run the quantized 65b models right now. So I spent slightly more than I normally do on this latest box and it's happily spitting out 10+ tokens a second. You're not going to beat GPT-4 yet, but you have direct control over where your info goes, what model you're running, compliance with work policies against using public AI, and relatively cheap fixed costs. Not to mention, the local version works with no internet and isn't subject to provider outages (not entirely true - but you're the provider and can resolve). Seems like an easy win for anyone who might be buying a desktop for graphic/gaming anyways.

xcskier56 3y ago
Super interesting! Can you point me to some of the models and repos you used to do this?
- horsawlarway 3y ago
  For base tooling, things like: https://huggingface.co/ https://huggingface.co/ (finding models and downloading them) https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp (llama) https://github.com/cmp-nct/ggllm.cpp https://github.com/cmp-nct/ggllm.cpp (falcon) For interactive work (art/chat/research/playing around), things like: https://github.com/oobabooga/text-generation-webui/blob/main/docs/LLaMA-model.md https://github.com/oobabooga/text-generation-webui/blob/main... (llama) (Also - they just added a decent chat server built into llama.cpp the project) https://github.com/invoke-ai/InvokeAI https://github.com/invoke-ai/InvokeAI (stable-diffusion) Plus a bunch of hacked together scripts. Some example models (I'm linking to quantized versions that someone else has made, but the tooling is in the above repos to create them from the published fp16 models) https://huggingface.co/TheBloke/llama-65B-GGML https://huggingface.co/TheBloke/llama-65B-GGML https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored-GPTQ https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored... etc. Hugging face has quite a number, although some require filling out forms for the base models for tuning/training.
  - xcskier56 3y ago
    Thank you!
generalizations 3y ago
How does it seem to compare with GPT 3.5? That really seems like the baseline for what's usable.
- horsawlarway 3y ago
  It depends heavily on the model you're running, and to some extent what you're doing with it. It also depends to on prompt effort. The quantized llama 65b model (you can do it yourself, or pull something like https://huggingface.co/TheBloke/llama-65B-GGML https://huggingface.co/TheBloke/llama-65B-GGML) is probably the highest quality for general purpose, but it does take a fair bit of effort to prompt since it's not tuned for a use-case. It's also not licensed commercially, so I avoid some things with it (ex: I do a lot of personal learning/investigation, but it doesn't touch or write anything related to work or personal projects) The open models are a little further behind, but it's interesting to see them spin off into niches where they have strengths based on tuning/training.
- gadflyinyoureye 3y ago
  I don’t know about self-hosted, but it the company I contact at has 3.5 instance. It has better understanding of code examples I provide than ChatGPT. The company hasn’t tweaked the code to our standards.
- root_axis 3y ago
  If you're a diligent tester, none of the open source models can touch GPT 3.5 yet, however, in practical terms, some of the 60b and 30b parameter models are almost indistinguishable from GPT 3.5 from the layperson's perspective. Now, if you consider the uncensored models, then you actually have some capabilities that GPT 3.5 and 4 are completely lacking. Based on the rate of progress in the open source world, it won't be more than a year before we have an open source model that is truly superior to GPT 3.5
  - horsawlarway 3y ago
    Yup - I'd say this feels about right. The commercial & api based models are still more capable general purpose tools. But the current open tooling can do some nifty stuff, and the community around it is moving at a breakneck speed still. In some areas, it's acceptably good. In some areas it's not. But it's getting better really fast.
joekim 3y ago
Would you be willing to create a guide? I think this would be of great help.
- horsawlarway 3y ago
  I started here https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp Which won't run everything, but will run model in the GGML format such as https://huggingface.co/TheBloke/llama-65B-GGML https://huggingface.co/TheBloke/llama-65B-GGML The steps are basically: 1. Download a model 2. Make sure you have the latest nvidia driver for your machine, along with the cuda toolkit. This will vary by OS but is fairly easy on most linux distros. 3. compile https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp following their instructions (in particular, look for LLAMA_CUBLAS for enabling GPU support) 4. Run the model following their instructions. There are several flags that are important, but you can also just use their server example that was added a few days ago - it gives a fairly solid chat interface.
- frognumber 3y ago
  I'll make a simpler guide: 1) Go to https://gpt4all.io/index.html https://gpt4all.io/index.html 2) Click the downloader for your OS 3) Run the installer 4) Run gpt4all, and wait for the obnoxiously slow startup time ... and that's it. On my machine, it works perfectly well -- about as fast as the web service version of GPT. I have a decent GPU, but I never checked if it's using it, since it's fast enough.
cvhashim04 3y ago
Sounds interesting. Would appreciate any tutorials or guides.
vladgur 3y ago
Can you describe your build?
- horsawlarway 3y ago
  2 x 3090 (renewed) ~1800 128gb ram ~400 reasonable processor/mobo/psu ~600 2Tb m2 drive ~94 In hindsight - I don't know that the second GPU was worth the spend. The c++ tooling is doing a very good job right now at spreading work between GPU vram and main ram and still being fast enough. Even ~4/5 tokens a second is fast enough to not feel like you're waiting. I'd suggest skipping the second card and dropping the price quite a bit (~2100 vs ~2900) unless you want to tune/train models.
  - frognumber 3y ago
    Are you using the second GPU at all? My experience is only a few systems will share load across GPUs. I didn't bother with dual GPUs for that reason. 4-5 tokens per second is slower than my system. I'm getting in the teens. I'm a little surprised since yours is newer, faster, and has way more RAM.
    - horsawlarway 3y ago
      Yes. I am definitely using both GPUs, I can run the quant 4 65b models entirely in VRAM (they use about 40GB). If I push everything into VRAM - I get 12.2 tokens on average running quant 4 llama 65b. If I run a smaller model I get considerably faster generation. Ex: llama 7b runs at 52 tokens/sec, but it's small enough I don't need the second GPU. Ex - here's my nvidia-smi output while 65b is running https://imgur.com/a/JnaieKg https://imgur.com/a/JnaieKg
bilsbie 3y ago
Does it ever refuse to answer you?

xcskier56 3y ago

Super interesting! Can you point me to some of the models and repos you used to do this?

horsawlarway 3y ago
For base tooling, things like: https://huggingface.co/ https://huggingface.co/ (finding models and downloading them) https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp (llama) https://github.com/cmp-nct/ggllm.cpp https://github.com/cmp-nct/ggllm.cpp (falcon) For interactive work (art/chat/research/playing around), things like: https://github.com/oobabooga/text-generation-webui/blob/main/docs/LLaMA-model.md https://github.com/oobabooga/text-generation-webui/blob/main... (llama) (Also - they just added a decent chat server built into llama.cpp the project) https://github.com/invoke-ai/InvokeAI https://github.com/invoke-ai/InvokeAI (stable-diffusion) Plus a bunch of hacked together scripts. Some example models (I'm linking to quantized versions that someone else has made, but the tooling is in the above repos to create them from the published fp16 models) https://huggingface.co/TheBloke/llama-65B-GGML https://huggingface.co/TheBloke/llama-65B-GGML https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored-GPTQ https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored... etc. Hugging face has quite a number, although some require filling out forms for the base models for tuning/training.
- xcskier56 3y ago
  Thank you!

generalizations 3y ago

How does it seem to compare with GPT 3.5? That really seems like the baseline for what's usable.

horsawlarway 3y ago
It depends heavily on the model you're running, and to some extent what you're doing with it. It also depends to on prompt effort. The quantized llama 65b model (you can do it yourself, or pull something like https://huggingface.co/TheBloke/llama-65B-GGML https://huggingface.co/TheBloke/llama-65B-GGML) is probably the highest quality for general purpose, but it does take a fair bit of effort to prompt since it's not tuned for a use-case. It's also not licensed commercially, so I avoid some things with it (ex: I do a lot of personal learning/investigation, but it doesn't touch or write anything related to work or personal projects) The open models are a little further behind, but it's interesting to see them spin off into niches where they have strengths based on tuning/training.
gadflyinyoureye 3y ago
I don’t know about self-hosted, but it the company I contact at has 3.5 instance. It has better understanding of code examples I provide than ChatGPT. The company hasn’t tweaked the code to our standards.
root_axis 3y ago
If you're a diligent tester, none of the open source models can touch GPT 3.5 yet, however, in practical terms, some of the 60b and 30b parameter models are almost indistinguishable from GPT 3.5 from the layperson's perspective. Now, if you consider the uncensored models, then you actually have some capabilities that GPT 3.5 and 4 are completely lacking. Based on the rate of progress in the open source world, it won't be more than a year before we have an open source model that is truly superior to GPT 3.5
- horsawlarway 3y ago
  Yup - I'd say this feels about right. The commercial & api based models are still more capable general purpose tools. But the current open tooling can do some nifty stuff, and the community around it is moving at a breakneck speed still. In some areas, it's acceptably good. In some areas it's not. But it's getting better really fast.

joekim 3y ago

Would you be willing to create a guide? I think this would be of great help.

horsawlarway 3y ago
I started here https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp Which won't run everything, but will run model in the GGML format such as https://huggingface.co/TheBloke/llama-65B-GGML https://huggingface.co/TheBloke/llama-65B-GGML The steps are basically: 1. Download a model 2. Make sure you have the latest nvidia driver for your machine, along with the cuda toolkit. This will vary by OS but is fairly easy on most linux distros. 3. compile https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp following their instructions (in particular, look for LLAMA_CUBLAS for enabling GPU support) 4. Run the model following their instructions. There are several flags that are important, but you can also just use their server example that was added a few days ago - it gives a fairly solid chat interface.
frognumber 3y ago
I'll make a simpler guide: 1) Go to https://gpt4all.io/index.html https://gpt4all.io/index.html 2) Click the downloader for your OS 3) Run the installer 4) Run gpt4all, and wait for the obnoxiously slow startup time ... and that's it. On my machine, it works perfectly well -- about as fast as the web service version of GPT. I have a decent GPU, but I never checked if it's using it, since it's fast enough.

cvhashim04 3y ago

Sounds interesting. Would appreciate any tutorials or guides.

vladgur 3y ago

Can you describe your build?

horsawlarway 3y ago
2 x 3090 (renewed) ~1800 128gb ram ~400 reasonable processor/mobo/psu ~600 2Tb m2 drive ~94 In hindsight - I don't know that the second GPU was worth the spend. The c++ tooling is doing a very good job right now at spreading work between GPU vram and main ram and still being fast enough. Even ~4/5 tokens a second is fast enough to not feel like you're waiting. I'd suggest skipping the second card and dropping the price quite a bit (~2100 vs ~2900) unless you want to tune/train models.
- frognumber 3y ago
  Are you using the second GPU at all? My experience is only a few systems will share load across GPUs. I didn't bother with dual GPUs for that reason. 4-5 tokens per second is slower than my system. I'm getting in the teens. I'm a little surprised since yours is newer, faster, and has way more RAM.
  - horsawlarway 3y ago
    Yes. I am definitely using both GPUs, I can run the quant 4 65b models entirely in VRAM (they use about 40GB). If I push everything into VRAM - I get 12.2 tokens on average running quant 4 llama 65b. If I run a smaller model I get considerably faster generation. Ex: llama 7b runs at 52 tokens/sec, but it's small enough I don't need the second GPU. Ex - here's my nvidia-smi output while 65b is running https://imgur.com/a/JnaieKg https://imgur.com/a/JnaieKg

bilsbie 3y ago

Does it ever refuse to answer you?