4 ms·

For my experiments with new self-hostable models on Linux, I've been using a script to download GGUF-models from TheBloke on HuggingFace (currently, TheBloke's

by kgeist 3y ago

For my experiments with new self-hostable models on Linux, I've been using a script to download GGUF-models from TheBloke on HuggingFace (currently, TheBloke's repository has 657 models in the GGUF format) which I feed to a simple program I wrote which invokes llama.cpp compiled with GPU support. The GGUF format and TheBloke are a blessing, because I'm able to check out new models basically on the day of their release (TheBloke is very fast) and without an issue. However, the only frontend I have is console. Judging by their site, their setup is exactly the same as mine (which I implemented over a weekend), except that they also added a React-based UI on top. I wonder, how they're planning to commercialize it, because it's pretty trivial to replicate, and there're already open-source UI's like oogabooga.

bambax 3y ago
I'd like to build myself a headless server to run models, that could be queried from various clients locally on my LAN, but am usure where to start and what the hardware requirements would be. Software can always be changed later but I'd rather buy the hardware parts only once. Do you have recommendations about this? or blog posts to get started? What would be a decent hardware configuration?
- fintechie 3y ago
  You can currently do this in an M2 Max with ollama and a Nextjs UI [0] running in a docker container. Any devices in the network can use the UI... and I guess if you want a LAN API you just need to run another container with with OAI compatible API that can query ollama.. eg [1] [0]https://github.com/ivanfioravanti/chatbot-ollama https://github.com/ivanfioravanti/chatbot-ollama [1]https://github.com/BerriAI/litellm https://github.com/BerriAI/litellm
- rsolva 3y ago
  Ollama does this. I run it in a container on my homelab (Proxmox on a HP EliteDesk SFF G2 800) and 7B models run decently fast on CPU-only. Ollama has a nice API and makes it easy to manage models. Together with ollama-webui, it can replace ChatGPT 3.5 for most tasks. I also use it in VSCode and nvim with plugins, works great! I have been meaning to write a short blog post about my setup...
  - Loic 3y ago
    If I may ask, which plugins are you using in VSCode?
    - rsolva 3y ago
      I'm using the extension Continue: https://marketplace.visualstudio.com/items?itemName=Continue.continue https://marketplace.visualstudio.com/items?itemName=Continue... The setup of connecting to Ollama is a bit clunky, but once it's set up it works well!
  - mfalcon 3y ago
    I've been trying Ollama locally. I've yet to know how it'll behave in a production setting.
    - kkielhofner 3y ago
      Depending on what you mean by "production" you'll probably want to look at "real" serving implementations like HF TGI, vLLM, lmdeploy, Triton Inference Server (tensorrt-llm), etc. There are also more bespoke implementations for things like serving large numbers of LoRA adapters[0]. These are heavily optimized for more efficient memory usage, performance, and responsiveness when serving large numbers of concurrent requests/users in addition to things like model versioning/hot load/reload/etc, Prometheus metrics, things like that. One major difference is at this level a lot of the more aggressive memory optimization techniques and support for CPU aren't even considered. Generally speaking you get GPTQ and possibly AWQ quantization + their optimizations + CUDA only. Their target users and their use cases are often using A100/H100 and just trying to need fewer of them. Support for lower VRAM cards, older CUDA compute architectures, etc come secondary to that (for the most part). [0] - https://github.com/S-LoRA/S-LoRA https://github.com/S-LoRA/S-LoRA
      - mfalcon 3y ago
        Thanks! Really helpful. I've a 3090 at home and my idea is to do some testing on a similar config in the cloud to have an idea of the amount of requests that could be served.
- akx 3y ago
  Just compile llama.cpp's server example, and you have a local HTTP API. It also has a simple UI (disclaimer: to which I've contributed). https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md https://github.com/ggerganov/llama.cpp/blob/master/examples/...
- guntars 3y ago
  LM Studio can start a local server with the APIs matching OpenAI. You can’t do concurrent requests, but that should get you started.
- Havoc 3y ago
  > usure where to start and what the hardware requirements would be Have a look at the localllama subreddit In short though dual 3090 is common, single 4090 or various flavours of M123 macs. Alternatively p40 can be jury-rigged too but research that carefully. In fact anything with more than one gpu is going to require careful research
ekianjo 3y ago
You dont need thebloke. Its trivial to make gguf files from bin models by yourself.
- cooperaustinj 3y ago
  What a comment. Why do it the easy way when the more difficult and slower way works ok it to the same result‽ For people who just want to USE models and not back at them, TheBloke is exactly the right place to go. Like telling someone interested in 3D printing minis to build a 3D printer instead of buying one. Obviously that helps them get to their goal of printing minis faster right?
  - paul_n 3y ago
    Actually, consider that the commenter may have helped un-obfuscate this world a little bit by saying that it is in fact easy. To be honest the hardest part about the local LLM scene is the absurd amount of jargon introduced - everything looks a bit more complex than it is. It’s really is easy with llama.cpp, someone even wrote a tutorial here: https://github.com/ggerganov/llama.cpp/discussions/2948 https://github.com/ggerganov/llama.cpp/discussions/2948 . But yes, TheBloke tends to have conversions up very quickly as well and has made a name for himself for doing this (+more)
  - ekianjo 3y ago
    This is a helpful comment because the Bloke only converts a small fraction of models and hardly ever updates them timely after the first release. So learn to cook.
deleted 3y ago
[deleted]

by kgeist 3y ago

bambax 3y ago

I'd like to build myself a headless server to run models, that could be queried from various clients locally on my LAN, but am usure where to start and what the hardware requirements would be. Software can always be changed later but I'd rather buy the hardware parts only once. Do you have recommendations about this? or blog posts to get started? What would be a decent hardware configuration?

fintechie 3y ago
You can currently do this in an M2 Max with ollama and a Nextjs UI [0] running in a docker container. Any devices in the network can use the UI... and I guess if you want a LAN API you just need to run another container with with OAI compatible API that can query ollama.. eg [1] [0]https://github.com/ivanfioravanti/chatbot-ollama https://github.com/ivanfioravanti/chatbot-ollama [1]https://github.com/BerriAI/litellm https://github.com/BerriAI/litellm
rsolva 3y ago
Ollama does this. I run it in a container on my homelab (Proxmox on a HP EliteDesk SFF G2 800) and 7B models run decently fast on CPU-only. Ollama has a nice API and makes it easy to manage models. Together with ollama-webui, it can replace ChatGPT 3.5 for most tasks. I also use it in VSCode and nvim with plugins, works great! I have been meaning to write a short blog post about my setup...
- Loic 3y ago
  If I may ask, which plugins are you using in VSCode?
  - rsolva 3y ago
    I'm using the extension Continue: https://marketplace.visualstudio.com/items?itemName=Continue.continue https://marketplace.visualstudio.com/items?itemName=Continue... The setup of connecting to Ollama is a bit clunky, but once it's set up it works well!
- mfalcon 3y ago
  I've been trying Ollama locally. I've yet to know how it'll behave in a production setting.
  - kkielhofner 3y ago
    Depending on what you mean by "production" you'll probably want to look at "real" serving implementations like HF TGI, vLLM, lmdeploy, Triton Inference Server (tensorrt-llm), etc. There are also more bespoke implementations for things like serving large numbers of LoRA adapters[0]. These are heavily optimized for more efficient memory usage, performance, and responsiveness when serving large numbers of concurrent requests/users in addition to things like model versioning/hot load/reload/etc, Prometheus metrics, things like that. One major difference is at this level a lot of the more aggressive memory optimization techniques and support for CPU aren't even considered. Generally speaking you get GPTQ and possibly AWQ quantization + their optimizations + CUDA only. Their target users and their use cases are often using A100/H100 and just trying to need fewer of them. Support for lower VRAM cards, older CUDA compute architectures, etc come secondary to that (for the most part). [0] - https://github.com/S-LoRA/S-LoRA https://github.com/S-LoRA/S-LoRA
    - mfalcon 3y ago
      Thanks! Really helpful. I've a 3090 at home and my idea is to do some testing on a similar config in the cloud to have an idea of the amount of requests that could be served.
akx 3y ago
Just compile llama.cpp's server example, and you have a local HTTP API. It also has a simple UI (disclaimer: to which I've contributed). https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md https://github.com/ggerganov/llama.cpp/blob/master/examples/...
guntars 3y ago
LM Studio can start a local server with the APIs matching OpenAI. You can’t do concurrent requests, but that should get you started.
Havoc 3y ago
> usure where to start and what the hardware requirements would be Have a look at the localllama subreddit In short though dual 3090 is common, single 4090 or various flavours of M123 macs. Alternatively p40 can be jury-rigged too but research that carefully. In fact anything with more than one gpu is going to require careful research

ekianjo 3y ago

You dont need thebloke. Its trivial to make gguf files from bin models by yourself.

cooperaustinj 3y ago
What a comment. Why do it the easy way when the more difficult and slower way works ok it to the same result‽ For people who just want to USE models and not back at them, TheBloke is exactly the right place to go. Like telling someone interested in 3D printing minis to build a 3D printer instead of buying one. Obviously that helps them get to their goal of printing minis faster right?
- paul_n 3y ago
  Actually, consider that the commenter may have helped un-obfuscate this world a little bit by saying that it is in fact easy. To be honest the hardest part about the local LLM scene is the absurd amount of jargon introduced - everything looks a bit more complex than it is. It’s really is easy with llama.cpp, someone even wrote a tutorial here: https://github.com/ggerganov/llama.cpp/discussions/2948 https://github.com/ggerganov/llama.cpp/discussions/2948 . But yes, TheBloke tends to have conversions up very quickly as well and has made a name for himself for doing this (+more)
- ekianjo 3y ago
  This is a helpful comment because the Bloke only converts a small fraction of models and hardly ever updates them timely after the first release. So learn to cook.

deleted 3y ago

[deleted]