5 ms·

Seemed like a great project. Hope to see it come back! There are some great open-source projects in this space – not quite the same – many are focused on local

Seemed like a great project. Hope to see it come back! There are some great open-source projects in this space – not quite the same – many are focused on local LLMs like Llama2 or Code Llama which was released last week: - https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama (download & run LLMs locally - I'm a maintainer) - https://github.com/simonw/llm https://github.com/simonw/llm (access LLMs from the cli - cloud and local) - https://github.com/oobabooga/text-generation-webui https://github.com/oobabooga/text-generation-webui (a web ui w/ different backends) - https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp (fast local LLM runner) - https://github.com/go-skynet/LocalAI https://github.com/go-skynet/LocalAI (has an openai-compatible api)

ajhai 3y ago
Adding to the list: - https://github.com/trypromptly/LLMStack https://github.com/trypromptly/LLMStack (build and run apps locally with LocalAI support - I'm a maintainer)
brucethemoose2 3y ago
Also https://github.com/LostRuins/koboldcpp https://github.com/LostRuins/koboldcpp The UI is relatively mature, as it predates llama. It includes upstream llama.cpp PRs, integrated AI horde support, lots of sampling tuning knobs, easy gpu/cpu offloading, and its basically dependency free.
- espe 3y ago
  yes. at first glance it looks like a windows app but it's actually very portable. it has some parameters for gpu offloading and extended context size that just work. it exposes an api endpoint. i use it on a workstation to serve larger llms locally and like the performance and ease of use.
jacquesm 3y ago
Ollama is very neat. Given how compressible the models are is there any work being done on using them in some kind of compressed format other than reducing the word size?
- nacs 3y ago
  Yes, AutoGPTQ supports this (8, 4, 3, and 2 bit quantization/"compression" of weights + inference). GPTQ has also been merged into Transformers library recently ( https://huggingface.co/blog/gptq-integration https://huggingface.co/blog/gptq-integration ). GGML quantization format used by llama.cpp also supports (8,6,5,4,3, and 2 bit quantization).
  - jacquesm 3y ago
    'other than'...
- jmorgan 3y ago
  There are different levels of quantization available for different models (if that's what you mean :). E.g. here are the versions available for Llama 2: https://ollama.ai/library/llama2/tags https://ollama.ai/library/llama2/tags which go down to 2-bit quantization (which surprisingly still happens to work reasonably well).
  - jacquesm 3y ago
    No, what I mean is that it seems as though there is quite a bit of sparseness to the matrix and I was wondering if that can somehow be used to further shrink the model, quantization is another effect (it leaves the shape of the various elements as they are but reduces their bit-depth).
    - jmorgan 3y ago
      Ah, gotcha! I thought you probably meant something else. I've been wondering this too, and it's something I've been meaning to look at. On a related note it doesn't seem like many local runners are leveraging techniques like PagedAttention yet (see https://vllm.ai/ https://vllm.ai/) which is inspired by operating system memory paging to reduce memory requirements for LLMs. It's not quite what you mentioned, but it might have a similar effect! Would love to know if you've seen other methods that might help reduce memory requirements.. it's one of the largest resource bottlenecks to running LLMs right now!

ajhai 3y ago

Adding to the list: - https://github.com/trypromptly/LLMStack https://github.com/trypromptly/LLMStack (build and run apps locally with LocalAI support - I'm a maintainer)

brucethemoose2 3y ago

Also https://github.com/LostRuins/koboldcpp https://github.com/LostRuins/koboldcpp The UI is relatively mature, as it predates llama. It includes upstream llama.cpp PRs, integrated AI horde support, lots of sampling tuning knobs, easy gpu/cpu offloading, and its basically dependency free.

espe 3y ago
yes. at first glance it looks like a windows app but it's actually very portable. it has some parameters for gpu offloading and extended context size that just work. it exposes an api endpoint. i use it on a workstation to serve larger llms locally and like the performance and ease of use.

jacquesm 3y ago

Ollama is very neat. Given how compressible the models are is there any work being done on using them in some kind of compressed format other than reducing the word size?

nacs 3y ago
Yes, AutoGPTQ supports this (8, 4, 3, and 2 bit quantization/"compression" of weights + inference). GPTQ has also been merged into Transformers library recently ( https://huggingface.co/blog/gptq-integration https://huggingface.co/blog/gptq-integration ). GGML quantization format used by llama.cpp also supports (8,6,5,4,3, and 2 bit quantization).
- jacquesm 3y ago
  'other than'...
jmorgan 3y ago
There are different levels of quantization available for different models (if that's what you mean :). E.g. here are the versions available for Llama 2: https://ollama.ai/library/llama2/tags https://ollama.ai/library/llama2/tags which go down to 2-bit quantization (which surprisingly still happens to work reasonably well).
- jacquesm 3y ago
  No, what I mean is that it seems as though there is quite a bit of sparseness to the matrix and I was wondering if that can somehow be used to further shrink the model, quantization is another effect (it leaves the shape of the various elements as they are but reduces their bit-depth).
  - jmorgan 3y ago
    Ah, gotcha! I thought you probably meant something else. I've been wondering this too, and it's something I've been meaning to look at. On a related note it doesn't seem like many local runners are leveraging techniques like PagedAttention yet (see https://vllm.ai/ https://vllm.ai/) which is inspired by operating system memory paging to reduce memory requirements for LLMs. It's not quite what you mentioned, but it might have a similar effect! Would love to know if you've seen other methods that might help reduce memory requirements.. it's one of the largest resource bottlenecks to running LLMs right now!