4 ms·

This looks great! If you're looking to do the same with open source code, you could likely run Ollama and a UI. https://github.com/jmorganca/ollama https://gi

This looks great! If you're looking to do the same with open source code, you could likely run Ollama and a UI. https://github.com/jmorganca/ollama https://github.com/jmorganca/ollama + https://github.com/ollama-webui/ollama-webui https://github.com/ollama-webui/ollama-webui

nigma 3y ago
I'm having a lot of fun chatting with characters using Faraday and koboldcpp. Faraday has a great UI that lets you adjust character profiles, generate alternative model responses, undo, or edit dialogue, and experiment with how models react to your input. There's also SillyTavern that I have yet to try out. - https://faraday.dev/ https://faraday.dev/ - https://github.com/LostRuins/koboldcpp https://github.com/LostRuins/koboldcpp - https://github.com/SillyTavern/SillyTavern https://github.com/SillyTavern/SillyTavern
eshack94 3y ago
Ollama is fantastic. I cannot recommend this project highly enough. Completely open-source, lightweight, and a great community.
activatedgeek 3y ago
I looked at Ollama before, but couldn't quite figure something out from the docs [1] It looks like a lot of the tooling is heavily engineered for a set of modern popular LLM-esque models. And looks like llama.cpp also supports LoRA models, so I'd assume there is a way to engineer a pipeline from LoRA to llama.cpp deployments, which probably covers quite a broad set of possibilities. Beyond llama.cpp, can someone point me to what the broader community uses for general PyTorch model deployments? I haven't quite ever self-hosted models, and am really keen to do one. Ideally, I am looking for something that stays close to the PyTorch core, and therefore allows me the flexibility to take any nn.Module to production. [1]: https://github.com/jmorganca/ollama/blob/main/docs/import.md https://github.com/jmorganca/ollama/blob/main/docs/import.md.
seydor 3y ago
oobabooga is no longer popular?
- oceanplexian 3y ago
  oobabooga is the king. As far as I know, ollama doesn’t support exllama, qlora fine tuning, multi-GPU, etc. Text-generation-webui might seem like a science project, but it’s leagues ahead (Like 2-4x faster inference with the right plugins) of everything else. Also has a nice openai mock API that works great.
  - jstarfish 3y ago
    oobabooga should be king, but it's unstable as hell. ooba's worth keeping an eye on, but koboldcpp is more stable, almost as versatile, and way less frustrating. It also still supports GGML.
  - LoganDark 3y ago
    In compatibility only. If you want keyboard shortcuts or a snappy interface, Python+gradio just isn't it. https://github.com/oobabooga/text-generation-webui/issues/4138 https://github.com/oobabooga/text-generation-webui/issues/41...
- Taek 3y ago
  I appreciate the ooba team for everything they've done but its not a great user experience
smcleod 3y ago
With a couple of other folks I'm currently working on an Ollama GUI, it's in early stages of development - https://github.com/ai-qol-things/rusty-ollama https://github.com/ai-qol-things/rusty-ollama

nigma 3y ago

I'm having a lot of fun chatting with characters using Faraday and koboldcpp. Faraday has a great UI that lets you adjust character profiles, generate alternative model responses, undo, or edit dialogue, and experiment with how models react to your input. There's also SillyTavern that I have yet to try out. - https://faraday.dev/ https://faraday.dev/ - https://github.com/LostRuins/koboldcpp https://github.com/LostRuins/koboldcpp - https://github.com/SillyTavern/SillyTavern https://github.com/SillyTavern/SillyTavern

eshack94 3y ago

Ollama is fantastic. I cannot recommend this project highly enough. Completely open-source, lightweight, and a great community.

activatedgeek 3y ago

I looked at Ollama before, but couldn't quite figure something out from the docs [1] It looks like a lot of the tooling is heavily engineered for a set of modern popular LLM-esque models. And looks like llama.cpp also supports LoRA models, so I'd assume there is a way to engineer a pipeline from LoRA to llama.cpp deployments, which probably covers quite a broad set of possibilities. Beyond llama.cpp, can someone point me to what the broader community uses for general PyTorch model deployments? I haven't quite ever self-hosted models, and am really keen to do one. Ideally, I am looking for something that stays close to the PyTorch core, and therefore allows me the flexibility to take any nn.Module to production. [1]: https://github.com/jmorganca/ollama/blob/main/docs/import.md https://github.com/jmorganca/ollama/blob/main/docs/import.md.

seydor 3y ago

oobabooga is no longer popular?

oceanplexian 3y ago
oobabooga is the king. As far as I know, ollama doesn’t support exllama, qlora fine tuning, multi-GPU, etc. Text-generation-webui might seem like a science project, but it’s leagues ahead (Like 2-4x faster inference with the right plugins) of everything else. Also has a nice openai mock API that works great.
- jstarfish 3y ago
  oobabooga should be king, but it's unstable as hell. ooba's worth keeping an eye on, but koboldcpp is more stable, almost as versatile, and way less frustrating. It also still supports GGML.
- LoganDark 3y ago
  In compatibility only. If you want keyboard shortcuts or a snappy interface, Python+gradio just isn't it. https://github.com/oobabooga/text-generation-webui/issues/4138 https://github.com/oobabooga/text-generation-webui/issues/41...
Taek 3y ago
I appreciate the ooba team for everything they've done but its not a great user experience

smcleod 3y ago

With a couple of other folks I'm currently working on an Ollama GUI, it's in early stages of development - https://github.com/ai-qol-things/rusty-ollama https://github.com/ai-qol-things/rusty-ollama