4 ms·

What’s the best chat interface for llama? I have a 3090 and would love to get one of the models running in my terminal for quick coding tasks.

ahoka 3y ago
This is the easiest to setup: https://faraday.dev/ https://faraday.dev/ I think Wizard is the “meta” for technical questions now.
- ForkMeOnTinder 3y ago
  Is this open source? I don't see a Linux download link and I want to try building it for Linux myself
reqo 3y ago
Here is an open source project that supports voice as well: https://github.com/cogentapps/chat-with-gpt https://github.com/cogentapps/chat-with-gpt I think it is meant to be used with ElevenLabs and OpenAI API, but might be easy to configure for use with local Whisper.cpp + llama?
ForkMeOnTinder 3y ago
ollama is dead easy to use. It's a single binary that downloads models on demand, like docker downloads images. pacman -S ollama ollama serve ollama run llama2:13b 'insert prompt' https://ollama.ai/ https://ollama.ai/
- riddleronroof 3y ago
  ollama wraps llama.cpp into a docker container, correct? Besides that it seeks like a go server for chat?
- _ea1k 3y ago
  The ollama shell is really nice.
evilduck 3y ago
It's not open source but it is still (for now) free: lmstudio.ai. Chat histories, a good config UI, easy prompt management, model management, model discovery, easy set up, cross platform, able to serve an API for connectivity to other tools. They're hiring and have no currently disclosed monetization strategy so I expect a rugpull soon where some now-free feature gets paywalled or purposefully crippled, but it's not like porcelain apps for free LLMs that rely entirely on llama.cpp to function can do vendor lock-in. I'd second ollama if OSS is a higher priority than features though.
oceanplexian 3y ago
Depends on what you mean by "best"? Absolute bleeding edge fastest possible inference? ExLlama or ExLlamaV2 on a 4090.

ahoka 3y ago

This is the easiest to setup: https://faraday.dev/ https://faraday.dev/ I think Wizard is the “meta” for technical questions now.

ForkMeOnTinder 3y ago
Is this open source? I don't see a Linux download link and I want to try building it for Linux myself

reqo 3y ago

Here is an open source project that supports voice as well: https://github.com/cogentapps/chat-with-gpt https://github.com/cogentapps/chat-with-gpt I think it is meant to be used with ElevenLabs and OpenAI API, but might be easy to configure for use with local Whisper.cpp + llama?

ForkMeOnTinder 3y ago

ollama is dead easy to use. It's a single binary that downloads models on demand, like docker downloads images. pacman -S ollama ollama serve ollama run llama2:13b 'insert prompt' https://ollama.ai/ https://ollama.ai/

riddleronroof 3y ago
ollama wraps llama.cpp into a docker container, correct? Besides that it seeks like a go server for chat?
_ea1k 3y ago
The ollama shell is really nice.

evilduck 3y ago

It's not open source but it is still (for now) free: lmstudio.ai. Chat histories, a good config UI, easy prompt management, model management, model discovery, easy set up, cross platform, able to serve an API for connectivity to other tools. They're hiring and have no currently disclosed monetization strategy so I expect a rugpull soon where some now-free feature gets paywalled or purposefully crippled, but it's not like porcelain apps for free LLMs that rely entirely on llama.cpp to function can do vendor lock-in. I'd second ollama if OSS is a higher priority than features though.

oceanplexian 3y ago

Depends on what you mean by "best"? Absolute bleeding edge fastest possible inference? ExLlama or ExLlamaV2 on a 4090.