4 ms·
I run Mistral 7b and Llama 3 locally using jani.ai on a 32GB Dell laptop and get about 6 tokens per second with a context window of 8k. It's definitely usable i
by onion2k 2y ago
I run Mistral 7b and Llama 3 locally using jani.ai on a 32GB Dell laptop and get about 6 tokens per second with a context window of 8k. It's definitely usable if you're patient. I'm glad I also have a Hugging Face account though.
- Liquix 2y agoseconded - IMHO Jan has the cleanest UI and most straightforward setup out of all LLM frontends available now. https://jan.ai/ https://jan.ai/ https://github.com/janhq/jan https://github.com/janhq/jan