4 ms·

There's a big community discussing exactly that over at https://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/LocalLLaMA/.

• smcleod 3y ago
  +1 r/localllama, 23GB should allow you to run 30b~ models, but honestly some of the new smaller models such as Mistral & friends (Zephyr etc..) are really interesting. You could also Give Mixtral a try if you get a low quant format such as this q3 - https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GPTQ/tree/gptq-3bit-128g-actorder_True https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-G...
  • dr_kiszonka 3y ago
    Do you know if there is a website or spreadsheet where I could enter my RAM, GPU, etc. and see what models I could run locally?
    • SkyMarshal 3y ago
      Yes, https://www.reddit.com/r/LocalLLaMA/ https://www.reddit.com/r/LocalLLaMA/. Just ask there and a bunch of Non-Artificial Intelligent agents will give you that answer ;).
    • unstuck3958 3y ago
      While I agree a website/spreadsheet would be convenient, it's not that complicated. As long as GPU is handling 50-75% of the LLM layers, you should get a decent tok/sec speed (unless you're running really really large models).
      • dr_kiszonka 3y ago
        Could you explain to me (in steps) how I would go about calculating how much VRAM I would need to run, say, Mistral 8x7B?