4 ms·
If you have a 24 gb 3090. Try out qwen:30b-a3b-instruct-2507-q4_K_M ( ollama ) It's pretty good.
by NicoJuicy 1y ago
If you have a 24 gb 3090. Try out qwen:30b-a3b-instruct-2507-q4_K_M ( ollama )
It's pretty good.
- jszymborski 1y agoDon't need a 3090, it runs really fast on an RTX 2080 too.
- naabb 1y agotbf I also run that on a 16GB 5070TI at 25T/S, it's amazing how fast it runs on consumer grade hardware. I think you could push up to a bigger model but I don't know enough about local llama.