6 ms·

If you want to run LLMs locally then the localllama community is your friend: https://old.reddit.com/r/LocalLLaMA/ https://old.reddit.com/r/LocalLLaMA/ In gene

If you want to run LLMs locally then the localllama community is your friend: https://old.reddit.com/r/LocalLLaMA/ https://old.reddit.com/r/LocalLLaMA/ In general there's no "best" LLM model, all of them will have some strengths and weaknesses. There are a bunch of good picks; for example: > DeepSeek-R1-0528-Qwen3-8B - https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B Released today; probably the best reasoning model in 8B size. > Qwen3 - https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2... Recently released. Hybrid thinking/non-thinking models with really great performance and plethora of sizes for every hardware. The Qwen3-30B-A3B can even run on CPU with acceptable speeds. Even the tiny 0.6B one is somewhat coherent, which is crazy.

deleted 1y ago
[deleted]
ignoramous 1y ago
> DeepSeek-R1-0528-Qwen3-8B https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B ... Released today; probably the best reasoning model in 8B size. ... we distilled the chain-of-thought from DeepSeek-R1-0528 to post-train Qwen3-8B Base, obtaining DeepSeek-R1-0528-Qwen3-8B ... on AIME 2024, surpassing Qwen3-8B by +10.0% & matching the performance of Qwen3-235B-thinking. Wild how effective distillation is turning out to be. No wonder, most shops have begun to "hide" CoT now: https://news.ycombinator.com/item?id=41525201 https://news.ycombinator.com/item?id=41525201
- arresin 1y ago
  > Beyond its improved reasoning capabilities, this version also offers a reduced hallucination rate, enhanced support for function calling, and better experience for vibe coding. Thank you for thinking of the vibe coders.
luke-stanley 1y ago
> Released today; probably the best reasoning model in 8B size. Actually DeepSeek-R1-0528-Qwen3-8B was uploaded Thursday (yesterday) at 11 AM UTC / 7 PM CST. I had to check if a new version came out since! I am waiting for the other sizes! ;D
ivape 1y ago
I'd also recommend you go with something like 8b, so you can have the other 8GB of vram for a decent sized context window. There's tons of good 8b ones, as mentioned above. If you go for the largest model you can fit, you'll have slower inference (as you pass in more tokens) and smaller context.
- svachalek 1y ago
  8b is the number of parameters. The most common quant is 4 bits per parameter so 8b params is roughly 4GB of VRAM. (Typically more like 4.5GB)
  - rhdunn 1y ago
    The number of quantized bits is a trade off between size and quality. Ideally you should be aiming for a 6-bit or 5-bit model. I've seen some models be unstable at 4-bit (where they will either repeat words or start generating random words). Anything below 4-bits is usually not worth it unless you want to experiment with running a 70B+ model -- though I don't have any experience of doing that, so I don't know how well the increased parameter size balances the quantization. See https://github.com/ggml-org/llama.cpp/pull/1684 https://github.com/ggml-org/llama.cpp/pull/1684 and https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9 https://gist.github.com/Artefact2/b5f810600771265fc1e3944228... for comparisons between quantization levels.
    - kouteiheika 1y ago
      > The number of quantized bits is a trade off between size and quality. Ideally you should be aiming for a 6-bit or 5-bit model. I've seen some models be unstable at 4-bit (where they will either repeat words or start generating random words). Note that that's a skill issue of whoever quantized the model. In general quantization even as low as 3-bit can be almost loseless when you do quantization-aware finetuning[1] (and apparently you don't even need that many training tokens), but even if you don't want to do any extra training you can be smart as to which parts of the model you're quantizing and by how much to minimize the damage (e.g. in the worst case over-quantizing even a single weight can have disastrous consequences[2]) Some time ago I ran an experiment where I finetuned a small model while quantizing parts of it to 2-bits to see which parts are most sensitive (the numbers are the final loss; lower is better): 1.5275 mlp.downscale 1.5061 mlp.upscale 1.4665 mlp.gate 1.4531 lm_head 1.3998 attn.out_proj 1.3962 attn.v_proj 1.3794 attn.k_proj 1.3811 input_embedding 1.3662 attn.q_proj 1.3397 unquantized baseline So as you can see quantizing some parts of the model affects it more strongly. The downprojection in the MLP layers is the most sensitive part of the model (which also matches with what [2] found), so it makes sense to quantize this part of the model less and instead quantize other parts more strongly. But if you'll just do the naive "quantize everything in 4-bit" then sure, you might get broken models. [1] - https://arxiv.org/pdf/2502.02631 https://arxiv.org/pdf/2502.02631 [2] - https://arxiv.org/pdf/2411.07191 https://arxiv.org/pdf/2411.07191
moffkalast 1y ago
Yes at this point it's starting to become almost a matter of how much you like the model's personality since they're all fairly decent. OP just has to start downloading and trying them out. With 16GB one can do partial DDR5 offloading with llama.cpp and run anything up to about 30B (even dense) or even more at a "reasonable" speed for chat purposes. Especially with tensor offload. I wouldn't count Qwen as that much of a conversationalist though. Mistral Nemo and Small are pretty decent. All of Llama 3.X are still very good models even by today's standards. Gemma 3s are great but a bit unhinged. And of course QwQ when you need GPT4 at home. And probably lots of others I'm forgetting.
diggan 1y ago
> If you want to run LLMs locally then the localllama community is your friend: https://old.reddit.com/r/LocalLLaMA/ https://old.reddit.com/r/LocalLLaMA/ For folks new to reddit, it's worth noting that LocalLlama, just like the rest of the internet but especially reddit, is filled with misinformed people spreading incorrect "facts" as truth, and you really can't use the upvote/downvote count as an indicator of quality or how truthful something is there. Something that is more accurate but put in a boring way will often be downvoted, while straight up incorrect but funny/emotional/"fitting the group think" comments usually get upvoted. For us who've spent a lot of time on the web, this sort of bullshit detector is basically built-in at this point, but if you're new to places where the group think is so heavy as on reddit, it's worth being careful taking anything at face value.
- ddoolin 1y ago
  This is entirely why I can't bring myself to use it. The groupthink and virtue signaling is intense, when it's not just extremely low effort crud that rises to the top. And yes, before anyone says, I know, "curate." No, thank you.
  - dingnuts 1y ago
    Friend, this website is EXACTLY the same
    - rafterydj 1y ago
      I understand that the core similarities are there, but I disagree. The comparisons have been around since I started browsing HN years ago. The moderation on this site, for one, emphasizes constructive conversation and discussion in a way that most subreddits can only dream of. It also helps that the target audience has been filtered with that moderation, so over time this site (on average) skews more technical and informed.
      - ffsm8 1y ago
        Frankly, no. As an obvious example that can be stated nowadays: musk has always been an over-promising liar. Eg just look at the 2012+ videos of thunderf00t. Yet people were literally banned here just for pointing out that he hasn't actually delivered on anything in the capacity he promised until he did the salute. It's pointless to list other examples, as this page is- as dingnuts pointed out - exactly the same and most people aren't actually willing to change their opinion based on arguments. They're set in their opinions and think everyone else is dumb.
nico 1y ago
What do you recommend for coding with aider or roo? Sometimes it’s hard to find models that can effectively use tools
- cchance 1y ago
  I havent found one good locally, i use DeepSeek r1 0528 its slow but free and really good at coding (openrouter has it free currently)
  - nico 1y ago
    Oh wow, just checked this leaderboard, r1 0528 looks really good https://leaderboard.techfren.net/ https://leaderboard.techfren.net/ Thanks for the recommendation, will try it out
xtracto 1y ago
There was this great post the other day [1] showing that with llama-cpp you could offload some specific tensors to the CPU and maintain good performance. That's a good way to use lare(ish) models in commodity hardware. Normally with llama-cpp you specifiy how many (full) layers you want to put in GPU (-ngl) . But CPU-offloading specific tensors that don't require heavy computation , saves GPU space without affecting speed that much. I've also read a paper on loading only "hot" neurons into the cpu [2] . The future of home AI looks so cool! [1] https://www.reddit.com/r/LocalLLaMA/comments/1ki7tg7/dont_offload_gguf_layers_offload_tensors_200_gen/ https://www.reddit.com/r/LocalLLaMA/comments/1ki7tg7/dont_of... [2] https://arxiv.org/abs/2312.12456 https://arxiv.org/abs/2312.12456

deleted 1y ago

[deleted]

ignoramous 1y ago

> DeepSeek-R1-0528-Qwen3-8B https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B ... Released today; probably the best reasoning model in 8B size. ... we distilled the chain-of-thought from DeepSeek-R1-0528 to post-train Qwen3-8B Base, obtaining DeepSeek-R1-0528-Qwen3-8B ... on AIME 2024, surpassing Qwen3-8B by +10.0% & matching the performance of Qwen3-235B-thinking. Wild how effective distillation is turning out to be. No wonder, most shops have begun to "hide" CoT now: https://news.ycombinator.com/item?id=41525201 https://news.ycombinator.com/item?id=41525201

arresin 1y ago
> Beyond its improved reasoning capabilities, this version also offers a reduced hallucination rate, enhanced support for function calling, and better experience for vibe coding. Thank you for thinking of the vibe coders.

luke-stanley 1y ago

> Released today; probably the best reasoning model in 8B size. Actually DeepSeek-R1-0528-Qwen3-8B was uploaded Thursday (yesterday) at 11 AM UTC / 7 PM CST. I had to check if a new version came out since! I am waiting for the other sizes! ;D

ivape 1y ago

I'd also recommend you go with something like 8b, so you can have the other 8GB of vram for a decent sized context window. There's tons of good 8b ones, as mentioned above. If you go for the largest model you can fit, you'll have slower inference (as you pass in more tokens) and smaller context.

svachalek 1y ago
8b is the number of parameters. The most common quant is 4 bits per parameter so 8b params is roughly 4GB of VRAM. (Typically more like 4.5GB)
- rhdunn 1y ago
  The number of quantized bits is a trade off between size and quality. Ideally you should be aiming for a 6-bit or 5-bit model. I've seen some models be unstable at 4-bit (where they will either repeat words or start generating random words). Anything below 4-bits is usually not worth it unless you want to experiment with running a 70B+ model -- though I don't have any experience of doing that, so I don't know how well the increased parameter size balances the quantization. See https://github.com/ggml-org/llama.cpp/pull/1684 https://github.com/ggml-org/llama.cpp/pull/1684 and https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9 https://gist.github.com/Artefact2/b5f810600771265fc1e3944228... for comparisons between quantization levels.
  - kouteiheika 1y ago
    > The number of quantized bits is a trade off between size and quality. Ideally you should be aiming for a 6-bit or 5-bit model. I've seen some models be unstable at 4-bit (where they will either repeat words or start generating random words). Note that that's a skill issue of whoever quantized the model. In general quantization even as low as 3-bit can be almost loseless when you do quantization-aware finetuning[1] (and apparently you don't even need that many training tokens), but even if you don't want to do any extra training you can be smart as to which parts of the model you're quantizing and by how much to minimize the damage (e.g. in the worst case over-quantizing even a single weight can have disastrous consequences[2]) Some time ago I ran an experiment where I finetuned a small model while quantizing parts of it to 2-bits to see which parts are most sensitive (the numbers are the final loss; lower is better): 1.5275 mlp.downscale 1.5061 mlp.upscale 1.4665 mlp.gate 1.4531 lm_head 1.3998 attn.out_proj 1.3962 attn.v_proj 1.3794 attn.k_proj 1.3811 input_embedding 1.3662 attn.q_proj 1.3397 unquantized baseline So as you can see quantizing some parts of the model affects it more strongly. The downprojection in the MLP layers is the most sensitive part of the model (which also matches with what [2] found), so it makes sense to quantize this part of the model less and instead quantize other parts more strongly. But if you'll just do the naive "quantize everything in 4-bit" then sure, you might get broken models. [1] - https://arxiv.org/pdf/2502.02631 https://arxiv.org/pdf/2502.02631 [2] - https://arxiv.org/pdf/2411.07191 https://arxiv.org/pdf/2411.07191

moffkalast 1y ago

Yes at this point it's starting to become almost a matter of how much you like the model's personality since they're all fairly decent. OP just has to start downloading and trying them out. With 16GB one can do partial DDR5 offloading with llama.cpp and run anything up to about 30B (even dense) or even more at a "reasonable" speed for chat purposes. Especially with tensor offload. I wouldn't count Qwen as that much of a conversationalist though. Mistral Nemo and Small are pretty decent. All of Llama 3.X are still very good models even by today's standards. Gemma 3s are great but a bit unhinged. And of course QwQ when you need GPT4 at home. And probably lots of others I'm forgetting.

diggan 1y ago

> If you want to run LLMs locally then the localllama community is your friend: https://old.reddit.com/r/LocalLLaMA/ https://old.reddit.com/r/LocalLLaMA/ For folks new to reddit, it's worth noting that LocalLlama, just like the rest of the internet but especially reddit, is filled with misinformed people spreading incorrect "facts" as truth, and you really can't use the upvote/downvote count as an indicator of quality or how truthful something is there. Something that is more accurate but put in a boring way will often be downvoted, while straight up incorrect but funny/emotional/"fitting the group think" comments usually get upvoted. For us who've spent a lot of time on the web, this sort of bullshit detector is basically built-in at this point, but if you're new to places where the group think is so heavy as on reddit, it's worth being careful taking anything at face value.

ddoolin 1y ago
This is entirely why I can't bring myself to use it. The groupthink and virtue signaling is intense, when it's not just extremely low effort crud that rises to the top. And yes, before anyone says, I know, "curate." No, thank you.
- dingnuts 1y ago
  Friend, this website is EXACTLY the same
  - rafterydj 1y ago
    I understand that the core similarities are there, but I disagree. The comparisons have been around since I started browsing HN years ago. The moderation on this site, for one, emphasizes constructive conversation and discussion in a way that most subreddits can only dream of. It also helps that the target audience has been filtered with that moderation, so over time this site (on average) skews more technical and informed.
    - ffsm8 1y ago
      Frankly, no. As an obvious example that can be stated nowadays: musk has always been an over-promising liar. Eg just look at the 2012+ videos of thunderf00t. Yet people were literally banned here just for pointing out that he hasn't actually delivered on anything in the capacity he promised until he did the salute. It's pointless to list other examples, as this page is- as dingnuts pointed out - exactly the same and most people aren't actually willing to change their opinion based on arguments. They're set in their opinions and think everyone else is dumb.

nico 1y ago

What do you recommend for coding with aider or roo? Sometimes it’s hard to find models that can effectively use tools

cchance 1y ago
I havent found one good locally, i use DeepSeek r1 0528 its slow but free and really good at coding (openrouter has it free currently)
- nico 1y ago
  Oh wow, just checked this leaderboard, r1 0528 looks really good https://leaderboard.techfren.net/ https://leaderboard.techfren.net/ Thanks for the recommendation, will try it out

xtracto 1y ago

There was this great post the other day [1] showing that with llama-cpp you could offload some specific tensors to the CPU and maintain good performance. That's a good way to use lare(ish) models in commodity hardware. Normally with llama-cpp you specifiy how many (full) layers you want to put in GPU (-ngl) . But CPU-offloading specific tensors that don't require heavy computation , saves GPU space without affecting speed that much. I've also read a paper on loading only "hot" neurons into the cpu [2] . The future of home AI looks so cool! [1] https://www.reddit.com/r/LocalLLaMA/comments/1ki7tg7/dont_offload_gguf_layers_offload_tensors_200_gen/ https://www.reddit.com/r/LocalLLaMA/comments/1ki7tg7/dont_of... [2] https://arxiv.org/abs/2312.12456 https://arxiv.org/abs/2312.12456