4 ms·

> Qwen3-Coder is available in multiple sizes, but we’re excited to introduce its most powerful variant first I'm most excited for the smaller sizes because I'm

• segmondy 1y ago
  small models can never match bigger models, the bigger models just know more and are smarter. the smaller models can get smarter, but as they do, the bigger models get smart too. HN is weird because at one point this was the location where I found the most technically folks, and now for LLM I find them at reddit. tons of folks are running huge models, get to researching and you will find out you can realistically host your own.
  • nico 1y ago
    > and now for LLM I find them at reddit. tons of folks are running huge models Very interesting. Any subs or threads you could recommend/link to? Thanks
    • ActorNightly 1y ago
      Basically just run ollama and run the quantized models. Don't expect high generation speeds though.
    • segmondy 1y ago
      join us at r/LocalLlama
  • BriggyDwiggs42 1y ago
    The small model only needs to get as good as the big model is today, not as the big model is in the future.
  • pxc 1y ago
    > small models can never match bigger models, the bigger models just know more and are smarter. They don't need to match bigger models, though. They just need to be good enough for a specific task! This is more obvious when you look at the things language models are best at, like translation. You just don't need a super huge model for translation, and in fact you might sometimes prefer a smaller one because being able to do something in real-time, or being able to run on a mobile device, is more important than marginal accuracy gains for some applications. I'll also say that due to the hallucination problem, beyond whatever knowledge is required for being more or less coherent and "knowing" what to write in web search queries, I'm not sure I find more "knowledgeable" LLMs very valuable. Even with proprietary SOTA models hosted on someone else's cloud hardware, I basically never want an LLM to answer "off the dome"; IME it's almost always wrong! (Maybe this is less true for others whose work focuses on the absolute most popular libraries and languages, idk.) And if an LLM I use is always going to be consulting documentation at runtime, maybe that knowledge difference isn't quite so vital— summarization is one of those things that seems much, much easier for language models than writing code or "reasoning". All of that is to say: Sure, bigger is better! But for some tasks, my needs are still below the ceiling of the capabilities of a smaller model, and that's where I'm focusing on local usage. For now that's mostly language-focused tasks entirely apart from coding (translation, transcription, TTS, maybe summarization). It may also include simple coding tasks today (e.g., fancy auto-complete, "ghost-text" style). I think it's reasonable to hope that it will eventually include more substantial programming tasks— even if larger models are still preferable for more sophisticated tasks (like "vibe coding", maybe). If I end up having a lot of fun, in a year or two I'll probably try to put together a machine that can indeed run larger models. :)
    • conradkay 1y ago
      For coding though it seems like people are willing to pay a lot more for a slightly better model.
      • omeid2 1y ago
        The problem with local vs remote isn't so much about paid. It is about compliance and privacy.
  • Eggpants 1y ago
    The large models are using tools/functions to make them useful. Sooner or later open source will provide a good set of tools/functions for coding as well.
  • mlyle 1y ago
    > HN is weird because at one point this was the location where I found the most technically folks, and now for LLM I find them at reddit. Is this an effort to chastise the viewpoint advanced? Because his viewpoint makes sense to me: I can run biggish models on my 128GB Macbook but not huge ones-- even 2b quantized ones suck too many resources. So I run a combination of local stuff and remote stuff depending upon various factors (cost, sensitivity of information, convenience/whether I'm at home, amount of battery left, etc ;) Yes, bigger models are better, but often smaller is good enough.
  • wkat4242 1y ago
    Well yes tons of people are running them but they're all pretty well off. I don't have 10-20k$ to spend on this stuff. Which is about the minimum to run a 480B model, with huge quantisation. And pretty slow because for that price all you get is an old Xeon with a lot of memory or some old nvidia datacenter cards. If you want a good setup it will cost a lot more. So small models it is. Sure, the bigger models are better but because the improvements come so fast it means I'm only 6 months to a year behind the big ones at any time. Is that worth 20k? For me no.
  • y1n0 1y ago
    I'd be interested in smaller models that were less general, with a training corpus more concentrated. A bash scripting model, or a clojure model, or a zig model, etc.
  • ActorNightly 1y ago
    Not really true. Gemma from Google with quantized aware training does an amazing job. Under the hood, the way it works, is that when you have final probabilities, it really doesn't matter if the most likely token is selected with 59% or 75% - in either case it gets selected. If the 59% case gets there with smaller amount of compute, and that holds across the board for the training set, the model will have similar performance. In theory, it should be possible to narrow down models even smaller to match the performance of big models, because I really doubt that you do need transformers for every single forward pass. There are probably plenty of shortcuts you can take in terms of compute for sets of tokens in the context. For example, coding structure is much more deterministic than natural text, so you probably don't need as much compute to generate accurate code. You do need a big model first to train a small model though. As for running huge models locally, its not enough to run them, you need good throughput as well. If you spend $2k on a graphics card, that is way more expensive than realistic usage with a paid API, and slower output as well.
  • ants_everywhere 1y ago
    There's a niche for small-and-cheap, especially if they're fast. I was surprised in the AlphaEvolve paper how much they relied on the flash model because they were optimizing for speed of generating ideas.
  • otabdeveloper4 1y ago
    > small models can never match bigger models, the bigger models just know more and are smarter Untrue. The big important issue for LLMs is hallucination, and making your model bigger does little to solve it. Increasing model size is a technological dead end. The future advanced LLM is not that.
  • giorgioz 1y ago
    which sub-reddits do you recommend?
• KronisLV 1y ago
  > I'm most excited for the smaller sizes because I'm interested in locally-runnable models that can sometimes write passable code, and I think we're getting close. Likewise, I found that the regular Qwen3-30B-A3B worked pretty well on a pair of L4 GPUs (60 tokens/second, 48 GB of memory) which is good enough for on-prem use where cloud options aren't allowed, but I'd very much like a similar code specific model, because the tool calling in something like RooCode just didn't work with the regular model. In those circumstances, it isn't really a comparison between cloud and on-prem, it's on-prem vs nothing.
  • NitpickLawyer 1y ago
    Give devstral a try, fp8 should fit in 48GB, it was surprisingly good for a 24B local model, w/ cline/roo. Handles itself well, doesn't get stuck much, most of the things work OK (considering the size ofc)
    • KronisLV 1y ago
      I did! I do think Mistral models are pretty okay, but even the 4-bit quantized version runs at about 16 tokens/second, more or less usable but a biiiig step down from the MoE options. Might have to swap out Ollama for vLLM though and see how different things are.
      • NitpickLawyer 1y ago
        > Might have to swap out Ollama for vLLM though and see how different things are. Oh, that might be it. Using gguf is slower than say AWQ if you want 4bit, or fp8 if you want the best quality (especially on Ada arch that I think your GPUs are). edit: vLLM is better for Tensor Parallel and also better for batched inference, some agentic stuff can do multiple queries in parallel. We run devstral fp8 on 2x A6000 (old, not even Ada) and even with marlin kernels we get ~35-40 t/s gen and 2-3k pp on a single session, with ~4 parallel sessions supported at full context. But in practice it can work with 6 people using it concurrently, as not all sessions get to the max context. You'd get 1/2 of that for 2x L4, but should see higher t/s in generation since you have Ada GPUs (native support for fp8).
• larodi 1y ago
  Been using ggerganov’s llama vscode plugin with the smaller 2.5 models and it actually works super nice on a M3 Max
  • pxc 1y ago
    What languages do you work in? How much code do you keep? Do you end up using it as scaffolding and rewriting it, it leaving most of it as is?
    • larodi 1y ago
      Languages: JS/TS, C/C++, Shader Code, Some ESP Arduino code. Not counting all the boilerplate and CSS that I dont care about too much. It very much reminds of tabbing autocomplete with IntelliSense step by step, but in a more diffusion-like way. but my tool-set is a mixture of agentic and autocomplete, not 100% of each. I try to keep a clear focus of the architecture, and actually own the code by reading most of it, keeping straight the parts of the code the way i like.
  • kimsia 1y ago
    I'm on a m1 max with 64gb ram, but i never use this vscode plugin before. Should I try? Is this the one? https://github.com/ggml-org/llama.vscode https://github.com/ggml-org/llama.vscode it sems to be built for code completion rather than outright agent mode
    • larodi 1y ago
      It is RAG for your codebase, and provides code completion. The gain is the local inference, and is actually useful with smaller models. The plugin itself provides chat also, but my gut feeling is that ggerganov runs several models at the some time, given he uses a 192gb machine. Have not tried this scenario yet, but looking at my API bill I’m probably going to try 100% local dev at some point. Besides vibe coding with existing tools seems to not work that good for enterprise size codebases.
• LinXitoW 1y ago
  Currently, the goal of everyone is creating one master model to rule them all, so we haven't seen too much specialization. I wonder how much more efficient smaller models could be if we created language specialized models. It feels intuitively obvious (so maybe wrong?) that a 32B Java Coder would be far better at coding Java than a generalist 32B Coder.
  • californical 1y ago
    I’ll fill the role to push back on your Java coder idea! First, Java code tends to be written a certain way, and for certain goals and business domains. Let’s say 90% of modern Java is a mix of: * students learning to program and writing algorithms * corporate legacy software from non-tech focused companies If you want to build something that is uncommon in that subset, it will likely struggle due to a lack of training data. And if you wanted to build something like a game, the majority of your training data is going to be based on ancient versions of Java, back when game development was more common in Java. Comparatively, including C in your training data gives you exposure to a whole separate set of domain data for training, like IoT devices, kernels, etc. Including Go will likely include a lot more networking and infrastructure code than Java would have had, which means there is also more context to pull from in what networking services expect. Code for those domains follow different patterns, but the concepts can still be useful in writing Java code. Now, there may be a middle ground where you could have a model that is still general for many coding languages, but given extra data and fine-tuning focused on domain-specific Java things — like more of a “32B CorporateJava Coder” model — based around the very specific architecture of Spring. And you’d be willing to accept that model to fail at writing games in Java. It’s interesting to think about for sure - but I do feel like domain-specific might be more useful than language-specific
    • pxc 1y ago
      Don't we also find with natural languages that focusing on training data from only a single language doesn't actually result in better writing in the target language, either?
  • 1899-12-30 1y ago