4 ms·

It's good to see more competition, and open source, but I'd be much more excited to see what level of coding and reasoning performance can be wrung out of a muc

It's good to see more competition, and open source, but I'd be much more excited to see what level of coding and reasoning performance can be wrung out of a much smaller LLM + agent as opposed to a trillion parameter one. The ideal case would be something that can be run locally, or at least on a modest/inexpensive cluster. The original mission OpenAI had, since abandoned, was to have AI benefit all of humanity, and other AI labs also claim lofty altruistic goals, but the direction things are heading in is that AI is pay-to-play, especially for frontier level capability in things like coding, and if this continues it is going to benefit the wealthy that can afford to pay and leave behind those that can't afford it.

pshirshov 1y ago
> The ideal case would be something that can be run locally, or at least on a modest/inexpensive cluster. 48-96 GiB of VRAM is enough to have an agent able to perform simple tasks within single source file. That's the sad truth. If you need more your only options are the cloud or somehow getting access to 512+ GiB
a-dub 1y ago
"open source" means there should be a script that downloads all the training materials and then spins up a pipeline that trains end to end. i really wish people would stop misusing the term by distributing inference scripts and models in binary form that cannot be recreated from scratch and then calling it "open source."
- danielmarkbruce 1y ago
  "open source" has come to mean "open weight" in model land. It is what it is. Words are used for communication, you are the one misusing the words. You can update the weights of the model, continue to train, whatever. Nobody is stopping you.
  - a-dub 1y ago
    it still doesn't sit right. sure it's different in terms of mutability from say, compiled software programs, but it still remains not end to end reproducible and available for inspection. these words had meaning long before "model land" became a thing. overloading them is just confusing for everyone.
    - danielmarkbruce 1y ago
      It's not confusing, no one is really confused except the people upset that the meaning is different in a different context. On top of that, in many cases a company/group/whoever can't even reproduce the model themselves. There are lots of sources of non-determinism even if folks are doing things in a very buttoned up manner. And, when you are training on trillions of tokens, you are likely training on some awful sounding stuff - "Facebook is trained llama 4 on nazi propaganda!" is not what they want to see published. How about just being thankful?
      - a-dub 1y ago
        i disagree. words matter. the whole point of open source is that anyone can look and see exactly how the sausage is made. that is the point. that is why the word "open" is used. ...and sure, compiling gcc is nondeterministic too, but i can still inspect the complete source from where it comes because it is open source, which means that all of the source materials are available for inspection.
        danielmarkbruce 1y ago
        The point of open source in software is as you say. It's just not the same thing though. Using words and phrases differently in different fields is common.
        a-dub 1y ago
        ...and my point is that it should be. the practice of science itself would be far stronger if it took more pages from open source software culture.
  - mensetmanusman 1y ago
    Weights are meaningless without training data and source.
    - antiframe 1y ago
      I get a lot of meaning out of weights and source (without the training data), not sure about you. Calling it meaningless seems like exaggeration.
      - mensetmanusman 1y ago
        Can you change the weights to improve?
        HarHarVeryFunny 1y ago
        You can fine tune without the original training data, which for a large LLM is typically going to mean using LoRA - keeping the original weights unchanged and adding separate fine-tuning weights.
        danielmarkbruce 1y ago
        it's a bunch of numbers. Of course you can change them.
- nyrp 1y ago
  > binary form that cannot be recreated from scratch Back in my day, we called it "freeware"
  - poly2it 1y ago
    You have more rights over a freely licensed binary file than over a freeware file.
- moffkalast 1y ago
  I'd agree but we're beyond hopelessly idealistic. That sort of approach only helps your competition who will use it to build a closed product and doesn't give anything of worth to people who want to actually use the model because they have no means to train it. Hell most people can barely scrape up enough hardware to even run inference. Reproducing models is also not very ecological in when it comes down to it, do we really all need to redo the training that takes absurd amounts of power just to prove that it works? At least change the dataset to try and get a better result and provide another datapoint, but most people don't have the knowhow for it anyway. Nvidia does try this approach sometimes funnily enough, they provide cool results with no model in hopes of getting people to buy their rented compute and their latest training platform as a service...
  - TheBicPen 1y ago
    > I'd agree but we're beyond hopelessly idealistic. That sort of approach only helps your competition who will use it to build a closed product That same argument can be applied to open-source (non-model) software, and is about as true there. It comes down to the business model. If anything, crating a closed-sourced copy of a piece of FOSS software is easier than an AI model since running a compiler doesn't cost millions of dollars.
- HarHarVeryFunny 1y ago
  Yeah, but "open weights" never seems to have taken off as a better description, and even if you did have the training data + recipe, the compute cost makes training it yourself totally impractical. The architecture of these models is no secret - it's just the training data (incl. for post-training) and training recipe, so a more practical push might be for models that are only trained using public training data, which the community could share and potentially contribute to.
- emsign 1y ago
  They'd have to publish or link the training data, which is full of copyrighted material. So yeah, calling it open source is weird, calling it warez would be appropriate.
  - oceanplexian 1y ago
    They should release it then. China doesn't have a problem stealing and distributing copyrighted material.
- andai 1y ago
  The meaning of Open Source 1990: Free Software 2000: Open Source: Finally we sanitized ourselves of that activism! It was scaring away customers! 2010: Source is available (under our very restrictive license) 2020: What source?
  - rurban 1y ago
    2025: What prompt?
pdyc 1y ago
I think that's where prompt engineering would be needed. Bigger models produce good output even with ambiguous prompts. Getting similar output from smaller models is art,
jstummbillig 1y ago
> I'd be much more excited to see what level of coding and reasoning performance can be wrung out of a much smaller LLM + agent Well, I think you are seeing that already? It's not like these models don't exist and they did not try to make them good, it's just that the results are not super great. And why would they be? Why would the good models (that are barely okay at coding) be big, if it was currently possible to build good models, that are small? Of course, new ideas will be found and this dynamic may drastically change in the future, but there is no reason to assume that people who work on small models find great optimizations that frontier models makers, who are very interested in efficient models, have not considered already.
- HarHarVeryFunny 1y ago
  Sure, but that's the point ... today's locally runnable models are a long way behind SOTA capability, so it'd be nice to see more research and experimentation in that direction. Maybe a zoo of highly specialized small models + agents for S/W development - one for planning, one for coding, etc?
  - Uehreka 1y ago
    If I understand transformers properly, this is unlikely to work. The whole point of “Large” Language Models is that you primarily make them better by making them larger, and when you do so, they get better at both general and specific tasks (so there isn’t a way to sacrifice generality but keep specific skills when training a small models). I know a lot of people want this (Apple really really wants this and is pouring money into it) but just because we want something doesn’t mean it will happen, especially if it goes against the main idea behind the current AI wave. I’d love to be wrong about this, but I’m pretty sure this is at least mostly right.
    - HarHarVeryFunny 1y ago
      Yeah - the whole business model of companies like OpenAI and Anthropic, at least at the moment, seems to be that the models are so big that you need to run them in the cloud with metered access. Maybe that could change in the future to sale or annual licence business model if running locally became possible. I think scale helps for general tasks where the breadth of capability may be needed, but it's not so clear that this needed for narrow verticals, especially something like coding (knowing how to fix car engines, or distinguish 100 breeds of dog is not of much use!).
pzo 1y ago
Even if pay-to-play companies like moonshootai help to pay less. You can run previous kimi k2 non-thinking model e.g. on groq with 720tok/s and for $1/$3 for million input/output tokens. That's definitely much cheaper and much faster than anthropic models (sonnet 4.5: 60tok/s, $3/$15)
stingraycharles 1y ago
With these things it’s always both at the same time: these super grandiose SOTA models are only making improvements mostly because of optimizations, and they’re just scaling our as far as they can. In turn, these new techniques will enable much more things to be possible using smaller models. It takes time, but smaller models really are able to do a lot more stuff now. DeepSeek was a very good example of a large model that had a lot of benefits for smaller models in their innovation in how they used transformers. Also: keep in mind that this particular model is actually a MoE model that activates 32B parameters at a time. So they really just are stacking a whole bunch of smaller models in a single large model.
ares623 1y ago
I don't understand. We already have that capability in our skulls. It's also "already there", so it would be a waste to not use it.
- HarHarVeryFunny 1y ago
  Software development is one of the areas where LLMs really are useful, whether that's vibe coding disposable software, or more structured use for serious development. I've been a developer for 40+ years, and very good at it, but for some tasks it's not about experience or overcoming complexity - just a bunch of grunt work that needs to come together. The other day I vibe coded a prototype app, just for one-time demo use, in less than 15 min that probably would have taken a week to write by hand, assuming one was already familiar with the tech stack. Developing is fun, and a brain is a terrible thing to waste, but today not using LLMs where appropriate for coding doesn't make any sense if you value your time whatsoever.
- tonyhart7 1y ago
  "I don't understand. We already have that capability in our skulls. It's also "already there", so it would be a waste to not use it." seems like you are here that not understand this Company want to replace human and won't need to pay massive salary
  - ares623 1y ago
    I understand the companies wanting it. I hate it, but I understand. I don’t understand the humans wanting to be replaced though.
    - tonyhart7 1y ago
      "I don’t understand the humans wanting to be replaced though." because human that replace these job isnt the same human that got cut???? human that can replace these jobs would be rich
maciejgryka 1y ago
I think it’s going to be a while before we see small models (defined roughly as “runnable on reasonable consumer hardware”) do a good job at general coding tasks. It’s a very broad area! You can do some specific tasks reasonably well (eg I distilled a toy git helper you can run locally here https://github.com/distil-labs/gitara https://github.com/distil-labs/gitara), but “coding” is such a big thing that you really need a lot of knowledge to do it well.
0xjmp 1y ago
This happens top down historically though, yes? Someone releases a maxed out parameter model. Another distillates it. Another bifurcates it. With some nuance sprinkled in.
wordpad 1y ago
The electricity cost to run these models locally is already more than equivalent API cost.
- ImPostingOnHN 1y ago
  Privacy is minimally valued by most, but not by all.
- HarHarVeryFunny 1y ago
  That's going to depend on how small the model can be made, and how much you are using it. If we assume that running locally meant running on a 500W consumer GPU, then the electricity cost to run this non-stop 8 hours a day for 20 days a month (i.e. "business hours") would be around $10-20. This is about the same as OpenAI or Anthropics $20/mo plans, but for all day coding you would want their $100 or $200/mo plans, and even these will throttle you and/or require you to switch to metered pricing when you hit plan limits.
  - wordpad 1y ago
    Neither $20 nor $200 plans cover any API costs. At $0.17 per million tokens for the smallest gpt model that's still faster rand more powerful than anything you can run locally and cheaper in kilowatts per hour than it would cost you to run locally even if you could.
twotwotwo 1y ago
I think there is a lot of progress on efficient useful models recently. I've seen GLM-4.6 getting mention for good coding results from a model that's much smaller than Kimi (~350b params) and seen it speculated that Windsurf based their new model on it. This Kimi release is natively INT4, with quantization-aware training. If that works--if you can get really good results from four-bit parameters--it seems like a really useful tool for any model creator wanting efficient inference. DeepSeek's v3.2-Exp uses their sparse attention technique to make longer-context training and inference more efficient. Its output's being priced at 60% less than v3.1 (though that's an imperfect indicator of efficiency). They've also quietly made 'thinking' mode need fewer tokens since R1, helping cost and latency. And though it's on the proprietary side, Haiku 4.5 approaching Sonnet 4 coding capability (at least on benches Anthropic released) also suggests legitimately useful models can be much smaller than the big ones. There's not yet a model at the level of any of the above that's practical for many people to run locally, though I think "efficient to run + open so competing inference providers can run it" is real progress. More important it seems like there's a good trendline towards efficiency, and a bunch of techniques are being researched and tested that, when used together, could make for efficient higher-quality models.
benjiro 1y ago
What i do not understand is why we are not seeing specialized models that go down to single experts. I do not need models that know how to program in Python, Rust, ... when i only use Go and Html. So we are we not seeing models that have very specialized experts, where for instance: * General interpreter model, that holds context/memory * Go Model * Html model if there is space in memory. * SQL model if there is space in memory. If there is no space, the GIM swamp out the Go model, for the HTML model, depending on where it is in Agent tasks or Edit/Ask code its overviewing. Because the models are going to be very small, switching in and out of memory will be ultra fast But most of the time we get very big Expert models, that still are very generalized over a entire field. This can then be extended that if you have the memory, models combine their output with tasks... Maybe i am just too much of a noob in the field of understanding how LLMs work, but it feels like people are too often running after large models that companies like Anthropic/OpenAI etc deploy. I understand why those big companies use insane big models. They have the money to load them up over a cluster, have the fast interconnect, and for them its more efficient. But from the bits and pieces that i see, people are more and more going to tons of small 1 a 2B models to produce better results. See my argument above. Like i said, never really gone beyond paying for my CoPilot subscription and running a bit of Ollama at home (don't have the time for the big stuff).
- EMM_386 1y ago
  I think one of the issues is that LLMs can't have a "Go" model and an "HTML model". I mean, they can but what would that contain? It's not the language-specific features that make models large. When models work on your code base, they do not "see" things like this, which is why they can go through an entire code base with variable names they have never seen before, function signatures they have never seen before, and directory structures that have never seen before and not have a problem. You need that "this is a variable, which is being passed to a function which recursively does ..." part. This is not something language specific, it's the high level understanding of how languages and systems operate. A variable is a variable whether in JavaScript or C++ and LLMs can "see" it as such. The details are different but it's that layer of "this is a software interface", "this is a function pointer" is outside of the "Go" or "Python" or "C#" model. I don't know how large the main model would have to be vs. the specialized models in order to pick this dynamic up.
esafak 1y ago
If NVIDIA had any competition we'd be able to run these larger models at home by now instead of being saddled with these 16GB midgets.
- selectodude 1y ago
  NVIDIA has tons of competition on inference hardware. They’re only a real monopoly when it comes to training new ones. And yet…
  - esafak 1y ago
    Those are for the enterprise. In the context of discussion, end users only have Apple, AMD, and Nvidia.
andai 1y ago
I used to be obsessed with what's the smartest LLM, until I tried actually using them for some tasks and realized that the smaller models did the same task way faster. So I switched my focus from "what's the smartest model" to "what's the smallest one that can do my task?" With that lens, "scores high on general intelligence benchmarks" actually becomes a measure of how overqualified the model is, and how much time, money and energy you are wasting.
- alansaber 1y ago
  What kind of task. Simple nlp, sure. Multi-hop or complex? Bigger is better.
robotresearcher 1y ago
Yes, I am also super interested in cutting the size of models. However, in a few years today’s large models will run locally anyhow. My home computer had 16KB RAM in 1983. My $20K research workstation had 192MB of RAM in 1995. Now my $2K laptop has 32GB. There is still such incredible pressure on hardware development that you can be confident that today’s SOTA models will be running at home before too long, even without ML architecture breakthroughs. Hopefully we will get both. Edit: the 90’s were exciting for compute per dollar improvements. That expensive Sun SPARC workstation I started my PhD with was obsolete three years later, crushed by a much faster $1K Intel Linux beige box. Linux installed from floppies…
- moregrist 1y ago
  > My home computer had 16KB RAM in 1983. My $20K research workstation had 192MB of RAM in 1995. Now my $2K laptop has 32GB. You’ve picked the wrong end of the curve there. Moore’s law was alive and kicking in the 90s. Every 1-3 years brought an order of magnitude better CPU and memory. Then we hit a wall. Measuring from the 2000s is more accurate. My desktop had 4GB of RAM in 2005. In 20 years it’s gone up by a factor of 8, but only by a factor of 2 in the past 10 years. I can kind of uncomfortably run a 24B parameter model on my MacBook Pro. That’s something like 50-200X smaller (depending on quantization) than a 1T parameter model. We’re a _long_ way from having enough RAM (let alone RAM in the GPU) for this size of model. If the 8x / 20 years holds, we’re talking 40-60 years. If 2X / 10 years holds, we’re talking considerably longer. If the curve continues to flatten, it’s even longer. Not to dampen anyone’s enthusiasm, but let’s be realistic about hardware improvements in the 2010s and 2020s. Smaller models will remain interesting for a very long time.
  - robotresearcher 1y ago
    Moore’s Law is about transistor density, not RAM in workstations. But yes, density is not doubling every two years any more. RAM growth slowed in laptops and workstations because we hit diminishing returns for normal-people applications. If local LLM applications are in demand, RAM will grow again. RAM doubled in Apple base models last year.
anabis 1y ago
>The ideal case would be something that can be run locally, or at least on a modest/inexpensive cluster. It's obviously valuable, so it should be coming. I expect 2 trends: - Local GPU/NPU will have a for-LLM version that has 50-100GB VRAM and runs MXFP4 etc. - Distillation will come for reasoning coding agents, probably one for each tech stack (LAMP, Android app, AWS, etc.)x business domain (gaming, social, finance, etc.)
pama 1y ago
It is not clear that a simple/small model with inference running on home hardware is energy or cost efficient compared to the scaled up inference of a large model with batch processing. There are dozens of optimizations possible when splitting an LLM on multiple tiny components on separate accelerator units and when one handles kv cache optimization at the data center level; these are simply not possible at home and would be a waste of effort and energy until you serve thousands to millions of requests in parallel.

pshirshov 1y ago

> The ideal case would be something that can be run locally, or at least on a modest/inexpensive cluster. 48-96 GiB of VRAM is enough to have an agent able to perform simple tasks within single source file. That's the sad truth. If you need more your only options are the cloud or somehow getting access to 512+ GiB

a-dub 1y ago

"open source" means there should be a script that downloads all the training materials and then spins up a pipeline that trains end to end. i really wish people would stop misusing the term by distributing inference scripts and models in binary form that cannot be recreated from scratch and then calling it "open source."

danielmarkbruce 1y ago
"open source" has come to mean "open weight" in model land. It is what it is. Words are used for communication, you are the one misusing the words. You can update the weights of the model, continue to train, whatever. Nobody is stopping you.
- a-dub 1y ago
  it still doesn't sit right. sure it's different in terms of mutability from say, compiled software programs, but it still remains not end to end reproducible and available for inspection. these words had meaning long before "model land" became a thing. overloading them is just confusing for everyone.
  - danielmarkbruce 1y ago
    It's not confusing, no one is really confused except the people upset that the meaning is different in a different context. On top of that, in many cases a company/group/whoever can't even reproduce the model themselves. There are lots of sources of non-determinism even if folks are doing things in a very buttoned up manner. And, when you are training on trillions of tokens, you are likely training on some awful sounding stuff - "Facebook is trained llama 4 on nazi propaganda!" is not what they want to see published. How about just being thankful?
    - a-dub 1y ago
      i disagree. words matter. the whole point of open source is that anyone can look and see exactly how the sausage is made. that is the point. that is why the word "open" is used. ...and sure, compiling gcc is nondeterministic too, but i can still inspect the complete source from where it comes because it is open source, which means that all of the source materials are available for inspection.
      - danielmarkbruce 1y ago
        The point of open source in software is as you say. It's just not the same thing though. Using words and phrases differently in different fields is common.
        a-dub 1y ago
        ...and my point is that it should be. the practice of science itself would be far stronger if it took more pages from open source software culture.
- mensetmanusman 1y ago
  Weights are meaningless without training data and source.
  - antiframe 1y ago
    I get a lot of meaning out of weights and source (without the training data), not sure about you. Calling it meaningless seems like exaggeration.
    - mensetmanusman 1y ago
      Can you change the weights to improve?
      - HarHarVeryFunny 1y ago
        You can fine tune without the original training data, which for a large LLM is typically going to mean using LoRA - keeping the original weights unchanged and adding separate fine-tuning weights.
      - danielmarkbruce 1y ago
        it's a bunch of numbers. Of course you can change them.
nyrp 1y ago
> binary form that cannot be recreated from scratch Back in my day, we called it "freeware"
- poly2it 1y ago
  You have more rights over a freely licensed binary file than over a freeware file.
moffkalast 1y ago
I'd agree but we're beyond hopelessly idealistic. That sort of approach only helps your competition who will use it to build a closed product and doesn't give anything of worth to people who want to actually use the model because they have no means to train it. Hell most people can barely scrape up enough hardware to even run inference. Reproducing models is also not very ecological in when it comes down to it, do we really all need to redo the training that takes absurd amounts of power just to prove that it works? At least change the dataset to try and get a better result and provide another datapoint, but most people don't have the knowhow for it anyway. Nvidia does try this approach sometimes funnily enough, they provide cool results with no model in hopes of getting people to buy their rented compute and their latest training platform as a service...
- TheBicPen 1y ago
  > I'd agree but we're beyond hopelessly idealistic. That sort of approach only helps your competition who will use it to build a closed product That same argument can be applied to open-source (non-model) software, and is about as true there. It comes down to the business model. If anything, crating a closed-sourced copy of a piece of FOSS software is easier than an AI model since running a compiler doesn't cost millions of dollars.
HarHarVeryFunny 1y ago
Yeah, but "open weights" never seems to have taken off as a better description, and even if you did have the training data + recipe, the compute cost makes training it yourself totally impractical. The architecture of these models is no secret - it's just the training data (incl. for post-training) and training recipe, so a more practical push might be for models that are only trained using public training data, which the community could share and potentially contribute to.
emsign 1y ago
They'd have to publish or link the training data, which is full of copyrighted material. So yeah, calling it open source is weird, calling it warez would be appropriate.
- oceanplexian 1y ago
  They should release it then. China doesn't have a problem stealing and distributing copyrighted material.
andai 1y ago
The meaning of Open Source 1990: Free Software 2000: Open Source: Finally we sanitized ourselves of that activism! It was scaring away customers! 2010: Source is available (under our very restrictive license) 2020: What source?
- rurban 1y ago
  2025: What prompt?

pdyc 1y ago

I think that's where prompt engineering would be needed. Bigger models produce good output even with ambiguous prompts. Getting similar output from smaller models is art,

jstummbillig 1y ago

> I'd be much more excited to see what level of coding and reasoning performance can be wrung out of a much smaller LLM + agent Well, I think you are seeing that already? It's not like these models don't exist and they did not try to make them good, it's just that the results are not super great. And why would they be? Why would the good models (that are barely okay at coding) be big, if it was currently possible to build good models, that are small? Of course, new ideas will be found and this dynamic may drastically change in the future, but there is no reason to assume that people who work on small models find great optimizations that frontier models makers, who are very interested in efficient models, have not considered already.

HarHarVeryFunny 1y ago
Sure, but that's the point ... today's locally runnable models are a long way behind SOTA capability, so it'd be nice to see more research and experimentation in that direction. Maybe a zoo of highly specialized small models + agents for S/W development - one for planning, one for coding, etc?
- Uehreka 1y ago
  If I understand transformers properly, this is unlikely to work. The whole point of “Large” Language Models is that you primarily make them better by making them larger, and when you do so, they get better at both general and specific tasks (so there isn’t a way to sacrifice generality but keep specific skills when training a small models). I know a lot of people want this (Apple really really wants this and is pouring money into it) but just because we want something doesn’t mean it will happen, especially if it goes against the main idea behind the current AI wave. I’d love to be wrong about this, but I’m pretty sure this is at least mostly right.
  - HarHarVeryFunny 1y ago
    Yeah - the whole business model of companies like OpenAI and Anthropic, at least at the moment, seems to be that the models are so big that you need to run them in the cloud with metered access. Maybe that could change in the future to sale or annual licence business model if running locally became possible. I think scale helps for general tasks where the breadth of capability may be needed, but it's not so clear that this needed for narrow verticals, especially something like coding (knowing how to fix car engines, or distinguish 100 breeds of dog is not of much use!).

pzo 1y ago

Even if pay-to-play companies like moonshootai help to pay less. You can run previous kimi k2 non-thinking model e.g. on groq with 720tok/s and for $1/$3 for million input/output tokens. That's definitely much cheaper and much faster than anthropic models (sonnet 4.5: 60tok/s, $3/$15)

stingraycharles 1y ago

With these things it’s always both at the same time: these super grandiose SOTA models are only making improvements mostly because of optimizations, and they’re just scaling our as far as they can. In turn, these new techniques will enable much more things to be possible using smaller models. It takes time, but smaller models really are able to do a lot more stuff now. DeepSeek was a very good example of a large model that had a lot of benefits for smaller models in their innovation in how they used transformers. Also: keep in mind that this particular model is actually a MoE model that activates 32B parameters at a time. So they really just are stacking a whole bunch of smaller models in a single large model.

ares623 1y ago

I don't understand. We already have that capability in our skulls. It's also "already there", so it would be a waste to not use it.

HarHarVeryFunny 1y ago
Software development is one of the areas where LLMs really are useful, whether that's vibe coding disposable software, or more structured use for serious development. I've been a developer for 40+ years, and very good at it, but for some tasks it's not about experience or overcoming complexity - just a bunch of grunt work that needs to come together. The other day I vibe coded a prototype app, just for one-time demo use, in less than 15 min that probably would have taken a week to write by hand, assuming one was already familiar with the tech stack. Developing is fun, and a brain is a terrible thing to waste, but today not using LLMs where appropriate for coding doesn't make any sense if you value your time whatsoever.
tonyhart7 1y ago
"I don't understand. We already have that capability in our skulls. It's also "already there", so it would be a waste to not use it." seems like you are here that not understand this Company want to replace human and won't need to pay massive salary
- ares623 1y ago
  I understand the companies wanting it. I hate it, but I understand. I don’t understand the humans wanting to be replaced though.
  - tonyhart7 1y ago
    "I don’t understand the humans wanting to be replaced though." because human that replace these job isnt the same human that got cut???? human that can replace these jobs would be rich

maciejgryka 1y ago

I think it’s going to be a while before we see small models (defined roughly as “runnable on reasonable consumer hardware”) do a good job at general coding tasks. It’s a very broad area! You can do some specific tasks reasonably well (eg I distilled a toy git helper you can run locally here https://github.com/distil-labs/gitara https://github.com/distil-labs/gitara), but “coding” is such a big thing that you really need a lot of knowledge to do it well.

0xjmp 1y ago

This happens top down historically though, yes? Someone releases a maxed out parameter model. Another distillates it. Another bifurcates it. With some nuance sprinkled in.

wordpad 1y ago

The electricity cost to run these models locally is already more than equivalent API cost.

ImPostingOnHN 1y ago
Privacy is minimally valued by most, but not by all.
HarHarVeryFunny 1y ago
That's going to depend on how small the model can be made, and how much you are using it. If we assume that running locally meant running on a 500W consumer GPU, then the electricity cost to run this non-stop 8 hours a day for 20 days a month (i.e. "business hours") would be around $10-20. This is about the same as OpenAI or Anthropics $20/mo plans, but for all day coding you would want their $100 or $200/mo plans, and even these will throttle you and/or require you to switch to metered pricing when you hit plan limits.
- wordpad 1y ago
  Neither $20 nor $200 plans cover any API costs. At $0.17 per million tokens for the smallest gpt model that's still faster rand more powerful than anything you can run locally and cheaper in kilowatts per hour than it would cost you to run locally even if you could.

twotwotwo 1y ago

I think there is a lot of progress on efficient useful models recently. I've seen GLM-4.6 getting mention for good coding results from a model that's much smaller than Kimi (~350b params) and seen it speculated that Windsurf based their new model on it. This Kimi release is natively INT4, with quantization-aware training. If that works--if you can get really good results from four-bit parameters--it seems like a really useful tool for any model creator wanting efficient inference. DeepSeek's v3.2-Exp uses their sparse attention technique to make longer-context training and inference more efficient. Its output's being priced at 60% less than v3.1 (though that's an imperfect indicator of efficiency). They've also quietly made 'thinking' mode need fewer tokens since R1, helping cost and latency. And though it's on the proprietary side, Haiku 4.5 approaching Sonnet 4 coding capability (at least on benches Anthropic released) also suggests legitimately useful models can be much smaller than the big ones. There's not yet a model at the level of any of the above that's practical for many people to run locally, though I think "efficient to run + open so competing inference providers can run it" is real progress. More important it seems like there's a good trendline towards efficiency, and a bunch of techniques are being researched and tested that, when used together, could make for efficient higher-quality models.

benjiro 1y ago

What i do not understand is why we are not seeing specialized models that go down to single experts. I do not need models that know how to program in Python, Rust, ... when i only use Go and Html. So we are we not seeing models that have very specialized experts, where for instance: * General interpreter model, that holds context/memory * Go Model * Html model if there is space in memory. * SQL model if there is space in memory. If there is no space, the GIM swamp out the Go model, for the HTML model, depending on where it is in Agent tasks or Edit/Ask code its overviewing. Because the models are going to be very small, switching in and out of memory will be ultra fast But most of the time we get very big Expert models, that still are very generalized over a entire field. This can then be extended that if you have the memory, models combine their output with tasks... Maybe i am just too much of a noob in the field of understanding how LLMs work, but it feels like people are too often running after large models that companies like Anthropic/OpenAI etc deploy. I understand why those big companies use insane big models. They have the money to load them up over a cluster, have the fast interconnect, and for them its more efficient. But from the bits and pieces that i see, people are more and more going to tons of small 1 a 2B models to produce better results. See my argument above. Like i said, never really gone beyond paying for my CoPilot subscription and running a bit of Ollama at home (don't have the time for the big stuff).

EMM_386 1y ago
I think one of the issues is that LLMs can't have a "Go" model and an "HTML model". I mean, they can but what would that contain? It's not the language-specific features that make models large. When models work on your code base, they do not "see" things like this, which is why they can go through an entire code base with variable names they have never seen before, function signatures they have never seen before, and directory structures that have never seen before and not have a problem. You need that "this is a variable, which is being passed to a function which recursively does ..." part. This is not something language specific, it's the high level understanding of how languages and systems operate. A variable is a variable whether in JavaScript or C++ and LLMs can "see" it as such. The details are different but it's that layer of "this is a software interface", "this is a function pointer" is outside of the "Go" or "Python" or "C#" model. I don't know how large the main model would have to be vs. the specialized models in order to pick this dynamic up.

esafak 1y ago

If NVIDIA had any competition we'd be able to run these larger models at home by now instead of being saddled with these 16GB midgets.

selectodude 1y ago
NVIDIA has tons of competition on inference hardware. They’re only a real monopoly when it comes to training new ones. And yet…
- esafak 1y ago
  Those are for the enterprise. In the context of discussion, end users only have Apple, AMD, and Nvidia.

andai 1y ago

I used to be obsessed with what's the smartest LLM, until I tried actually using them for some tasks and realized that the smaller models did the same task way faster. So I switched my focus from "what's the smartest model" to "what's the smallest one that can do my task?" With that lens, "scores high on general intelligence benchmarks" actually becomes a measure of how overqualified the model is, and how much time, money and energy you are wasting.

alansaber 1y ago
What kind of task. Simple nlp, sure. Multi-hop or complex? Bigger is better.

robotresearcher 1y ago

Yes, I am also super interested in cutting the size of models. However, in a few years today’s large models will run locally anyhow. My home computer had 16KB RAM in 1983. My $20K research workstation had 192MB of RAM in 1995. Now my $2K laptop has 32GB. There is still such incredible pressure on hardware development that you can be confident that today’s SOTA models will be running at home before too long, even without ML architecture breakthroughs. Hopefully we will get both. Edit: the 90’s were exciting for compute per dollar improvements. That expensive Sun SPARC workstation I started my PhD with was obsolete three years later, crushed by a much faster $1K Intel Linux beige box. Linux installed from floppies…

moregrist 1y ago
> My home computer had 16KB RAM in 1983. My $20K research workstation had 192MB of RAM in 1995. Now my $2K laptop has 32GB. You’ve picked the wrong end of the curve there. Moore’s law was alive and kicking in the 90s. Every 1-3 years brought an order of magnitude better CPU and memory. Then we hit a wall. Measuring from the 2000s is more accurate. My desktop had 4GB of RAM in 2005. In 20 years it’s gone up by a factor of 8, but only by a factor of 2 in the past 10 years. I can kind of uncomfortably run a 24B parameter model on my MacBook Pro. That’s something like 50-200X smaller (depending on quantization) than a 1T parameter model. We’re a _long_ way from having enough RAM (let alone RAM in the GPU) for this size of model. If the 8x / 20 years holds, we’re talking 40-60 years. If 2X / 10 years holds, we’re talking considerably longer. If the curve continues to flatten, it’s even longer. Not to dampen anyone’s enthusiasm, but let’s be realistic about hardware improvements in the 2010s and 2020s. Smaller models will remain interesting for a very long time.
- robotresearcher 1y ago
  Moore’s Law is about transistor density, not RAM in workstations. But yes, density is not doubling every two years any more. RAM growth slowed in laptops and workstations because we hit diminishing returns for normal-people applications. If local LLM applications are in demand, RAM will grow again. RAM doubled in Apple base models last year.

anabis 1y ago

>The ideal case would be something that can be run locally, or at least on a modest/inexpensive cluster. It's obviously valuable, so it should be coming. I expect 2 trends: - Local GPU/NPU will have a for-LLM version that has 50-100GB VRAM and runs MXFP4 etc. - Distillation will come for reasoning coding agents, probably one for each tech stack (LAMP, Android app, AWS, etc.)x business domain (gaming, social, finance, etc.)

pama 1y ago

It is not clear that a simple/small model with inference running on home hardware is energy or cost efficient compared to the scaled up inference of a large model with batch processing. There are dozens of optimizations possible when splitting an LLM on multiple tiny components on separate accelerator units and when one handles kv cache optimization at the data center level; these are simply not possible at home and would be a waste of effort and energy until you serve thousands to millions of requests in parallel.