5 ms·

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was before "thinking" models, and yet using that library I was able to "guide" the model in the required "prompt / instruct" context such that it was working towards completion, and I saw the first things like we see now in the thinking traces "oh, test x doesn't pass because blah, I need to..." and so on. Anyway, the revelation was "even if the models never improve, I'll have years of fun finding out all the ways I can use these things". And, obviously, the models improved a lot since then. But I think that revelation can still be applied, as a sort of "truism". We have, right now, access to things that 10-20 years ago would be considered magic. We are still finding ways of cobbling together systems with glue, duct tape and prayers and find new things they can do. I think the "good-enough" stage has come not just for API models (cheap, fast, etc) but for local as well. Even if slower, even if clunkier, but they are good enough for a set of ever increasing tasks, and what's more it's incredibly fun to work with them.

LoveMistral 1mo ago
Same. Mistral 7b has been more than I ever needed for text for years now. Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid. Mistral 7b can do anything, and it’s basically instant even on an M3
- frigidwalnut 1mo ago
  Sounds interesting. Can you give more details on your workflow and what tasks you use it for?
  - LoveMistral 1mo ago
    Code, creative writing, email summaries, automated email replies, and I prefill my invoice notes and daily updates for work. Actually built a full invoicing product for that, using it too. I use Mistral 7b and LlamaIndexTS on Node, I run it on a MacBook M3 and on a Linux server with only 8GB VRAM (old gaming PC). Basically flawless, runs very fast and I don’t even know what paying for “tokens” is :)
- Almondsetat 1mo ago
  What kind of work are you doing? For example, if I have some code in the hot path and I want to do all the usual tricks to help the compiler vectorize it, such a small model is not able to do much.
  - LoveMistral 1mo ago
    RAG is your friend (or any vector db). No model can vectorize an entire codebase in context. Even a big mainstream product (like Gemini) cannot handle more than ~1k lines without missing details and making mistakes. And about every 1k lines, it seems to forget the previous 1k, doesn’t it? So you can never hold more than a file or 2 (or 3) in context at a time without losing details. What you find is that the big models like Gemini are doing vector storage and retrieval too, and breaking prompts down into chunks for various models to handle to assemble a thorough response. If you want that kind of control in your outputs, and be able to hold a lot in your inputs, I don’t see any other way regardless of which model you use.
    - usef- 1mo ago
      Out of interest, have you tried the newer models? You are not describing my experience recently.
      - LoveMistral 1mo ago
        Yes - you are experiencing a mix of context caching and db retrieval from these mainstream model experiences. Even the best models available lose a ton of detail over time if you were to paste in tens of thousands of lines of code. The only way to hold huge amounts of context with a high degree of accuracy is to store it using various mechanisms (one of which is RAG). On “effectiveness”, I mean end use case effectiveness in the tasks at hand, not whatever benchmark the model developer or vendor themselves come up with - which may or may not be useful to the work I’m doing.
- casper14 1mo ago
  What are some limitations you have found with using a smaller model like that?
  - LoveMistral 1mo ago
    As a developer, none related to the model. It’s faster, lighter, makes less assumptions. For a casual consumer, it is no doubt a worse chat buddy. Knows less historical facts à la Wikipedia. But I’m not really using LLMs for that kind of entertainment and I don’t rely on them for fact-checking anyway. It’s almost worse to rely on a smoother talker for something it can’t possibly know. For feeding in thousands of daily updates and getting it to predict the next one based on a crude list of tasks, it’s great. For completing code files in my style, also great. It can also handle most small customer service issues and refunds related to my apps on App Store, the back and forth to collect data from users. For a booking engine I run with a partner, it is great at confirming bookings and following up. It handles cancellations which are about 40% of confirmed bookings due to the nature of the business. I used to feel like I was wasting my time with them - there is no way we can charge for it. Having basically a smart bot do it makes everyone happier (even the customer who knows they aren’t hurting anyone’s feelings). For invoices, I prefill notes and things based on previous ones. It’s fine. I don’t need a massive model that takes 30s to reply etc. it’s way overkill. Maybe worse than overkill - off in a direction I don’t need.
jermaustin1 1mo ago
To me, most local models work just fine for anything you can be patient for. If I want something quicker, I will go to a SOTA model via API, but with multiple 3090s, I have never really needed a hosted model for a lot of my experiments. For code, they are great, but for creativity for NPC controllers, they leave something to be desired, but work well enough for testing, so I don't burn tokens until I'm actually playing my games. But nothing one-shots a prototype better than Fable 5. I can have a prototype built in 30 minutes, hooked up to my local LLMs and Claude Code is very good at testing the interactions and even tuning the prompts of the NPCs for better experiences.
- __float 1mo ago
  "with multiple 3090s" is quite a bit of burying the lede for "most local models work just fine", don't you think?
  - jermaustin1 1mo ago
    Having multiple 6 year old cards doesn't seem like it's that big of burden for local LLMs. I get that a lot of people don't have them. And a single one can be VERY performant. And the smaller models like a 7B can run on much smaller hardware like a mid-range [3|4|5]060. My entire AI Dev Box cost $4500 in parts. 128GB RAM, i7-10700, 1TB and 2TB SSD, and 2x 3090s. Today's prices and inflation have definitely made that price tag seem a lot better than it was, but it was an investment in all things GPU that were happening in 2020 (crypto, blender, image gen), then LLMs exploded.
    - thayne 1mo ago
      A single, used 3090 costs more than I have ever spent on a computer.
      - wafflemaker 1mo ago
        My single 3080 runs so hot I don't need to warm my room in winter, and have to play games in my underwear in summer.
deleted 1mo ago
[deleted]
ksec 1mo ago
While they are improving rapidly, or as you say even if they don't. The next stage is for hardware companies ( cough Apple cough ) to ship these Local Model ready hardware in their products. It will be interesting to track the improvements of these 7B model over time. There will be a turning point in the next few years where it attract enough consumer attention to create yet another Smartphone and PC super cycle.
riazrizvi 1mo ago
I think there's something subtle about language and ambiguity that means they aren't designed to become superintelligent autonomous machines. They're value is as information repositories that actual intelligent autonomous machines (us) mine and string together.
- dgellow 1mo ago
  Yes LLMs are a beautiful way to compact knowledge. It would be such a cool technology to develop and worked with if it wasn’t linked to such a toxic industry
  - riazrizvi 1mo ago
    I think you're just observing ppl in one of these rare instances where enough of them come together because they are motivated. 'Toxic' is the clamoring sound of a crowded room where what gets through to your ears are just the most annoying snippets of incomplete conversations. I dare you to hang out with any actual people here, understand their viewpoint and listen to what they actually have to say in person, within the context of watching them do it.
    - dgellow 1mo ago
      I know those people. Lots of them are fantastic humans. That doesn’t change the fact the AI industry is extremely toxic
    - shakna 1mo ago
      By annoying snippets of incomplete conversations, do you mean Musk's clamoring for legalisation of child porn, or Altman's obsession with doomsday in talking about AI, or Zuckerberg's "Aut Zuck Aut Nihil"? The people involved at the leadership, in this wave, or incredibly toxic individuals. When the leaders behave a certain way, it tends to be reflected by those following after - intentional or not.
    - pjc50 1mo ago
      The "toxicity" was: - ingesting all the current knowledge without regard for intellectual property or the work of people that went into it; then - claiming that AI would make all those people who put in the work redundant It's not really surprising that when the sales pitch is "this will eliminate human creative work in all writing and illustration centric industries", people got angry.
cyanydeez 1mo ago
I've amassed access to 4 different GPU rigs with 128GB to 72GB; I didn't this before I event touched an agentic engineering harness. It was sometime in February/March when I set them to first tackle small problems, and now with deer-flow, they're scaffolding full project/scope implementation and I'm finishing off the fine details around the problematic edges.
swatcoder 1mo ago
Yes. The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words. The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of restrained but efficient model+harness-tuples that have been distilled, finetuned, and rigged to deliver on narrowly scoped but idiosyncratically-shaped tasks with incredible efficiency and erogonomics.
- jimmaswell 1mo ago
  This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always performed the best at all tasks. Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence. http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html Recent comment touching on this in relation to LLM's in more depth: https://news.ycombinator.com/item?id=49322695#49323341 https://news.ycombinator.com/item?id=49322695#49323341
  - applfanboysbgon 1mo ago
    This idea has not failed to pan out at all. I work for a startup that is exactly what GP described, and am set for life because of how wildly successful it is. Notably, we are successful, in a genuine sense of the word: we bootstrapped from running tiny models to larger and larger models on our own slowly improving fleet of GPUs, and now have millions in revenue without a single dime of outside investment. Conversely, you cannot call taking on ~1 trillion in debt and purchase commitments to scale "success". OpenAI and Anthropic are underwater financially. To be precise, they're in the Mariana Trench.
    - 1mo ago
viscousviolin 1mo ago
If someone has an old GPU laying around, say a GTX 1080 with 8 GB of memory, would that be enough to get a (small?) local model running?
- bityard 1mo ago
  A small model, yes! But not necessarily a good model. With the additional caveat that I don't know whether that specific card is supported by modern drivers. You'd be looking at one in the 6B or 7B parameters range at FP8. Or smaller. It's been quite some time since a recognizable company in the AI space released a model that small. You can try larger model that has been quantized down to that size, but they don't always fare well with that. Modern text-to-speech and speech-to-text models also fit well into modest amounts of VRAM.
  - avadodin 1mo ago
    You're arguing for a very specific range of weights but many slightly smaller and slightly larger models have been released including QAT and MoE versions. An old nVidia brand card with 8GB is more than enough to see those models running at usable speeds and accuracy.
    - bityard 1mo ago
      I wasn't arguing for or against anything, just trying to help answer the person's question.
- trucks-refinish 1mo ago
  I'm able to run qwen 3.6 35b a3b with my rtx 2060 super which also has 8gb of vram. It's a very capable model, not as smart as a frontier model but it is still very useful. I use an 6bit quant and get around 20 tokens per second.
- tobiasu 1mo ago
  Sure. The fastest small coding model is probably Mellum2 12B-A2.5 by Jetbrains. It matches or beats all Qwen models in this class. Can even run on a notebook CPU and comes in Base (best for FIM), Instruct and Thinking variants. mradermacher has imatrix quants for people who can't run it at Q8. IQ4 should fit, but even if it doesn't, llama.cpp has options to partially offload models to system memory.
eqmvii 1mo ago
I see it in a slightly opposite way: even the good models are relatively cheap, and so I worry what we might miss by spending too much time playing with the Sonnets of the world when the Opuses are still objectively a bargain for the power they bring.
- zahlman 1mo ago
  > when the Opuses are still objectively a bargain for the power they bring. The cost isn't just what you're billed. There are security, privacy etc. concerns.
  - Foobar8568 1mo ago
    I know companies that are using github, even using public repo, and request their teams to not use SOTA models, but are ok with local models. Just stupid policy.
    - avadodin 1mo ago
      If Orang mane bans GitHub they've got their local clones and can whip out a local server and a CI solution. If Orang mane bans Claude, they've got their local models. The latter has already happened too so I'd say their risk modeling is spot on.
    - zahlman 1mo ago
      If you're writing open-source code then there's obviously nothing wrong with publishing it in a public repo. "Using Github" doesn't require you to use their CI, but even then, a human managing secrets for GitHub CI is worlds apart from trying to make sure an internet-connected agent doesn't leak secrets. And if you have sensitive data that you can't send to a remote model but you would benefit from the technology, then processing it with a local model without network access is the obvious way to address that.
dominotw 1mo ago
ppl keep talking about the supposed unexplored and untapped "model overhang" but very few things in the world are where you can write elaborate test criteria to before using ai. A sales person sending a prospect email doesnt have a way to write a test harness for it. Yet these tasks dominate what humans do compared to writing a crud app . otherwise anthropic wouldnt have trillions dollar valuation
Der_Einzige 1mo ago
BTW structured/constrained generation has so many places to trivially enable jailbreaking/alignment/safety problems that closed source models heavily limit the full expresivity of grammars and capabilities, particular of on-the-fly dynamic grammar construction/reconstruction.
keeda 1mo ago
Yep, I've been having excellent experiences with the models even from the 2023 era. They required a lot of "holding it right" (mostly: being very precise in what went into the context) but their raw coding capabilities were astonishingly good even then. However, back then I was getting the AI to write individual functions or classes or a test suite. I was decomposing the larger task into smaller tasks, delegating some of them to the AI, reviewing the results and composing the codebase from those. I was also essentially the harness. Today the models can write and test and deploy an entire project. In terms of the code quality, I actually don't think today's frontier models would have written it much better than the 2023 models did. So in terms of raw coding capabilities i.e. converting a high-level specification into working code, I think we hit the peak way back in 2024 itself. What has changed is the AI has learned how to do the task I was doing (besides being the "harness"!), which was the mid-to-higher level "engineering" aspects like decomposing a task, specifying it to a reasonable level, reviewing the outputs, and course correcting as needed. I'm not sure if that is something the AI labs explicitly focused on during training (which may be why Meta is having its highly paid engineers do annotation work), or an emergent property of "better reasoning" (which I believe Dario implied in a podcast), or some mix of both. But the fact remains that even the weaker models are more capable than we realize, and many being open weights, are here to stay.
- mw888 1mo ago
  It does have to be said that if LLMs keep becoming better coders at some point the bottleneck on quality is prompting. Good ideas have many hidden assumptions you think are procedural but often are pivotal to your broader vision. I find that when I give an LLM my full handcrafted codebase, it does very well. It follows my conventions, sees the intent and can coherently build within its scope. It writes much better code than a 'vibe' prompt. It is always tempting and I myself will continue pushing the boundaries, but when you keep an LLM in reasonable scope (that may be one line, function, file at a time, depending on your idea of reasonable), you, by definition, can get sound utility out of them.
nowittyusername 1mo ago
There's A LOT low hanging fruit still out there for sure. And with antigenic systems being able to do the boring repetitive work of looking for that low hanging fruit I think we will see interesting things indeed. Also I think heuristics is where its at for such things. Once you describe some good heutistical structures for the research models to always follow related to "creativity" and such things, thats where we will see biggest difference. The agentic systems know the scientific method well and can follow it they just need the ability to be "creative" so their sampling becomes less rigid.
gozzoo 1mo ago
> We have, right now, access to things that 10-20 years ago would be considered magic These things would be considered magic even 4 years ago!
QuercusMax 1mo ago
Just being able to instantly generate a complicated query expression to pull specific bits out of a JSON blob sold me. It's awesome that I can ask Claude to build a whole feature and it will often one-shot it for me, but generating utility bash / python scripts or little throwaway utility webapps is what really excites me.
apatheticonion 1mo ago
Agreed. I've been "guide coding" in my editor (Zed / VSCode) for a while now and it's really enjoyable. DeepSeek v4 flash has been dirt cheap and so fast that my development loop is; - small prompt - review - small prompt - review I build software with the same quality I normally would but it's way faster to produce and I think more about architecture and flows than I do about implementation details. The small diffs let me accept / modify / veto diffs and if the model struggles, I just write it by hand. It prevents compounding defects from leading the model astray (like you see in vibe coding). In some cases vibe coding is useful, like when the complete specification is available (e.g. creating a JavaScript engine that implements the standard) - but anything that requires iterative development sees vibe coding break down pretty quickly (you could argue that is the case for a JavaScript engine). I feel energised by AI assisted coding rather than drained, as it's a force multiplier for my skills and it lets me build more than I could by myself. That said, most of my team vibe codes and reviewing their work is like pulling teeth.
- ianmarcinkowski 1mo ago
  Basically how I feel, but I use the low-powered API models like Sonnet. I feel more energized and focused on the bigger picture than getting stuck on stupid implementation of micro-problems. I think the muscle memory of doing those tiny problems is good for our minds, but solving larger-scale issues is also challenging. I'm on vacation right now and getting claude to build a mostly-throwaway e2e testing harness (admittedly not small-prompt-review-repeat) for a backend API to speed up our existing e2e test suites which do click-ops to set up tests 8-10 years ago, we had a team who spent 3-4 months every year maintaining our E2E suite and people would do rotations on there to spread the knowledge. I basically want an industry standard practice implemented on my team of 4 devs who are too busy doing other things.
- mdh6 1mo ago
  [dead]
HyperAI 1mo ago
[flagged]

LoveMistral 1mo ago

Same. Mistral 7b has been more than I ever needed for text for years now. Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid. Mistral 7b can do anything, and it’s basically instant even on an M3

frigidwalnut 1mo ago
Sounds interesting. Can you give more details on your workflow and what tasks you use it for?
- LoveMistral 1mo ago
  Code, creative writing, email summaries, automated email replies, and I prefill my invoice notes and daily updates for work. Actually built a full invoicing product for that, using it too. I use Mistral 7b and LlamaIndexTS on Node, I run it on a MacBook M3 and on a Linux server with only 8GB VRAM (old gaming PC). Basically flawless, runs very fast and I don’t even know what paying for “tokens” is :)
Almondsetat 1mo ago
What kind of work are you doing? For example, if I have some code in the hot path and I want to do all the usual tricks to help the compiler vectorize it, such a small model is not able to do much.
- LoveMistral 1mo ago
  RAG is your friend (or any vector db). No model can vectorize an entire codebase in context. Even a big mainstream product (like Gemini) cannot handle more than ~1k lines without missing details and making mistakes. And about every 1k lines, it seems to forget the previous 1k, doesn’t it? So you can never hold more than a file or 2 (or 3) in context at a time without losing details. What you find is that the big models like Gemini are doing vector storage and retrieval too, and breaking prompts down into chunks for various models to handle to assemble a thorough response. If you want that kind of control in your outputs, and be able to hold a lot in your inputs, I don’t see any other way regardless of which model you use.
  - usef- 1mo ago
    Out of interest, have you tried the newer models? You are not describing my experience recently.
    - LoveMistral 1mo ago
      Yes - you are experiencing a mix of context caching and db retrieval from these mainstream model experiences. Even the best models available lose a ton of detail over time if you were to paste in tens of thousands of lines of code. The only way to hold huge amounts of context with a high degree of accuracy is to store it using various mechanisms (one of which is RAG). On “effectiveness”, I mean end use case effectiveness in the tasks at hand, not whatever benchmark the model developer or vendor themselves come up with - which may or may not be useful to the work I’m doing.
casper14 1mo ago
What are some limitations you have found with using a smaller model like that?
- LoveMistral 1mo ago
  As a developer, none related to the model. It’s faster, lighter, makes less assumptions. For a casual consumer, it is no doubt a worse chat buddy. Knows less historical facts à la Wikipedia. But I’m not really using LLMs for that kind of entertainment and I don’t rely on them for fact-checking anyway. It’s almost worse to rely on a smoother talker for something it can’t possibly know. For feeding in thousands of daily updates and getting it to predict the next one based on a crude list of tasks, it’s great. For completing code files in my style, also great. It can also handle most small customer service issues and refunds related to my apps on App Store, the back and forth to collect data from users. For a booking engine I run with a partner, it is great at confirming bookings and following up. It handles cancellations which are about 40% of confirmed bookings due to the nature of the business. I used to feel like I was wasting my time with them - there is no way we can charge for it. Having basically a smart bot do it makes everyone happier (even the customer who knows they aren’t hurting anyone’s feelings). For invoices, I prefill notes and things based on previous ones. It’s fine. I don’t need a massive model that takes 30s to reply etc. it’s way overkill. Maybe worse than overkill - off in a direction I don’t need.

jermaustin1 1mo ago

To me, most local models work just fine for anything you can be patient for. If I want something quicker, I will go to a SOTA model via API, but with multiple 3090s, I have never really needed a hosted model for a lot of my experiments. For code, they are great, but for creativity for NPC controllers, they leave something to be desired, but work well enough for testing, so I don't burn tokens until I'm actually playing my games. But nothing one-shots a prototype better than Fable 5. I can have a prototype built in 30 minutes, hooked up to my local LLMs and Claude Code is very good at testing the interactions and even tuning the prompts of the NPCs for better experiences.

__float 1mo ago
"with multiple 3090s" is quite a bit of burying the lede for "most local models work just fine", don't you think?
- jermaustin1 1mo ago
  Having multiple 6 year old cards doesn't seem like it's that big of burden for local LLMs. I get that a lot of people don't have them. And a single one can be VERY performant. And the smaller models like a 7B can run on much smaller hardware like a mid-range [3|4|5]060. My entire AI Dev Box cost $4500 in parts. 128GB RAM, i7-10700, 1TB and 2TB SSD, and 2x 3090s. Today's prices and inflation have definitely made that price tag seem a lot better than it was, but it was an investment in all things GPU that were happening in 2020 (crypto, blender, image gen), then LLMs exploded.
  - thayne 1mo ago
    A single, used 3090 costs more than I have ever spent on a computer.
    - wafflemaker 1mo ago
      My single 3080 runs so hot I don't need to warm my room in winter, and have to play games in my underwear in summer.

deleted 1mo ago

[deleted]

ksec 1mo ago

While they are improving rapidly, or as you say even if they don't. The next stage is for hardware companies ( cough Apple cough ) to ship these Local Model ready hardware in their products. It will be interesting to track the improvements of these 7B model over time. There will be a turning point in the next few years where it attract enough consumer attention to create yet another Smartphone and PC super cycle.

riazrizvi 1mo ago

I think there's something subtle about language and ambiguity that means they aren't designed to become superintelligent autonomous machines. They're value is as information repositories that actual intelligent autonomous machines (us) mine and string together.

dgellow 1mo ago
Yes LLMs are a beautiful way to compact knowledge. It would be such a cool technology to develop and worked with if it wasn’t linked to such a toxic industry
- riazrizvi 1mo ago
  I think you're just observing ppl in one of these rare instances where enough of them come together because they are motivated. 'Toxic' is the clamoring sound of a crowded room where what gets through to your ears are just the most annoying snippets of incomplete conversations. I dare you to hang out with any actual people here, understand their viewpoint and listen to what they actually have to say in person, within the context of watching them do it.
  - dgellow 1mo ago
    I know those people. Lots of them are fantastic humans. That doesn’t change the fact the AI industry is extremely toxic
  - shakna 1mo ago
    By annoying snippets of incomplete conversations, do you mean Musk's clamoring for legalisation of child porn, or Altman's obsession with doomsday in talking about AI, or Zuckerberg's "Aut Zuck Aut Nihil"? The people involved at the leadership, in this wave, or incredibly toxic individuals. When the leaders behave a certain way, it tends to be reflected by those following after - intentional or not.
  - pjc50 1mo ago
    The "toxicity" was: - ingesting all the current knowledge without regard for intellectual property or the work of people that went into it; then - claiming that AI would make all those people who put in the work redundant It's not really surprising that when the sales pitch is "this will eliminate human creative work in all writing and illustration centric industries", people got angry.

cyanydeez 1mo ago

I've amassed access to 4 different GPU rigs with 128GB to 72GB; I didn't this before I event touched an agentic engineering harness. It was sometime in February/March when I set them to first tackle small problems, and now with deer-flow, they're scaffolding full project/scope implementation and I'm finishing off the fine details around the problematic edges.

swatcoder 1mo ago

Yes. The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words. The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of restrained but efficient model+harness-tuples that have been distilled, finetuned, and rigged to deliver on narrowly scoped but idiosyncratically-shaped tasks with incredible efficiency and erogonomics.

jimmaswell 1mo ago
This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always performed the best at all tasks. Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence. http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html Recent comment touching on this in relation to LLM's in more depth: https://news.ycombinator.com/item?id=49322695#49323341 https://news.ycombinator.com/item?id=49322695#49323341
- applfanboysbgon 1mo ago
  This idea has not failed to pan out at all. I work for a startup that is exactly what GP described, and am set for life because of how wildly successful it is. Notably, we are successful, in a genuine sense of the word: we bootstrapped from running tiny models to larger and larger models on our own slowly improving fleet of GPUs, and now have millions in revenue without a single dime of outside investment. Conversely, you cannot call taking on ~1 trillion in debt and purchase commitments to scale "success". OpenAI and Anthropic are underwater financially. To be precise, they're in the Mariana Trench.
  - 1mo ago

viscousviolin 1mo ago

If someone has an old GPU laying around, say a GTX 1080 with 8 GB of memory, would that be enough to get a (small?) local model running?

bityard 1mo ago
A small model, yes! But not necessarily a good model. With the additional caveat that I don't know whether that specific card is supported by modern drivers. You'd be looking at one in the 6B or 7B parameters range at FP8. Or smaller. It's been quite some time since a recognizable company in the AI space released a model that small. You can try larger model that has been quantized down to that size, but they don't always fare well with that. Modern text-to-speech and speech-to-text models also fit well into modest amounts of VRAM.
- avadodin 1mo ago
  You're arguing for a very specific range of weights but many slightly smaller and slightly larger models have been released including QAT and MoE versions. An old nVidia brand card with 8GB is more than enough to see those models running at usable speeds and accuracy.
  - bityard 1mo ago
    I wasn't arguing for or against anything, just trying to help answer the person's question.
trucks-refinish 1mo ago
I'm able to run qwen 3.6 35b a3b with my rtx 2060 super which also has 8gb of vram. It's a very capable model, not as smart as a frontier model but it is still very useful. I use an 6bit quant and get around 20 tokens per second.
tobiasu 1mo ago
Sure. The fastest small coding model is probably Mellum2 12B-A2.5 by Jetbrains. It matches or beats all Qwen models in this class. Can even run on a notebook CPU and comes in Base (best for FIM), Instruct and Thinking variants. mradermacher has imatrix quants for people who can't run it at Q8. IQ4 should fit, but even if it doesn't, llama.cpp has options to partially offload models to system memory.

eqmvii 1mo ago

I see it in a slightly opposite way: even the good models are relatively cheap, and so I worry what we might miss by spending too much time playing with the Sonnets of the world when the Opuses are still objectively a bargain for the power they bring.

zahlman 1mo ago
> when the Opuses are still objectively a bargain for the power they bring. The cost isn't just what you're billed. There are security, privacy etc. concerns.
- Foobar8568 1mo ago
  I know companies that are using github, even using public repo, and request their teams to not use SOTA models, but are ok with local models. Just stupid policy.
  - avadodin 1mo ago
    If Orang mane bans GitHub they've got their local clones and can whip out a local server and a CI solution. If Orang mane bans Claude, they've got their local models. The latter has already happened too so I'd say their risk modeling is spot on.
  - zahlman 1mo ago
    If you're writing open-source code then there's obviously nothing wrong with publishing it in a public repo. "Using Github" doesn't require you to use their CI, but even then, a human managing secrets for GitHub CI is worlds apart from trying to make sure an internet-connected agent doesn't leak secrets. And if you have sensitive data that you can't send to a remote model but you would benefit from the technology, then processing it with a local model without network access is the obvious way to address that.

dominotw 1mo ago

ppl keep talking about the supposed unexplored and untapped "model overhang" but very few things in the world are where you can write elaborate test criteria to before using ai. A sales person sending a prospect email doesnt have a way to write a test harness for it. Yet these tasks dominate what humans do compared to writing a crud app . otherwise anthropic wouldnt have trillions dollar valuation

Der_Einzige 1mo ago

BTW structured/constrained generation has so many places to trivially enable jailbreaking/alignment/safety problems that closed source models heavily limit the full expresivity of grammars and capabilities, particular of on-the-fly dynamic grammar construction/reconstruction.

keeda 1mo ago

Yep, I've been having excellent experiences with the models even from the 2023 era. They required a lot of "holding it right" (mostly: being very precise in what went into the context) but their raw coding capabilities were astonishingly good even then. However, back then I was getting the AI to write individual functions or classes or a test suite. I was decomposing the larger task into smaller tasks, delegating some of them to the AI, reviewing the results and composing the codebase from those. I was also essentially the harness. Today the models can write and test and deploy an entire project. In terms of the code quality, I actually don't think today's frontier models would have written it much better than the 2023 models did. So in terms of raw coding capabilities i.e. converting a high-level specification into working code, I think we hit the peak way back in 2024 itself. What has changed is the AI has learned how to do the task I was doing (besides being the "harness"!), which was the mid-to-higher level "engineering" aspects like decomposing a task, specifying it to a reasonable level, reviewing the outputs, and course correcting as needed. I'm not sure if that is something the AI labs explicitly focused on during training (which may be why Meta is having its highly paid engineers do annotation work), or an emergent property of "better reasoning" (which I believe Dario implied in a podcast), or some mix of both. But the fact remains that even the weaker models are more capable than we realize, and many being open weights, are here to stay.

mw888 1mo ago
It does have to be said that if LLMs keep becoming better coders at some point the bottleneck on quality is prompting. Good ideas have many hidden assumptions you think are procedural but often are pivotal to your broader vision. I find that when I give an LLM my full handcrafted codebase, it does very well. It follows my conventions, sees the intent and can coherently build within its scope. It writes much better code than a 'vibe' prompt. It is always tempting and I myself will continue pushing the boundaries, but when you keep an LLM in reasonable scope (that may be one line, function, file at a time, depending on your idea of reasonable), you, by definition, can get sound utility out of them.

nowittyusername 1mo ago

There's A LOT low hanging fruit still out there for sure. And with antigenic systems being able to do the boring repetitive work of looking for that low hanging fruit I think we will see interesting things indeed. Also I think heuristics is where its at for such things. Once you describe some good heutistical structures for the research models to always follow related to "creativity" and such things, thats where we will see biggest difference. The agentic systems know the scientific method well and can follow it they just need the ability to be "creative" so their sampling becomes less rigid.

gozzoo 1mo ago

> We have, right now, access to things that 10-20 years ago would be considered magic These things would be considered magic even 4 years ago!

QuercusMax 1mo ago

Just being able to instantly generate a complicated query expression to pull specific bits out of a JSON blob sold me. It's awesome that I can ask Claude to build a whole feature and it will often one-shot it for me, but generating utility bash / python scripts or little throwaway utility webapps is what really excites me.

apatheticonion 1mo ago

Agreed. I've been "guide coding" in my editor (Zed / VSCode) for a while now and it's really enjoyable. DeepSeek v4 flash has been dirt cheap and so fast that my development loop is; - small prompt - review - small prompt - review I build software with the same quality I normally would but it's way faster to produce and I think more about architecture and flows than I do about implementation details. The small diffs let me accept / modify / veto diffs and if the model struggles, I just write it by hand. It prevents compounding defects from leading the model astray (like you see in vibe coding). In some cases vibe coding is useful, like when the complete specification is available (e.g. creating a JavaScript engine that implements the standard) - but anything that requires iterative development sees vibe coding break down pretty quickly (you could argue that is the case for a JavaScript engine). I feel energised by AI assisted coding rather than drained, as it's a force multiplier for my skills and it lets me build more than I could by myself. That said, most of my team vibe codes and reviewing their work is like pulling teeth.

ianmarcinkowski 1mo ago
Basically how I feel, but I use the low-powered API models like Sonnet. I feel more energized and focused on the bigger picture than getting stuck on stupid implementation of micro-problems. I think the muscle memory of doing those tiny problems is good for our minds, but solving larger-scale issues is also challenging. I'm on vacation right now and getting claude to build a mostly-throwaway e2e testing harness (admittedly not small-prompt-review-repeat) for a backend API to speed up our existing e2e test suites which do click-ops to set up tests 8-10 years ago, we had a team who spent 3-4 months every year maintaining our E2E suite and people would do rotations on there to spread the knowledge. I basically want an industry standard practice implemented on my team of 4 devs who are too busy doing other things.
mdh6 1mo ago
[dead]

HyperAI 1mo ago

[flagged]