4 ms·

My understanding is that local LLMs are mostly just toys that output basic responses, and simply can’t compete with full LLMs trained with $60 million+ worth of

My understanding is that local LLMs are mostly just toys that output basic responses, and simply can’t compete with full LLMs trained with $60 million+ worth of compute time, and that no matter how good hardware gets, larger companies will always have even better hardware and resources to output even better results, so basically this is pointless for anything competitive or serious. Is this accurate?

quest88 2y ago
That's a strange question. It sounds like you're saying the author isn't serious in their use of LLMs. Can you describe what you mean by competitive and serious? You're commenting on a post about how the author runs LLMs locally because they find them useful. Do you think they would run them and write an entire post on how they use it if they didn't find it useful? The author is seriously using them.
- prophesi 2y ago
  It's the missing piece of the article. How is their experience running LLM's locally compared to GPT-4o / Sonnet 3.5 / etc, outside of data privacy?
  - th0ma5 2y ago
    In my experience, if you can make a reasonable guess as to the total contained information about your topic within the given size of model, then you can be very specific with your context and ask and it is generally as good as naive use of any of the larger models. Honestly, Llama3.2 is as accurate as anything and runs reasonably quick on a CPU. Larger models to me mostly increase the surface area for potential error and get exhausting to read even when you ask them to get on with it.
    - momo_O 2y ago
      > if you can make a reasonable guess as to the total contained information about your topic within the given size of model Can you expand a bit on how you might do this?
      - th0ma5 2y ago
        There's something you don't know that it may know and you want to see what it knows. This is like just a sentence or maybe a few both input and output. All the other talk about this model vs that model vs agents vs rag vs prompt engineering is all about practitioner worries. Keep in mind the thing is probably wrong as you would with any of them. Or that they are subliminally telling you something wrong you may accidentally repeat in front of someone at some time where it really matters and you're going to let everyone and yourself down. Which is current state of all of these things, so, if you're not building them, or are an NLP specialist working with multidisciplinary researchers on a specific goal of pushing research, then these things all have the same utility at the end of the day. Some of the most short sighted systems advice seems to just spill out of Claude unsolicited, so, whatever the big models are up to isn't entirely helpful in my opinion. Hopefully they'll be pressured to reveal their prompts and other safety measures.
pickettd 2y ago
Depends on what benchmarks/reports you trust I guess (and how much hardware you have for local models either in-person or in-cloud). https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ has Deepseek v3 scoring higher than most closed LLMs on coding (but it is a huge local model). And https://livebench.ai https://livebench.ai has QwQ scoring quite high in the reasoning category (and that is relatively easy to run locally but it doesn't score super high in other categories).
alkonaut 2y ago
If big models require big hardware how come there are so many free LLMs? How much compute is one ChatGPT interaction with the default model? Is it a massive loss leader for OpenAI?
- drillsteps5 2y ago
  A big part of it is the fact that Meta tries very hard to not lose the relevance against OpenAI, so they keep building and releasing a lot of their models for free. I would imagine they're taking huge losses on these but it's a price they seem to be willing to pay for being late to the party. The same applies (to smaller extent) to Google, and more recently to Alibaba. A lot of free (not open-source, mind you) LLMs are either from Meta or built by other smaller outfits by tweaking Meta's LLMs (there are exceptions, like Mistral). One thing to keep in mind is that while training a model requires A LOT of data, manual intervention, hardware, power, and time, using these LLMs (inference, or forward pass in the neural network) is really not. Right now it typically requires NVidia GPU with lots of VRAM but I have no doubt in just a few months/maybe a year someone will find much easier way to do that without sacrificing the speed.
  - alkonaut 2y ago
    Yes I get that using the models costs a lot more than training. But using them isn’t free. Even using a large nVidia chip for a fraction of a second is expensive. So where does this money come from? Gains from paid tiers, investor money?
    - drillsteps5 2y ago
      You got it backwards. Training is extremely costly in many different ways, while running inference is much less expensive. Meta, Google, Alibaba, and some others train their models and put them into the open for free burning through their cash reserves trying hard not to lose it all to OpenAI (and MSFT).
      - alkonaut 2y ago
drillsteps5 2y ago
This is not my experience, but it depends on what you consider "competitive" and "serious". While it does appear that a large number of people are using these for RP and... um... similar stuff, I do find code generation to be fairly good, esp with some recent Qwens (from Alibaba). Full disclaimer: I do use this sparingly, either to get the boilerplate, or to generate a complete function/module with clearly defined specs, and I sometimes have to re-factor the output to fit my style and preferences. I also use various general models (mostly Meta's) fairly regularly to come up with and discuss various business and other ideas, and get general understanding in the areas I want to get more knowledge of (both IT and non-IT), which helps when I need to start digging deeper into the details. I usually run quantized versions (I have older GPU with lower VRAM). Wordsmithing and generating illustrations for my blog articles (I prefer Plex, it's actually fairly good with adding various captions to illustrations; the built-in image generator in ChatGPT was still horrible at it when I tried it a few months ago). Some resume tweaking as part of manual workflow (so multiple iterations with checking back and forth between what LLM gave me and my version). So it's mostly stuff for my own personal consumption, that I don't necessarily trust the cloud with. If you have a SaaS idea with an LLM or other generative AI at its core, processing the requests locally is probably not the best choice. Unless you're prototyping, in which case it can help.
- meta_x_ai 2y ago
  Why would you waste time in Good models when there are great models?
  - drillsteps5 2y ago
    Good models are good enough for me, meta_x_ai, I gain experience by setting them up and following up on industry trends, and I don't trust OpenAI (or MSFT, or Google, or whatever) with my information. No, I do not do anything illegal or unethical, but it's not the point.
  - KetoManx64 2y ago
    The good local model isn't creating a profile about me, my preferences, my health issues, political leanings, and other info like the "great" Google and OpenAI are most likely doing based on the questions you ask the models. Just imagine if one day there's a data breach and your profile ends up on the dark web for future employers to find.
pavlov 2y ago
The leading provider of open models used for local LLM setups is Meta (with the Llama series). They have enormous resources to train these models. They’re giving away these expensive models because it doesn’t hurt Meta’s ad business but reduces risk that competitors could grow moats using closed models. The old “commoditize your complements” play.
johnklos 2y ago
There's a huge difference between training and running models. The multimillion (or billion) dollar collections of hardware assemble the datasets that we (people who run LLMs locally) run. The non-open datasets that we-host-LLMs-for-money companies do the same, and their data isn't all that much fancier. Open LLMs are catching up to closed ones, and the competition means everyone wins except for the victims of Nvidia's price gouging. This is a bit like confusing the resources that go in to making a game with the resources needed to run and play that game. They're worlds different.
- prettyStandard 2y ago
  Even Nvidia's victims are better off... (O)llama runs on AMD GPUs
kolbe 2y ago
I think the bigger issue is hardware utilization. Meta and Qwen's medium sized models are fantastic and competitive with 6 month old offerings from OpenAI and Claude, but you need to have $2500 of hardware to run it. If you're going to be spinning this 24/7, sure the API costs for OpenAI and Anthropic would be insane, but as far as normal personal usage patterns go, you'd probably spend $20/month. So, either spend $2500 that will depreciate to $1000 in two years or $580 in API costs?
philjohn 2y ago
Not entirely. TRAINING an LLM requires a lot of compute. Running inference on a pre-trained LLM is less computationally expensive, to the point where you can run LLAMA (cost $$$ to train on Meta's GPU cluster) with CPU-based inference.
dboreham 2y ago
This isn't accurate.

quest88 2y ago

That's a strange question. It sounds like you're saying the author isn't serious in their use of LLMs. Can you describe what you mean by competitive and serious? You're commenting on a post about how the author runs LLMs locally because they find them useful. Do you think they would run them and write an entire post on how they use it if they didn't find it useful? The author is seriously using them.

prophesi 2y ago
It's the missing piece of the article. How is their experience running LLM's locally compared to GPT-4o / Sonnet 3.5 / etc, outside of data privacy?
- th0ma5 2y ago
  In my experience, if you can make a reasonable guess as to the total contained information about your topic within the given size of model, then you can be very specific with your context and ask and it is generally as good as naive use of any of the larger models. Honestly, Llama3.2 is as accurate as anything and runs reasonably quick on a CPU. Larger models to me mostly increase the surface area for potential error and get exhausting to read even when you ask them to get on with it.
  - momo_O 2y ago
    > if you can make a reasonable guess as to the total contained information about your topic within the given size of model Can you expand a bit on how you might do this?
    - th0ma5 2y ago
      There's something you don't know that it may know and you want to see what it knows. This is like just a sentence or maybe a few both input and output. All the other talk about this model vs that model vs agents vs rag vs prompt engineering is all about practitioner worries. Keep in mind the thing is probably wrong as you would with any of them. Or that they are subliminally telling you something wrong you may accidentally repeat in front of someone at some time where it really matters and you're going to let everyone and yourself down. Which is current state of all of these things, so, if you're not building them, or are an NLP specialist working with multidisciplinary researchers on a specific goal of pushing research, then these things all have the same utility at the end of the day. Some of the most short sighted systems advice seems to just spill out of Claude unsolicited, so, whatever the big models are up to isn't entirely helpful in my opinion. Hopefully they'll be pressured to reveal their prompts and other safety measures.

pickettd 2y ago

Depends on what benchmarks/reports you trust I guess (and how much hardware you have for local models either in-person or in-cloud). https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ has Deepseek v3 scoring higher than most closed LLMs on coding (but it is a huge local model). And https://livebench.ai https://livebench.ai has QwQ scoring quite high in the reasoning category (and that is relatively easy to run locally but it doesn't score super high in other categories).

alkonaut 2y ago

If big models require big hardware how come there are so many free LLMs? How much compute is one ChatGPT interaction with the default model? Is it a massive loss leader for OpenAI?

drillsteps5 2y ago
A big part of it is the fact that Meta tries very hard to not lose the relevance against OpenAI, so they keep building and releasing a lot of their models for free. I would imagine they're taking huge losses on these but it's a price they seem to be willing to pay for being late to the party. The same applies (to smaller extent) to Google, and more recently to Alibaba. A lot of free (not open-source, mind you) LLMs are either from Meta or built by other smaller outfits by tweaking Meta's LLMs (there are exceptions, like Mistral). One thing to keep in mind is that while training a model requires A LOT of data, manual intervention, hardware, power, and time, using these LLMs (inference, or forward pass in the neural network) is really not. Right now it typically requires NVidia GPU with lots of VRAM but I have no doubt in just a few months/maybe a year someone will find much easier way to do that without sacrificing the speed.
- alkonaut 2y ago
  Yes I get that using the models costs a lot more than training. But using them isn’t free. Even using a large nVidia chip for a fraction of a second is expensive. So where does this money come from? Gains from paid tiers, investor money?
  - drillsteps5 2y ago
    You got it backwards. Training is extremely costly in many different ways, while running inference is much less expensive. Meta, Google, Alibaba, and some others train their models and put them into the open for free burning through their cash reserves trying hard not to lose it all to OpenAI (and MSFT).
    - alkonaut 2y ago

drillsteps5 2y ago

This is not my experience, but it depends on what you consider "competitive" and "serious". While it does appear that a large number of people are using these for RP and... um... similar stuff, I do find code generation to be fairly good, esp with some recent Qwens (from Alibaba). Full disclaimer: I do use this sparingly, either to get the boilerplate, or to generate a complete function/module with clearly defined specs, and I sometimes have to re-factor the output to fit my style and preferences. I also use various general models (mostly Meta's) fairly regularly to come up with and discuss various business and other ideas, and get general understanding in the areas I want to get more knowledge of (both IT and non-IT), which helps when I need to start digging deeper into the details. I usually run quantized versions (I have older GPU with lower VRAM). Wordsmithing and generating illustrations for my blog articles (I prefer Plex, it's actually fairly good with adding various captions to illustrations; the built-in image generator in ChatGPT was still horrible at it when I tried it a few months ago). Some resume tweaking as part of manual workflow (so multiple iterations with checking back and forth between what LLM gave me and my version). So it's mostly stuff for my own personal consumption, that I don't necessarily trust the cloud with. If you have a SaaS idea with an LLM or other generative AI at its core, processing the requests locally is probably not the best choice. Unless you're prototyping, in which case it can help.

meta_x_ai 2y ago
Why would you waste time in Good models when there are great models?
- drillsteps5 2y ago
  Good models are good enough for me, meta_x_ai, I gain experience by setting them up and following up on industry trends, and I don't trust OpenAI (or MSFT, or Google, or whatever) with my information. No, I do not do anything illegal or unethical, but it's not the point.
- KetoManx64 2y ago
  The good local model isn't creating a profile about me, my preferences, my health issues, political leanings, and other info like the "great" Google and OpenAI are most likely doing based on the questions you ask the models. Just imagine if one day there's a data breach and your profile ends up on the dark web for future employers to find.

pavlov 2y ago

The leading provider of open models used for local LLM setups is Meta (with the Llama series). They have enormous resources to train these models. They’re giving away these expensive models because it doesn’t hurt Meta’s ad business but reduces risk that competitors could grow moats using closed models. The old “commoditize your complements” play.

johnklos 2y ago

There's a huge difference between training and running models. The multimillion (or billion) dollar collections of hardware assemble the datasets that we (people who run LLMs locally) run. The non-open datasets that we-host-LLMs-for-money companies do the same, and their data isn't all that much fancier. Open LLMs are catching up to closed ones, and the competition means everyone wins except for the victims of Nvidia's price gouging. This is a bit like confusing the resources that go in to making a game with the resources needed to run and play that game. They're worlds different.

prettyStandard 2y ago
Even Nvidia's victims are better off... (O)llama runs on AMD GPUs

kolbe 2y ago

I think the bigger issue is hardware utilization. Meta and Qwen's medium sized models are fantastic and competitive with 6 month old offerings from OpenAI and Claude, but you need to have $2500 of hardware to run it. If you're going to be spinning this 24/7, sure the API costs for OpenAI and Anthropic would be insane, but as far as normal personal usage patterns go, you'd probably spend $20/month. So, either spend $2500 that will depreciate to $1000 in two years or $580 in API costs?

philjohn 2y ago

Not entirely. TRAINING an LLM requires a lot of compute. Running inference on a pre-trained LLM is less computationally expensive, to the point where you can run LLAMA (cost $$$ to train on Meta's GPU cluster) with CPU-based inference.

dboreham 2y ago

This isn't accurate.