16 ms·

Seems to completely ignore usage of local/free models as well as anything but Sonnet/ChatGPT. So my confidence in the good faith of the author is... heavily res

Seems to completely ignore usage of local/free models as well as anything but Sonnet/ChatGPT. So my confidence in the good faith of the author is... heavily restricted.

pistoriusp 1y ago
Do you use a local/ free model?
- busymom0 1y ago
  I am currently using a local model qwen3:8b running on a 2020 (2018 intel chip) Mac mini for classifying news headlines and it's working decently well for my task. Each headline takes about 2-3 seconds but is pretty accurate. Uses about 5.3 gigs of ram.
  - darkwater 1y ago
    Can you expand a bit on your software setup? I thought running local models was restricted to having expensive GPUs or latest Apple Silicon with unified memory. I have a Intel 11th gen home server which I would like to use to run some local model for tinkering if possible.
    - marmarama 1y ago
      It's really just a performance tradeoff, and where your acceptable performance level is. Ollama, for example, will let you run any available model on just about any hardware. But using the CPU alone is _much_ slower than running it on any reasonable GPU, and obviously CPU performance varies massively too. You can even run models that are bigger than available RAM too, but performance will be terrible. The ideal case is to have a fast GPU and run a model that fits entirely within the GPU's memory. In these cases you might measure the model's processing speed in tens of tokens per second. As the idealness decreases, the processing speed decreases. On a CPU only with a model that fits in RAM, you'd be maxing out in the low single digit tokens per second, and on lower performance hardware, you start talking about seconds over token instead. If the model does not fit in RAM, then the measurement is minutes per token. For most people, their minimum acceptable performance level is in the double digit tokens per second range, which is why people optimize for that with high-end GPUs with as much memory as possible, and choose models that fit inside the GPU's RAM. But in theory you can run large models on a potato, if you're prepared to wait until next week for an answer.
      - mark_l_watson 1y ago
        +1 > It's really just a performance tradeoff, and where your acceptable performance level is. I am old enough to remember developers respecting the economics of running the software they create. Ollama running locally paired occasionally with using Ollama Cloud when required is a nice option if you use it enough. I have twice signed up and paid $20/month for Ollama Cloud, love the service, but use it so rarely (because local models so often are sufficient) that I cancelled both times. If Ollama ever implements a pay as you go API for Ollama Cloud, then I will be a long term customer. I like the business model of OpenRouter but I enjoy using Ollama Cloud more. I am probably in the minority, but I wish subscription plans would go away and Claude Code, gemini-cli, codex, etc. would all be only available pay as you go, with ‘anti dumping’ laws applied to running unsustainable businesses. I don’t mean to pick on OpenAI, but I think the way they fund their operations actually helps threaten the long term viability of our economy. Our government making the big all-in bet on AI dominance seems crazy to me.
    - simonw 1y ago
      Those little 4B and 8B models will run on almost anything. They're really fun to try out but severely limited in comparison to the larger ones - classifying headlines to categories should work well but I wouldn't trust them to refactor code! If you have 8GB of RAM you can even try running them directly in Chrome via WebAssembly. Here's a demo running a model that's less than 1GB to load, entirely in your browser (and it worked for me in mobile safari just now): https://huggingface.co/spaces/cfahlgren1/Qwen-2.5-WebLLM https://huggingface.co/spaces/cfahlgren1/Qwen-2.5-WebLLM
    - busymom0 1y ago
      It really is a very simple setup. I basically had an old Intel based Mac mini from 2020. The intel chip inside it is from 2018). It's a 3 GHz 6-core Core i5. I had upgraded the ram on it to 32 GB when I bought it. However, the ollama only uses about 5.5 gigs of it. So it can be run on 16gb Mac too. The Qwen model I am using is fairly small but does the job I need it to for classifying headlines pretty decently. All I ask it to do is whether a specific headline is political or not. It only responds to me with in True or False. I access this model from an app (running locally) using the `http://localhost:11434/api/generate` REST api with `think` set to false. Note that this qwen model is a `thinking` model. So disabling it is important. Otherwise it takes very long to respond. Note that I tested this on my newer M4 Mac mini too and there, the performance is a LOT faster. Also, on my new M4 Mac, I originally tried using the Apple's built in Foundation Models for this task and while it was decent, many times, it was hitting Apple's guardrails and refusing to respond because it claimed the headline was too sensitive. So I switched to the Qwen model which didn't have this problem. Note that while this does the job I need it to, as another comment said, it won't be much help for things like coding.
- s1mplicissimus 1y ago
  Yes, for the little it's good I'm currently using LMStudio with varying models
NumerousProcess 1y ago
Augment doesn't support local models or anything else other than Claude/GPT
nicce 1y ago
Most people can’t affort the GPUs for local models if you want to get close to cloud capabilities.
- Tepix 1y ago
  Isn't the point that you don't need SOTA capabilities all the time?
- s1mplicissimus 1y ago
  Most people I know can't afford to leak business insider information to 3rd party SaaS providers, so it's unfortunately not really an option.
  - ruszki 1y ago
    But… they do all the time. Almost everybody uses some mix of Office, Slack, Notion, random email providers, random “security” solutions etc. The exception is the opposite. The only thing prevents info leaking is ToS, and there are options for that even with LLMs. Nothing changed regarding that.
    - Antibabelic 1y ago
      In my personal experience, it's very common for big companies to host email, messengers, conferencing software on their own servers.
      - ruszki 1y ago
        Yes, then they use Outlook for example. Have you checked the ToS of the new Outlook version for commoners? They flat out state that they can use all of your emails for whatever they want. Also, companies host for example an Exchange server on prem; and guess, what it connects to? Why you can usually access account at outlook.com?
        consonaut 1y ago
moffkalast 1y ago
I think it's also true for many local models. People still use NeMo, QwQ, Llama3 for use cases that fit them despite there being replacements that do better on "benchmarks". Not to mention relics like BERT that are still tuned for classification even today. ML models always have weird behaviours and a successor is unlikely to be better in literally every way, once you have something that works well enough it's hard to upgrade without facing different edge cases. Inference for new releases is routinely bugged for at least a month or two as well, depending on how active the devs of a specific inference engine are and how much model creators collaborate. Personally, I hate how data from GPT's few week (and arguably somewhat ongoing) sycophancy rampage has leaked into datasets that are used for training local models, making a lot of new LLM releases insufferable to use.

Seems to completely ignore usage of local/free models as well as anything but Sonnet/ChatGPT. So my confidence in the good faith of the author is... heavily restricted.

pistoriusp 1y ago

Do you use a local/ free model?

busymom0 1y ago
I am currently using a local model qwen3:8b running on a 2020 (2018 intel chip) Mac mini for classifying news headlines and it's working decently well for my task. Each headline takes about 2-3 seconds but is pretty accurate. Uses about 5.3 gigs of ram.
- darkwater 1y ago
  Can you expand a bit on your software setup? I thought running local models was restricted to having expensive GPUs or latest Apple Silicon with unified memory. I have a Intel 11th gen home server which I would like to use to run some local model for tinkering if possible.
  - marmarama 1y ago
    It's really just a performance tradeoff, and where your acceptable performance level is. Ollama, for example, will let you run any available model on just about any hardware. But using the CPU alone is _much_ slower than running it on any reasonable GPU, and obviously CPU performance varies massively too. You can even run models that are bigger than available RAM too, but performance will be terrible. The ideal case is to have a fast GPU and run a model that fits entirely within the GPU's memory. In these cases you might measure the model's processing speed in tens of tokens per second. As the idealness decreases, the processing speed decreases. On a CPU only with a model that fits in RAM, you'd be maxing out in the low single digit tokens per second, and on lower performance hardware, you start talking about seconds over token instead. If the model does not fit in RAM, then the measurement is minutes per token. For most people, their minimum acceptable performance level is in the double digit tokens per second range, which is why people optimize for that with high-end GPUs with as much memory as possible, and choose models that fit inside the GPU's RAM. But in theory you can run large models on a potato, if you're prepared to wait until next week for an answer.
    - mark_l_watson 1y ago
      +1 > It's really just a performance tradeoff, and where your acceptable performance level is. I am old enough to remember developers respecting the economics of running the software they create. Ollama running locally paired occasionally with using Ollama Cloud when required is a nice option if you use it enough. I have twice signed up and paid $20/month for Ollama Cloud, love the service, but use it so rarely (because local models so often are sufficient) that I cancelled both times. If Ollama ever implements a pay as you go API for Ollama Cloud, then I will be a long term customer. I like the business model of OpenRouter but I enjoy using Ollama Cloud more. I am probably in the minority, but I wish subscription plans would go away and Claude Code, gemini-cli, codex, etc. would all be only available pay as you go, with ‘anti dumping’ laws applied to running unsustainable businesses. I don’t mean to pick on OpenAI, but I think the way they fund their operations actually helps threaten the long term viability of our economy. Our government making the big all-in bet on AI dominance seems crazy to me.
  - simonw 1y ago
    Those little 4B and 8B models will run on almost anything. They're really fun to try out but severely limited in comparison to the larger ones - classifying headlines to categories should work well but I wouldn't trust them to refactor code! If you have 8GB of RAM you can even try running them directly in Chrome via WebAssembly. Here's a demo running a model that's less than 1GB to load, entirely in your browser (and it worked for me in mobile safari just now): https://huggingface.co/spaces/cfahlgren1/Qwen-2.5-WebLLM https://huggingface.co/spaces/cfahlgren1/Qwen-2.5-WebLLM
  - busymom0 1y ago
    It really is a very simple setup. I basically had an old Intel based Mac mini from 2020. The intel chip inside it is from 2018). It's a 3 GHz 6-core Core i5. I had upgraded the ram on it to 32 GB when I bought it. However, the ollama only uses about 5.5 gigs of it. So it can be run on 16gb Mac too. The Qwen model I am using is fairly small but does the job I need it to for classifying headlines pretty decently. All I ask it to do is whether a specific headline is political or not. It only responds to me with in True or False. I access this model from an app (running locally) using the `http://localhost:11434/api/generate` REST api with `think` set to false. Note that this qwen model is a `thinking` model. So disabling it is important. Otherwise it takes very long to respond. Note that I tested this on my newer M4 Mac mini too and there, the performance is a LOT faster. Also, on my new M4 Mac, I originally tried using the Apple's built in Foundation Models for this task and while it was decent, many times, it was hitting Apple's guardrails and refusing to respond because it claimed the headline was too sensitive. So I switched to the Qwen model which didn't have this problem. Note that while this does the job I need it to, as another comment said, it won't be much help for things like coding.
s1mplicissimus 1y ago
Yes, for the little it's good I'm currently using LMStudio with varying models

NumerousProcess 1y ago

Augment doesn't support local models or anything else other than Claude/GPT

nicce 1y ago

Most people can’t affort the GPUs for local models if you want to get close to cloud capabilities.

Tepix 1y ago
Isn't the point that you don't need SOTA capabilities all the time?
s1mplicissimus 1y ago
Most people I know can't afford to leak business insider information to 3rd party SaaS providers, so it's unfortunately not really an option.
- ruszki 1y ago
  But… they do all the time. Almost everybody uses some mix of Office, Slack, Notion, random email providers, random “security” solutions etc. The exception is the opposite. The only thing prevents info leaking is ToS, and there are options for that even with LLMs. Nothing changed regarding that.
  - Antibabelic 1y ago
    In my personal experience, it's very common for big companies to host email, messengers, conferencing software on their own servers.
    - ruszki 1y ago
      Yes, then they use Outlook for example. Have you checked the ToS of the new Outlook version for commoners? They flat out state that they can use all of your emails for whatever they want. Also, companies host for example an Exchange server on prem; and guess, what it connects to? Why you can usually access account at outlook.com?
      - consonaut 1y ago

moffkalast 1y ago

I think it's also true for many local models. People still use NeMo, QwQ, Llama3 for use cases that fit them despite there being replacements that do better on "benchmarks". Not to mention relics like BERT that are still tuned for classification even today. ML models always have weird behaviours and a successor is unlikely to be better in literally every way, once you have something that works well enough it's hard to upgrade without facing different edge cases. Inference for new releases is routinely bugged for at least a month or two as well, depending on how active the devs of a specific inference engine are and how much model creators collaborate. Personally, I hate how data from GPT's few week (and arguably somewhat ongoing) sycophancy rampage has leaked into datasets that are used for training local models, making a lot of new LLM releases insufferable to use.