5 ms·

Usage-based pricing killing your vibe, here's how to roll your own local AI

46 points by Bender 6mo ago

_345 6mo ago
It's a seriously degraded experience from a developer's perspective. Okay you've got one local LLM installed finally after configuring everything perfectly, what happens when you want to run a second instance? Now you've blown past your vram and system ram limits, and you're stuck to just one. Furthermore, the model they recommend doesn't quite reach ~gpt-5.4-mini level performance- that quality dip means you may as well just pay for something like Kimi K2.6 via openrouter if you want a something ~>= sonnet 4.6 in performance as a backup for when you run out of anthropic/openai usage.
- 2ndorderthought 6mo ago
  Why are you running 2 instances anyways? If you want that workflow just rent a few ec2 gpu instances and fire away?
  - vidarh 6mo ago
    If you're going to rent a few ec2 gpu instances you might as well funnel things through openrouter. Not that many of us have workflows where trusting an LLM provider is a problem but sending the data to EC2 is not. As for why, why would you not? Sitting around waiting for a single assistant is inefficient use of time; I tend to have more like 4-10 instances running in parallel.
    - jen20 6mo ago
      > Not that many of us have workflows where trusting an LLM provider is a problem but sending the data to EC2 is not. I'd imagine plenty of people have a problem with trusting fly-by-night inference providers or model owners with opt-out policies [1] [2] about training on your data, who would be more than happy to send data to EC2, or even the same models in Amazon Bedrock. [1]: https://github.blog/news-insights/company-news/updates-to-github-copilot-interaction-data-usage-policy/ https://github.blog/news-insights/company-news/updates-to-gi... [2]: https://help.openai.com/en/articles/5722486-how-your-data-is-used-to-improve-model-performance https://help.openai.com/en/articles/5722486-how-your-data-is...
    - 2ndorderthought 6mo ago
      I absolutely see no reason to send company IP, future plans, and current code base to any other company. I also do not run 10 agents at the same time. There's no way I could keep up with the volume of work from doing that in any meaningful way
      - killingtime74 6mo ago
        Does your company self host everything though. Many are already in the cloud, why single out llms to not use cloud for
        2ndorderthought 6mo ago
        I trust most cloud providers more than most LLMs providers but I still don't trust them much. Anything I can keep safeguarded on premises I do.
        killingtime74 6mo ago
        I understand that most of the cloud providers run the llms on their own infra, like AWS Bedrock https://aws.amazon.com/bedrock/pricing/ https://aws.amazon.com/bedrock/pricing/
      - gowld 6mo ago
        Nobody wants or needs your company IP, future plans, and current code base. You don't run 10 agents to get more volume of work. You run 10 agents to get better quality work
- 0xbadcafebee 6mo ago
  Not sure why you got downvoted. 95% of people should be paying for a subscription. It's far cheaper, far more scalable, and far less hassle. Local AI only makes sense for a couple of use cases: - Privacy - Constant churning on tokens - Latency - Availability Local AI is "cheaper" when you already have the hardware sitting around, like an old MacBook or gaming GPU, or the API cost (subscriptions will all run out if you churn 24/7) is too high to bare. I'm surprised companies are still selling their old MacBooks to employees, when they could be turning them into Beowulf clusters for cheap AI compute on long-running jobs (the cost is just electricity) If usage-based pricing is killing your vibe, find a cheaper subscription with higher limits. Here's a list of them compared on price-per-request-limit: https://codeberg.org/mutablecc/calculate-ai-cost/src/branch/main/subscription_vs_api_comparison.csv https://codeberg.org/mutablecc/calculate-ai-cost/src/branch/...
  - xscott 6mo ago
    I think you're right about the cost/benefit trade-off in general, but I do wonder how much "compaction" Codex and Claude do is to keep context fresh and how much is to save (them) runtime costs. If you've got a 1M token context, but they constantly summarize it down to something much smaller, is it really 1M tokens of benefit? With a local model, you can use all 256k tokens on your own terms. However, I don't have any benchmarks to know.
    - 0xbadcafebee 6mo ago
      I think you might be confused a bit about compaction? The LLM API endpoint does not do compaction, it's an external agent harness that does it. And the Codex/Claude agents aren't constantly summarizing it down, they generally wait until you get within 3/4 of the max of the context size. Compaction doesn't save them money, it just makes it possible for you to continue a session. If you compact a session too many times, besides the fact that the model basically stops being useful, you eventually just cannot do anything else in the session because all the context is taken up by compaction notes. But if you don't compact it, pretty soon the session is completely unusable because it can't output any more tokens. You can disable compaction in those agents if you want to see the difference. Also, using a lot of context can make the model perform poorly, so compaction can improve results. If you have a much larger context size, it means you have more headroom before the model starts to perform poorly (as it grows closer to max context size). A larger context also lets you do things like handle larger documents or reason over a larger amount of data without having to break it up into subtasks. Eventually we want models' context to get much bigger so we can do more things in a session. (Some research is being done to see if we can get rid of the limit entirely)
- xscott 6mo ago
  Your point about caliber/quality is fair, but I have been pretty astonished by some of the newer/better models (Gemma 4 variants, GPT-OSS before that). However, there's not a lot of memory increase to have multiple sessions in parallel with one model. It's an HTTP server, and other than some caching, basically stateless.
  - iib 6mo ago
    Doesn't llama.cpp (or similar) have to evict the kv cache for this, so that performance is degraded when running multiple sessions? Or how do you load a model in memory and then use it in multiple sessions? I am still learning this stuff
    - tredre3 6mo ago
      The model is loaded once and can be used for multiple sessions, and even parallel requests. llama.cpp uses a unified KV cache that is shared between requests (be they happening in parallel or not). As new requests come in, they'll evict no longer referenced branches, then move to evict the least recently used entry, and so on. If you come back to a session that's been evicted it will just be parsed again. This is a problem only on very long context sessions, but it can still be a problem to you. So one way to reduce such evictions (and reduce KV cache size significantly as a bonus) is to reduce the number of kv cache checkpoints. Checkpoints allow you to branch a session at any point and not have to recompute it from the start. If you find that you rarely branch a conversation, or if you rely entirely on a coding harness, then setting ctx-checkpoints to 0 or 1 will save tons of VRAM and allow more different sessions to stay in VRAM. This is especially true for models with very large checkpoints (such as Gemma 4).
    - xscott 6mo ago
      There are so many flags to llama.ccp that I won't try to say anything too strong, but I believe things related to `--kv-offload` mean you can have the KV cache in GPU VRAM, regular GPU RAM, paged to disk, etc... I'm on a Mac with unified memory, so I can't easily benchmark it for you, but I think a PC with 64GB of regular RAM and a 24GB gaming card could swap between multiple sessions without too much pain. The weights could stay resident on the GPU. On the other hand, I did just dump some Project Gutenberg texts into a prompt, and building that cache in the first place was slower than I though it would be.
- cyanydeez 6mo ago
  you e gott token addiction.
janice1999 6mo ago
A 24GB Nvidia RTX 3090 TI is ~2000 euro.
- 2ndorderthought 6mo ago
  Which is how many months of Claude or Claude + chatgpt when Claude is down? And do you own anything after using those subscriptions? Can you pick and choose from dozens of models and whatever comes next? Can you play video games with your Claude subscription?
  - beej71 6mo ago
    Believe me when I say that I want to run local models, and I do. But in my testing, 24 GB doesn't get you much brainpower.
    - 2ndorderthought 6mo ago
      Have you tried the latest qwen3.6 models? For most of my questions and 8-9b model works great. Upshot is not having chatgpt/meta sell my data or target me with random thoughts later.
      - ekjhgkejhgk 6mo ago
        We're in the same boat. I would rather have NO llm, than an llm that collects my data (which you should assume is all of them, unless you've been asleep for the last 20 years). Fortunately, I don't have to pick one or the other - instead I run Qwen 3.6 35B A3B. It's a bit slow with my 8gb GPU (I'm in the process of getting a bigger one) but again, to me the choice isn't "what's the best I can get", it's "what's the best local I can get".
      - datadrivenangel 6mo ago
        Inference speed is still slow in a meaningfully different way. The models are good, but not great, and much slower, which for coding means a 2-3 minute task with claude code and opus takes an hour and has a higher chance of being wrong.
efficax 6mo ago
qwen3.6 does a good job locally except it can take 20-30 minutes to respond to a prompt on a mac studio with 32gb of ram.
- 2ndorderthought 6mo ago
  Yea you probably do want to use a GPU for models of that size. I also wonder what quantization you are using? If you haven't tried other quants I really would
  - efficax 6mo ago
    This is qwen3.6:27b-coding-nvfp4. It's only an M1. If they ever ship an M5 studio with 96GB of ram, that's my next upgrade path for the local llm experiments. You can get work done with them if you have a harness that can drive outcomes without needing feedback (I've been building a tdd red to green agent harness lately that is very effective if given a good plan upfront). So if you can stand waiting a few days to see results that would only take hours with a model deployed to frontier nvidia hardware, you can get results this way.
    - datadrivenangel 6mo ago
      The time delay is the real issue. Much much slower wall clock time.
- smcleod 6mo ago
  Apple Silicon before the M4 does not have matmul instructions which causes the prompt processing to be very slow. It's quite different on the M5, much like using a nvidia GPU
roscas 6mo ago
Local AI does not mean privacy or offline. Claude code does not run offline. It needs an internet connection. "./claude-2.1.126-linux-x64 Welcome to Claude Code v2.1.126 Unable to connect to Anthropic services Failed to connect to api.anthropic.com: ECONNREFUSED Please check your internet connection and network settings. Note: Claude Code might not be available in your country. Check supported countries at https://anthropic.com/supported-countries https://anthropic.com/supported-countries" Let me also add that most of services that are private, will connect to the internet. LMStudio and many others will try to get a connection and all others. I don't remember a single one that does not connect to their servers and send some kind of information.
roscas 6mo ago
BTW, LMStudio and a few others are really amazing. They allow you to download models from HF and manage many details before load them. A medium pc with an 8 or 10gb graphics card is already a nice setup to run many models, that are really good. You can also run Ollama that is very simple to use and help you code on vscodium with Continue. Pretty nice!
- sieabahlpark 6mo ago
  [dead]
AussieWog93 6mo ago
I've tried these small models and they're nowhere near as good as Claude or GPT-5. The new ones running on a 16GB M1 are maybe GPT-4 level (with decent performance to be fair). I wonder if it's possible to make some hyper-overturned model that, say, does nothing but program in Python get SOTA-ish performance in that narrow task.
trashface 6mo ago
I like how in copilot now, I need to consider in vscode whether to accept a tab-complete, because if its coming from copilot it will count against my usage, whereas if it is coming from the ide tools it will not. So I'm like, making individual decisions on whether to type something myself or just "use up" some completion budget. Funny to get nickel and dimed like this by one of the biggest companies in the world.
- NemoNobody 6mo ago
  I was very happy with Copilot but they changed it so it's obviously making me the product bc I'm not paying for it directly anymore. I started using the local deepseek model I setup a year and half ago but never opened. It's wildly outdated but I really, really like that's it local, it feels different. I'm now in the process of getting a more updated model running locally and phasing put ChatGPT and unfortunately Copilot too. Tbh, Steam changed everything - I am actually considering Linux for the first time in my life.
gowld 6mo ago
$20/month cloud plan is definitely better than anything you'll get locally. Cost is not a reason to go local.

46 points by Bender 6mo ago

_345 6mo ago

It's a seriously degraded experience from a developer's perspective. Okay you've got one local LLM installed finally after configuring everything perfectly, what happens when you want to run a second instance? Now you've blown past your vram and system ram limits, and you're stuck to just one. Furthermore, the model they recommend doesn't quite reach ~gpt-5.4-mini level performance- that quality dip means you may as well just pay for something like Kimi K2.6 via openrouter if you want a something ~>= sonnet 4.6 in performance as a backup for when you run out of anthropic/openai usage.

2ndorderthought 6mo ago
Why are you running 2 instances anyways? If you want that workflow just rent a few ec2 gpu instances and fire away?
- vidarh 6mo ago
  If you're going to rent a few ec2 gpu instances you might as well funnel things through openrouter. Not that many of us have workflows where trusting an LLM provider is a problem but sending the data to EC2 is not. As for why, why would you not? Sitting around waiting for a single assistant is inefficient use of time; I tend to have more like 4-10 instances running in parallel.
  - jen20 6mo ago
    > Not that many of us have workflows where trusting an LLM provider is a problem but sending the data to EC2 is not. I'd imagine plenty of people have a problem with trusting fly-by-night inference providers or model owners with opt-out policies [1] [2] about training on your data, who would be more than happy to send data to EC2, or even the same models in Amazon Bedrock. [1]: https://github.blog/news-insights/company-news/updates-to-github-copilot-interaction-data-usage-policy/ https://github.blog/news-insights/company-news/updates-to-gi... [2]: https://help.openai.com/en/articles/5722486-how-your-data-is-used-to-improve-model-performance https://help.openai.com/en/articles/5722486-how-your-data-is...
  - 2ndorderthought 6mo ago
    I absolutely see no reason to send company IP, future plans, and current code base to any other company. I also do not run 10 agents at the same time. There's no way I could keep up with the volume of work from doing that in any meaningful way
    - killingtime74 6mo ago
      Does your company self host everything though. Many are already in the cloud, why single out llms to not use cloud for
      - 2ndorderthought 6mo ago
        I trust most cloud providers more than most LLMs providers but I still don't trust them much. Anything I can keep safeguarded on premises I do.
        killingtime74 6mo ago
        I understand that most of the cloud providers run the llms on their own infra, like AWS Bedrock https://aws.amazon.com/bedrock/pricing/ https://aws.amazon.com/bedrock/pricing/
    - gowld 6mo ago
      Nobody wants or needs your company IP, future plans, and current code base. You don't run 10 agents to get more volume of work. You run 10 agents to get better quality work
0xbadcafebee 6mo ago
Not sure why you got downvoted. 95% of people should be paying for a subscription. It's far cheaper, far more scalable, and far less hassle. Local AI only makes sense for a couple of use cases: - Privacy - Constant churning on tokens - Latency - Availability Local AI is "cheaper" when you already have the hardware sitting around, like an old MacBook or gaming GPU, or the API cost (subscriptions will all run out if you churn 24/7) is too high to bare. I'm surprised companies are still selling their old MacBooks to employees, when they could be turning them into Beowulf clusters for cheap AI compute on long-running jobs (the cost is just electricity) If usage-based pricing is killing your vibe, find a cheaper subscription with higher limits. Here's a list of them compared on price-per-request-limit: https://codeberg.org/mutablecc/calculate-ai-cost/src/branch/main/subscription_vs_api_comparison.csv https://codeberg.org/mutablecc/calculate-ai-cost/src/branch/...
- xscott 6mo ago
  I think you're right about the cost/benefit trade-off in general, but I do wonder how much "compaction" Codex and Claude do is to keep context fresh and how much is to save (them) runtime costs. If you've got a 1M token context, but they constantly summarize it down to something much smaller, is it really 1M tokens of benefit? With a local model, you can use all 256k tokens on your own terms. However, I don't have any benchmarks to know.
  - 0xbadcafebee 6mo ago
    I think you might be confused a bit about compaction? The LLM API endpoint does not do compaction, it's an external agent harness that does it. And the Codex/Claude agents aren't constantly summarizing it down, they generally wait until you get within 3/4 of the max of the context size. Compaction doesn't save them money, it just makes it possible for you to continue a session. If you compact a session too many times, besides the fact that the model basically stops being useful, you eventually just cannot do anything else in the session because all the context is taken up by compaction notes. But if you don't compact it, pretty soon the session is completely unusable because it can't output any more tokens. You can disable compaction in those agents if you want to see the difference. Also, using a lot of context can make the model perform poorly, so compaction can improve results. If you have a much larger context size, it means you have more headroom before the model starts to perform poorly (as it grows closer to max context size). A larger context also lets you do things like handle larger documents or reason over a larger amount of data without having to break it up into subtasks. Eventually we want models' context to get much bigger so we can do more things in a session. (Some research is being done to see if we can get rid of the limit entirely)
xscott 6mo ago
Your point about caliber/quality is fair, but I have been pretty astonished by some of the newer/better models (Gemma 4 variants, GPT-OSS before that). However, there's not a lot of memory increase to have multiple sessions in parallel with one model. It's an HTTP server, and other than some caching, basically stateless.
- iib 6mo ago
  Doesn't llama.cpp (or similar) have to evict the kv cache for this, so that performance is degraded when running multiple sessions? Or how do you load a model in memory and then use it in multiple sessions? I am still learning this stuff
  - tredre3 6mo ago
    The model is loaded once and can be used for multiple sessions, and even parallel requests. llama.cpp uses a unified KV cache that is shared between requests (be they happening in parallel or not). As new requests come in, they'll evict no longer referenced branches, then move to evict the least recently used entry, and so on. If you come back to a session that's been evicted it will just be parsed again. This is a problem only on very long context sessions, but it can still be a problem to you. So one way to reduce such evictions (and reduce KV cache size significantly as a bonus) is to reduce the number of kv cache checkpoints. Checkpoints allow you to branch a session at any point and not have to recompute it from the start. If you find that you rarely branch a conversation, or if you rely entirely on a coding harness, then setting ctx-checkpoints to 0 or 1 will save tons of VRAM and allow more different sessions to stay in VRAM. This is especially true for models with very large checkpoints (such as Gemma 4).
  - xscott 6mo ago
    There are so many flags to llama.ccp that I won't try to say anything too strong, but I believe things related to `--kv-offload` mean you can have the KV cache in GPU VRAM, regular GPU RAM, paged to disk, etc... I'm on a Mac with unified memory, so I can't easily benchmark it for you, but I think a PC with 64GB of regular RAM and a 24GB gaming card could swap between multiple sessions without too much pain. The weights could stay resident on the GPU. On the other hand, I did just dump some Project Gutenberg texts into a prompt, and building that cache in the first place was slower than I though it would be.
cyanydeez 6mo ago
you e gott token addiction.

janice1999 6mo ago

A 24GB Nvidia RTX 3090 TI is ~2000 euro.

2ndorderthought 6mo ago
Which is how many months of Claude or Claude + chatgpt when Claude is down? And do you own anything after using those subscriptions? Can you pick and choose from dozens of models and whatever comes next? Can you play video games with your Claude subscription?
- beej71 6mo ago
  Believe me when I say that I want to run local models, and I do. But in my testing, 24 GB doesn't get you much brainpower.
  - 2ndorderthought 6mo ago
    Have you tried the latest qwen3.6 models? For most of my questions and 8-9b model works great. Upshot is not having chatgpt/meta sell my data or target me with random thoughts later.
    - ekjhgkejhgk 6mo ago
      We're in the same boat. I would rather have NO llm, than an llm that collects my data (which you should assume is all of them, unless you've been asleep for the last 20 years). Fortunately, I don't have to pick one or the other - instead I run Qwen 3.6 35B A3B. It's a bit slow with my 8gb GPU (I'm in the process of getting a bigger one) but again, to me the choice isn't "what's the best I can get", it's "what's the best local I can get".
    - datadrivenangel 6mo ago
      Inference speed is still slow in a meaningfully different way. The models are good, but not great, and much slower, which for coding means a 2-3 minute task with claude code and opus takes an hour and has a higher chance of being wrong.

efficax 6mo ago

qwen3.6 does a good job locally except it can take 20-30 minutes to respond to a prompt on a mac studio with 32gb of ram.

2ndorderthought 6mo ago
Yea you probably do want to use a GPU for models of that size. I also wonder what quantization you are using? If you haven't tried other quants I really would
- efficax 6mo ago
  This is qwen3.6:27b-coding-nvfp4. It's only an M1. If they ever ship an M5 studio with 96GB of ram, that's my next upgrade path for the local llm experiments. You can get work done with them if you have a harness that can drive outcomes without needing feedback (I've been building a tdd red to green agent harness lately that is very effective if given a good plan upfront). So if you can stand waiting a few days to see results that would only take hours with a model deployed to frontier nvidia hardware, you can get results this way.
  - datadrivenangel 6mo ago
    The time delay is the real issue. Much much slower wall clock time.
smcleod 6mo ago
Apple Silicon before the M4 does not have matmul instructions which causes the prompt processing to be very slow. It's quite different on the M5, much like using a nvidia GPU

roscas 6mo ago

Local AI does not mean privacy or offline. Claude code does not run offline. It needs an internet connection. "./claude-2.1.126-linux-x64 Welcome to Claude Code v2.1.126 Unable to connect to Anthropic services Failed to connect to api.anthropic.com: ECONNREFUSED Please check your internet connection and network settings. Note: Claude Code might not be available in your country. Check supported countries at https://anthropic.com/supported-countries https://anthropic.com/supported-countries" Let me also add that most of services that are private, will connect to the internet. LMStudio and many others will try to get a connection and all others. I don't remember a single one that does not connect to their servers and send some kind of information.

roscas 6mo ago

BTW, LMStudio and a few others are really amazing. They allow you to download models from HF and manage many details before load them. A medium pc with an 8 or 10gb graphics card is already a nice setup to run many models, that are really good. You can also run Ollama that is very simple to use and help you code on vscodium with Continue. Pretty nice!

sieabahlpark 6mo ago
[dead]

AussieWog93 6mo ago

I've tried these small models and they're nowhere near as good as Claude or GPT-5. The new ones running on a 16GB M1 are maybe GPT-4 level (with decent performance to be fair). I wonder if it's possible to make some hyper-overturned model that, say, does nothing but program in Python get SOTA-ish performance in that narrow task.

trashface 6mo ago

I like how in copilot now, I need to consider in vscode whether to accept a tab-complete, because if its coming from copilot it will count against my usage, whereas if it is coming from the ide tools it will not. So I'm like, making individual decisions on whether to type something myself or just "use up" some completion budget. Funny to get nickel and dimed like this by one of the biggest companies in the world.

NemoNobody 6mo ago
I was very happy with Copilot but they changed it so it's obviously making me the product bc I'm not paying for it directly anymore. I started using the local deepseek model I setup a year and half ago but never opened. It's wildly outdated but I really, really like that's it local, it feels different. I'm now in the process of getting a more updated model running locally and phasing put ChatGPT and unfortunately Copilot too. Tbh, Steam changed everything - I am actually considering Linux for the first time in my life.

gowld 6mo ago

$20/month cloud plan is definitely better than anything you'll get locally. Cost is not a reason to go local.