4 ms·
The secret is to not run out of quota. Instead have Claude know when to offload work to local models and what model is best suited for the job. It will shape t
by EagnaIonat 9mo ago
The secret is to not run out of quota.
Instead have Claude know when to offload work to local models and what model is best suited for the job. It will shape the prompt for the model. Then have Claude review the results. Massive reduction in costs.
btw, at least on Macbooks you can run good models with just M1 32GB of memory.
- BuildTheRobots 9mo agoI don't suppose you could point to any resources on where I could get started. I have a M2 with 64gb of unified memory and it'd be nice to make it work rather than burning Github credits.
- EagnaIonat 9mo agohttps://ollama.com https://ollama.com Although I'm starting to like LMStudio more, as it has more features that Ollama is missing. https://lmstudio.ai https://lmstudio.ai You can then get Claude to create the MCP server to talk to either. Then a CLAUDE.md that tells it to read the models you have downloaded, determine their use and when to offload. Claude will make all that for you as well.
- shen 9mo agoWhich local models are you using for the 32gb MacBooks?
- EagnaIonat 9mo agoMainly gpt-oss-20b as the thinking mode is really good. I occasionally use granite4 as it is a very fast model. But any 4GB model should easily be used.
- eek2121 9mo agoLM Studio is fantastic for playing with local models.
- kilroy123 9mo agoI strongly think you're on to something here. I wish Apple would invest heavily in something like this. The big powerful models think about tasks, then offload some stuff to a drastically cheaper cloud model or the model running on your hardware.