3 ms·
My 2c: I think the "cloud vs local" debate is (maybe) a false dichotomy. In my experience, I use a hybrid approach and I've seen a huge productivity boost from
by ngxson 4mo ago
My 2c: I think the "cloud vs local" debate is (maybe) a false dichotomy. In my experience, I use a hybrid approach and I've seen a huge productivity boost from it.
The cloud-based models are fine for big and complex tasks, but the pricing is ridiculous for small stuff—like summarizing a discussion or fixing a small bug. And cloud and privacy have never been a good match.
As an example, this comment itself was written with the help of Qwen3.5-4B running locally with an extension on top of llama.cpp default web UI [1]. The extension injects my browser's context directly into the conversation, which allows me to summarize things and draft up comments quickly. Speed is pretty acceptable for the size: ~5s TTFT and ~100 t/s generation, all running on a Macbook M5.
And when I want to run bigger tasks, I don't just stick to one provider. Apart from well-known closed-weight providers like OpenAI or Anthropic, I also experiment with open-weight models like GLM-5.1, DeepSeek V4, and Qwen3.6-27B, which provide quite good results for the price.
I'd argue both have value, and I don't see why anyone needs to choose one exclusively. Anyone else doing this?
[1]: https://github.com/ngxson/llama-companion https://github.com/ngxson/llama-companion
- phainopepla2 4mo agoWhy not just use DS V4 Flash for the small stuff? Very fast and extremely cheap.
- ngxson 4mo agoThe dsv4 flash is 158B params in total. It is possible to run locally but will require all my system RAM. Also, a lot of my day-to-day tasks perform the same on both small and bigger models: summarize a web page, draft a response, translations, quick web search, etc.
- phainopepla2 4mo agoSorry, I meant non-locally. I'm assuming privacy is not a concern since you mentioned using Deepseek already. The cost of V4 Flash for small tasks is so minuscule as to be almost free, and you don't have to deal with a churning laptop (or even buying a high-end laptop, for someone who doesn't already have one). I guess what I'm really asking is, what's the advantage of using these small local models if privacy isn't a concern?
- ngxson 4mo agoI do use both DSv4 the "normal" and the flash variant, non-locally. It works well, not exceptionally. And while it's cheap, I'd say that the difference between $1 per month vs $5 per month is not a big concern to me. IMO pricing is pretty competitive among open-weight models: https://huggingface.co/inference/models https://huggingface.co/inference/models Depending on use cases, but for me I found 2 use cases where a local model is a must and not optional: - Running offline without internet access: for example, I have this project that allow transcribe and summarize audio in real time. I already used it in some events where wifi is not available: https://github.com/ngxson/llama.cpp-realtime-audio-recap https://github.com/ngxson/llama.cpp-realtime-audio-recap - Handle private personal data, for example health records. This is the same category of "privacy" that you mentioned, but I just want to bring up the fact that people value their privacy differently.
- coder543 4mo agodsv4 flash has 284 billion parameters, not 158 billion. Huggingface's little parameter count badge seems unreliable.
- deleted 4mo ago[deleted]