11 ms·
Most people can’t affort the GPUs for local models if you want to get close to cloud capabilities.
by nicce 11mo ago
Most people can’t affort the GPUs for local models if you want to get close to cloud capabilities.
- Tepix 11mo agoIsn't the point that you don't need SOTA capabilities all the time?
- s1mplicissimus 11mo agoMost people I know can't afford to leak business insider information to 3rd party SaaS providers, so it's unfortunately not really an option.
- ruszki 11mo agoBut… they do all the time. Almost everybody uses some mix of Office, Slack, Notion, random email providers, random “security” solutions etc. The exception is the opposite. The only thing prevents info leaking is ToS, and there are options for that even with LLMs. Nothing changed regarding that.
- Antibabelic 11mo agoIn my personal experience, it's very common for big companies to host email, messengers, conferencing software on their own servers.
- ruszki 11mo agoYes, then they use Outlook for example. Have you checked the ToS of the new Outlook version for commoners? They flat out state that they can use all of your emails for whatever they want. Also, companies host for example an Exchange server on prem; and guess, what it connects to? Why you can usually access account at outlook.com?
- consonaut 11mo agoYour on premise exchange server has zero connections to outlook.com. OWA (Outlook Web Access) looks similar to outlook.com but has otherwise nothing to do with it.
- ruszki 11mo agoThen let’s call outlook.office.com, and that’s an OWA, and you’re redirected there if you login on outlook.com. And that’s the exception when on prem Exchange servers really work as more than mere proxies nowadays. I’m quite sure that there are still real fully on prem solutions, but it’s laughable these opinions, that most company really care about this. They simply don’t.
- infecto 11mo agoIn my experience it’s very common for big companies to not host. Think Fortune 500 type companies. Most are legally happy with their MSA and reasonably confident in security standards.
- s1mplicissimus 11mo ago> In my personal experience, it's very common for big companies to host email, messengers, conferencing software on their own servers. Mind sharing a clarification on your understanding of "common" and "big"?
- omgmajk 11mo agoAll of those things are hosted on-prem in the bigger orgs I have worked in.
- simonw 11mo agoI don't think Slack or Notion have on-prem/self-hosted options.
- infecto 11mo agoThis is a poor take imo. Depends on the industry but the worlds businesses run on the shoulders of companies like Microsoft and heavily use OneDrive/Sharepoint. Most entities, even those with sensitive information are legally comfortable with that arrangement. Using a LLM does not change much so long as the MSA is similar.
- s1mplicissimus 11mo ago> Depends on the industry but the worlds businesses run on the shoulders of companies like Microsoft and heavily use OneDrive/Sharepoint I am sure MS employees need to tell themselves that to sleep well. The statement itself doesn't seem to hold much epistemological value above that though.
- infecto 11mo agoIt goes in direct conflict with your idea. I am sure you know some people within your circle that say they cannot leak data but the fact remains. Over 85% of Fortune 500 companies use some combo of OneDrive or Sharepoint. The companies have already gotten familiar with the risks and legally are comfortable with the MSAs. So I am not sure what legs you are standing on. Absolutely there are specific companies or industries where they think the risk is too great but for many, outsourcing the process is either the same or less risk then doing it all inhouse.
- rhdunn 11mo agoA 4090 has 24GB of VRAM allowing you to run a 22B model entirely in memory at FP8 and 24B models at Q6_K (~19GB). A 5090 has 32GB of VRAM allowing you to run a 32B model in memory at Q6_K. You can run larger models by splitting the GPU layers that are run in VRAM vs stored in RAM. That is slower, but still viable. This means that you can run the Qwen3-Coder-30B-A3B model locally on a 4090 or 5090. That model is a Mixture of Experts model with 3B active parameters, so you really only need a card with 3B of VRAM so you could run it on a 3090. The Qwen3-Coder-480B-A35B model could also be run on a 4090 or 5090 by splitting the active 35B parameters across VRAM and RAM. Yes, it will be slower than running it in the cloud. But you can get a long way with a high-end gaming rig.
- Foobar8568 11mo agoYes but they are really less performant than claude code or codex. I really cried with the 20-25GB models ( 30b Qwen, Devstral etc). They really don't hold a candle, I didn't think the gap was this large or maybe Claude code and GPT performs much better than I imagined.
- jen729w 11mo agoHonestly though how many people reading this do you think have that setup vs. 85% of us being on a MBx? > The Qwen3-Coder-480B-A35B model could also be run on a 4090 or 5090 by splitting the active 35B parameters across VRAM and RAM. Reminds me of running Doom when I had to hack config.sys to forage 640KB of memory. Less than 0.1% of the people reading this are doing that. Me, I gave $20 to some cloud service and I can do whatever the hell I want from this M1 MBA in a hotel room in Japan.
- reaslonik 11mo ago>I can do whatever the hell I want from this M1 MBA in a hotel room in Japan. As long as it's within terms and conditions of whatever agreement you made for that $20. I can run queries on my own inference setup from remote locations too
- deleted 11mo ago
- EagnaIonat 11mo agoThe more recent LLMs work fine on an M1 mac. Can't speak for Windows/Linux. There was even a recent release of Granite4 that runs on a Raspberry Pi. https://github.com/Jewelzufo/granitepi-4-nano https://github.com/Jewelzufo/granitepi-4-nano For my local work I use Ollama. (M4 Max 128GB) - gpt-oss. 20b or 120b depending on complexity of use cases. - granite4 for speed and lower complexity (around the same as gpt20b).
- whitehexagon 11mo agoAgreed, GPU is the expensive route, especially when I was looking at external GPU solutions. Using Qwen3:32b on a 32GB M1 Pro may not be "close to cloud capabilities" but it is more than powerful enough for me, and most importantly, local and private. As a bonus, running Asahi Linux feels like I own my Personal Computer once again.
- mark_l_watson 11mo agoI agree with you (I have a 32G M2Pro) and I like to mix using local models running with Ollama and LM Studio with using gemini-cli (used to also occasionally use codex but I just cancelled my $20/month OpenAI subscription - I like their products but I don’t like their business model, so I lose out now on that option). Running smaller models on Apple Silicon is kinder on the environment/energy use and has privacy benefits for corporate use. Using a hybrid approach makes sense for many use cases. Everyone gets to make their own decisions; for me, I like to factor in externalities like social benefit, environment, and wanting the economy to do as well as it can in our new post-mono polar world.