3 ms·
What's the benefit to running LLMs locally? Data is already remote, LLM inferencing isn't particularly constrained by Internet latency. So you get worse models,
by kcb 1y ago
What's the benefit to running LLMs locally? Data is already remote, LLM inferencing isn't particularly constrained by Internet latency. So you get worse models, performance, and battery life. Local compute on a power constrained mobile device is required for applications that require low latency or significant data throughput and LLM inferencing is neither.
- fkyoureadthedoc 1y ago> What's the benefit to running LLMs locally? At work: That I don't rent $30,000 a month of PTUs from Microsoft. That I can put more restricted data classifications into it. > LLM inferencing isn't particularly constrained by Internet latency But user experience is
- kcb 1y ago30k in a month is an enormous amount of tokens with Claude through AWS Bedrock. And companies already commonly trust AWS with their most sensitive data.
- mrweasel 1y agoThe data you need is mostly not remote. A friend works at a software development company, they can use LLMs, but only local ones (local as in their datacenter) and it can only be trained on their code base). Customer service LLMs need to be trained on in-house material, not generic Internet sources. The general advantage is that you know that you're not leaking information, because there's nowhere to leak it to. You know the exact input, because you provided it. You also get the benefit of being able to have on device encryption, the data is no good in the datacenter if it's encrypted.
- kcb 1y agoLocal as in datacenter is the key there. The original comment was about end user devices.