3 ms·
> What's the benefit to running LLMs locally? At work: That I don't rent $30,000 a month of PTUs from Microsoft. That I can put more restricted data classific
by fkyoureadthedoc 1y ago
> What's the benefit to running LLMs locally?
At work:
That I don't rent $30,000 a month of PTUs from Microsoft. That I can put more restricted data classifications into it.
> LLM inferencing isn't particularly constrained by Internet latency
But user experience is
- kcb 1y ago30k in a month is an enormous amount of tokens with Claude through AWS Bedrock. And companies already commonly trust AWS with their most sensitive data.