3 ms·
I don't see why you couldn't see improvements in self-hosted or hosting-as-a-service model throughput? Basically API-style support for a company's internal LLM
by jsrozner 1mo ago
I don't see why you couldn't see improvements in self-hosted or hosting-as-a-service model throughput? Basically API-style support for a company's internal LLM system. Why not? Or secure infra offered by AWS to self-host your own models that get served to the company just like any other company internal service can be hosted on AWS or similar?
It won't match Anthropic or OpenAI, but it could be economic?
- hypfer 1mo ago> Or secure infra offered by AWS to self-host your own models There is no such thing as "secure infra" hosted by someone else. This might still be fine, depending on your threat model, of course, but if your weights absolutely must never leave the confines of your org, you cannot use any shared hosting provider, because they just offer legal coverage of incidents. But if your moat is your knowledge, legal doesn't matter as much as the knowledge being suddenly unmoated.
- porridgeraisin 1mo agoIt's possible, but it depends a lot on the nature of the data sovereignty guarantees. Today for many companies, the kind of guarantees they have with AWS amounts to basically legal coverage. And that is fine when the company is itself protecting data only due to legal or regulatory reasons. But if the company is protecting it for commercial reasons... then a different model is needed. It could happen, but it has to be worked out.