4 ms·

What about inference providers like Baseten, Modal, Fireworks, Together, etc? I thought one of their value propositions was inference (using open weights models

by ssivark 21d ago

What about inference providers like Baseten, Modal, Fireworks, Together, etc? I thought one of their value propositions was inference (using open weights models) that guarantees with crisp terms that they will not use your data.

hazard 21d ago
I worked very briefly at Baseten, and I can say that it was a perpetual annoyance (from an engineering perspective) that customers would complain about issues with their models but we couldn't actually see the inputs/outputs. I don't know about the other providers, but at Baseten they literally weren't stored anywhere.
lynndotpy 21d ago
I don't have any much exposure to the attitudes people have around them, and I haven't worked with them. So I can't really say
jmalicki 21d ago
> using open weights models AWS and Azure give you the same thing for Claude and ChatGPT, no need to be stuck with open weights. They might sometimes store some of it for other purposes (I don't know the specifics), but it is emphatically not being fed back to OpenAI or Anthropic.
ljlolel 21d ago
A provider can genuinely avoid storing inputs, as the Baseten engineer below describes. That is still different from proving what code received the prompt or protecting plaintext while it runs; I built TrustedRouter to separate ZDR, attestation, and confidential routes: https://trustedrouter.com/blog/attestation-is-all-you-need?utm_source=hackernews&utm_medium=organic_reply&utm_campaign=openrouter_conversations_202609&utm_content=20260913_hackernews_inference_provider_zdr_terms https://trustedrouter.com/blog/attestation-is-all-you-need?u...
- ssivark 20d ago
  > to separate ZDR, attestation, and confidential routes Could you please clarify what that means? Given what I've been searching for, I might in principle be part of your intended customer profile, but I can't figure out whether you are merely doing routing (alternative to OpenRouter) or also inference (alternative to the names I've mentioned above). If it's merely routing, then how do you protect me from any potential misbehavior on the part of the inference provider? Just feedback for what you're building, so please take this in a positive spirit... I'm an AI researcher and not quite an infra guy, and I'm making recommendations on token APIs for several less knowledgeable around me (I've gotten a few people set up with Baseten recently), and I couldn't figure out whether/why I would be interested in TrustedRouter. You should communicate the story better :-) EDIT: Here's what I now understand after some digging; please correct if wrong. There are some M token providers (not the names I listed above?) who provide cryptographic guarantees about inference services. But somebody still needs to verify what they do on each request. For an individual running a single harness, that harness would be a logical place to perform this verification if possible. For an org with N users each running their own harness, TrustedRouter solves the N*M problem and becomes the single gateway for trusted inference -- provided one somehow trusts/verifies TrustedRouter.
  - ljlolel 20d ago
    yes, and we are also a router for the people just wanting routing and only want zdr or uncaring about privacy it’s all transparent and on github. i’d recommend just pointing your agent at trustedrouter.com since its well documented but quite a large product

by ssivark 21d ago

hazard 21d ago

I worked very briefly at Baseten, and I can say that it was a perpetual annoyance (from an engineering perspective) that customers would complain about issues with their models but we couldn't actually see the inputs/outputs. I don't know about the other providers, but at Baseten they literally weren't stored anywhere.

lynndotpy 21d ago

I don't have any much exposure to the attitudes people have around them, and I haven't worked with them. So I can't really say

jmalicki 21d ago

> using open weights models AWS and Azure give you the same thing for Claude and ChatGPT, no need to be stuck with open weights. They might sometimes store some of it for other purposes (I don't know the specifics), but it is emphatically not being fed back to OpenAI or Anthropic.

ljlolel 21d ago

A provider can genuinely avoid storing inputs, as the Baseten engineer below describes. That is still different from proving what code received the prompt or protecting plaintext while it runs; I built TrustedRouter to separate ZDR, attestation, and confidential routes: https://trustedrouter.com/blog/attestation-is-all-you-need?utm_source=hackernews&utm_medium=organic_reply&utm_campaign=openrouter_conversations_202609&utm_content=20260913_hackernews_inference_provider_zdr_terms https://trustedrouter.com/blog/attestation-is-all-you-need?u...

ssivark 20d ago
> to separate ZDR, attestation, and confidential routes Could you please clarify what that means? Given what I've been searching for, I might in principle be part of your intended customer profile, but I can't figure out whether you are merely doing routing (alternative to OpenRouter) or also inference (alternative to the names I've mentioned above). If it's merely routing, then how do you protect me from any potential misbehavior on the part of the inference provider? Just feedback for what you're building, so please take this in a positive spirit... I'm an AI researcher and not quite an infra guy, and I'm making recommendations on token APIs for several less knowledgeable around me (I've gotten a few people set up with Baseten recently), and I couldn't figure out whether/why I would be interested in TrustedRouter. You should communicate the story better :-) EDIT: Here's what I now understand after some digging; please correct if wrong. There are some M token providers (not the names I listed above?) who provide cryptographic guarantees about inference services. But somebody still needs to verify what they do on each request. For an individual running a single harness, that harness would be a logical place to perform this verification if possible. For an org with N users each running their own harness, TrustedRouter solves the N*M problem and becomes the single gateway for trusted inference -- provided one somehow trusts/verifies TrustedRouter.
- ljlolel 20d ago
  yes, and we are also a router for the people just wanting routing and only want zdr or uncaring about privacy it’s all transparent and on github. i’d recommend just pointing your agent at trustedrouter.com since its well documented but quite a large product