3 ms·
I would love for things like this to be accessible via hubs like open router or AWS bedrock. It's hard to justify adding new model vendors directly with all the
by mushufasa 18d ago
I would love for things like this to be accessible via hubs like open router or AWS bedrock. It's hard to justify adding new model vendors directly with all the heightened concerns about privacy and security, but if bold new capabilities are added to a centralized already-vendor like AWS, technical people can adopt them without going through a whole compliance/purchasing/vendor review process. And an extra middleman tax is well worth it when the cost savings of the model itself can be one-two orders of magnitude.
- oblio 18d agoThe thing is, in this climate it's hard to believe such tech will remain secret for long. So, assuming this is not vaporware, this would raise the tide for everyone because it shows what's possible.
- cheeze 18d agoIsn't openrouter the exact opposite of caring about security and privacy? I guess you can choose your provider still? But isn't the point that the lowest bidder is doing inference?
- ajmurmann 18d agoYou can set privacy requirements and define an allow list. To me the main value prop is that I get one bill for all models and can quickly try new models without signing up anywhere or changing my code. Oh! Also you can pass an array of models and if the first provider is down it automatically falls through to the next provider. More useful than it should be...
- hobofan 18d agoThat's still ultimately privacy by contract (where you have to trust the inference providers to uphold their end of the deal), rather than privacy by design.
- LeBit 18d agoI always setup guard rails so that only zdr providers are used.
- CompleteSkeptic 18d agoThey don't like adding stealth startups :(
- varenc 18d agoI think the trouble is that Typesafe APIs don't fit into the normal OpenAI-style API that every other regular LLM provider users. You're not just providing unstructured text and getting unstructured text back. It would take a different request and response format than every other model on Open Router. Though you could shoe-horn it in some way, it'd be hacky. But agreed it'd be very useful to see it deployed on other hubs, and it seems worth it to provide the bespoke API format. Perhaps Typesafe's API will end up becoming the standard for a new type of structured model, the way OpenAI's API did.
- anshumankmr 18d agoIt can be shoehorned to work with OpenAI's newer Responses format.
- deleted 18d ago[deleted]