3 ms·
Maybe a little off topic, but is there a way for a distributed app to connect to one of the LLM companies (OpenAI, etc.) without the unencrypted data hitting an
by lowbatt 2y ago
Maybe a little off topic, but is there a way for a distributed app to connect to one of the LLM companies (OpenAI, etc.) without the unencrypted data hitting an in-between proxy server?
An app I'm building uses LLMs to process messages. I don’t want the unencrypted message to hit my server - and ideally I wouldn’t have the ability to decrypt it. But I can’t communicate directly from client -> LLM Service without leaking the API key.
- simonw 2y ago"But I can’t communicate directly from client -> LLM Service without leaking the API key." There is a way you can do that right now: the OpenAI WebRTC API introduced the idea of an "ephemeral key": https://platform.openai.com/docs/guides/realtime-webrtc https://platform.openai.com/docs/guides/realtime-webrtc This provides a way for your server to create a limited-time API key for a user which their browser can then use to talk to OpenAI's API directly without proxying through you. I love this idea, but I want it for way more than just the WebRTC API, and I'd like it for other API providers too. My ideal version would be a way to create an ephemeral API key that's only allowed to talk to a specific model with a specific pre-baked system prompt (and maybe tool configuration and suchlike) and that only works for a limited time and has a limited token budget.
- lowbatt 2y agointeresting, will check that out. thanks!
- michaelmrose 2y agoWill such processing be cheap enough to be done by a box that plugs into a customers router to handle such? Would they buy them? Notably not just for this use case but others
- whyage 2y agoCheck out https://www.opaque.co/ https://www.opaque.co/