4 ms·
I don't understand how this works? Is it another proxy on top? What stops the provider from reading/storing the prompts at the LLM execution level?
by XCSme 2mo ago
I don't understand how this works?
Is it another proxy on top? What stops the provider from reading/storing the prompts at the LLM execution level?
- ljlolel 2mo agoit's confidential compute, it's open source and you can verify yourself that it's not reading the prompts
- XCSme 2mo agoOpen source doesn't matter if someone else is running it, right? They can change it? As long as the prompt is not encrypted at some point, and I don't think LLMs can run on encrypted prompts, then it can be read.
- ljlolel 2mo agoWith confidential compute / TEEs you can guarantee that the code is running, it's verifiable with remote attestation
- XCSme 2mo agoSo where does the guarantee stop? At the GPU driver level? Firmware level? What if the GPU has a custom bios flash that somehow logs the unencrypted prompts?
- ljlolel 2mo agogoes all the way to keys owned by Nvidia and Intel
- inigyou 2mo agoSGX has been cracked
- ljlolel 2mo agodepends on your threat model