3 ms·

FHE is the future of AI. I predict local models with encrypted weights will become the norm. Both privacy preserving (insofar as anything on our devices can be)

FHE is the future of AI. I predict local models with encrypted weights will become the norm. Both privacy preserving (insofar as anything on our devices can be) and locked down to prevent misuse. It may not be pretty but I think this is where we will end up.

boramalper 7mo ago
If you're interested in "private AI", see Confer [0] by Moxie Marlinspike, the founder of Signal private messaging app. They go into more detail in their blog. [1] [0] https://confer.to/ https://confer.to/ [1] https://confer.to/blog/2025/12/confessions-to-a-data-lake/ https://confer.to/blog/2025/12/confessions-to-a-data-lake/
- CamperBob2 7mo ago
  I don't get how this can work, and Moxie (or rather his LLM) never bothers to explain. How can an LLM possibly exchange encrypted text with the user without decrypting it? The correct solution isn't yet another cloud service, but rather local models.
  - boramalper 7mo ago
    They explain it in Private inference [0] if you want to read about it. [0] https://confer.to/blog/2026/01/private-inference/ https://confer.to/blog/2026/01/private-inference/
  - FrasiertheLion 7mo ago
    The model is running in a secure enclave that spans the GPU using NVIDIA Confidential Computing: https://www.nvidia.com/en-us/data-center/solutions/confidential-computing/ https://www.nvidia.com/en-us/data-center/solutions/confident.... The connection is encrypted with a key that is only accessible inside the enclave. Within the enclave itself, DRAM and PCIe connections between the CPU and GPU are encrypted, but the CPU registers and the GPU onboard memory are plaintext. So the computation is happening on plaintext data, it’s just extremely difficult to access it from even the machine running the enclave.
    - olejorgenb 7mo ago
      How is it then much different than trusting the policies of Anthropic etc? To be fair you need some enterprise deal to get the truly zero retention policy.
      - FrasiertheLion 7mo ago
        Enclaves have a property that allows the hardware to compute a measurement (a cryptographic hash) of everything running inside it, such as the firmware, system software such as the operating system and drivers, the application code, the security configuration. This is signed by the hardware manufacturer (Intel/AMD + NVIDIA). Then, verification involves a three part approach. Disclaimer: I'm the cofounder of Tinfoil: https://tinfoil.sh/ https://tinfoil.sh/, we also run inference inside secure enclaves. So I'll explain this as we do it. First, you open source the code that's running in the enclave, and pin a commitment to it to a transparency log (in our case, Sigstore). Then, when a client connects to the server (that's running in the enclave), the enclave computes the measurement of its current state and returns that to the client. This process is called remote attestation. The client then fetches the pinned measurements from Sigstore and compares it against the fetched measurements from the enclave. This guarantees that the code running in the enclave is the same as the code that was committed to publicly. So if someone claimed they were only analyzing aggregated metrics, they could not suddenly start analyzing individual request metrics because the code would change -> hash changes -> verification fails.
        olejorgenb 7mo ago
        Thanks for explaining :) > First, you open source the code that's running in the enclave, and pin a commitment to it to a transparency log (in our case, Sigstore). This means you have reproducible builds as well? (source+build-artifacts is signed) Also - even if there are still some risk that the link is not 100% safe, maybe it's safe to assume vendors like yourself going through all that trouble are honorable? (alternatively - they are very curious of what "paranoid" people would send through LLMs :sweatsmile:)
Foobar8568 7mo ago
FHE is impractical by all means. Either it's trivially broken and unsecured or the space requirements go beyond anything usable. There is basically no business demand beside from sellers and scholars.
Reptur 7mo ago
If encrypted outputs can be viewed or used, they can be reverse-engineered through that same interface. FHE shifts the attack surface, it does not eliminate it.
- Chance-Device 7mo ago
  If you know how to reverse engineer weights or even hidden states through simple text output without logprobs I’d be interested in hearing about it. I imagine a lot of other people would be too.
- anon291 7mo ago
  I mean, no they cannot be viewed at any point once encrypted unless you have the key. That's the point. Even the intermediate steps are random gibberish unless you have the key
eulgro 7mo ago
In science fiction maybe. We're hitting real limits on compute while AI is still far from a level where it would harmful, and FHE is orders of magnitude less efficient than direct calculation.

boramalper 7mo ago

If you're interested in "private AI", see Confer [0] by Moxie Marlinspike, the founder of Signal private messaging app. They go into more detail in their blog. [1] [0] https://confer.to/ https://confer.to/ [1] https://confer.to/blog/2025/12/confessions-to-a-data-lake/ https://confer.to/blog/2025/12/confessions-to-a-data-lake/

CamperBob2 7mo ago
I don't get how this can work, and Moxie (or rather his LLM) never bothers to explain. How can an LLM possibly exchange encrypted text with the user without decrypting it? The correct solution isn't yet another cloud service, but rather local models.
- boramalper 7mo ago
  They explain it in Private inference [0] if you want to read about it. [0] https://confer.to/blog/2026/01/private-inference/ https://confer.to/blog/2026/01/private-inference/
- FrasiertheLion 7mo ago
  The model is running in a secure enclave that spans the GPU using NVIDIA Confidential Computing: https://www.nvidia.com/en-us/data-center/solutions/confidential-computing/ https://www.nvidia.com/en-us/data-center/solutions/confident.... The connection is encrypted with a key that is only accessible inside the enclave. Within the enclave itself, DRAM and PCIe connections between the CPU and GPU are encrypted, but the CPU registers and the GPU onboard memory are plaintext. So the computation is happening on plaintext data, it’s just extremely difficult to access it from even the machine running the enclave.
  - olejorgenb 7mo ago
    How is it then much different than trusting the policies of Anthropic etc? To be fair you need some enterprise deal to get the truly zero retention policy.
    - FrasiertheLion 7mo ago
      Enclaves have a property that allows the hardware to compute a measurement (a cryptographic hash) of everything running inside it, such as the firmware, system software such as the operating system and drivers, the application code, the security configuration. This is signed by the hardware manufacturer (Intel/AMD + NVIDIA). Then, verification involves a three part approach. Disclaimer: I'm the cofounder of Tinfoil: https://tinfoil.sh/ https://tinfoil.sh/, we also run inference inside secure enclaves. So I'll explain this as we do it. First, you open source the code that's running in the enclave, and pin a commitment to it to a transparency log (in our case, Sigstore). Then, when a client connects to the server (that's running in the enclave), the enclave computes the measurement of its current state and returns that to the client. This process is called remote attestation. The client then fetches the pinned measurements from Sigstore and compares it against the fetched measurements from the enclave. This guarantees that the code running in the enclave is the same as the code that was committed to publicly. So if someone claimed they were only analyzing aggregated metrics, they could not suddenly start analyzing individual request metrics because the code would change -> hash changes -> verification fails.
      - olejorgenb 7mo ago
        Thanks for explaining :) > First, you open source the code that's running in the enclave, and pin a commitment to it to a transparency log (in our case, Sigstore). This means you have reproducible builds as well? (source+build-artifacts is signed) Also - even if there are still some risk that the link is not 100% safe, maybe it's safe to assume vendors like yourself going through all that trouble are honorable? (alternatively - they are very curious of what "paranoid" people would send through LLMs :sweatsmile:)

Foobar8568 7mo ago

FHE is impractical by all means. Either it's trivially broken and unsecured or the space requirements go beyond anything usable. There is basically no business demand beside from sellers and scholars.

Reptur 7mo ago

If encrypted outputs can be viewed or used, they can be reverse-engineered through that same interface. FHE shifts the attack surface, it does not eliminate it.

Chance-Device 7mo ago
If you know how to reverse engineer weights or even hidden states through simple text output without logprobs I’d be interested in hearing about it. I imagine a lot of other people would be too.
anon291 7mo ago
I mean, no they cannot be viewed at any point once encrypted unless you have the key. That's the point. Even the intermediate steps are random gibberish unless you have the key

eulgro 7mo ago

In science fiction maybe. We're hitting real limits on compute while AI is still far from a level where it would harmful, and FHE is orders of magnitude less efficient than direct calculation.