10 ms·
The flip is also true: LLM inference is very nearly maximally optimal for FHE. Thee costly bit, multiplication of a bunch of compressed floating point numbers,
by adastra22 2mo ago
The flip is also true: LLM inference is very nearly maximally optimal for FHE. Thee costly bit, multiplication of a bunch of compressed floating point numbers, translates 1:1. The only bit I’m not sure about is the softmax sampling at the end, but that’s just once per token.
On the other hand, each FHE step is a LOT more costly (e.g. elliptic curve exponentiation) than a vectorized BF8 multiply on GPU.
Unlike the sorting number case, it is probably same complexity. But no hardware support AND a massive slow down constant even if you were comparing apples to apples on hardware.
Disclaimer: I spent a decade working on crypto systems but I am not familiar with TFA’s research.
- skew-aberration 2mo agoSoftmax is once per token per a layer, and growing linearly with context window size (therefore quadratic over full input).
- yorwba 2mo agoThat's softmax dot-product attention. It's quadratic even without fully-homomorphic encryption, but at least it's not inherently branchy, so won't necessarily slow down much more than other floating point operations under encryption. But softmax sampling, where you pick a single output token at the end and feed it back in to generate the next one, is branchy, so you need to do some extra encrypted computation to avoid leaking which token was sampled.
- covoeus 2mo agoAssuming the client is online, you could offload the costly FHE sampling by handing them the raw logits.