2 ms·
Softmax is once per token per a layer, and growing linearly with context window size (therefore quadratic over full input).
by skew-aberration 2mo ago
Softmax is once per token per a layer, and growing linearly with context window size (therefore quadratic over full input).
- yorwba 2mo agoThat's softmax dot-product attention. It's quadratic even without fully-homomorphic encryption, but at least it's not inherently branchy, so won't necessarily slow down much more than other floating point operations under encryption. But softmax sampling, where you pick a single output token at the end and feed it back in to generate the next one, is branchy, so you need to do some extra encrypted computation to avoid leaking which token was sampled.
- covoeus 2mo agoAssuming the client is online, you could offload the costly FHE sampling by handing them the raw logits.