4 ms·
Anyone want to bet that much like speculative execution, speculative decoding is going to introduce a whole slew of vulnerabilities in the ways LLMs work?
by lightedman 3mo ago
Anyone want to bet that much like speculative execution, speculative decoding is going to introduce a whole slew of vulnerabilities in the ways LLMs work?
- skirmish 3mo agoDon't think so because all tokens predicted speculatively are still validated against the main model (which is faster than predicting them from scratch) and only accepted if they match exactly.
- lightedman 3mo agoIf your main model is inherently-busted does validation actually matter?
- skirmish 3mo agoBut then it's not a problem of speculative decoding, fix your dam main model. BTW, in case there is confusion, we are not talking about CPU speculative execution affecting model inference at all, just about this specific technique: predict tokens via a smaller drafter model, then validate them against the main model in batch.