2 ms·

> Speculative decoding is the process of guessing which tokens a model might generate, then validating those guesses. As a computer engineer, it’s always inter

> Speculative decoding is the process of guessing which tokens a model might generate, then validating those guesses. As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack. Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design. Then in the mid-2000s the Speculator[0] paper brought that concept to distributed systems, which we’re still seeing work on[1][2]. Everything old is new again (: [0]https://www.cs.princeton.edu/courses/archive/fall07/cos518/papers/spec-execution.pdf https://www.cs.princeton.edu/courses/archive/fall07/cos518/p... [1]https://www.usenix.org/system/files/osdi25-shen-weihai.pdf https://www.usenix.org/system/files/osdi25-shen-weihai.pdf [2] https://www.microsoft.com/en-us/research/publication/distributed-speculative-execution-for-resilient-cloud-applications/ https://www.microsoft.com/en-us/research/publication/distrib...

freakynit 2mo ago
Can we expect similar issues such as spectre and meltdown that intel experienced with speculative execution.. but, in the form of prompt injection/poisoning?
- rf15 2mo ago
  Ok, I'll bite: no, considering these are very different domains and you don't get system access by getting the wrong speculative branch for your next text token, you just get a slightly different (but probably still related enough) text.
  - StevenWaterman 2mo ago
    Speculative decoding is lossless because the main model checks whether it agrees with what the drafter outputted
  - ranger_danger 2mo ago
    > you don't get system access by getting the wrong speculative branch for your next text token I think you could if the client supports tool/MCP calls.
- Lerc 2mo ago
  No, spectre is based upon speculative execution of code generated by another party. The breach is from the protection stopping that code from doing anything bad. Speculatve execution making things faster by varying amounts is used to turn those differences in timing into a signal. You could possibly in-pronciple detect what a model with a censorship filter was actually saying, but not really. The signal is weak and needs lots of samples to get anything worth having. You just don't have that level of control to set things up.
mathisfun123 2mo ago
> optimizations applied at different levels of the stack That's because it's just "guess and check" not some deep universal insight.
- jumploops 2mo ago
  Simple optimizations are often the best :)
- wongarsu 2mo ago
  It's closer to "we can make this highly parallel for not that much cost, but we struggle to use that concurrency. So what if we just guess what the next step is going to be? If we are right we get a big speedup, if we are wrong we just throw that work away". Which I would classify as a notable insight. Doing work that you are 50% certain is useless is not the most obvious thing
bessbd 2mo ago
> guessing which tokens a model might generate A transformer that generates tokens based on pre-training? We could call it a GPT for short.
- polynomial 2mo ago
  Yo dawg I heard you liked transformers
  - kridsdale1 2mo ago
    RIP Optimus Prime.

freakynit 2mo ago

Can we expect similar issues such as spectre and meltdown that intel experienced with speculative execution.. but, in the form of prompt injection/poisoning?

rf15 2mo ago
Ok, I'll bite: no, considering these are very different domains and you don't get system access by getting the wrong speculative branch for your next text token, you just get a slightly different (but probably still related enough) text.
- StevenWaterman 2mo ago
  Speculative decoding is lossless because the main model checks whether it agrees with what the drafter outputted
- ranger_danger 2mo ago
  > you don't get system access by getting the wrong speculative branch for your next text token I think you could if the client supports tool/MCP calls.
Lerc 2mo ago
No, spectre is based upon speculative execution of code generated by another party. The breach is from the protection stopping that code from doing anything bad. Speculatve execution making things faster by varying amounts is used to turn those differences in timing into a signal. You could possibly in-pronciple detect what a model with a censorship filter was actually saying, but not really. The signal is weak and needs lots of samples to get anything worth having. You just don't have that level of control to set things up.

mathisfun123 2mo ago

> optimizations applied at different levels of the stack That's because it's just "guess and check" not some deep universal insight.

jumploops 2mo ago
Simple optimizations are often the best :)
wongarsu 2mo ago
It's closer to "we can make this highly parallel for not that much cost, but we struggle to use that concurrency. So what if we just guess what the next step is going to be? If we are right we get a big speedup, if we are wrong we just throw that work away". Which I would classify as a notable insight. Doing work that you are 50% certain is useless is not the most obvious thing

bessbd 2mo ago

> guessing which tokens a model might generate A transformer that generates tokens based on pre-training? We could call it a GPT for short.

polynomial 2mo ago
Yo dawg I heard you liked transformers
- kridsdale1 2mo ago
  RIP Optimus Prime.