4 ms·
The article is a bit naive: > However, as previously mentioned, Jalapeño’s results are obtained without speculative decoding and Vera Rubin’s results use specu
by bjourne 2mo ago
The article is a bit naive:
> However, as previously mentioned, Jalapeño’s results are obtained without speculative decoding and Vera Rubin’s results use speculative decoding. Speculative decoding leads to a ~3-5x reduction in cost per token. When speculative decoding is implemented on Jalapeño, this will enable Jalapeño to serve tokens even more cost effectively.
How much speculative decoding improves throughput is workload-dependent. Yes, it can improve performance by 5x, but it can also slow down performance by 2x. It also requires a vastly more complex decode loop than the standard token-by-token decode. The tech journos didn't ask themselves if speculative decoding improves perf so much why wasn't it on by default? Perhaps, because speculative decoding is not optimal for OpenAI's ASIC? Perhaps that is also why they were only able to benchmark the not-very-representative single-user-decode case?
- dist-epoch 2mo agoIn the slides on twitter you can see Jalapeno CAN do speculative decoding. In fact they explicitly mention how compute is disaggregated 3 ways now: prefill, predict, decode, and how a huge Jalapeno advantage is that it uses dark sillicon to switch between these without having to move the KV cache which remains local.
- bjourne 2mo agoThen the article contradicts the slides because it states that OpenAI choose not to disaggregate prefill and decode. Idk you men with "predict"---conventional LLM serving comprises only two phases.
- dist-epoch 2mo agosorry, my mistake, I meant draft not predict > it states that OpenAI choose not to disaggregate prefill and decode They disaggregate INSIDE the chip, not by having separate machines for the 3 phases. the slides: https://x.com/beffjezos/status/2092416851737518190 https://x.com/beffjezos/status/2092416851737518190