3 ms·
In the slides on twitter you can see Jalapeno CAN do speculative decoding. In fact they explicitly mention how compute is disaggregated 3 ways now: prefill, pre
by dist-epoch 2mo ago
In the slides on twitter you can see Jalapeno CAN do speculative decoding. In fact they explicitly mention how compute is disaggregated 3 ways now: prefill, predict, decode, and how a huge Jalapeno advantage is that it uses dark sillicon to switch between these without having to move the KV cache which remains local.
- bjourne 2mo agoThen the article contradicts the slides because it states that OpenAI choose not to disaggregate prefill and decode. Idk you men with "predict"---conventional LLM serving comprises only two phases.
- dist-epoch 2mo agosorry, my mistake, I meant draft not predict > it states that OpenAI choose not to disaggregate prefill and decode They disaggregate INSIDE the chip, not by having separate machines for the 3 phases. the slides: https://x.com/beffjezos/status/2092416851737518190 https://x.com/beffjezos/status/2092416851737518190