4 ms·

> It is also a bit weird that they are not incorporating speculative decoding Wouldn’t speculative decoding decrease overall throughput, but optimise (perceive

by jeeeb 7mo ago

> It is also a bit weird that they are not incorporating speculative decoding Wouldn’t speculative decoding decrease overall throughput, but optimise (perceived) responsiveness?

YetAnotherNick 7mo ago
For compute bound region(high batch size) yes, but for low batch size it could improve the throughput.