3 ms·
i'm not willing to trade any more quality for performance. no draft, no cache for kv either. i'll take the performance cost, it just makes me think carefully ab
by segmondy 1y ago
i'm not willing to trade any more quality for performance. no draft, no cache for kv either. i'll take the performance cost, it just makes me think carefully about my prompt. i rarely every need more than one prompt to get my answers. :D
- zackangelo 1y agoDraft model doesn’t degrade quality!
- segmondy 1y agoI beg to differ, especially when it comes to code.
- jychang 1y agoSpeculative decoding doesn't change output tokens.