24 ms·
You don't actually need 480GB of RAM, but if you want at least 3 tokens / s, it's a must. If you have 500GB of SSD, llama.cpp does disk offloading -> it'll be
by danielhanchen 1y ago
You don't actually need 480GB of RAM, but if you want at least 3 tokens / s, it's a must.
If you have 500GB of SSD, llama.cpp does disk offloading -> it'll be slow though less than 1 token / s
- UncleOxidant 1y ago> but if you want at least 3 tokens / s 3 t/s isn't going to be a lot of fun to use.
- segmondy 1y agobeg to differ, I'm living fine with 1.5tk/sec
- danielhanchen 1y agoSpec decoding on a small draft model could help increase it by say 30 to 50%!
- segmondy 1y agoi'm not willing to trade any more quality for performance. no draft, no cache for kv either. i'll take the performance cost, it just makes me think carefully about my prompt. i rarely every need more than one prompt to get my answers. :D
- zackangelo 1y agoDraft model doesn’t degrade quality!
- segmondy 1y agoI beg to differ, especially when it comes to code.
- jychang 1y agoSpeculative decoding doesn't change output tokens.