3 ms·
> the generation speed on this build is 6 to 8 tokens per second > ...if you want Q8 you'll need >700GB of GPU memory, which will probably cost $100k+
by _giorgio_ 2y ago
> the generation speed on this build is 6 to 8 tokens per second
> ...if you want Q8 you'll need >700GB of GPU memory, which will probably cost $100k+
- alias_neo 2y agoI clicked through because that $6000 price tag seemed insane, achievable even. Now it makes sense. Still undecided how I feel about having the ability to use all that quality in the full size model if one could only retrieve it at 6-8 tokens per second.