3 ms·
Running something like Qwq 32b q4 with a ~50k context will use up those 128GB with the large KV cache.
by Tepix 2y ago
Running something like Qwq 32b q4 with a ~50k context will use up those 128GB with the large KV cache.
- aurareturn 2y agoSo what makes you think Strix Halo with such a weak GPU and slow memory bandwidth can handle 50k context with a usable experience for a 32B model? Let's be realistic here. The compute, bandwidth, capacity (if 128GB) are completely imbalanced for Strix Halo. M4 Pro with 64GB is much more balanced.
- Tepix 2y agoYou're probably right. However with sparse models and MoE, 128GB may be useful.