2 ms·
So what makes you think Strix Halo with such a weak GPU and slow memory bandwidth can handle 50k context with a usable experience for a 32B model? Let's be rea
by aurareturn 2y ago
So what makes you think Strix Halo with such a weak GPU and slow memory bandwidth can handle 50k context with a usable experience for a 32B model?
Let's be realistic here.
The compute, bandwidth, capacity (if 128GB) are completely imbalanced for Strix Halo. M4 Pro with 64GB is much more balanced.
- Tepix 2y agoYou're probably right. However with sparse models and MoE, 128GB may be useful.