3 ms·
For learning it's all you need. Pick a good small model, I would recommend Qwen3.5 there are models ranging from 0.8b params (will run on a smart phone) all the
by spottedmarley 2mo ago
For learning it's all you need. Pick a good small model, I would recommend Qwen3.5 there are models ranging from 0.8b params (will run on a smart phone) all the way up to 397b params .. I run the 122b param model daily with 128GB vram and get 20+tk/sec. regardless of the model, you can learn all about how to host and harness a model with any size
- scott01 2mo agoWith this amount of VRAM, do you happen to be using a Strix Halo or a Mac?
- spottedmarley 2mo agoIt's a Dell Pro Max GB10. I have two that I can bridge together to run larger models across the 256GB pool but I only bridge them when I am experimenting with larger models and they will run fine but at lower tk/s with really only marginal reasoning gains, so I tend to stick to medium models (122b) on one GB10 and I use the other to serve gen models for music, image and video gen.