3 ms·
> Do you think it the models you’re using could be quantized more that they could be downloaded on first run using Background Assets? I first tried the Qwen 3.
by karimf 6mo ago
> Do you think it the models you’re using could be quantized more that they could be downloaded on first run using Background Assets?
I first tried the Qwen 3.5 0.8B Q4_K_S and the model couldn't hold a basic conversation. Although I haven't tried lower quants on 2B.
I'm also interested on the Apple Foundation models, and it's something I plan to try next. AFAIK it's on par with Qwen-3-4B [0]. The biggest upside as you alluded to is that you don't need to download it, which is huge for user onboarding.
[0] https://machinelearning.apple.com/research/apple-foundation-models-2025-updates https://machinelearning.apple.com/research/apple-foundation-...
- Patrick_Devine 6mo agoTry it with mxfp8 or bf16. It's a decent model for doing tool calling, but I wouldn't recommend using it with 4 bit quantization.
- podlp 6mo agoSubjectively, AFM isn’t even close to Qwen. It’s one of the weakest models I’ve used. I’m not even sure how many people have Apple Intelligence enabled. But I agree, there must be a huge onboarding win long-term using (and adapting) a model that’s already optimized for your machine. I’ve learned how to navigate most of its shortcomings, but it’s not the most pleasant to work with.