You don't mention that MoE models horrible damage logprobs (by definition) and basically need a whole new theory of LLM sampling written for them. Dense models

You don't mention that MoE models horrible damage logprobs (by definition) and basically need a whole new theory of LLM sampling written for them. Dense models are better for a reason, and the idea that "everyone is doing MoE now and dense models are dead" is total bunk nonsense. You can quantize dense models, and 4 bit quantized Qwen 32B is still better than full precision GPT-OSS. Luckily Unsloth even gives you tools to go down to 1.58bits!