3 ms·

I have compared instruction following of stock gpt-OSS against stock qwen and the 20B outperformed all others, intelligently following instructions and reasonin

I have compared instruction following of stock gpt-OSS against stock qwen and the 20B outperformed all others, intelligently following instructions and reasoning about tool calling correctly for tools it hasn’t been trained on. Additionally, it performs like a 3B model because it uses 32 experts. I don’t know where this claim that it sucks comes from but my evaluation of similarly competitive models showed it leading the pack by a lot.

Der_Einzige 1y ago
Even if you're right about all of this it still doesn't refute my "fine-tuning doesn't work for 99% of customers including you" thesis. Also, most mainstream AI benchmarks do not agree with you: LLMarena (https://lmarena.ai/leaderboard https://lmarena.ai/leaderboard) has GPT-OSS 120B as #53 and GPT-OSS 20B as #69 (nice), which is extremely far from leading. DeepSeek V3.1 is ranked #9, and is a solid 60+ elo points above GPT-OSS. I know you're going to link some of the "ya but chatbot arena sucks cus of theoretical attacks against it" paper and the llama4 debacle, but here's more evidence that GPT-OSS blows: https://livebench.ai/#/?q=GPT-oss https://livebench.ai/#/?q=GPT-oss GPT-oss global average: 54.60 Deepseek V3.1 thinking global average: 70.75 Qwen 3 32B global average: 63.71 So bring receipts next time because I did.
- vlovich123 1y ago
  Qwen 32B is a dense model. The competitors for GPT-OSS 20B are max 8B dense models. You’re comparing it against models it’s not competing with and calling it crap. That’s like saying Ferrari is better than Toyota. Sure, but only if we’re comparing 0-60. If we add a budget of 20K, suddenly the Toyota starts looking more competitive. I never claimed it was a frontier model. Just best in class for the performance it can achieve and the memory footprint it can fit in. And btw OSS did super well on domain specific tests without fine tuning. A model I don’t need to fine tune beats one that does.
  - Der_Einzige 1y ago
    You don't mention that MoE models horrible damage logprobs (by definition) and basically need a whole new theory of LLM sampling written for them. Dense models are better for a reason, and the idea that "everyone is doing MoE now and dense models are dead" is total bunk nonsense. You can quantize dense models, and 4 bit quantized Qwen 32B is still better than full precision GPT-OSS. Luckily Unsloth even gives you tools to go down to 1.58bits!
  - stingraycharles 1y ago
    Excuse my ignorance, but what is a “dense” model? Why is comparing GPT-OSS 20B against 8B dense models “fair” but not when comparing it against 32B models?
    - jychang 1y ago
      So gpt-oss-20b is a sparse MoE model Which means it has ~3b parameters active per token. Qwen3-32b has 32b params active per token
    - spwa4 1y ago
      An MoE model will not activate most of the model for any given query, so there is zero compute happening across an increasing part of the model. Dense model means 32B parameters => 32B get used in calculation for every token. Every calculation takes time, and assuming similar latent space size (which they all have). For example Qwen-32B MoE model has for example 80B parameters, but only 3B get used in calculation for any given token. For example Qwen3-Next 80B A3B Performance comparison: Qwen-32B => 56 tok/sec, 32 GB of VRAM Qwen3-Next 80B A3B => 167 tok/sec, 85 GB of VRAM So despite being close to 3x "bigger", Qwen3-Next is more than 3 times faster with the same compute capacity. There's a but though. But because what gets activated from one token to the next is a different subset of the model, it is still critical to have all 80B parameters loaded into memory. So MoE performs much better with less compute, at the cost of more memory. It also performs better on benchmarks, it is a better model. Similar techniques have long been used in ML to great success, rather than trying to create one brilliant model, create many that each have pros and cons, and then train a second model to figure out the best model for the task in front of you. There's even a name for the practice "ensemble models". MoE only kind-of fits (because you can't easily swap out models) There's other factors, the big one being attention. That's why non-attention models, like MAMBA, will wipe the floor in terms of performance per flop (a compute unit), with anything else. When it comes to intelligence however ...
- mountainriver 1y ago
  I literally talked to 5 customers last week that needed fine tuning, legitimately needed it. I get if you’re just doing basic RAG on text you generally don’t but that’s only part of the ecosystem
  - vlovich123 1y ago
    I didn’t say it didn’t need it, just that it needed it less. For example, previous models needed a lot of careful fine tuning to properly do tool calling. OSS does not need that.
asabla 1y ago
I'm always so confused by those statements as well. Because just like you, I feel that the 20B version is really good at following instructions. Some of the qwen models are too, but they seem to need a bit more handholding. This is of course just anecdotal from my end. And I've been slacking on keeping up with evals while testing at home
corlinp 1y ago
The performance is great, but the censorship is ridiculous for me. I tried it as a backend for my game Guessix[1], but it would refuse for ridiculous reasons like "Cannot answer questions about copyrighted works like Harry Potter." 1. https://guessix.com/ https://guessix.com/
- BoorishBears 1y ago
  Use constrained generation
  - artdigital 1y ago
    Mind explaining?
    - 7thpower 1y ago
      Curious as well
    - BoorishBears 1y ago
      If you constrain the model to a JSON schema, most frivolous refusals go away. And if you finetune on a few formatted examples the effect is even greater
  - corlinp 1y ago
    Do you mean like structured outputs? Unfortunately here the model is guided to explicitly tell you when you violate the rules and why, it can confuse it's system rules with the game rules and say you're not allowed to ask a question about copyrighted material etc.
- artdigital 1y ago
  Try the uncensored/jailbroken variants like openai-gpt-oss-20b-abliterated-uncensored-neo-imatrix I just tried to ask it how to make crystal meth and it generated a very detailed step by step guide
  - 1y ago

Der_Einzige 1y ago

Even if you're right about all of this it still doesn't refute my "fine-tuning doesn't work for 99% of customers including you" thesis. Also, most mainstream AI benchmarks do not agree with you: LLMarena (https://lmarena.ai/leaderboard https://lmarena.ai/leaderboard) has GPT-OSS 120B as #53 and GPT-OSS 20B as #69 (nice), which is extremely far from leading. DeepSeek V3.1 is ranked #9, and is a solid 60+ elo points above GPT-OSS. I know you're going to link some of the "ya but chatbot arena sucks cus of theoretical attacks against it" paper and the llama4 debacle, but here's more evidence that GPT-OSS blows: https://livebench.ai/#/?q=GPT-oss https://livebench.ai/#/?q=GPT-oss GPT-oss global average: 54.60 Deepseek V3.1 thinking global average: 70.75 Qwen 3 32B global average: 63.71 So bring receipts next time because I did.

vlovich123 1y ago
Qwen 32B is a dense model. The competitors for GPT-OSS 20B are max 8B dense models. You’re comparing it against models it’s not competing with and calling it crap. That’s like saying Ferrari is better than Toyota. Sure, but only if we’re comparing 0-60. If we add a budget of 20K, suddenly the Toyota starts looking more competitive. I never claimed it was a frontier model. Just best in class for the performance it can achieve and the memory footprint it can fit in. And btw OSS did super well on domain specific tests without fine tuning. A model I don’t need to fine tune beats one that does.
- Der_Einzige 1y ago
  You don't mention that MoE models horrible damage logprobs (by definition) and basically need a whole new theory of LLM sampling written for them. Dense models are better for a reason, and the idea that "everyone is doing MoE now and dense models are dead" is total bunk nonsense. You can quantize dense models, and 4 bit quantized Qwen 32B is still better than full precision GPT-OSS. Luckily Unsloth even gives you tools to go down to 1.58bits!
- stingraycharles 1y ago
  Excuse my ignorance, but what is a “dense” model? Why is comparing GPT-OSS 20B against 8B dense models “fair” but not when comparing it against 32B models?
  - jychang 1y ago
    So gpt-oss-20b is a sparse MoE model Which means it has ~3b parameters active per token. Qwen3-32b has 32b params active per token
  - spwa4 1y ago
    An MoE model will not activate most of the model for any given query, so there is zero compute happening across an increasing part of the model. Dense model means 32B parameters => 32B get used in calculation for every token. Every calculation takes time, and assuming similar latent space size (which they all have). For example Qwen-32B MoE model has for example 80B parameters, but only 3B get used in calculation for any given token. For example Qwen3-Next 80B A3B Performance comparison: Qwen-32B => 56 tok/sec, 32 GB of VRAM Qwen3-Next 80B A3B => 167 tok/sec, 85 GB of VRAM So despite being close to 3x "bigger", Qwen3-Next is more than 3 times faster with the same compute capacity. There's a but though. But because what gets activated from one token to the next is a different subset of the model, it is still critical to have all 80B parameters loaded into memory. So MoE performs much better with less compute, at the cost of more memory. It also performs better on benchmarks, it is a better model. Similar techniques have long been used in ML to great success, rather than trying to create one brilliant model, create many that each have pros and cons, and then train a second model to figure out the best model for the task in front of you. There's even a name for the practice "ensemble models". MoE only kind-of fits (because you can't easily swap out models) There's other factors, the big one being attention. That's why non-attention models, like MAMBA, will wipe the floor in terms of performance per flop (a compute unit), with anything else. When it comes to intelligence however ...
mountainriver 1y ago
I literally talked to 5 customers last week that needed fine tuning, legitimately needed it. I get if you’re just doing basic RAG on text you generally don’t but that’s only part of the ecosystem
- vlovich123 1y ago
  I didn’t say it didn’t need it, just that it needed it less. For example, previous models needed a lot of careful fine tuning to properly do tool calling. OSS does not need that.

asabla 1y ago

I'm always so confused by those statements as well. Because just like you, I feel that the 20B version is really good at following instructions. Some of the qwen models are too, but they seem to need a bit more handholding. This is of course just anecdotal from my end. And I've been slacking on keeping up with evals while testing at home

corlinp 1y ago

The performance is great, but the censorship is ridiculous for me. I tried it as a backend for my game Guessix[1], but it would refuse for ridiculous reasons like "Cannot answer questions about copyrighted works like Harry Potter." 1. https://guessix.com/ https://guessix.com/

BoorishBears 1y ago
Use constrained generation
- artdigital 1y ago
  Mind explaining?
  - 7thpower 1y ago
    Curious as well
  - BoorishBears 1y ago
    If you constrain the model to a JSON schema, most frivolous refusals go away. And if you finetune on a few formatted examples the effect is even greater
- corlinp 1y ago
  Do you mean like structured outputs? Unfortunately here the model is guided to explicitly tell you when you violate the rules and why, it can confuse it's system rules with the game rules and say you're not allowed to ask a question about copyrighted material etc.
artdigital 1y ago
Try the uncensored/jailbroken variants like openai-gpt-oss-20b-abliterated-uncensored-neo-imatrix I just tried to ask it how to make crystal meth and it generated a very detailed step by step guide
- 1y ago