6 ms·

Open Source isn't even within 50% of what the SOTA models are. Benchmarks are toys, real world use is vastly different, and that's where they seriously lag. Wh

by echelon 6mo ago

Open Source isn't even within 50% of what the SOTA models are. Benchmarks are toys, real world use is vastly different, and that's where they seriously lag. Why should anyone waste time on poorer results? I'd rather pay my $200/mo because my time matters. I'm not a poor college student anymore, and I need more return on my time. I'm not shitting on open weights here - I want open source to win. I just don't see how that's possible. It's like Photoshop vs. Gimp. Not only is the Gimp UX awful, but it didn't even offer (maybe still doesn't?) full bit depth support. For a hacker with free time, that's fine. But if my primary job function is to transform graphics in exchange for money, I'm paying for the better tool. Gimp is entirely a no-go in a professional setting. Or it's like Google Docs / Microsoft Office vs. LibreOffice. LibreOffice is still pretty trash compared to the big tools. It's not just that Google and Microsoft have more money, but their products are involved in larger scale feedback loops that refine the product much more quickly. But with weights it's even worse than bad UX. These open weights models just aren't as smart. They're not getting RLHF'd on real world data. The developers of these open weights models can game benchmarks, but the actual intelligence for real world problems is lacking. And that's unfortunately the part that actually matters. Again, to be clear: I hate this. I want open. I just don't see how it will ever be able to catch up to full-featured products.

MostlyStable 6mo ago
I think that there will come a point when open source models are "good enough" for many tasks (they probably already are for some tasks; or at least, some small number of people seem happy with them), but, as you suggest, it will likely always (for the forseeable future at least) be the case that closed SOTA models are significantly ahead of open models, and any task which can still benefit from a smarter model (which will probably always remain some large subset of tasks) will be better done on a closed model. The trick is going to be recognizing tasks which have some ceiling on what they need and which will therefore eventually be doable by open models, and those which can always be done better if you add a bit more intelligence.
Someone1234 6mo ago
> Open Source isn't even within 50% of what the SOTA models are. When was the last time you used any of them? Because, a lot of people are actively using them for 9-5 work today, I count myself in that group. That opinion feels outdated, like it was formed a year ago+ and held onto. Or based on highly quantized versions and or small non-Thinking models. Do you really think Qwen3.6 for a specific example is "50%" as good as Opus4.7? Opus4.7 is clearly and objectively better, no debate on that, but the gap isn't anywhere near that wide. I'd call "20%" hyperbole, the true difference is difficult to exactly measure but sub-10% for their top-tier Thinking models is likely.
- vlovich123 6mo ago
  Qwen3.6 at which model size and quantization? I already think Opus 4.6 is usable but still dumb as bricks. A 20% cut off that feels like it would still be unusable. And that's not even getting to the annoyance of setting everything up to run locally & getting HW that can run it locally which basically looks like a Macbook M4 these days as the x86 side is ridiculously pricey to get decent performance out of models.
  - Someone1234 6mo ago
    At their highest model size and quant. We are discussing price and quality at the top, not what you can run on the lower end. So the starting point is Opus 4.7 pricing and we're contrasting alternatives near the top end (offered across multiple providers). Also I said 20% was hyperbole, meaning far too high.
    - vlovich123 6mo ago
      That makes no sense because the largest Qwen models are not even open weight so I’m not sure how that’s any different.
      - Someone1234 6mo ago
        Right, which isn't what we're discussing, since I mentioned "across multiple providers" in every comment about this topic. Those closed weight models aren't available like we're discussing. They're only available from the vendor that created them.
kube-system 6mo ago
There's going to be a day when we look back at $200/mo price tags and say "wow that was cheap". The breakeven at this price is 6 minutes of productivity per work day for an engineer making $200k.
- cheschire 6mo ago
  Okay, but then by that logic a person making only $20k would break even at about an hour. Are you suggesting that someone making $20k should be spending $200/mo on Claude?
  - kube-system 6mo ago
    I'm talking about the cost of labor. If you pay someone $20,000 for labor, and they save 65 minutes worth of labor per day using a $200/mo Claude subscription, you are better off buying the Claude subscription.
    - kuboble 6mo ago
      I think if you (a company) pay someone for labor, your labor cannot use personal subscription and you have to pay considerably higher api prices.
      - hrimfaxi 6mo ago
        Most companies don't provide a corporate cell phone and have no problems with answering emails from a personal account. Can't have it both ways.
      - kube-system 6mo ago
        You could it’s just against ToS. But the specific numbers in my prior comments aren’t really relevant to my point. Adjust for whatever numbers you want.
        kuboble 6mo ago
brazukadev 6mo ago
> Open Source isn't even within 50% of what the SOTA models are Who said so? GLM 5.1 is 90% Opus, at least. Some people quite happy with Kimi 2.6 too. I did not try Deepseek 4 yet but also hearing it is as good as Opus. You might be confusing open source models with local models. It is not easy to run a 1.6T model locally, but they are not 50% of SOTA models.
bandrami 6mo ago
> Why should anyone waste time on poorer results? Because in almost no real-world project is "programming time" the limiting factor?
- dymk 6mo ago
  No, it's rate at which you can solve problems, and weaker models waste your time because they don't solve problems at the same speed.
  - hunterpayne 6mo ago
    No, its the number of debug cycles you need to solve said problems. That's the major attribute that controls dev time. And models require far more than I need. You are paying money to take longer and produce worse code. If its different for you, that's a you problem.
- bdangubic 6mo ago
  amazing how often is this repeated on here are some sort of a gospel SWEs pass down to one another to continue this charade. I have worked in this industry for 30+ years on countless projects, last decade+ as consultant - at every single project (every single one) programming time was the limiting factor. there is a whole industry inside our industry dealing with “processes” and “how to estimate” (apparently we are incapable of doing that) and whatnot, all because the actual programming time is always a limiting factor and there isn’t an even close 2nd
  - bandrami 6mo ago
    That's just not my experience. Making the software in the first place is never even the cost center.
  - hypnoce_fr 6mo ago
    What counts as programming time ? Writing ? Reviewing ? Compiling ? Debugging ? It also depends the industry. From idea to production, the limiting factor is not always writing the code, and in my experience (15years in fintech) it almost has never been. Discussion, alignment, compilation, heavy testing pipelines, shipping, all of this on a 30million line monorepo. On a greenfield 10k line repo, yes, AI really shines. In other cases, it’s currently just a helper on very specific narrow tasks, that is not always programming.
twobitshifter 6mo ago
Unless you are getting outside of your comfort zone and taking a month off from your $200 subscription, every other month, I can’t see how you can make the universal claim that the open weights models are all 50% as good. Just today, DeepSeek released a new model, so nobody knows how that will compare, a week ago it was Gemma 4, etc. I’m okay with you making a comparison, but state the model and the timeframe in which it was tested that you are basing your conclusions on.
bachmeier 6mo ago
> Benchmarks are toys, real world use is vastly different...Why should anyone waste time on poorer results? I'd rather pay my $200/mo because my time matters. This kind of rhetoric is not helpful. If you want to make a point, then make one, but this adds nothing to the conversation. Maybe open source models don't work for you. They work very well for me.
oceanplexian 6mo ago
> Benchmarks are toys, real world use is vastly different, and that's where they seriously lag. I'm not disagreeing per-se but if you think the benchmarks are flawed and "my real world usage" is more reflective of model capabilities, why not write some benchmarks of your own? You stand to make a lot of money and gain a lot of clout in the industry if you've figured out a better way to measure model capability, maybe the frontier labs would hire you.
conrs 6mo ago
IMO It's a different and new model. We're engineers, and we're rich. It's not going to be good enough for us. But the much larger market by far is all the people who used to HAVE to work with engineers. They now have optionality; the pendulum is going to swing.
swader999 6mo ago
Also, this space will (and perhaps already is for some of us) be an arms race. Sure you can go local but hosted will always be able to offer more and if you want to be competitive, you'll need to be using the most capable.
nancyminusone 6mo ago
People pirate photoshop and office if they don't want to pay for it, making it as "free" as GIMP. If there is a free option people will use it. never underestimate the cheapskates.
kardos 6mo ago
If sharing all of your code with the closed providers is OK then it works. If that is a blocker, open weights becomes much more compelling...
joquarky 6mo ago
What will you do when they stop burning cash and the $200 plan becomes $2000?
jawilson2 6mo ago
I think the problem is that we're all waiting for the patented Silicon Value Rug Pull and ensuing enshittification, where there are a dozen tiers of products, you need 4 of them, and they now cost $2000/month. I want to hedge against that.
lelanthran 6mo ago
> Open Source isn't even within 50% of what the SOTA models are. The gap has been shrinking with each release, and the SOTA has already run into diminishing returns for each extra unit of data+computation it uses. Do you really want to bet that the gap will not eventually be a hairs breadth?

by echelon 6mo ago

MostlyStable 6mo ago

I think that there will come a point when open source models are "good enough" for many tasks (they probably already are for some tasks; or at least, some small number of people seem happy with them), but, as you suggest, it will likely always (for the forseeable future at least) be the case that closed SOTA models are significantly ahead of open models, and any task which can still benefit from a smarter model (which will probably always remain some large subset of tasks) will be better done on a closed model. The trick is going to be recognizing tasks which have some ceiling on what they need and which will therefore eventually be doable by open models, and those which can always be done better if you add a bit more intelligence.

Someone1234 6mo ago

> Open Source isn't even within 50% of what the SOTA models are. When was the last time you used any of them? Because, a lot of people are actively using them for 9-5 work today, I count myself in that group. That opinion feels outdated, like it was formed a year ago+ and held onto. Or based on highly quantized versions and or small non-Thinking models. Do you really think Qwen3.6 for a specific example is "50%" as good as Opus4.7? Opus4.7 is clearly and objectively better, no debate on that, but the gap isn't anywhere near that wide. I'd call "20%" hyperbole, the true difference is difficult to exactly measure but sub-10% for their top-tier Thinking models is likely.

vlovich123 6mo ago
Qwen3.6 at which model size and quantization? I already think Opus 4.6 is usable but still dumb as bricks. A 20% cut off that feels like it would still be unusable. And that's not even getting to the annoyance of setting everything up to run locally & getting HW that can run it locally which basically looks like a Macbook M4 these days as the x86 side is ridiculously pricey to get decent performance out of models.
- Someone1234 6mo ago
  At their highest model size and quant. We are discussing price and quality at the top, not what you can run on the lower end. So the starting point is Opus 4.7 pricing and we're contrasting alternatives near the top end (offered across multiple providers). Also I said 20% was hyperbole, meaning far too high.
  - vlovich123 6mo ago
    That makes no sense because the largest Qwen models are not even open weight so I’m not sure how that’s any different.
    - Someone1234 6mo ago
      Right, which isn't what we're discussing, since I mentioned "across multiple providers" in every comment about this topic. Those closed weight models aren't available like we're discussing. They're only available from the vendor that created them.

kube-system 6mo ago

There's going to be a day when we look back at $200/mo price tags and say "wow that was cheap". The breakeven at this price is 6 minutes of productivity per work day for an engineer making $200k.

cheschire 6mo ago
Okay, but then by that logic a person making only $20k would break even at about an hour. Are you suggesting that someone making $20k should be spending $200/mo on Claude?
- kube-system 6mo ago
  I'm talking about the cost of labor. If you pay someone $20,000 for labor, and they save 65 minutes worth of labor per day using a $200/mo Claude subscription, you are better off buying the Claude subscription.
  - kuboble 6mo ago
    I think if you (a company) pay someone for labor, your labor cannot use personal subscription and you have to pay considerably higher api prices.
    - hrimfaxi 6mo ago
      Most companies don't provide a corporate cell phone and have no problems with answering emails from a personal account. Can't have it both ways.
    - kube-system 6mo ago
      You could it’s just against ToS. But the specific numbers in my prior comments aren’t really relevant to my point. Adjust for whatever numbers you want.
      - kuboble 6mo ago

brazukadev 6mo ago

> Open Source isn't even within 50% of what the SOTA models are Who said so? GLM 5.1 is 90% Opus, at least. Some people quite happy with Kimi 2.6 too. I did not try Deepseek 4 yet but also hearing it is as good as Opus. You might be confusing open source models with local models. It is not easy to run a 1.6T model locally, but they are not 50% of SOTA models.

bandrami 6mo ago

> Why should anyone waste time on poorer results? Because in almost no real-world project is "programming time" the limiting factor?

dymk 6mo ago
No, it's rate at which you can solve problems, and weaker models waste your time because they don't solve problems at the same speed.
- hunterpayne 6mo ago
  No, its the number of debug cycles you need to solve said problems. That's the major attribute that controls dev time. And models require far more than I need. You are paying money to take longer and produce worse code. If its different for you, that's a you problem.
bdangubic 6mo ago
amazing how often is this repeated on here are some sort of a gospel SWEs pass down to one another to continue this charade. I have worked in this industry for 30+ years on countless projects, last decade+ as consultant - at every single project (every single one) programming time was the limiting factor. there is a whole industry inside our industry dealing with “processes” and “how to estimate” (apparently we are incapable of doing that) and whatnot, all because the actual programming time is always a limiting factor and there isn’t an even close 2nd
- bandrami 6mo ago
  That's just not my experience. Making the software in the first place is never even the cost center.
- hypnoce_fr 6mo ago
  What counts as programming time ? Writing ? Reviewing ? Compiling ? Debugging ? It also depends the industry. From idea to production, the limiting factor is not always writing the code, and in my experience (15years in fintech) it almost has never been. Discussion, alignment, compilation, heavy testing pipelines, shipping, all of this on a 30million line monorepo. On a greenfield 10k line repo, yes, AI really shines. In other cases, it’s currently just a helper on very specific narrow tasks, that is not always programming.

twobitshifter 6mo ago

Unless you are getting outside of your comfort zone and taking a month off from your $200 subscription, every other month, I can’t see how you can make the universal claim that the open weights models are all 50% as good. Just today, DeepSeek released a new model, so nobody knows how that will compare, a week ago it was Gemma 4, etc. I’m okay with you making a comparison, but state the model and the timeframe in which it was tested that you are basing your conclusions on.

bachmeier 6mo ago

> Benchmarks are toys, real world use is vastly different...Why should anyone waste time on poorer results? I'd rather pay my $200/mo because my time matters. This kind of rhetoric is not helpful. If you want to make a point, then make one, but this adds nothing to the conversation. Maybe open source models don't work for you. They work very well for me.

oceanplexian 6mo ago

> Benchmarks are toys, real world use is vastly different, and that's where they seriously lag. I'm not disagreeing per-se but if you think the benchmarks are flawed and "my real world usage" is more reflective of model capabilities, why not write some benchmarks of your own? You stand to make a lot of money and gain a lot of clout in the industry if you've figured out a better way to measure model capability, maybe the frontier labs would hire you.

conrs 6mo ago

IMO It's a different and new model. We're engineers, and we're rich. It's not going to be good enough for us. But the much larger market by far is all the people who used to HAVE to work with engineers. They now have optionality; the pendulum is going to swing.

swader999 6mo ago

Also, this space will (and perhaps already is for some of us) be an arms race. Sure you can go local but hosted will always be able to offer more and if you want to be competitive, you'll need to be using the most capable.

nancyminusone 6mo ago

People pirate photoshop and office if they don't want to pay for it, making it as "free" as GIMP. If there is a free option people will use it. never underestimate the cheapskates.

kardos 6mo ago

If sharing all of your code with the closed providers is OK then it works. If that is a blocker, open weights becomes much more compelling...

joquarky 6mo ago

What will you do when they stop burning cash and the $200 plan becomes $2000?

jawilson2 6mo ago

I think the problem is that we're all waiting for the patented Silicon Value Rug Pull and ensuing enshittification, where there are a dozen tiers of products, you need 4 of them, and they now cost $2000/month. I want to hedge against that.

lelanthran 6mo ago

> Open Source isn't even within 50% of what the SOTA models are. The gap has been shrinking with each release, and the SOTA has already run into diminishing returns for each extra unit of data+computation it uses. Do you really want to bet that the gap will not eventually be a hairs breadth?