5 ms·

I just ran this through a simple change I’ve asked Sonnet 4 and Opus 4.1, and it fails too. It’s a simple substitution request where I provide a Lint error tha

I just ran this through a simple change I’ve asked Sonnet 4 and Opus 4.1, and it fails too. It’s a simple substitution request where I provide a Lint error that suggests the correct change. All the models fail. I could ask someone with no development experience to do this change and they could. I worry everyone is chasing benchmarks to the detriment of general performance. Or the next token weight for the incorrect change outweigh my simple but precise instructions. Either way it’s no good Edit: With a followup “please do what I asked” sort of prompt it came through, while Opus just loops. So theres that at least

darksaints 1y ago
> I worry everyone is chasing benchmarks to the detriment of general performance. I've been worried about this for a while. I feel like Claude in particular took a step back in my own subjective performance evaluation in the switch from 3.7 to 4, while the benchmark scores leaped substantially. To be fair, benchmarking has always been the most difficult problem to solve in this space, so it's not surprising that benchmark development isn't exactly keeping pace with all of the modeling/training development happening.
- GoatInGrey 1y ago
  Not that it was better at programming, but I really miss Sonnet 3.5 for educational discussions. I've sometimes considered that what I actually miss was the improvement 3.5 delivered over other models at that time. Though since my system message for Sonnet since 3.7 has been primarily instructing it to behave like a human and have a personality, I really think we lost something.
  - walthamstow 1y ago
    I still use 3.5 today in Cursor. It's still the best model they've produced for my workflow. It's twice as fast as 4 and doesn't vomit pointless comments all over my code.
MichealCodes 1y ago
More like churning benchmarks... Release new model at max power, get all the benchmark glory, silently reduce model capability in the following weeks, repeat by releasing newer, smarter model.
- zamadatix 1y ago
  That (thankfully) can't compound, so would never be more than a one time offset. E.g. if you report a score of 60% SWE-bench verified for new model A, dumb A down to score 50%, and report a 20% improvement over A with new model B then it's pretty obvious when your last two model blogposts say 60%. The only way around this is to never report on the same benchmark versions twice, which they include too many to realistically do every release.
  - MichealCodes 1y ago
    The benchmarks are not typically ongoing, we do not often see comparisons between week 1 and week 8. Sprinkle a bit of training on the benchmarks in and you can ensure higher scores for the next model. A perfect scam loop to keep the people happy until they wise up.
    - zamadatix 1y ago
      > The benchmarks are not typically ongoing, we do not often see comparisons between week 1 and week 8 You don't need to compare "A (Week 1)" to "A (Week 8)" to be able to show "B (Week 1)" is genuinely x% better than "A (Week 1)".
      - MichealCodes 1y ago
        As I said sprinkle a bit of benchmarks polluting the training and you have your loop. Each iteration will be better at benchmarks if that's the goal and that goal/context reinforces.
        zamadatix 1y ago
Cthulhu_ 1y ago
That's what I was thinking too; the models have the same data sources (they have all scraped the internet, github, book repositories, etc), they all optimize for the same standardized tests. Other than marginally better scores in those tests (and they will cherry-pick them to make them look better), how do the various competitors differentiate from each other still? What's the USP?
- cies 1y ago
  LLM (the model) is not the agent (ClaudeCode) that uses LLMs. LLMs improve slowly, but the agents are where the real value is produced: when should it write tests, when should it try to compile, how to move fwd from a compile error, can it click on your web app to test its own work, etc. etc.
itsoktocry 1y ago
>It’s a simple substitution request where I provide a Lint error that suggests the correct change. All the models fail. I could ask someone with no development experience to do this change and they could. I don't understand why this kind of thing is useful. Do the thing yourself and move on. For every one problem like this, AI can do 10 better/faster than I can.
- beefnugs 1y ago
  You dont understand how complete unreliability is a problem? So instead of just "doing things" you want a world where you try it ai-way, fail, then "do thing" 47 times in a row, then 3 ai-way saved you 5 minutes. Then 7 ai-way fail, then try to remember hmm did this work last time or not? ai-way fails another 3 times. "do thing" 3 times. How many ai-way failed today? oh it wasted 30% of the day and i forget which ways worked or not, i better start writing that all down. Lets call it the MAGIC TOME of incantations. oh i have to rewrite the tome again the model changed
- bobbylarrybobby 1y ago
  One reason is to simply say “fix all lints” and have the model do it
- stefs 1y ago
  How can I trust it to do the complicated task well when it fails to do the simple thing?
  - enraged_camel 1y ago
    It's not about simple vs. complex. It's about the types of tasks the AI has been trained on: pattern-matching, thinking, reasoning, research. Tasks like linting and formatting a block of code are pretty simple, but also very specialized. You're much better off using formatters/linters than an AI.
    - ewoodrich 1y ago
      An agent being tasked to resolve simple issues from a compiler/test suite/linter/etc is pretty typical use case. It's not clear in this example if the linter was capable of auto fixing the problem, so ordinarily this would be a case where you'd hope an LLM would shine given specific, accurate context and known solution.
la_fayette 1y ago
At this point it would be an interesting idea, to collect examples, in a form of a community database, were LLMs miserably fail. I have examples myself...
- vunderba 1y ago
  Any such examples are often "closely guarded secrets" to prevent them from being benchmaxxed and gamed - which is absolutely what would happen if you consolidated them in a publicly available centralized repository.
  - squigz 1y ago
    This seems like a non-issue, unless I'm misunderstanding. If failures can be used to help game benchmarks, companies are doing so. They don't need us to avoid compiling such information, which would be helpful to actual users.
    - dns_snek 1y ago
      People might want to use the same test scenario in the future to see how much the models have improved. We can't do that if the example gets scraped into the training data set.
  - la_fayette 1y ago
    Since such a database should evolve continuously, I wouldn't see that as a problem. The important thing is, that each example is somehow verifiable, in the form of a unmodifiable test setup. So the LLM provides a solution, which is executed against the test to verify. Something like ACID3 Tests... But sure it can be gamed somehow in probably all setups...
Forgeties79 1y ago
> I worry everyone is chasing benchmarks to the detriment of general performance. I’m not sure this is entirely what you’re driving at, but the example I always think of in my head is “I want an AI agent that will scan through my 20 to 30,000 photos, remove all the duplicates, then organize them all in some coherent fashion.” that’s the kind of service I need right now, and it feels like something AI should be able to do, yet I have not encountered anything that remotely accomplishes this task. I’m still using Dupe Guru and depending on the ref system to not scatter my stuff all over further. Sidebar, if anybody has any recommendations for this, I would love to hear them lol
- mh- 1y ago
  The reality of that specific ask is it would not be difficult to build, but I believe it would be extremely difficult to build and offer at a price that users would pay for. So you're unlikely to find a commercial offering that does that using a (V)LM.
  - Forgeties79 1y ago
    Yeah I imagine so. Hell I would pay like $100 for them to just do it once. If they really could do it with like 99% accuracy I would pay upwards of $300 tbh. Still, that’s probably not good enough lol
    - kirkoplamen 1y ago
      Hey bro, I'd like to take this project using Claude for $300 :) Do you mind contacting me? stxcth9aoj at mozmail.com
- fullstackchris 1y ago
  azure vision / "cognitive services" can do this for literally a few bucks am i even on hacker news? how do people not know there are optimized models for specific use cases? not everything (nor should it) has to run through an LLM https://azure.microsoft.com/en-us/pricing/details/cognitive-services/computer-vision/ https://azure.microsoft.com/en-us/pricing/details/cognitive-...
PunchTornado 1y ago
Downvoted because you didn’t mention the prompt and the issue.

darksaints 1y ago

> I worry everyone is chasing benchmarks to the detriment of general performance. I've been worried about this for a while. I feel like Claude in particular took a step back in my own subjective performance evaluation in the switch from 3.7 to 4, while the benchmark scores leaped substantially. To be fair, benchmarking has always been the most difficult problem to solve in this space, so it's not surprising that benchmark development isn't exactly keeping pace with all of the modeling/training development happening.

GoatInGrey 1y ago
Not that it was better at programming, but I really miss Sonnet 3.5 for educational discussions. I've sometimes considered that what I actually miss was the improvement 3.5 delivered over other models at that time. Though since my system message for Sonnet since 3.7 has been primarily instructing it to behave like a human and have a personality, I really think we lost something.
- walthamstow 1y ago
  I still use 3.5 today in Cursor. It's still the best model they've produced for my workflow. It's twice as fast as 4 and doesn't vomit pointless comments all over my code.

MichealCodes 1y ago

More like churning benchmarks... Release new model at max power, get all the benchmark glory, silently reduce model capability in the following weeks, repeat by releasing newer, smarter model.

zamadatix 1y ago
That (thankfully) can't compound, so would never be more than a one time offset. E.g. if you report a score of 60% SWE-bench verified for new model A, dumb A down to score 50%, and report a 20% improvement over A with new model B then it's pretty obvious when your last two model blogposts say 60%. The only way around this is to never report on the same benchmark versions twice, which they include too many to realistically do every release.
- MichealCodes 1y ago
  The benchmarks are not typically ongoing, we do not often see comparisons between week 1 and week 8. Sprinkle a bit of training on the benchmarks in and you can ensure higher scores for the next model. A perfect scam loop to keep the people happy until they wise up.
  - zamadatix 1y ago
    > The benchmarks are not typically ongoing, we do not often see comparisons between week 1 and week 8 You don't need to compare "A (Week 1)" to "A (Week 8)" to be able to show "B (Week 1)" is genuinely x% better than "A (Week 1)".
    - MichealCodes 1y ago
      As I said sprinkle a bit of benchmarks polluting the training and you have your loop. Each iteration will be better at benchmarks if that's the goal and that goal/context reinforces.
      - zamadatix 1y ago

Cthulhu_ 1y ago

That's what I was thinking too; the models have the same data sources (they have all scraped the internet, github, book repositories, etc), they all optimize for the same standardized tests. Other than marginally better scores in those tests (and they will cherry-pick them to make them look better), how do the various competitors differentiate from each other still? What's the USP?

cies 1y ago
LLM (the model) is not the agent (ClaudeCode) that uses LLMs. LLMs improve slowly, but the agents are where the real value is produced: when should it write tests, when should it try to compile, how to move fwd from a compile error, can it click on your web app to test its own work, etc. etc.

itsoktocry 1y ago

>It’s a simple substitution request where I provide a Lint error that suggests the correct change. All the models fail. I could ask someone with no development experience to do this change and they could. I don't understand why this kind of thing is useful. Do the thing yourself and move on. For every one problem like this, AI can do 10 better/faster than I can.

beefnugs 1y ago
You dont understand how complete unreliability is a problem? So instead of just "doing things" you want a world where you try it ai-way, fail, then "do thing" 47 times in a row, then 3 ai-way saved you 5 minutes. Then 7 ai-way fail, then try to remember hmm did this work last time or not? ai-way fails another 3 times. "do thing" 3 times. How many ai-way failed today? oh it wasted 30% of the day and i forget which ways worked or not, i better start writing that all down. Lets call it the MAGIC TOME of incantations. oh i have to rewrite the tome again the model changed
bobbylarrybobby 1y ago
One reason is to simply say “fix all lints” and have the model do it
stefs 1y ago
How can I trust it to do the complicated task well when it fails to do the simple thing?
- enraged_camel 1y ago
  It's not about simple vs. complex. It's about the types of tasks the AI has been trained on: pattern-matching, thinking, reasoning, research. Tasks like linting and formatting a block of code are pretty simple, but also very specialized. You're much better off using formatters/linters than an AI.
  - ewoodrich 1y ago
    An agent being tasked to resolve simple issues from a compiler/test suite/linter/etc is pretty typical use case. It's not clear in this example if the linter was capable of auto fixing the problem, so ordinarily this would be a case where you'd hope an LLM would shine given specific, accurate context and known solution.

la_fayette 1y ago

At this point it would be an interesting idea, to collect examples, in a form of a community database, were LLMs miserably fail. I have examples myself...

vunderba 1y ago
Any such examples are often "closely guarded secrets" to prevent them from being benchmaxxed and gamed - which is absolutely what would happen if you consolidated them in a publicly available centralized repository.
- squigz 1y ago
  This seems like a non-issue, unless I'm misunderstanding. If failures can be used to help game benchmarks, companies are doing so. They don't need us to avoid compiling such information, which would be helpful to actual users.
  - dns_snek 1y ago
    People might want to use the same test scenario in the future to see how much the models have improved. We can't do that if the example gets scraped into the training data set.
- la_fayette 1y ago
  Since such a database should evolve continuously, I wouldn't see that as a problem. The important thing is, that each example is somehow verifiable, in the form of a unmodifiable test setup. So the LLM provides a solution, which is executed against the test to verify. Something like ACID3 Tests... But sure it can be gamed somehow in probably all setups...

Forgeties79 1y ago

> I worry everyone is chasing benchmarks to the detriment of general performance. I’m not sure this is entirely what you’re driving at, but the example I always think of in my head is “I want an AI agent that will scan through my 20 to 30,000 photos, remove all the duplicates, then organize them all in some coherent fashion.” that’s the kind of service I need right now, and it feels like something AI should be able to do, yet I have not encountered anything that remotely accomplishes this task. I’m still using Dupe Guru and depending on the ref system to not scatter my stuff all over further. Sidebar, if anybody has any recommendations for this, I would love to hear them lol

mh- 1y ago
The reality of that specific ask is it would not be difficult to build, but I believe it would be extremely difficult to build and offer at a price that users would pay for. So you're unlikely to find a commercial offering that does that using a (V)LM.
- Forgeties79 1y ago
  Yeah I imagine so. Hell I would pay like $100 for them to just do it once. If they really could do it with like 99% accuracy I would pay upwards of $300 tbh. Still, that’s probably not good enough lol
  - kirkoplamen 1y ago
    Hey bro, I'd like to take this project using Claude for $300 :) Do you mind contacting me? stxcth9aoj at mozmail.com
fullstackchris 1y ago
azure vision / "cognitive services" can do this for literally a few bucks am i even on hacker news? how do people not know there are optimized models for specific use cases? not everything (nor should it) has to run through an LLM https://azure.microsoft.com/en-us/pricing/details/cognitive-services/computer-vision/ https://azure.microsoft.com/en-us/pricing/details/cognitive-...

PunchTornado 1y ago

Downvoted because you didn’t mention the prompt and the issue.