4 ms·

We're also in benchmark saturation territory. I heard it speculated that Anthropic emphasizes benchmarks less in their publications because internally they don'

by verdverm 11mo ago

We're also in benchmark saturation territory. I heard it speculated that Anthropic emphasizes benchmarks less in their publications because internally they don't care about them nearly as much as making a model that works well on the day-to-day

quantumHazer 11mo ago
Seems pretty false if you look at the model card and web site of Opus 4.5 that is… (check notes) their latest model.
- verdverm 11mo ago
  Building a good model generally means it will do well on benchmarks too. The point of the speculation is that Anthropic is not focused on benchmaxxing which is why they have models people like to use for their day-to-day. I use Gemini, Anthropic stole $50 from me (expired and kept my prepaid credits) and I have not forgiven them yet for it, but people rave about claude for coding so I may try the model again through Vertex Ai... The person who made the speculation I believe was more talking about blog posts and media statements than model cards. Most ai announcements come with benchmark touting, Anthropic supposedly does less / little of this in their announcements. I haven't seen or gathered the data to know what is truth
  - elcritch 11mo ago
    You could try Codex cli. I prefer it over Claude code now, but only slightly.
    - verdverm 11mo ago
      No thanks, not touching anything Oligarchy Altman is behind
Mistletoe 11mo ago
How do you measure whether it works better day to day without benchmarks?
- standardUser 11mo ago
  Subscriptions.
  - mrguyorama 11mo ago
    Ah yes, humans are famously empirical in their behavior and we definitely do not have direct evidence of the "best" sports players being much more likely than the average to be superstitious or do things like wear "lucky underwear" or buy right into scam bracelets that "give you more balance" using a holographic sticker.
    - standardUser 11mo ago
      It's all the shareholders care about. These are not research institutions.
- bulbar 11mo ago
  Manually labeling answers maybe? There exist a lot of infrastructure built around and as it's heavily used for 2 decades and it's relatively cheap. That's still benchmarking of course, but not utilizing any of the well known / public ones.
- verdverm 11mo ago
  Internal evals, Big AI certainly has good, proprietary training and eval data, it's one reason why their models are better
  - aydyn 11mo ago
    Then publish the results of those internal evals. Public benchmark saturation isn't an excuse to be un-quantitative.
brokensegue 11mo ago
how do you quantitatively measure day-to-day quality? only thing i can think is A/B tests which take a while to evaluate
- verdverm 11mo ago
  more or less this, but also synthetic if you think about GANs, it's all the same concept 1. train model (agent) 2. train another model (agent) to do something interesting with/to the main model 3. gain new capabilities 4. iterate You can use a mix of both real and synthetic chat sessions or whatever you want your model to be good at. Mid/late training seems to be where you start crafting personality and expertises. Getting into the guts of agentic systems has me believing we have quite a bit of runway for iteration here, especially as we move beyond single model / LLM training. I still need to get into what all is de jour in the RL / late training, that's where a lot of opportunity lies from my understanding so far Nathan Lambert (https://bsky.app/profile/natolambert.bsky.social https://bsky.app/profile/natolambert.bsky.social) from Ai2 (https://allenai.org/ https://allenai.org/) & RLHF Book (https://rlhfbook.com/ https://rlhfbook.com/) has a really great video out yesterday about the experience training Olmo 3 Think https://www.youtube.com/watch?v=uaZ3yRdYg8A https://www.youtube.com/watch?v=uaZ3yRdYg8A
HDThoreaun 11mo ago
Arc-AGI is just an iq test. I don’t see the problem with training it to be good at iq tests because that’s a skill that translates well.
- CamperBob2 11mo ago
  Exactly. In principle, at least, the only way to overfit to Arc-AGI is to actually be that smart. Edit: if you disagree, try actually TAKING the Arc-AGI 2 test, then post.
  - npinsker 11mo ago
    Completely false. This is like saying being good at chess is equivalent to being smart. Look no farther than the hodgepodge of independent teams running cheaper models (and no doubt thousands of their own puzzles, many of which surely overlap with the private set) that somehow keep up with SotA, to see how impactful proper practice can be. The benchmark isn’t particularly strong against gaming, especially with private data.
    - CamperBob2 11mo ago
      Completely false. This is like saying being good at chess is equivalent to being smart. No, it isn't. Go take the test yourself and you'll understand how wrong that is. Arc-AGI is intentionally unlike any other benchmark.
      - fwip 11mo ago
        Took a couple just now. It seems like a straight-forward generalization of the IQ tests I've taken before, reformatted into an explicit grid to be a little bit friendlier to machines. Not to humble-brag, but I also outperform on IQ tests well beyond my actual intelligence, because "find the pattern" is fun for me and I'm relatively good at visual-spatial logic. I don't find their ability to measure 'intelligence' very compelling.
        11mo ago
stego-tech 11mo ago
These models still consistently fail the only benchmark that matters: if I give you a task, can you complete it successfully without making shit up? Thus far they all fail. Code outputs don’t run, or variables aren’t captured correctly, or hallucinations are stated as factual rather than suspect or “I don’t know.” It’s 2000’s PC gaming all over again (“gotta game the benchmark!”).
- verdverm 11mo ago
  I'm not sure, here's my anecdotal counter example, was able to get gemini-2.5-flash, in two turns, to understand and implement something I had done separately first, and it found another bug (also that I had fixed, but forgot was in this path) That I was able to have a flash model replicate the same solution I had, to two problems in two turns, it's just the opposite experience of your consistency argument. I'm using tasks I've already solved as the evals while developing my custom agentic setup (prompts/tools/envs). They are able to do more of them today then they were even 6-12 months ago (pre-thinking models). https://bsky.app/profile/verdverm.com/post/3m7p7gtwo5c2v https://bsky.app/profile/verdverm.com/post/3m7p7gtwo5c2v
  - stego-tech 11mo ago
    And therein lies the rub for why I still approach this technology with caution, rather than charge in full steam ahead: variable outputs based on immensely variable inputs. I read stories like yours all the time, and it encourages me to keep trying LLMs from almost all the major vendors (Google being a noteworthy exception while I try and get off their platform). I want to see the magic others see, but when my IT-brain starts digging in the guts of these things, I’m always disappointed at how unstructured and random they ultimately are. Getting back to the benchmark angle though, we’re firmly in the era of benchmark gaming - hence my quip about these things failing “the only benchmark that matters.” I meant for that to be interpreted along the lines of, “trust your own results rather than a spreadsheet matrix of other published benchmarks”, but I clearly missed the mark in making that clear. That’s on me.
    - verdverm 11mo ago

quantumHazer 11mo ago

Seems pretty false if you look at the model card and web site of Opus 4.5 that is… (check notes) their latest model.

verdverm 11mo ago
Building a good model generally means it will do well on benchmarks too. The point of the speculation is that Anthropic is not focused on benchmaxxing which is why they have models people like to use for their day-to-day. I use Gemini, Anthropic stole $50 from me (expired and kept my prepaid credits) and I have not forgiven them yet for it, but people rave about claude for coding so I may try the model again through Vertex Ai... The person who made the speculation I believe was more talking about blog posts and media statements than model cards. Most ai announcements come with benchmark touting, Anthropic supposedly does less / little of this in their announcements. I haven't seen or gathered the data to know what is truth
- elcritch 11mo ago
  You could try Codex cli. I prefer it over Claude code now, but only slightly.
  - verdverm 11mo ago
    No thanks, not touching anything Oligarchy Altman is behind

Mistletoe 11mo ago

How do you measure whether it works better day to day without benchmarks?

standardUser 11mo ago
Subscriptions.
- mrguyorama 11mo ago
  Ah yes, humans are famously empirical in their behavior and we definitely do not have direct evidence of the "best" sports players being much more likely than the average to be superstitious or do things like wear "lucky underwear" or buy right into scam bracelets that "give you more balance" using a holographic sticker.
  - standardUser 11mo ago
    It's all the shareholders care about. These are not research institutions.
bulbar 11mo ago
Manually labeling answers maybe? There exist a lot of infrastructure built around and as it's heavily used for 2 decades and it's relatively cheap. That's still benchmarking of course, but not utilizing any of the well known / public ones.
verdverm 11mo ago
Internal evals, Big AI certainly has good, proprietary training and eval data, it's one reason why their models are better
- aydyn 11mo ago
  Then publish the results of those internal evals. Public benchmark saturation isn't an excuse to be un-quantitative.

brokensegue 11mo ago

how do you quantitatively measure day-to-day quality? only thing i can think is A/B tests which take a while to evaluate

verdverm 11mo ago
more or less this, but also synthetic if you think about GANs, it's all the same concept 1. train model (agent) 2. train another model (agent) to do something interesting with/to the main model 3. gain new capabilities 4. iterate You can use a mix of both real and synthetic chat sessions or whatever you want your model to be good at. Mid/late training seems to be where you start crafting personality and expertises. Getting into the guts of agentic systems has me believing we have quite a bit of runway for iteration here, especially as we move beyond single model / LLM training. I still need to get into what all is de jour in the RL / late training, that's where a lot of opportunity lies from my understanding so far Nathan Lambert (https://bsky.app/profile/natolambert.bsky.social https://bsky.app/profile/natolambert.bsky.social) from Ai2 (https://allenai.org/ https://allenai.org/) & RLHF Book (https://rlhfbook.com/ https://rlhfbook.com/) has a really great video out yesterday about the experience training Olmo 3 Think https://www.youtube.com/watch?v=uaZ3yRdYg8A https://www.youtube.com/watch?v=uaZ3yRdYg8A

HDThoreaun 11mo ago

Arc-AGI is just an iq test. I don’t see the problem with training it to be good at iq tests because that’s a skill that translates well.

CamperBob2 11mo ago
Exactly. In principle, at least, the only way to overfit to Arc-AGI is to actually be that smart. Edit: if you disagree, try actually TAKING the Arc-AGI 2 test, then post.
- npinsker 11mo ago
  Completely false. This is like saying being good at chess is equivalent to being smart. Look no farther than the hodgepodge of independent teams running cheaper models (and no doubt thousands of their own puzzles, many of which surely overlap with the private set) that somehow keep up with SotA, to see how impactful proper practice can be. The benchmark isn’t particularly strong against gaming, especially with private data.
  - CamperBob2 11mo ago
    Completely false. This is like saying being good at chess is equivalent to being smart. No, it isn't. Go take the test yourself and you'll understand how wrong that is. Arc-AGI is intentionally unlike any other benchmark.
    - fwip 11mo ago
      Took a couple just now. It seems like a straight-forward generalization of the IQ tests I've taken before, reformatted into an explicit grid to be a little bit friendlier to machines. Not to humble-brag, but I also outperform on IQ tests well beyond my actual intelligence, because "find the pattern" is fun for me and I'm relatively good at visual-spatial logic. I don't find their ability to measure 'intelligence' very compelling.
      - 11mo ago

stego-tech 11mo ago

These models still consistently fail the only benchmark that matters: if I give you a task, can you complete it successfully without making shit up? Thus far they all fail. Code outputs don’t run, or variables aren’t captured correctly, or hallucinations are stated as factual rather than suspect or “I don’t know.” It’s 2000’s PC gaming all over again (“gotta game the benchmark!”).

verdverm 11mo ago
I'm not sure, here's my anecdotal counter example, was able to get gemini-2.5-flash, in two turns, to understand and implement something I had done separately first, and it found another bug (also that I had fixed, but forgot was in this path) That I was able to have a flash model replicate the same solution I had, to two problems in two turns, it's just the opposite experience of your consistency argument. I'm using tasks I've already solved as the evals while developing my custom agentic setup (prompts/tools/envs). They are able to do more of them today then they were even 6-12 months ago (pre-thinking models). https://bsky.app/profile/verdverm.com/post/3m7p7gtwo5c2v https://bsky.app/profile/verdverm.com/post/3m7p7gtwo5c2v
- stego-tech 11mo ago
  And therein lies the rub for why I still approach this technology with caution, rather than charge in full steam ahead: variable outputs based on immensely variable inputs. I read stories like yours all the time, and it encourages me to keep trying LLMs from almost all the major vendors (Google being a noteworthy exception while I try and get off their platform). I want to see the magic others see, but when my IT-brain starts digging in the guts of these things, I’m always disappointed at how unstructured and random they ultimately are. Getting back to the benchmark angle though, we’re firmly in the era of benchmark gaming - hence my quip about these things failing “the only benchmark that matters.” I meant for that to be interpreted along the lines of, “trust your own results rather than a spreadsheet matrix of other published benchmarks”, but I clearly missed the mark in making that clear. That’s on me.
  - verdverm 11mo ago