4 ms·

The simplest explanation is that 'Fable-level' doesn't mean anything; it's just hype, and there's not much difference in capability. All you need to have Fable

The simplest explanation is that 'Fable-level' doesn't mean anything; it's just hype, and there's not much difference in capability. All you need to have Fable-level AI is to announce it, and have enough fans shift from insisting that model Y is the best now, way better than model X.

cromka 2mo ago
Especially it today's Fable, which was running extremely slow and was clearly dumbed down as they faced outages.
matheusmoreira 2mo ago
> The simplest explanation is that 'Fable-level' doesn't mean anything; it's just hype, and there's not much difference in capability. Couldn't be further from the truth. The models can be tested and statistically evaluated. I ran a massive Fable max code review on my lone lisp codebase. Now that I have switched to OpenAI, I decided to run an equivalent review using Sol max and compare them. I'm keeping all data so I can thoroughly evaluate their performance in multiple areas such as correctness, rigor, performance, security, maintainability, consistency, among others. Fable pass is 100% done and I'm around 70% done with the Sol pass. Preliminary results are already becoming clear: Sol is capable of reproducing around 70% to 90% of Fable's performance. Haven't tested open weight models but I'd wager they have the same performance as Sol if not lower. It seems Fable is still king, I'm afraid. It's undeniable that OpenAI is providing huge value here: up to 90% Fable performance at multiple times the usage on a subscription than what Anthropic offers us is a phenomenal deal. However, if one desires the best model, to me it looks like Fable is still it.
- Planktonne 2mo ago
  You sound very certain, but so do all the people who disagree with you, and they've got their own private benchmarks. You'll forgive me if I remain unconvinced.
  - matheusmoreira 2mo ago
    If I sounded certain, it was not intentional. I made sure to hedge my statistical claims with "seems" and "looks like". I'm no AI lab, I'm just a random subscription user trying to get the most value out of them. I'm just saying it's not wise to simply put all these models in the same bucket and say any differences are due to vibes or hype. They are clearly different. We can and should scrutinize the testing methodology but it's not exactly fair to just ignore the results. I don't intend for my benchmark to be private. The core component of my test is my parallel code review skill which is already on my GitHub. I'll be publishing the results on my website when it's done. Anyone could take the skill and reproduce the test using multiple models against any codebase out there, then analyse the depth of each model's findings.
usef- 2mo ago
Are you just guessing this? Using it, it's was clearly a jump in intelligence over previous models. It (mythos) was first made public in April so it's not a surprise that others would catch up, though.
- Planktonne 2mo ago
  > it's was clearly a jump in intelligence over previous models. Some people thought this. Some people didn't. Some people thought it was a step backwards. We don't have a solid ground-truth way of estimating this.
  - usef- 2mo ago
    In theory that's what benchmarks are for. If you're assuming they're "benchmaxxed", note that new benchmarks have been released after the model came out that it did well on without being trained. Do you have any links to credible claims or independent benchmarks that found they were a step down? Or a specific task that worked worse for you? My private benchmark tasks, and independent evaluators I've seen all overwhelmingly showed improvement. Every model released for the past four years has had claims on the internet of getting worse. But transcripts are permanent so it should be easy to give a side by side of an earlier task that is now worse. I don't ever see people do that. Instead I see that every single task on a computer that is verifiable is now night-and-day better. I'm genuinely curious if you've used them yourself or you're judging this based on internet commentary?

cromka 2mo ago

Especially it today's Fable, which was running extremely slow and was clearly dumbed down as they faced outages.

matheusmoreira 2mo ago

> The simplest explanation is that 'Fable-level' doesn't mean anything; it's just hype, and there's not much difference in capability. Couldn't be further from the truth. The models can be tested and statistically evaluated. I ran a massive Fable max code review on my lone lisp codebase. Now that I have switched to OpenAI, I decided to run an equivalent review using Sol max and compare them. I'm keeping all data so I can thoroughly evaluate their performance in multiple areas such as correctness, rigor, performance, security, maintainability, consistency, among others. Fable pass is 100% done and I'm around 70% done with the Sol pass. Preliminary results are already becoming clear: Sol is capable of reproducing around 70% to 90% of Fable's performance. Haven't tested open weight models but I'd wager they have the same performance as Sol if not lower. It seems Fable is still king, I'm afraid. It's undeniable that OpenAI is providing huge value here: up to 90% Fable performance at multiple times the usage on a subscription than what Anthropic offers us is a phenomenal deal. However, if one desires the best model, to me it looks like Fable is still it.

Planktonne 2mo ago
You sound very certain, but so do all the people who disagree with you, and they've got their own private benchmarks. You'll forgive me if I remain unconvinced.
- matheusmoreira 2mo ago
  If I sounded certain, it was not intentional. I made sure to hedge my statistical claims with "seems" and "looks like". I'm no AI lab, I'm just a random subscription user trying to get the most value out of them. I'm just saying it's not wise to simply put all these models in the same bucket and say any differences are due to vibes or hype. They are clearly different. We can and should scrutinize the testing methodology but it's not exactly fair to just ignore the results. I don't intend for my benchmark to be private. The core component of my test is my parallel code review skill which is already on my GitHub. I'll be publishing the results on my website when it's done. Anyone could take the skill and reproduce the test using multiple models against any codebase out there, then analyse the depth of each model's findings.

usef- 2mo ago

Are you just guessing this? Using it, it's was clearly a jump in intelligence over previous models. It (mythos) was first made public in April so it's not a surprise that others would catch up, though.

Planktonne 2mo ago
> it's was clearly a jump in intelligence over previous models. Some people thought this. Some people didn't. Some people thought it was a step backwards. We don't have a solid ground-truth way of estimating this.
- usef- 2mo ago
  In theory that's what benchmarks are for. If you're assuming they're "benchmaxxed", note that new benchmarks have been released after the model came out that it did well on without being trained. Do you have any links to credible claims or independent benchmarks that found they were a step down? Or a specific task that worked worse for you? My private benchmark tasks, and independent evaluators I've seen all overwhelmingly showed improvement. Every model released for the past four years has had claims on the internet of getting worse. But transcripts are permanent so it should be easy to give a side by side of an earlier task that is now worse. I don't ever see people do that. Instead I see that every single task on a computer that is verifiable is now night-and-day better. I'm genuinely curious if you've used them yourself or you're judging this based on internet commentary?