3 ms·

Funny, I find Codex to still be better at Coding than Opus or Fable. I A/B tested on a whole array of prompts between Codex and Fable, and Fable almost always

Funny, I find Codex to still be better at Coding than Opus or Fable. I A/B tested on a whole array of prompts between Codex and Fable, and Fable almost always found that Codex had produced a better plan and covered more edge cases than it did itself. For every problem I gave the exact same prompt to both models, then I had each analyze the other's output. For roughly 80% of the prompts, Fable acknowledged that Codex's output was an improvement on its own, for 20% the converse situation occurred. There was one egregious case where Fable suggested deploying code which would have resulted in a production bug, an edge-case which Codex identified and proposed a fix. Note: this is all for optimized Rust code designed to be highly CPU and memory efficient. I do prefer Anthropic's models for any tasks with front-end/design work needed. But I don't do much of that kind of work usually.

hodgehog11 4mo ago
I've used them back to back as well. Codex is good at specific tasks; it doesn't try to go big, it does what it's told provided the task is relatively procedural. If Codex can make progress on a task, why would I give it to Fable? Fable fumbled the one simple task that I gave it too. I gave it multiple very hard open-ended tasks (effectively math tasks) involving research code and it crushed them. It's the first model I've seen that can do that. The current Codex will never produce the type of code Fable gave me no matter how many times I run the same problem at it, because it won't stop trying naive rubbish. And if I tell Codex to try to improve the code, it can't figure out why trying the same classical tricks isn't making it work better, regardless of what I tell it. Opus is marginally better because it can at least recognize some subtleties over time, but still disappointing because it has no idea how to deal with them. Most programmers want precision instruments for their workflow. That's fine, use the right tool for the job. In my line of work, I need crazy solutions because the obvious stuff doesn't work. That's where Fable shined for me.
- saberience 4mo ago
  I can buy the crazy solutions part. Codex is highly steerable I find whereas Opus (and Fable in a different way) worked for longer and often did MORE than I wanted it to do. Fable was absolutely maximalist. We tested it on a fairly innocuous looking prompt and it ended up working for an hour and costing roughly 1000 dollars in tokens. So I think Fable probably has it's place, but if you know what you want and roughly how to get it, it seems like it would waste too many tokens/money.

hodgehog11 4mo ago

I've used them back to back as well. Codex is good at specific tasks; it doesn't try to go big, it does what it's told provided the task is relatively procedural. If Codex can make progress on a task, why would I give it to Fable? Fable fumbled the one simple task that I gave it too. I gave it multiple very hard open-ended tasks (effectively math tasks) involving research code and it crushed them. It's the first model I've seen that can do that. The current Codex will never produce the type of code Fable gave me no matter how many times I run the same problem at it, because it won't stop trying naive rubbish. And if I tell Codex to try to improve the code, it can't figure out why trying the same classical tricks isn't making it work better, regardless of what I tell it. Opus is marginally better because it can at least recognize some subtleties over time, but still disappointing because it has no idea how to deal with them. Most programmers want precision instruments for their workflow. That's fine, use the right tool for the job. In my line of work, I need crazy solutions because the obvious stuff doesn't work. That's where Fable shined for me.

saberience 4mo ago
I can buy the crazy solutions part. Codex is highly steerable I find whereas Opus (and Fable in a different way) worked for longer and often did MORE than I wanted it to do. Fable was absolutely maximalist. We tested it on a fairly innocuous looking prompt and it ended up working for an hour and costing roughly 1000 dollars in tokens. So I think Fable probably has it's place, but if you know what you want and roughly how to get it, it seems like it would waste too many tokens/money.