3 ms·

Sol easily outperforms Fable on every task I've tried it on.

enraged_camel 29d ago
I can't speak for others but I have a feeling you're in the very small minority with this take. You could say Sol is faster and cheaper and that's true. Outperforms Fable? Impossible to believe without hard evidence.
- athrowaway3z 29d ago
  I dont think that feeling is entirely useful. Because Claude doesn't allow third party harnesses on their subscriptions I doubt the majority of signals you're getting are actually that significant on pure model quality. I suspect you're right on Sol not outperforming Fable; but i've not used Fable that much. --- But, fwiw, in my custom harness between Sol & Opus 4.8 - then Sol wins by a ridiculous margin as Opus keeps claiming slightly wrong things with certainty much more.
  - andxor 29d ago
    This is not saying much. Opus 4.8 is ancient history.
andxor 29d ago
That's not my experience and I suspect it's not most people's experience. Out of curiosity, what's the hardest task you tried?
- Koffiepoeder 29d ago
  For me something the likes of: design a CDM for integrating these 5 logistical systems, with full docs and examples provided for each, as well as modeled transports specific to our business. Prompt was of course much longer. Both failed spectacularly. But sol's output at least contained interesting findings and some useful parts, as well as not being 20000 words of unbearable language.
  - Demiurge 29d ago
    You’re thinking long horizon tasks. I agree that Sol is great at it. I don’t think it’s smarter in quick win tasks that are still difficult. This is where intelligence is not one of a kind, these systems have different pros and cons. I use Sol as an architect and fable as a brilliant single task solver.
- CSMastermind 28d ago
  I have both set up with full access to all the repos at the company, infrastructure, deployment pipelines, etc. I can tell Sol, "Hey we need to update this core database schema to handle this new use case" and it will masterfully handle the update, version the API, roll the consumers over, including versioning the Kafka schemas, deploying things in sequence, watching the deployments to make sure the new services act actually active before cutting over consumers, exercising the website and mobile apps in staging environments before releasing to production, etc. Fable just falls over on long horizon tasks, it does partial implementations, it cuts corners, it gives up, it doesn't verify it's work, it loses track of what it's doing, etc. It's fine for specific well scoped tasks but can't take high level guidance for complex updates.
adonese 29d ago
what i found to work well with me is Fable for design / ideas and Sol for implementation. Codex models just tend to be more attentive and follow through instructions. Whereby claude models are weaker on this area (they tend to cut corners).
- glub 29d ago
  Claude models tend to cut corners during design/ideation too. It becomes especially visible once you pair Sol as advisor to Fable. Sol will start going crazy - "hey, you said this, and it's actually false, i checked that", or "you need a hash chained, triple encrypted, secure-enclave backed storage for this". But I actually prefer it this way. Sol is a master of overengineering and being overly scrupulous, so Fable balances this out, and I can always say "don't listen to Sol's advisory" about 50% of the time.

I can't speak for others but I have a feeling you're in the very small minority with this take. You could say Sol is faster and cheaper and that's true. Outperforms Fable? Impossible to believe without hard evidence.

athrowaway3z 29d ago
I dont think that feeling is entirely useful. Because Claude doesn't allow third party harnesses on their subscriptions I doubt the majority of signals you're getting are actually that significant on pure model quality. I suspect you're right on Sol not outperforming Fable; but i've not used Fable that much. --- But, fwiw, in my custom harness between Sol & Opus 4.8 - then Sol wins by a ridiculous margin as Opus keeps claiming slightly wrong things with certainty much more.
- andxor 29d ago
  This is not saying much. Opus 4.8 is ancient history.

andxor 29d ago

That's not my experience and I suspect it's not most people's experience. Out of curiosity, what's the hardest task you tried?

Koffiepoeder 29d ago
For me something the likes of: design a CDM for integrating these 5 logistical systems, with full docs and examples provided for each, as well as modeled transports specific to our business. Prompt was of course much longer. Both failed spectacularly. But sol's output at least contained interesting findings and some useful parts, as well as not being 20000 words of unbearable language.
- Demiurge 29d ago
  You’re thinking long horizon tasks. I agree that Sol is great at it. I don’t think it’s smarter in quick win tasks that are still difficult. This is where intelligence is not one of a kind, these systems have different pros and cons. I use Sol as an architect and fable as a brilliant single task solver.
CSMastermind 28d ago
I have both set up with full access to all the repos at the company, infrastructure, deployment pipelines, etc. I can tell Sol, "Hey we need to update this core database schema to handle this new use case" and it will masterfully handle the update, version the API, roll the consumers over, including versioning the Kafka schemas, deploying things in sequence, watching the deployments to make sure the new services act actually active before cutting over consumers, exercising the website and mobile apps in staging environments before releasing to production, etc. Fable just falls over on long horizon tasks, it does partial implementations, it cuts corners, it gives up, it doesn't verify it's work, it loses track of what it's doing, etc. It's fine for specific well scoped tasks but can't take high level guidance for complex updates.

adonese 29d ago

what i found to work well with me is Fable for design / ideas and Sol for implementation. Codex models just tend to be more attentive and follow through instructions. Whereby claude models are weaker on this area (they tend to cut corners).

glub 29d ago
Claude models tend to cut corners during design/ideation too. It becomes especially visible once you pair Sol as advisor to Fable. Sol will start going crazy - "hey, you said this, and it's actually false, i checked that", or "you need a hash chained, triple encrypted, secure-enclave backed storage for this". But I actually prefer it this way. Sol is a master of overengineering and being overly scrupulous, so Fable balances this out, and I can always say "don't listen to Sol's advisory" about 50% of the time.