3 ms·
I've been making an auction site and have been using an AI swarm to test it: sellers, intermediaries, buyers, market practices/norms etc. I was mostly using GPT
by m101 4mo ago
I've been making an auction site and have been using an AI swarm to test it: sellers, intermediaries, buyers, market practices/norms etc. I was mostly using GPT 5.5 xhigh to code up the scenario, and looping over it to check with opus 4.8.
Out of curiosity I asked Fable to review it all and I was shocked to find that there were a lot of blindingly obvious common sense mistakes that got through, for example:
- all intermediaries were given the prices of all buyers up front
- private price information in certain auction types was actually being broadcast to everyone
- multiple contradictions in instructions
If it was any one of these things then I might have understood - but the fact that so many got passed both Opus and GPT 5.5 makes me think that Fable has something special. This is a common sense type thing, that I think you only get to notice when your task doesn't involve a measurable metric, but rather some sort of real world fuzzy task.
There's clearly a problem with all these measures of performance when the difference between these models was night and day in my specific task.
- throwwwll 4mo agoMaybe you are something special by letting those slip through in the first place?..
- OsrsNeedsf2P 4mo agoGP literally caught them?
- cadamsdotcom 4mo agoPrompt: can you reformat your sentence to be less unkind?
- m101 4mo agoThe point is that there's a difference in these models and everyone is looking for where the differences are. stop being an arse.
- inglor_cz 4mo agoThis conversation is about capabilities of Fable 5 vs. older models, not about the GP's abilities.
- pleasstopnw 4mo ago[dead]
- TacticalCoder 4mo ago> ... and I was shocked to find that there were a lot of blindingly obvious common sense mistakes that got through Wait... Are you telling me models everybody told me were better than coders up to just one month ago are actually making lots of mistakes? This is shocking.
- kristofferR 4mo agoThis seems like the exact project you should try out Codex Security for. It catches a lot of stuff: https://chatgpt.com/codex/cloud/security/ https://chatgpt.com/codex/cloud/security/
- port3000 4mo agoIt's just much more thorough and spins up a lot of subagents to basically do a lot more E2E testing. Not necessarily smarter, imo you could get the same result with a lesser model by procedurally prompting, but a lot more compute and orchestration.
- m101 4mo agoi had to specifically tell fable not to use a bunch of subagents in order to preserve my token allowance.
- rob 4mo agoUnless you're coming up with a deterministic set of criteria for evaluating these bugs and issues, every single model is going to keep telling you it finds new things and to fix them. I'm sure you said the same "find mistakes please" thing to Opus 4.8 and GPT 5.5 when you were using $previous_amazing_latest_model, and they also found and fixed them. Once the next "Fable"-type model comes out I'm sure it's going to find even more mistakes that the "special" Fable made. You're using these models to make mistakes and then using upgraded versions of them to find their previous mistakes and fix them, until a new version comes along that can magically fix even more mistakes their previous versions made. There's no end to it.
- m101 4mo agoYes - I was thinking this - however I had already worked on it so many times with opus and gpt that I thought they had enough time to realise some common sense things that fable just got and understood first time, on the first pass. The difference seemed significant enough to comment about.