6 ms·

I think Fable's dominance is overstated. It definitely has the lead, but quantifying what that lead actually is is really hard. I'm using GPT 5.6 Sol to do some

by jchw 2mo ago

I think Fable's dominance is overstated. It definitely has the lead, but quantifying what that lead actually is is really hard. I'm using GPT 5.6 Sol to do some shit that I personally would consider "crazy" - low level undocumented hardware driver alchemy, reverse engineering highly obfuscated code, even a bit of screwing around with a rendering engine in Vulkan, really just about the most complex tasks I can get any model to do, and it does great. For the more advanced stuff, it definitely needs the effort bumped. But even with the effort bumped, the token usage really doesn't seem to skyrocket too badly until at least you hit xhigh and max, which really only seem to be necessary if you are doing genuine crazy stuff, so it's not that bad. I did similar stuff with Fable. In fact, I went directly from an Anthropic subscription with Fable to an OpenAI subscription with Sol, more or less, and it really felt pretty seamless. If anything, I was thrilled to realize how much I actually preferred Codex CLI, to the point where I started using it at work too. Fable seems to be generally more impressive at outputting one-shot web apps. I'm not really saying that to try to downplay what Fable can do, it's just that if I compare the two, this is one of the few definitely noticeable areas that you can easily demonstrate. Obviously, one-shotting programs is much better as a demonstration of a model's capabilities than it is practically useful (not that it is useless, but hopefully my point is understood). However, whatever Fable truly is better at, one thing I really like about GPT 5.6 Sol is even harder to quantify: taste. GPT 5.6 Sol outputs are still LLM outputs and they contain many things that people would probably consider "Claude-isms" for better or worse, but overall I really prefer the GPT 5.6 Sol output. I find it to be generally more tasteful. Hard to quantify, but when talking to people I've had enough people seemingly agree with me to convince me that it really is true.

andrewingram 2mo ago
I used Sol to extract the remaining decryption keys from the Super Mario Maker 2 (Switch) game files. Someone had previously extracted all the keys from the original release, but not any of the new ones from updates. Not only did it succeed, but it helped me understand the data sufficiently to add support for “Super World” rendering to my level viewer (which I made back in 2021), eg the little widget at the top of https://www.smm2-viewer.com/players/B16-306-GVG https://www.smm2-viewer.com/players/B16-306-GVG I was very pleasantly surprised to find Sol wasn’t obstructive over what was clearly a very grey area endeavour.
- mjhagen 2mo ago
  I was having it look at creating a driver for some old scanner and it actively looked up exactly where that gray area for my country was wrt decompilation.
- alchemist1e9 2mo ago
  Fable is almost unusable for anything but super boring mainstream stuff. I was getting safeguard flagged so often I’ve significantly reduced my usage out of fear they will blacklist/ban me. Some of the topics it’s flagged have been hard for me to understand what it seeing that can be remotely concerning in my requests.
  - andai 2mo ago
    The safety is really funny to me. I ask it a lot of extreme stuff and it goes through, but I ask it mundane stuff and hit the filters all the time.
    - vintermann 2mo ago
      It has learned a little too well how it works in human society.
    - vjvjvjvjghv 2mo ago
      Reminds me of the URL blocking of my company. Nature.com is being blocked but I can access a ton of super sketchy download sites.
  - xscott 2mo ago
    I've gotten flagged for asking questions about tokens and tensors. That makes me believe it's not about safety, it's about protecting their turf. I cancelled my subscription - same fear about getting flagged too much leading to a ban.
    - KronisLV 2mo ago
      OpenAI and Kimi are both pretty okay alternatives! I guess GLM 5.3 on Max reasoning as well but for more limited domains.
calvinmorrison 2mo ago
things that are alchemical are rarely alchemy. That is to say things are very fiddly but stick a room of monkeys on typewriters, a schizophrenic developer with HolyC and adderall or an LLM, persistence is the key to many of these things like drivers, extracting keys from vintage security domains, etc. Dropping into xdd to a human is a chore, not for an LLM.
- jchw 2mo ago
  Although I am not exactly sure what you mean, I am not really claiming it is doing anything I couldn't do - but yes, it does so with much less effort. For example, I can have it set up probes and tracing on Linux that I personally would have to consult documentation to do. It might not even have to consult the documentation due to having the information on-tap, but even if it does, it's nothing that would cause it any fatigue, it's just going to keep moving forward in a loop until it is satisfied that it meets the criteria. I could've done all of this alone - I really could have. I just would not have. Being able to do something 10 times faster or with 10 times less effort is, in some senses, sometimes more impactful than being able to do entirely new things you couldn't do before.
  - saghm 2mo ago
    This is a really good point that I definitely failed to grasp when first hearing about these tools. At least for me, the best way to use these tools is as a way to free myself from having to spend time thinking about the things that aren't worthwhile so I can focus on the things that truly are. I've had times in my life spending hours reading documentation and googling random things to try to tease out the correct sequence of commands or the exact right shape of an API to be able to make things work to know that it doesn't make me more productive to do that myself rather than point an LLM at the thing and let it spit out the answer after a few minutes. Meanwhile, I can spend that time thinking about what comes next, or what the correct way to take that one-off output and abstract it to something that can be used meaningfully in more flexible ways. The only obvious objection I can think of to this line of thinking (at least from a technical perspective) is "how does someone build up the knowledge to be able to use a tool effectively in that way if not by doing things by hand at first?" The honest answer that is "I don't know, but that's also pretty much exactly the type of thing my employers have never been paying me to solve in the first place". Even just a decade into my career, there have already been plenty of times in my career I've struggle to convince people that we should do stuff in a way that won't bite us in the ass a month or two down the line, and in the times I've managed to succeed, it's usually only by putting in more of my own time and effort to make the initial investment seem more palatable. Luckily right now I'm not in one of those times when I'm having to go full throttle to keep the lights on a few months from now, but I don't have enough fuel in reserves to work on a plan for when we need to build a new rocket in another ten years. Maybe ask me next month.
edg5000 2mo ago
FYI I run it consistently in xhigh regardless of difficulty of the task at hand. I remember high being very fast, but I'd rather wait a bit more and get better output. AIs are insanely fast compared to me anyway, even on xhigh. Consumes more usage, but even at 100 EUR/m I don't hit limits.
- jchw 2mo ago
  To be fair, I actually do run xhigh as my default. However, for the first time in my experience of trying and using LLMs, with Sol.. sometimes I feel confident enough to set the effort level to "Low". I just had Sol prototype some AWS stuff on low earlier. Great result, did exactly what I wanted.
- saghm 2mo ago
  After hitting the session limit on my company's plan so many times with Claude when I was using it, I mostly keep Codex on "high" rather than "xhigh" as a way to leave the tokens for my more ambitious coworkers. It's possible that having it higher might end up with better output, but so far at least I've yet to see a way to get any model to do 100% of what I need up front without any need for me to make changes that end up being more tedious to do via interaction than by hand, and it doesn't feel worth spending a bunch more tokens trying to figure out how to better communicate to it up front how the dominoes get set up so they fall in place properly the next time.
- FooBarWidget 2mo ago
  How do you handle context limits? With more thinking tokens you fill it up earlier. Compaction degrades performance too. What's your strategy?
  - edg5000 2mo ago
    Initially I was planning heavily around context limits, but I've learned to just ignore it completely. Compaction is seamless for me. If details are lost in compaction, the model just re-reads what's needed. My conclusion is that at least for Sol, the summaries (which I've never seen) must be amazing. Every now and then a detail gets lost and I have to repeat it. I don't think there is performance degration, because the model is smart enough to re-read relevant files as needed.
cgannett 2mo ago
And Mai-Code-1.1-Flash seems like a really good cooperative player to GPT 5.6 Sol. You get Sol to help you make a detailed plan, and Mai codes it up and you can get pretty decent code out the other end without too many tokens if you are careful.
- manmal 2mo ago
  Why wouldn’t you use Luna for that? It’s super cheap.
  - alex7o 2mo ago
    Taste I suppose?
  - locknitpicker 2mo ago
    > Why wouldn’t you use Luna for that? It’s super cheap. MAI also offers a ultra cheap version that's competitive with Luna. So much so that the models look like they were designed by a product manager explicitly to eat away OpenAI's market share. Vscode even pushed them quite hard onto users with the latest release, going to the extent of putting up a modal to convince users to try them out.
- swingboy 2mo ago
  How detailed of a plan? Are you including code snippets or just behavior and letting the lesser model decide how to implement?
hatthew 2mo ago
I feel like those examples are considered difficult because they're niche topics, but aren't actually all that difficult in a general sense. What I consider truly difficult are things like taking a ticket and implementing it in a preexisting codebase, using a clean and reasonable design that fits the existing style and makes sense to a human, and avoids the footguns I learned by working with the codebase for over a day.
- jchw 2mo ago
  If you said this in 2025 I would've 100% understood, but to be honest getting AI models to do a pretty good job on day-to-day ticket work has become so boring that we don't even bother using the top tier models and higher effort slots for that anymore. I personally wind up tweaking the results a lot and recursively having fresh agents review the diff, but that's just because I'm picky; in a lot of cases the first diff is actually pretty damn decent. Compared to what I am doing at home experimentally, I feel like day-to-day work is absolutely nothing. Not only am I also working with existing codebases in my experimental prototyping, but I am also doing things vastly more complex with vastly harder constraints.
  - com2kid 2mo ago
    All non-trivial code terra has generated for me has had at least one serious bug in it. Typically caught by a review from myself or Sol. But I wouldn't trust lower tier models for end to end solutions.
    - jchw 2mo ago
      Personally I wouldn't want bots running autonomously on a repo, even if there were other bots cross checking them. But that having been said, I'd also say that my experience was similar with human code: it is rare to not find at least some issue worth at least pointing out. The only real difference is that the LLMs have vastly different holes than people do, making the real challenge trying to make sure you're covering them. In most cases for me the best solution seems to be just giving them a way to test and attempt to prove things out in a realistic environment. But clearly, we haven't really left the era of having humans in the loop. Fully vibe-coded codebases clearly suffer from a myriad of issues.
wonnage 2mo ago
AI-pilled obsession with "taste" is bordering on insanity It's just vibes
- mfru 2mo ago
  It's vibes all the way down. It's... really just vibes? Always has been.
- perching_aix 2mo ago
  That's what they're always going to be, so not sure what would be "insane" about it. They literally feed on and emit natural language, and are put to work on informally defined, arbitrary tasks. When people figure out any reliable strategies to test and benchmark them, that's insane, and in the positive sense. This very same issue has been a thing for humans as well forever, and remains only very questionably solved (IQ, academic tests). This is not easy.
  - wonnage 2mo ago
    This sounds about as unhinged as Google saying Material Design 3 is 30% more rebellious
    - deleted 2mo ago
      [deleted]
    - perching_aix 2mo ago
      Doesn't just sound like it, it is. That's life for you. That's what I'm pointing out to you. [0] Just consider your own example. Do you think a less or more "rebellious look" is not something designers can actually ellicit? Less so in software design, sure, but in character design for example? Or general product design? Do you think e.g. Monster energy drinks are branded the way they are completely due to happenstance or something? Except people don't usually put numbers to it, because they understand that that's hard to defend. You're the one who's describing such an idea, and wants such a thing to happen, classifying anything else as just vibes (that's the point!) and unhingedness. You're handwaving the difficulty and fundamentally limited nature of that, assuming that it is some laziness or mental delusion that's preventing it instead. You're also pretending as if it was somehow not real as a result. What I'm telling you is that you're wrong about that. Any kind of qualitative analysis that's actually defensible with these is genuinely difficult and limited in nature. See also all the opining about benchmaxxing. It also doesn't mean they're useless though, see also benchmarking. The guy above didn't put numbers to his vibe assessment, they just drew a comparison, exactly because they know that there's not much else they can earnestly offer. You're sulking at them not lying to you by overstating their rigor, and you're flipping the arrow as if this limitation was some sort of mistake, not a necessary and intrinsic property, which it absolutely is. Natural language is an inherently subjective medium. [0] In fancier and more mathematical terms: https://abeljansma.nl/2026/07/10/truth-is-not-a-direction.html https://abeljansma.nl/2026/07/10/truth-is-not-a-direction.ht...

by jchw 2mo ago

andrewingram 2mo ago

I used Sol to extract the remaining decryption keys from the Super Mario Maker 2 (Switch) game files. Someone had previously extracted all the keys from the original release, but not any of the new ones from updates. Not only did it succeed, but it helped me understand the data sufficiently to add support for “Super World” rendering to my level viewer (which I made back in 2021), eg the little widget at the top of https://www.smm2-viewer.com/players/B16-306-GVG https://www.smm2-viewer.com/players/B16-306-GVG I was very pleasantly surprised to find Sol wasn’t obstructive over what was clearly a very grey area endeavour.

mjhagen 2mo ago
I was having it look at creating a driver for some old scanner and it actively looked up exactly where that gray area for my country was wrt decompilation.
alchemist1e9 2mo ago
Fable is almost unusable for anything but super boring mainstream stuff. I was getting safeguard flagged so often I’ve significantly reduced my usage out of fear they will blacklist/ban me. Some of the topics it’s flagged have been hard for me to understand what it seeing that can be remotely concerning in my requests.
- andai 2mo ago
  The safety is really funny to me. I ask it a lot of extreme stuff and it goes through, but I ask it mundane stuff and hit the filters all the time.
  - vintermann 2mo ago
    It has learned a little too well how it works in human society.
  - vjvjvjvjghv 2mo ago
    Reminds me of the URL blocking of my company. Nature.com is being blocked but I can access a ton of super sketchy download sites.
- xscott 2mo ago
  I've gotten flagged for asking questions about tokens and tensors. That makes me believe it's not about safety, it's about protecting their turf. I cancelled my subscription - same fear about getting flagged too much leading to a ban.
  - KronisLV 2mo ago
    OpenAI and Kimi are both pretty okay alternatives! I guess GLM 5.3 on Max reasoning as well but for more limited domains.

calvinmorrison 2mo ago

things that are alchemical are rarely alchemy. That is to say things are very fiddly but stick a room of monkeys on typewriters, a schizophrenic developer with HolyC and adderall or an LLM, persistence is the key to many of these things like drivers, extracting keys from vintage security domains, etc. Dropping into xdd to a human is a chore, not for an LLM.

jchw 2mo ago
Although I am not exactly sure what you mean, I am not really claiming it is doing anything I couldn't do - but yes, it does so with much less effort. For example, I can have it set up probes and tracing on Linux that I personally would have to consult documentation to do. It might not even have to consult the documentation due to having the information on-tap, but even if it does, it's nothing that would cause it any fatigue, it's just going to keep moving forward in a loop until it is satisfied that it meets the criteria. I could've done all of this alone - I really could have. I just would not have. Being able to do something 10 times faster or with 10 times less effort is, in some senses, sometimes more impactful than being able to do entirely new things you couldn't do before.
- saghm 2mo ago
  This is a really good point that I definitely failed to grasp when first hearing about these tools. At least for me, the best way to use these tools is as a way to free myself from having to spend time thinking about the things that aren't worthwhile so I can focus on the things that truly are. I've had times in my life spending hours reading documentation and googling random things to try to tease out the correct sequence of commands or the exact right shape of an API to be able to make things work to know that it doesn't make me more productive to do that myself rather than point an LLM at the thing and let it spit out the answer after a few minutes. Meanwhile, I can spend that time thinking about what comes next, or what the correct way to take that one-off output and abstract it to something that can be used meaningfully in more flexible ways. The only obvious objection I can think of to this line of thinking (at least from a technical perspective) is "how does someone build up the knowledge to be able to use a tool effectively in that way if not by doing things by hand at first?" The honest answer that is "I don't know, but that's also pretty much exactly the type of thing my employers have never been paying me to solve in the first place". Even just a decade into my career, there have already been plenty of times in my career I've struggle to convince people that we should do stuff in a way that won't bite us in the ass a month or two down the line, and in the times I've managed to succeed, it's usually only by putting in more of my own time and effort to make the initial investment seem more palatable. Luckily right now I'm not in one of those times when I'm having to go full throttle to keep the lights on a few months from now, but I don't have enough fuel in reserves to work on a plan for when we need to build a new rocket in another ten years. Maybe ask me next month.

edg5000 2mo ago

FYI I run it consistently in xhigh regardless of difficulty of the task at hand. I remember high being very fast, but I'd rather wait a bit more and get better output. AIs are insanely fast compared to me anyway, even on xhigh. Consumes more usage, but even at 100 EUR/m I don't hit limits.

jchw 2mo ago
To be fair, I actually do run xhigh as my default. However, for the first time in my experience of trying and using LLMs, with Sol.. sometimes I feel confident enough to set the effort level to "Low". I just had Sol prototype some AWS stuff on low earlier. Great result, did exactly what I wanted.
saghm 2mo ago
After hitting the session limit on my company's plan so many times with Claude when I was using it, I mostly keep Codex on "high" rather than "xhigh" as a way to leave the tokens for my more ambitious coworkers. It's possible that having it higher might end up with better output, but so far at least I've yet to see a way to get any model to do 100% of what I need up front without any need for me to make changes that end up being more tedious to do via interaction than by hand, and it doesn't feel worth spending a bunch more tokens trying to figure out how to better communicate to it up front how the dominoes get set up so they fall in place properly the next time.
FooBarWidget 2mo ago
How do you handle context limits? With more thinking tokens you fill it up earlier. Compaction degrades performance too. What's your strategy?
- edg5000 2mo ago
  Initially I was planning heavily around context limits, but I've learned to just ignore it completely. Compaction is seamless for me. If details are lost in compaction, the model just re-reads what's needed. My conclusion is that at least for Sol, the summaries (which I've never seen) must be amazing. Every now and then a detail gets lost and I have to repeat it. I don't think there is performance degration, because the model is smart enough to re-read relevant files as needed.

cgannett 2mo ago

And Mai-Code-1.1-Flash seems like a really good cooperative player to GPT 5.6 Sol. You get Sol to help you make a detailed plan, and Mai codes it up and you can get pretty decent code out the other end without too many tokens if you are careful.

manmal 2mo ago
Why wouldn’t you use Luna for that? It’s super cheap.
- alex7o 2mo ago
  Taste I suppose?
- locknitpicker 2mo ago
  > Why wouldn’t you use Luna for that? It’s super cheap. MAI also offers a ultra cheap version that's competitive with Luna. So much so that the models look like they were designed by a product manager explicitly to eat away OpenAI's market share. Vscode even pushed them quite hard onto users with the latest release, going to the extent of putting up a modal to convince users to try them out.
swingboy 2mo ago
How detailed of a plan? Are you including code snippets or just behavior and letting the lesser model decide how to implement?

hatthew 2mo ago

I feel like those examples are considered difficult because they're niche topics, but aren't actually all that difficult in a general sense. What I consider truly difficult are things like taking a ticket and implementing it in a preexisting codebase, using a clean and reasonable design that fits the existing style and makes sense to a human, and avoids the footguns I learned by working with the codebase for over a day.

jchw 2mo ago
If you said this in 2025 I would've 100% understood, but to be honest getting AI models to do a pretty good job on day-to-day ticket work has become so boring that we don't even bother using the top tier models and higher effort slots for that anymore. I personally wind up tweaking the results a lot and recursively having fresh agents review the diff, but that's just because I'm picky; in a lot of cases the first diff is actually pretty damn decent. Compared to what I am doing at home experimentally, I feel like day-to-day work is absolutely nothing. Not only am I also working with existing codebases in my experimental prototyping, but I am also doing things vastly more complex with vastly harder constraints.
- com2kid 2mo ago
  All non-trivial code terra has generated for me has had at least one serious bug in it. Typically caught by a review from myself or Sol. But I wouldn't trust lower tier models for end to end solutions.
  - jchw 2mo ago
    Personally I wouldn't want bots running autonomously on a repo, even if there were other bots cross checking them. But that having been said, I'd also say that my experience was similar with human code: it is rare to not find at least some issue worth at least pointing out. The only real difference is that the LLMs have vastly different holes than people do, making the real challenge trying to make sure you're covering them. In most cases for me the best solution seems to be just giving them a way to test and attempt to prove things out in a realistic environment. But clearly, we haven't really left the era of having humans in the loop. Fully vibe-coded codebases clearly suffer from a myriad of issues.

wonnage 2mo ago

AI-pilled obsession with "taste" is bordering on insanity It's just vibes

mfru 2mo ago
It's vibes all the way down. It's... really just vibes? Always has been.
perching_aix 2mo ago
That's what they're always going to be, so not sure what would be "insane" about it. They literally feed on and emit natural language, and are put to work on informally defined, arbitrary tasks. When people figure out any reliable strategies to test and benchmark them, that's insane, and in the positive sense. This very same issue has been a thing for humans as well forever, and remains only very questionably solved (IQ, academic tests). This is not easy.
- wonnage 2mo ago
  This sounds about as unhinged as Google saying Material Design 3 is 30% more rebellious
  - deleted 2mo ago
    [deleted]
  - perching_aix 2mo ago
    Doesn't just sound like it, it is. That's life for you. That's what I'm pointing out to you. [0] Just consider your own example. Do you think a less or more "rebellious look" is not something designers can actually ellicit? Less so in software design, sure, but in character design for example? Or general product design? Do you think e.g. Monster energy drinks are branded the way they are completely due to happenstance or something? Except people don't usually put numbers to it, because they understand that that's hard to defend. You're the one who's describing such an idea, and wants such a thing to happen, classifying anything else as just vibes (that's the point!) and unhingedness. You're handwaving the difficulty and fundamentally limited nature of that, assuming that it is some laziness or mental delusion that's preventing it instead. You're also pretending as if it was somehow not real as a result. What I'm telling you is that you're wrong about that. Any kind of qualitative analysis that's actually defensible with these is genuinely difficult and limited in nature. See also all the opining about benchmaxxing. It also doesn't mean they're useless though, see also benchmarking. The guy above didn't put numbers to his vibe assessment, they just drew a comparison, exactly because they know that there's not much else they can earnestly offer. You're sulking at them not lying to you by overstating their rigor, and you're flipping the arrow as if this limitation was some sort of mistake, not a necessary and intrinsic property, which it absolutely is. Natural language is an inherently subjective medium. [0] In fancier and more mathematical terms: https://abeljansma.nl/2026/07/10/truth-is-not-a-direction.html https://abeljansma.nl/2026/07/10/truth-is-not-a-direction.ht...