3 ms·

Maybe it's because I don't use it in Codex, but I don't like working with Sol. It CONSTANTLY omits things it shouldn't, and is always dispatching sub-agents to

Maybe it's because I don't use it in Codex, but I don't like working with Sol. It CONSTANTLY omits things it shouldn't, and is always dispatching sub-agents to do what I tell it to do, that don't have all the necessary context, and so they go on and do the research that was already done by the top-level agent. It's maddening. I tried it again today because of the discount, it told me it couldn't run acceptance tests because a .env file did not exist, and when I showed it the damn file it went "ah, it's there now". I think it was the first time I've ever had an agent try to gaslight me.

stillpointlab 1mo ago
I'd give it some more time. My first interactions with Sol were not good. It cut a dangerous corner and I had to call it out. But I continued to work with it and found that it was mostly my own style of interacting that needed to change. In a way it is similar to a new co-worker, they have their own personality and ways of working. Once I figured that out I have been able to get very good work out of Sol. Sol seems to work better when you are clear, precise, direct and unambiguous. The model seems annoyed if things aren't spelled out. Not micro-managing, it seems to have a high bar for specific intent. When I get Fable to write out specs for Sol, I tell Fable that Sol is a nit-picking literalist that is exceptional at instruction following. So far this description has lead Fable to generate specs that Sol implements at a high quality.
giwook 1mo ago
> I think it was the first time I've ever had an agent try to gaslight me. You must not have been using agents for very long then because this behavior has been around for some time now.
- VulgarExigency 1mo ago
  I've been using them for a bit over an year now, although I've ramped up usage in the past few months (like everyone else, I suppose). They've hallucinated and lied, but none have ever been as persistent liars as Sol has to me.
  - giwook 1mo ago
    Let's be careful about anthropomorphizing large language models. They are probabilistic models that are simply giving you words in a sequence they think makes the most sense to them. No one is "lying" to you :)
    - VulgarExigency 1mo ago
      Well, yes, it does not know it is lying (it does not know anything, in fact). But for some reason, with Sol specifically, it has persistently written false information when it had the correct information within its context to a degree I've not experienced with other models.

stillpointlab 1mo ago

I'd give it some more time. My first interactions with Sol were not good. It cut a dangerous corner and I had to call it out. But I continued to work with it and found that it was mostly my own style of interacting that needed to change. In a way it is similar to a new co-worker, they have their own personality and ways of working. Once I figured that out I have been able to get very good work out of Sol. Sol seems to work better when you are clear, precise, direct and unambiguous. The model seems annoyed if things aren't spelled out. Not micro-managing, it seems to have a high bar for specific intent. When I get Fable to write out specs for Sol, I tell Fable that Sol is a nit-picking literalist that is exceptional at instruction following. So far this description has lead Fable to generate specs that Sol implements at a high quality.

giwook 1mo ago

> I think it was the first time I've ever had an agent try to gaslight me. You must not have been using agents for very long then because this behavior has been around for some time now.

VulgarExigency 1mo ago
I've been using them for a bit over an year now, although I've ramped up usage in the past few months (like everyone else, I suppose). They've hallucinated and lied, but none have ever been as persistent liars as Sol has to me.
- giwook 1mo ago
  Let's be careful about anthropomorphizing large language models. They are probabilistic models that are simply giving you words in a sequence they think makes the most sense to them. No one is "lying" to you :)
  - VulgarExigency 1mo ago
    Well, yes, it does not know it is lying (it does not know anything, in fact). But for some reason, with Sol specifically, it has persistently written false information when it had the correct information within its context to a degree I've not experienced with other models.