4 ms·

My biggest problem with working LLMs is that they don't understand negatives and they also fail to remember their previous instructions somehow. For example:

My biggest problem with working LLMs is that they don't understand negatives and they also fail to remember their previous instructions somehow. For example: If I tell it to not use X, it will do X. When I point it out, it fixes it. Then a few prompts later, it will use X again. Another issue is the hallucinations. Even if you provide it the entire schema (I did this for a toy app I was working with), it kept on making up "columns" that don't exist. My Invoice model has no STATUS column, why do you keep assuming it's there in the code? I found them useful for generating the initial version of a new simple feature, but they are not very good for making changes to an existing ones. I've tried many models, Sonnet is the better one at coding, 3.7 at least, I am not impressed with 4.

CamperBob2 1y ago
I've tried many models, Sonnet is the better one at coding, 3.7 at least, I am not impressed with 4. If Sonnet 3.7 is the best you've found, then no, you haven't tried many models. At least not lately. For coding, I'd suggest Gemini 2.5 Pro, o3-mini-high, or Opus 4. I've heard good things about Grok 4 as well, so if you're OK with that whole scene and the guy who runs it, maybe give it a shot. If you have already done so and still think Sonnet 3.7 is better than any of them, then the most likely explanation is that you got incredibly lucky with Claude and incredibly unlucky with the others. LLMs aren't parrots, but they are definitely stochastic.