4 ms·

I use Opus 4.8 and GPT 5.5 and haven't suffered from hallucinations in months. But we also put a lot of effort into our harness.

Loic 5mo ago
Sometimes the harness can only be a human. And this is fine. Developing new software with a really smart intern is the same, you, as an expert, need to bring your experience/expertise on the table to have everything right. Because experience needs time.
Aeolun 5mo ago
Opus 4.8 and gpt constantly hallucinate stuff as well. If you haven’t encountered or caught it that’s something different. Of course these days it’s mostly confidently asserting a wrong thing.
- deleted 5mo ago
  [deleted]
- rfgplk 5mo ago
  I primarily use Opus for a Lisp-like DSL codebase (non public, closed source) and it genuinely has _never_ hallucinated. All it pulls from is BNF, language spec + examples. So I have no idea how people are getting it to hallucinate on _popular_ languages.
  - Chu4eeno 5mo ago
    I think you forget that they really are stochastic (I'd wager it has hallucinated things to you that wasn't important so you missed it, or you've just been very lucky), and the people you're arguing with are forgetting that there is significant difference in when and how often even frontier LLMs hallucinate. I catch claude every now and then, but isn't the measured hallucination rates down in low single digit percent for claude now?
- latentsea 5mo ago
  They said that they hadn't "suffered" from hallucinations in months due to effort they put into harness engineering. It's not quite the same thing as saying hallucinations are never happening for them. I find hallucinations happen for us with those models, but we've worked on baking in guardrails and fact checking against sources of truth, so that it's less of a problem. We're engineers. We just try to engineer out the failure modes.
rfgplk 5mo ago
Those models simply don't hallucinate if you use them properly in any form. The only way they _might_ hallucinate is if you use the web based chat interface and give them zero context.
- vips7L 5mo ago
  Ah the old “you’re holding it wrong” argument.
  - JoeyJoJoJr 5mo ago
    Sorry, but we aren’t at the “press a button to build a game/app” yet.

Loic 5mo ago

Sometimes the harness can only be a human. And this is fine. Developing new software with a really smart intern is the same, you, as an expert, need to bring your experience/expertise on the table to have everything right. Because experience needs time.

Aeolun 5mo ago

Opus 4.8 and gpt constantly hallucinate stuff as well. If you haven’t encountered or caught it that’s something different. Of course these days it’s mostly confidently asserting a wrong thing.

deleted 5mo ago
[deleted]
rfgplk 5mo ago
I primarily use Opus for a Lisp-like DSL codebase (non public, closed source) and it genuinely has _never_ hallucinated. All it pulls from is BNF, language spec + examples. So I have no idea how people are getting it to hallucinate on _popular_ languages.
- Chu4eeno 5mo ago
  I think you forget that they really are stochastic (I'd wager it has hallucinated things to you that wasn't important so you missed it, or you've just been very lucky), and the people you're arguing with are forgetting that there is significant difference in when and how often even frontier LLMs hallucinate. I catch claude every now and then, but isn't the measured hallucination rates down in low single digit percent for claude now?
latentsea 5mo ago
They said that they hadn't "suffered" from hallucinations in months due to effort they put into harness engineering. It's not quite the same thing as saying hallucinations are never happening for them. I find hallucinations happen for us with those models, but we've worked on baking in guardrails and fact checking against sources of truth, so that it's less of a problem. We're engineers. We just try to engineer out the failure modes.

rfgplk 5mo ago

Those models simply don't hallucinate if you use them properly in any form. The only way they _might_ hallucinate is if you use the web based chat interface and give them zero context.

vips7L 5mo ago
Ah the old “you’re holding it wrong” argument.
- JoeyJoJoJr 5mo ago
  Sorry, but we aren’t at the “press a button to build a game/app” yet.