2 ms·

Claude does ask questions for clarification or asks me to provide something it does not know though, at least it has happened many times to me. At other times I

Claude does ask questions for clarification or asks me to provide something it does not know though, at least it has happened many times to me. At other times I will have to ask if it needs X or Y to be able to answer more accurately, although this may be the same case with other LLMs, too. The former though was quite a surprise to me, coming from GPT.

ColinEberhardt 2y ago
Ah, interesting - I’ve not had much experience with Claude, will give it a go. Thanks.
zamalek 2y ago
I am working on a pet project, using tactile "premium" 4/5-way switches in super-ergonomic form-factor keyboard (initially like the logitech vertical mouse, but that turned out awful). The only model to not get hung up on Cherry MX and hallucinate 4-way cherry switches has been Claude (the others did make attempts at other manufacturers, but hallucinated part numbers). It is significantly ahead of the competition.
jug 2y ago
On this topic, SimpleQA benchmark has a component measuring hallucination rate vs ”know” vs ”don’t know”. OpenAI models have often been more troubled than the rest. See also, from the paper: https://imgur.com/7NDZ0ON https://imgur.com/7NDZ0ON (you want a low ”Incorrect” score as it’s an attempted answer, but wrong) I wish hallucination benchmarks were far more popular.
bamboozled 2y ago
On 3.7 Sonny in "slow mode", still give me some seriously deceptive answers, it can be, even great, so good I've grown to over trust it and it's smashed me a few times. I notice it a lot when coding with static typed languages, you paste your code and it will tell you that you were "deceived" very quickly. It gives extremely confident but wrong answers, I've found it to just be way more convincing.
- deleted 2y ago
  [deleted]
- johnisgood 2y ago
  I have only used 3.7 Sonnet for programming, and mostly with what I already know (apart from 1-2 obscure languages) but I always fed it their docs. It still got some things wrong and took a couple reiteration though, but GPT performed much much worse in this regard to the point of being useless while with 3.7 Sonnet I wrote functioning stuff in those obscure languages.

ColinEberhardt 2y ago

Ah, interesting - I’ve not had much experience with Claude, will give it a go. Thanks.

zamalek 2y ago

I am working on a pet project, using tactile "premium" 4/5-way switches in super-ergonomic form-factor keyboard (initially like the logitech vertical mouse, but that turned out awful). The only model to not get hung up on Cherry MX and hallucinate 4-way cherry switches has been Claude (the others did make attempts at other manufacturers, but hallucinated part numbers). It is significantly ahead of the competition.

jug 2y ago

On this topic, SimpleQA benchmark has a component measuring hallucination rate vs ”know” vs ”don’t know”. OpenAI models have often been more troubled than the rest. See also, from the paper: https://imgur.com/7NDZ0ON https://imgur.com/7NDZ0ON (you want a low ”Incorrect” score as it’s an attempted answer, but wrong) I wish hallucination benchmarks were far more popular.

bamboozled 2y ago

On 3.7 Sonny in "slow mode", still give me some seriously deceptive answers, it can be, even great, so good I've grown to over trust it and it's smashed me a few times. I notice it a lot when coding with static typed languages, you paste your code and it will tell you that you were "deceived" very quickly. It gives extremely confident but wrong answers, I've found it to just be way more convincing.

deleted 2y ago
[deleted]
johnisgood 2y ago
I have only used 3.7 Sonnet for programming, and mostly with what I already know (apart from 1-2 obscure languages) but I always fed it their docs. It still got some things wrong and took a couple reiteration though, but GPT performed much much worse in this regard to the point of being useless while with 3.7 Sonnet I wrote functioning stuff in those obscure languages.