3 ms·

So I downloaded that report which of course doesn't contain the most relevant information (the questions) but it contains some examples of wrong answers. I fed

So I downloaded that report which of course doesn't contain the most relevant information (the questions) but it contains some examples of wrong answers. I fed the first question to Grok (which they claimed they tested as well) and it answered it correctly in detail. I repeated it with another one - again correct answer. I then selected the question they said Grok specifically answered incorrectly and it again answered it correctly. I am sticking with my first intuition: people are terrible at testing tools and probably wanted them to answer incorrectly/not fully (the questions are constructed in a way to make it difficult as well). They also have vested interest in the conclusion (they are financial advisory firm) so there is that to consider. People reading ft will now think chat boxes are bad at answering financial questions while they are pretty good at it. Zero consequences for spreading fake news for Financial Times there but good for financial advisors I guess.

Dwedit 14d ago
LLMs use random numbers, so a single test won't necessarily match someone else's experience.
- bluecalm 14d ago
  They use random numbers so the answers sounds a bit different but in my experience you won't be able to get a wrong answer to a simple question no matter how many times you try it.
ModernMech 14d ago
> each LLM was tested 600 times, and in total over 10,000 questions and answers were assessed. Okay but why do you feel 3 trials say as much as 10,000?
- bluecalm 14d ago
  >>Okay but why do you feel 3 trials say as much as 10,000? I don't trust them so I've used 3 examples in incorrect questions/answers they have given and I got correct answers. I spend enough time with LLMs to know that if Grok answered it correctly and in detail then it wouldn't be a problem for GPT or Claude either. The questions are also constructed in a way that it's easy to answer not fully (which they qualify as wrong). LLMs still answer them correctly and in detail though.
  - deleted 13d ago
    [deleted]

Dwedit 14d ago

LLMs use random numbers, so a single test won't necessarily match someone else's experience.

bluecalm 14d ago
They use random numbers so the answers sounds a bit different but in my experience you won't be able to get a wrong answer to a simple question no matter how many times you try it.

ModernMech 14d ago

> each LLM was tested 600 times, and in total over 10,000 questions and answers were assessed. Okay but why do you feel 3 trials say as much as 10,000?

bluecalm 14d ago
>>Okay but why do you feel 3 trials say as much as 10,000? I don't trust them so I've used 3 examples in incorrect questions/answers they have given and I got correct answers. I spend enough time with LLMs to know that if Grok answered it correctly and in detail then it wouldn't be a problem for GPT or Claude either. The questions are also constructed in a way that it's easy to answer not fully (which they qualify as wrong). LLMs still answer them correctly and in detail though.
- deleted 13d ago
  [deleted]