3 ms·
> I stress test commercially deployed LLMs like Gemini and Claude with trivial tasks: sports trivia, fixing recipes, explaining board game rules, etc. It works
by simianwords 7mo ago
> I stress test commercially deployed LLMs like Gemini and Claude with trivial tasks: sports trivia, fixing recipes, explaining board game rules, etc. It works well like 95% of the time. That's fine for inconsequential things. But you'd have to be deeply irresponsible to accept that kind of error rate on things that actually matter.
95% is not my experience and frankly dishonest.
I have ChatGPT open right now, can you give me examples where it doesn't work but some other source may have got it correct?
I have tested it against a lot of examples - it barely gets anything wrong with a text prompt that fits a few pages.
> The most intellectually honest way to evaluate these things is how they behave now on real tasks
A falsifiable way is to see how it is used in real life. There are loads of serious enterprise projects that are mostly done by LLMs. Almost all companies use AI. Either they are irresponsible or you are exaggerating.
Lets be actually intellectually honest here.
- qsera 7mo ago>95% is not my experience and frankly dishonest. Quite frankly, this is exactly like how two people can use the same compression program on two different files and get vastly different compression ratios (because one has a lot of redundancy and the other one has not).
- simianwords 7mo agoI'm asking for a single example.
- qsera 7mo agoBut why do you need an example? Isn't it pretty well understood that LLMS will have trouble responding to stuff that is under represented in the training data? You will just won't have any clue what that could be.
- simianwords 7mo agofair so it must be easy to give an example? I have ChatGPT open with 5.4-thinking. I'm honestly curious about what you can suggest since I have not been able to get it to bullshit easily.
- qsera 7mo agoI am not the OP, an I have only used ChatGPT free version. Last day I asked it something. It answered. Then I asked it to provide sources. Then it provided sources, and also changed its original answer. When I checked the new answers it was wrong, and when I checked sources, it didn't actually contain the information that I asked for, and thus it hallucinated the answers as well as the sources...
- simianwords 7mo agoI trust you. If it were happening so frequently you may be able to give me a single prompt to get it to bullshit?
- the_snooze 7mo agoI did this in one attempt just now: https://gemini.google.com/share/b4e016be1f69 https://gemini.google.com/share/b4e016be1f69 #8 has an incorrect answer (3 appearances according to Gemini, 2 according to reality https://en.wikipedia.org/wiki/Bowl_championship_series#BCS_appearances_by_team https://en.wikipedia.org/wiki/Bowl_championship_series#BCS_a...) So it works well 95% of the time for literally a trivial use case. Imagine if any other tech tool had that kind of reliability: `ls` displays 95% of your files, your phone successfully sends and receives 95% of text messages, or Microsoft Word saving 95% of the characters you typed in. That's just not acceptable.
- simianwords 7mo agoHi! The challenge was ChatGPT but even then it looks like you used the weakest version of Gemini.