3 ms·
The test is rigged because they used non thinking models.
by tuhgdetzhh 9mo ago
The test is rigged because they used non thinking models.
- felix089 9mo agoThese are reasoning / thinking models
- tuhgdetzhh 9mo agoSource?
- tverbeure 9mo agoI don't know, but model names such as "kimi-k2-thinking" in the test set might offset a clue.
- etyhhgfff 9mo agoYes, there are some exceptions where it clearly states that a thinking model has been chosen like for kimi, but there is no such indicator for the GPT family from OpenAI and other major models.
- handoflixue 9mo agoTesting some subset X does not mean the test is rigged unless they failed to disclose that. But also: GPT 5.2 Thinking, Standard Effort: Walk - https://chatgpt.com/share/699d38cb-e560-8012-8986-d27428de8a01 https://chatgpt.com/share/699d38cb-e560-8012-8986-d27428de8a... I'm assuming "GPT 5.2 Thinking" is, in fact, a thinking model?
- randomtoast 9mo agoThe problem is you haven't used the API, but you have used your ChatGPT subscriptions with personality, memories and possible customization. I can see for instance that your ChatGPT answers with emojis, while my ChatGPT subscription never does. If you ask GPT 5.2 with high reasoning efforts in the API, you get 10 out of 10: drive.
- handoflixue 9mo agoIf it doesn't work at all using the most popular pricing plans (subscription), AND it doesn't work on the most popular way of accessing it (web), then it seems fair to say there's a problem. And the problem is NOT that I'm using a product in the advertised, intended way.