3 ms·
this has reasoning disabled everywhere, making it a pretty bad benchmark. the argument given is that's the "default consumer experience" that might be generall
by spongebobstoes 4mo ago
this has reasoning disabled everywhere, making it a pretty bad benchmark. the argument given is that's the "default consumer experience"
that might be generally true, but I think chatgpt has reasoning enabled for free accounts. regardless, reasoning is the state of the art, and disabling it reduces the value of this research to predict the future
it's also not clear if this is using the API or the product model, when both exist. they behave differently
lastly, the actual model details are very much buried. I am relieved to see opus 4.8 and chatgpt 5.5 were used, but this information should be presented more clearly. a brand is not a model, and models change quickly
- ImJamal 4mo agoWhy would it be a bad benchmark to explore what the average person will experience? If 99% of people will experience X then it makes sense to look into X instead of Y. I agree other things should be evaluated and more clearly presented.
- spongebobstoes 4mo agobecause it's a dead end of the technology. reasoning will be everywhere soon enough, even for free accounts to know what most people experience, this research should be based on the in-app experience. it isn't, so it misses that mark too