3 ms·
The super impressive stuff is more subtle: 1. Price is half for a better performing model. A 1000x1000 image costs $0.003. 2. Cognitive ability on visuals wen
by muzani 2y ago
The super impressive stuff is more subtle:
1. Price is half for a better performing model. A 1000x1000 image costs $0.003.
2. Cognitive ability on visuals went up sharply. https://github.com/kagisearch/llm-chess-puzzles https://github.com/kagisearch/llm-chess-puzzles
It solves twice as much despite a minor update. It could just be better trained on chess though, but this would be amazing if it could be applied to the medical field as well. I might use it as a budget art director too - it's more capable of knowing the difference in subtle changes in color and dealing with highlights.
- mlyle 2y agoI'm not sure for text it's a better performing model. I was just testing GPT-4o on a use case (generating AP MCQ questions) and -4o is repeatedly generating questions with multiple correct answers and will not fix it when prompted. (Providing the history to GPT-4Turbo results in it fixing the MCQ just fine).
- muzani 2y agoAfter some testing, I find it's not as good as code too. It is better at some things, but benchmarks don't tell the whole story apparently.
- llm_trw 2y agoYes, I've been using phind extensively as a google replacement and after moving to gpt4o the responses have gotten so much dumber. I guess time to build something that lets you select which model to use after a google search.
- maeil 2y agoThe benchmark you're linking in 2 is genuinely meaningless due to it being 1 specific task. I can easily make a benchmark for another task (that I'm personally working on) where e.g. Gemini is much better than GPT4-Vision and any Claude model (not sure about GPT-4o yet) and then post that as a benchmark. Does that mean Gemini is better at image reasoning? No. These benchmarks are really missing the mark and I hope people here are smart enough to do their own testing or rely on tests with a much bigger variety of tasks if they want to measure overall performance. Because currently we're at a point where the big 3 (GPT, Claude, Gemini) each have tasks that they beat the other two at.
- muzani 2y agoIt's a test used for humans. I personally am not a big fan of the popular benchmarks because they are, ironically, the narrow tasks that these models are trained on. In fact, GPT-4o performance on key benchmarks has been higher, but on real world tasks, it has flopped on everything we used other models on. They're best tested on the kinds of tasks you would give humans . GPT-4 is still the best contender on AP Biology, which is a legitimately difficult benchmark. GPT tends to work with whatever you throw at it while Gemini just hides behind arbitrary benchmarks. If there are tasks that some models are better than others at, than by all means let's highlight them, rather than acting defensive when another model does much better at a certain task.