4 ms·
The Gemini benchmarks appear to be heavily gamed. They seem to be carefully presented and use different prompting techniques to show Gemini beating GPT-4. Since
by dougmwne 3y ago
The Gemini benchmarks appear to be heavily gamed. They seem to be carefully presented and use different prompting techniques to show Gemini beating GPT-4. Since no one can actually use Gemini Ultra to confirm, I think it’s safe to assume this is fluff for shareholders and that once the public gets hands on it will be apparent that GPT-4 is still the better model and better aligned with fewer hallucinations.
There’s very little information on how soon we might see GPT-5. OpenAI seems to have thrown the kitchen sink at GPT-4, so without any new algorithm breakthroughs (which they are rumored to have made with Q*, but that might be just PR) I don’t know that we should expect miracles.
I’m bearish on LLMs in the next 1 year but bullish over the next 5 years.
- nolist_policy 3y agoOTOH, if rumors are true GPT4 is multiple models in a trenchcoat.
- PartiallyTyped 3y agoI thought that had been established :thinking:
- geniium 3y ago> I’m bearish on LLMs in the next 1 year but bullish over the next 5 years. I find this sentence particuarly interesting.
- rkangel 3y agoThis is a good bet, because as a rule people overestimate progress in the short term, and underestimate it in the long term (and at the moment 5 years is at least medium term for LLMs! https://quoteinvestigator.com/2019/01/03/estimate/ https://quoteinvestigator.com/2019/01/03/estimate/
- monkeydust 3y agoYea if Gartner did one good thing is the hype curve which supports this thinking also https://en.wikipedia.org/wiki/Gartner_hype_cycle https://en.wikipedia.org/wiki/Gartner_hype_cycle
- Cannabat 3y agoEven if the benchmarks are accurate - or even overly conservative - it's possible that these are the "uncensored" versions of the models, and what the public gets access to in the end is far less capable.
- freedomben 3y agoAh yes! That's probably be what's going on! That would explain a lot of things, especially why they're letting the beautiful people have access now but not us plebs. They want a few months to work on censoring it before the masses get it and make it say something offensive.
- kartoolOz 3y agoAlphacode 2 technical paper claims to solve 43% of problems (77 problems from 12 codeforce competitions) performing 85%ile on all human participants. Caveat is deep in the technical paper, 1) generates 1m candidates from N different prior models (fine tuned on previous codeforces) 2) throws away 95%ile candidates (doesn't fit the test case + no compile) 3) groups semantically similar candidates 4) scores candidates from each group (Based on another scoring model, probably latency + descriptions etc) 5) Picks top 10 Makes 10 submissions and finally gets the score .. sure this is how humans solve problems ... totally awed by AGI /s
- dougmwne 3y ago10 submissions is rather arbitrary. Why not just do 1 billion submissions? AI magic!