8 ms·
I've tested bard/gemini extensively on tasks that I routinely get very helpful results from GPT-4 with, and bard consistently, even dramatically underperforms.
by resters 3y ago
I've tested bard/gemini extensively on tasks that I routinely get very helpful results from GPT-4 with, and bard consistently, even dramatically underperforms.
It pains me to say this but it appears that bard/gemini is extraordinarily overhyped. Oddly it has seemed to get even worse at straightforward coding tasks that GPT-4 manages to grok and complete effortlessly.
The other day I asked bard to do some of these things and it responded with a long checklist of additional spec/reqiurement information it needed from me, when I had already concisely and clearly expressed the problem and addressed most of the items in my initial request.
It was hard to say if it was behaving more like a clerk in a bureaucratic system or an employee that was on strike.
At first I thought the underperformance of bard/gemini was due to Google trying to shoehorn search data into the workflow in some kind of effort to keep search relevant (much like the crippling MS did to GPT-4 in it's bingified version) but now I have doubts that Google is capable of competing with OpenAI.
- huytersd 3y agoI guess Pro is not supposed to be on par with GPT4. That would be Ultra coming out sometime in the first quarter. I’m going to reserve judgement till that is released.
- YetAnotherNick 3y agoIn my experience, Bard is not comparable to GPT-3.5 in terms of instruction following and it sometimes gets lost in complex situations and then the response quality drops significantly. While GPT-3.5 is a much better feel, if that is a word for evaluating LLMs. And Bard is just annoying if it can't complete a task. Also hallucinations are wild in Gemini pro compared to GPT-3.5.
- AuthConnectFail 3y agoany examples of that? my experience has been other way round (i don't have gpt-4 access so i am comparing chatgpt-3.5 with bard)
- ipsum2 3y agoI don't know why you were down voted for sharing your opinion on bard. I agree with you that bard is significantly worse than gpt 3.5.
- nycdatasci 3y agoPer LLM leaderboard, Bard (jan 24 - Gemini Pro) is on par with GPT 4: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... I think there’s bias in the types of prompts they’re getting. In my personal experience, Bard is useful for creative use cases but not good with reasoning or facts.
- EvgeniyZh 3y agoThey are getting whatever you give them ;)
- skissane 3y agoHere is a simple maths problem that GPT-4 gets right but Bard (even the Gemini Pro version) consistently gets wrong: “What is one (short scale) centillion divided by the cube of a googol?” But you are right, we don’t know the types of prompts Chatbot Arena users are submitting. Maths problems like that are probably a small minority of usage. One other thing I notice: if you ask about controversial issues, both GPT-3.5/4 and Bard can get a bit “preachy” from a progressive perspective - but I personally find Bard to be noticeably more “preachy” than OpenAI at this (while still not reaching Llama levels)
- shortrounddev2 3y agoBy comparison I find bing image generator kicks dall-es ass
- jjackson5324 3y ago> By comparison I find bing image generator kicks dall-es ass Huh? Doesn't bing image generator just use the DALL-E api?
- Filligree 3y agoIt's the same generator, yeah. And I find the Bing version of it has so heavy censoring, I can never make it actually draw anything I want...
- mavamaarten 3y agoSame. I wanted to create some silly images but literally almost everything I tried was censored.
- novagameco 3y agoApparently, but when I use Dall-e 3 on OpenAI, the images it generates look like shit. Under-developed, with crappy eyes and hands, the kind of typical mutant stuff you see with AI generated Images. Bing seems to be much better at those types of details out of the box
- distances 3y agoI get good results through ChatGPT image generation but mostly disappointing when using DALL-E directly. Not sure if my prompt game is just sorely lacking or if there's something else being involved via ChatGPT.
- sfmike 3y agois it free
- deleted 3y ago
- sroussey 3y agoHow were you able to test Gemini Pro before today? Are you able to test Gemini Ultra?
- dchest 3y agoFrom the linked article: "Last December, we brought Gemini Pro into Bard in English..."
- mensetmanusman 3y agoGoogle just has to pay their AI scientist eight figures to catch up.
- behnamoh 3y ago> I've tested bard/gemini extensively on tasks that I routinely get very helpful results from GPT-4 with, and bard consistently, even dramatically underperforms. Yes. And I don't buy the lmsys leaderboard results where Google somehow shoved a mysterious gemini-pro model to be better than GPT-4. In my experience, its answers looked very much like GPT-4 (even the choice of words) so it could be that Bard was finetuned on GPT-4 data. Shady business when Google's Bard service is miles behind GPT-4.
- resters 3y agoTrue, what is most puzzling about it is the effort Google is putting into generating hype for something that is at best months away (by which time OpenAI will likely have released a better model)... My best guess is that Google realizes that something like GPT-4 is a far superior interface to interact with the world's information than search, and since most of Google's revenue comes from search, the handwriting is on the wall that Google's profitability will be completely destroyed in a few years once the world catches on. MS seeems to have had that same paranoia with the bingified GPT-4. What I found most remarkable about it was how much worse it performed seemingly because it was incorporating the top n bing results into the interaction. Obviously there are a lot of refinements to how a RAG or similar workflow might actually generate helpful queries and inform the AI behind the scenes with relevant high quality context. I think GPT-4 probably does this to some extent today. So what is remarkable is how far behind Google (and even MS via it's bingified version) are from what OpenAI has already available for $20 per month. Google started out free of spammy ads and has increasingly become more and more like the kind of ads everywhere in your face, spammy stuff that it replaced. GPT-4 is such a refreshingly simple and to the point way to interact with information. This is antithetical to what funds Google's current massive business... namely ads that distract from what the user wanted in hopes of inspiring a transaction that can be linked to the ad via a massive surveillance network and behavioral profiling model. I would not be surprised if within Google the product vision for the ultimate AI assistant is one that gently mentions various products and services as part of every interaction.
- xiphias2 3y agoThe ad model already went to take attribution / conversion from different sources into account (although there's a lot of spammy implementations), but it took many years for Google to make youtube / mobile ads profitable, and now adoption is much faster.
- gundmc 3y agoI don't think Google has released the version of Gemini that is supposed to compete with GPT4 yet. The current version is apparently more on the level of GPT 3.5, so your observations don't surprise me
- CSMastermind 3y agoI will say as someone who tries to regularly evaluate all the models Google's censorship is much worse than other companies. I routinely get "I can't do that" messages from Bard and no one else when testing queries. As an example, I had a photo of a beach I wanted to see if it knew the location of and it was blocked for inappropriate content. I stared at the picture for like 5 minutes confused until I blacked out the woman in a bikini standing on the beach and resubmitted the query at which point it processed it. It's refused to do translation for me because the text contains 'rude language'. It's blocked my requests on copyright grounds. I don't at all understand the heavy-handed censorship they're applying when they're behind in the market.
- jonplackett 3y agoI just tried to get it to write me some code that queries an API and it refused. I asked if it was not allowed to write any code for any APIs and it said yes that’s true. FFS
- nuclearnice3 3y agoIt won't surprise me if the photo or similar ends up banning your entire Google life with no reasonable appeal possible.
- outside415 3y agotheir censorship is the worst of any platform. being killed from within by the woke mob apparently. it's a pity for google employees, they're going to be undergoing cost cutting/perpetual lay offs for the foreseeable future as other players eat their advertising lunch.
- mike10921 3y agoOn the flip side, I find that GPT4 is constantly getting degraded. It intentionally only returns partial answers even when I direct it specifically not to do so. My guess is, that they are trying to save on CPU consumption by generating shorter responses.
- resters 3y agoI think at high traffic times it gets slightly different parameters that make it more likely to do that. I've had the best results during what I think are off-peak hours.
- nomel 3y agoIs this with the API or web interface?
- sebzim4500 3y agoMy feeling is that it got worse but then it got better again over the last few months. I don't have data to back this of course.
- zellyk 3y agoBard has been dead to me the second I saw it was not available in Canada... GPT all the way to be honest.
- replwoacause 3y agoTrust me you’re not missing anything. Google sucks at AI.
- sjwhevvvvvsj 3y agoMy personal favorite Bard failure mode is when I need help with Google Cloud and Bard has no idea what to do but GPT tells me *exactly* what I need. If you can’t even support your own products…I’m not sure what I’m supposed to do with this pos.
- 762236 3y agoMy experience is the opposite. I'm really tired of fighting ChatGPT.
- maxglute 3y agoFor whatever reason Bard doing pretty good with Google Sheet script suggestions than GPT4 for me. ALmost everything else is subpar.