15 ms·
Google's Bard shows big leap on LLM performance leaderboard
- lordswork 3y agoAnyone know the difference between Bard Gemini Pro and Gemini Pro Dev API on the leaderboard?
- refulgentis 3y agoShipping the org chart: tl;dr hosted via Bard org via Search versus via Cloud. And it's likely they're actually different.
- vineyardmike 3y agoWhile the other response was unflattering but true, a better answer would be that Bard ___ is trained to be a general purpose chat bot, specific for the Bard user experience. While Gemini ___ API is a developer focused API for LLM use. Similar to how ChatGPT performs different to the OpenAI APIs or Copilot in Bing, or whatever. Basically, they’re fine tuned and prompted to respond differently.
- brokensegue 3y agothis leaderboard seems easily cheated/gamed. once enough eyes are on it it will be worthless
- bearjaws 3y agoKind of, ChatBot Arena was updated on Dec 7th, so hard to know for sure if Bard is just fine tuned against it. That being said Chatbot Arena is a pretty wide variety of test scenarios. If fine tuning made the model perfect, all the small models would get similar scores to GPT4, which they don't. Essentially it ranks how people believe a ChatBot should respond, rather than just zero shot, 1 shot and COT type benchmarks.
- deleted 3y ago[deleted]
- karmasimida 3y agoI don't think Google would specifically targeting this leaderboard just for marketing purpose though.
- kossTKR 3y agoWhy not? Their entire presentation a month ago was fake.
- karmasimida 3y agoBecause this leaderboard has a very limited audience, and second only to GPT-4 turbo isn't good marketing regardless.
- khimaros 3y agothese scores are based on feedback from users who write any prompt they like and then blindly choose the best response. nothing to game.
- deleted 3y ago[deleted]
- modeless 3y agoHow would you game it? I think it is clearly the least gameable leaderboard we have. A more valid criticism might be that you don't like the metric it's measuring, but I think it is a useful metric, though certainly not the only useful metric.
- pptr 3y agoI think there is a theoretical threat for models hosted on an external server. You could ask a question on lmsys, check your server logs for the generated response, go back to lmsys and pick the response that your model generated. Maybe you could also use a better model for requests from lmsys. E. g. use an unquantized model, disable censorship, etc. I doubt any of the big players are doing that, but you never know.
- visarga 3y agoI could tell from the second interaction which was Bard Pro because it uses a specific formatting with 2 levels of indentation I never saw in other models.
- vinni2 3y agoComparatively BARD models have relatively lower number of votes. I would wait until number of votes are in the same magnitude as other models.
- danans 3y agoThat's why the site also shows the confidence interval.
- jsnell 3y ago3000 matches should be more than enough for an Elo-style ranking system to converge. But the notebook[0] linked to from the site includes a section with an equal number of matches being sampled for each pairwise comparison, and the results are pretty much the same. [0] https://colab.research.google.com/drive/1KdwokPjirkTmpO_P1WByFNFiqxWQquwH#scrollTo=5lBa9pdtzMA3 https://colab.research.google.com/drive/1KdwokPjirkTmpO_P1WB...
- lumuba 3y agohow is GPT4-Turbo higher than GPT-4?
- refulgentis 3y agoIt's actually better. All the people claiming unannounced updates make OpenAI model significantly worse were misled by their own lying eyes. It's so hard to believe, I know, but it's true.
- qeternity 3y agoIt isn’t, because performance isn’t scalar. It seems to be a more preferable chatbot in this arena. It is objectively less capable for many other domain specific tasks.
- refulgentis 3y agoHow does the arena work? ;)
- qeternity 3y agoLol it’s literally called the “LMSYS Chatbot Arena”. But yeah we tried some things.
- joshuanapoli 3y agoIt depends on the size of the response that you need…
- apapapa 3y agoChatGPT was best when first released and absent of censorship ... Probably 10x worst right now
- ravenstine 3y agoNo clue, but it wouldn't surprised me if they identified inputs and training that weren't actually helping. Just as human beings aren't necessarily more helpful by having more varied input, the same seems to apply to LLMs. The interesting thing is it still has a 128k context length. This is awesome because GPT became way more useful to me once it reached this level of context.
- tiltowait 3y agoGemini was a bit disappointing on launch. Good to see them improving. My own experience doesn't really match the arena's leaderboard. I find myself using Claude 2 much more than GPT 4/Turbo. I prefer its out-of-the-box response style, and its answers for my queries seem just as good, if not better. Interesting, Kagi (where I use the chatbots) ranks Claude 1 as equal to (and faster than) GPT 4 (non-Turbo), and marks Claude 2's quality as on-par with 4 Turbo (albeit slower). Worth noting it's a simple 1-4-star ranking, unlike the arena's ELO numbers.
- npinsker 3y agoCompletely concur with your experience. I've always suspected that the questions asked on these arena sites tend towards being a bit different qualitatively than real-world use cases.
- vineyardmike 3y agoI rarely hear people talk about liking Claude. I’m curious, what do you use it for? I personally use bard for general internet search stuff, because I like how google has set up the linking (and the UI). I use GPT-* for technical stuff (eg GitHub codepilot) because I find it can synthesize code better.
- random3 3y agoI'm curious about how the benchmark is done. I suspect it can be improved in order to represent user's / usability expectations. I gave Bard a go, after seeing Jeff Dean's tweet. It's just as frustrating as it was, compared to GPT-4. It's simply off the question and unable to realize it's off. I asked it to generate a chart and 3 times it came back with "here's a chart" with no chart, finally saying it doesn't have that ability.
- qeternity 3y agoIt’s based on users blind rating the output of two LLMs given the same prompt.
- kozikow 3y agoWhere GPT really improved for me subjectively is long context - at least since launch. Such ranking can't compare it. Also I felt I keep getting more personalized results, i.e. models are somehow biased towards user. I heard they plan it, I don't know it's launched, but I feel it. And there's also fine-tuning in the other direction - my brain got used to ways of interacting with GPT. Same as with Google, I just somehow subconsciously know how to write prompts that get me what I want.
- nemonemo 3y agoTypically benchmarks have limited aspects they are measuring. I can imagine another suite of benchmarks with longer contexts, but in that case, it might be more difficult to do it in a blind comparison form. At the least, it would be quite costly to run such benchmarks.
- bdhcuidbebe 3y ago> It's just as frustrating as it was, compared to GPT-4. It's simply off the question and unable to realize it's off. If this aint anecdotal evidence, I dont know what is. You need to assume a whole lot to draw these conclusions from prompt output comparison. Heck, just compare a couple outcomes from ChatGPT and you should conclude the same thing, assuming rationality of course.
- modeless 3y agoWow. I've suspected for a while that Bard's performance has been limited mostly by cost. Google isn't charging for Bard and they didn't want to run a gigantic model for everyone for free forever. Maybe they made a breakthrough in inference cost for their better models? Or maybe they got tired of everyone clowning on them for being behind and decided to eat the cost for a while. I still think they ought to launch a subscription so we can see their absolute best model running in public.
- ysofunny 3y agoI think their play is the same as always it's better to let more people interact with it because this will help training the model (get more data) so it must be free to use.
- j-b 3y agoThe trick is to access the "bard-jan-24-gemini-pro" model, available in direct chat mode here: https://chat.lmsys.org/ https://chat.lmsys.org/. Significantly better than the prior model.
- jxy 3y agohow odd! What exactly is lmsys using? Some hidden API that google give them so they can have a better ranking there?
- declaredapple 3y ago> Some hidden API that google give them so they can have a better ranking there? I don't know about that second part - but it would make sense that google (and others) may want to use lmsys's arena to benchmark their models. After all, Human A/B tests are far better then the current automated benchmarks. I would like more info from lmsys as to how they're accessing these though.
- j-b 3y agoMost likely through this platform: https://console.cloud.google.com/vertex-ai https://console.cloud.google.com/vertex-ai
- tempsy 3y agoBard is far less moderated than GPT-4 eg far fewer requests feel restricted. That alone makes it far better than GPT-4.
- kccqzy 3y agoBy moderation you mean safety filters?
- _nalply 3y agoYes. Moderation means no sex, hate, illegal things and religion. I tried to talk about Allah being merciful and got my ass moderated away. I am Buddhist and when I talked about reincarnation being misunderstood that was fine. So the limits are not clear.
- laborcontract 3y agoExcept when you run afoul of bard verus gpt-4, you get banned from the rest of your livelihood versus just openai?
- m3kw9 3y agoIt’s pretty good got newer info than gpt4, but refuses on stuff that doesn’t make sense when I ask it to write code using a certain public library, it refuses due to privacy reasons. Where as gpt4 will do it. Getting close, but this will force OpenAI to come out with gpt5 for another big round of catch up
- cryptozeus 3y agoFrom all free LLMs I find bard to be most useful. Chatgpt 3.5 is not even close and it lazy.
- warkdarrior 3y agoLazy??? What does it mean for a model to be lazy?
- jerrygenser 3y agoDoes not completely answer the question, all aspects of a question, or has a response otherwise cut off. This was common for gpt4 and coding questions it would simply stop responding in the middl
- pb7 3y agoChatGPT, is that you?
- vendiddy 3y agoSometimes I'd have GPT tell me to how to do something when it could have done it for me. For example it might tell me to read some documentation and find the answer myself. So I'm thinking "Yes well, what do I need you for then?"
- mike_hearn 3y agoQ: Please write me a function that makes a couple of text edits and a button. A: Certainly! Here is a function that makes a couple of text edits and a button. function renderForm() { var edit = document.createElement('input'); // The rest of the code is similar. }
- cookie_monsta 3y agoI don't get what's lazy about this. Is it because it's not appending the elements?
- seydor 3y agoIt used to be responsive to my requests, it did not deny me things. It seesm they have added a censorship layer. I miss slutty bard
- vdaea 3y agoGood for those who can access the Gemini model. I'm still stuck with PaLM2.
- vivekjibw 3y agoI asked Bard to rewrite an email and it's response included how to respond to the email.
- ipsum2 3y agoIt's not a valid comparison. Bard uses Google (ie the Internet) to answer things, while GPT4 doesn't. So bard can answer things like, "what's the weather in SF today" or "who won the basketball game last night".
- 2OEH8eoCRo0 3y agoI'm not sure what you mean. ChatGPT with GPT4 uses Bing to search the web.
- ipsum2 3y agoSee reply to sibling comment.
- robinjhuang 3y agoChat GPT4 does do internet searches now.
- ipsum2 3y agoThe model being compared on the LLM Arena does not use ChatGPT4, it uses GPT4 API, which does not have access to the internet by default.
- deleted 3y ago[deleted]
- summerlight 3y agohttps://twitter.com/JeffDean/status/1750930658900517157 https://twitter.com/JeffDean/status/1750930658900517157 > Bard, powered by the Gemini Pro-scale model, debuts at the #2 position on the independent lmsys leaderboard. According to Jeff Dean's tweet, it looks like they have a new "Gemini Pro-scale model" being rolled out, not sure what it means by "Pro-scale" though. Also not sure if everyone already got it...
- ijhuygft776 3y agofor me Bard is much better than ChatGPT... I wish I would be using an uncensored Mistral though.
- visarga 3y agoYeah, I was evaluating Bard Pro vs Starling and up to the end I wasn't sure what to pick. Starling 7B was almost as good. People don't realize there are ways to "mommy, can we get GPT?" "we have GPT at home darling"
- deleted 3y ago[deleted]
- nblgbg 3y agoSome how my experience doesn't reflect this. I keep going back to GPT if i need an answer. I even say most of the time I prefer 3.5 better than Google Bard. GPT 4 is definitely way better than Bard for sure.