5 ms·

I'm curious about how the benchmark is done. I suspect it can be improved in order to represent user's / usability expectations. I gave Bard a go, after seeing

I'm curious about how the benchmark is done. I suspect it can be improved in order to represent user's / usability expectations. I gave Bard a go, after seeing Jeff Dean's tweet. It's just as frustrating as it was, compared to GPT-4. It's simply off the question and unable to realize it's off. I asked it to generate a chart and 3 times it came back with "here's a chart" with no chart, finally saying it doesn't have that ability.

qeternity 3y ago
It’s based on users blind rating the output of two LLMs given the same prompt.
- kozikow 3y ago
  Where GPT really improved for me subjectively is long context - at least since launch. Such ranking can't compare it. Also I felt I keep getting more personalized results, i.e. models are somehow biased towards user. I heard they plan it, I don't know it's launched, but I feel it. And there's also fine-tuning in the other direction - my brain got used to ways of interacting with GPT. Same as with Google, I just somehow subconsciously know how to write prompts that get me what I want.
  - nemonemo 3y ago
    Typically benchmarks have limited aspects they are measuring. I can imagine another suite of benchmarks with longer contexts, but in that case, it might be more difficult to do it in a blind comparison form. At the least, it would be quite costly to run such benchmarks.
bdhcuidbebe 3y ago
> It's just as frustrating as it was, compared to GPT-4. It's simply off the question and unable to realize it's off. If this aint anecdotal evidence, I dont know what is. You need to assume a whole lot to draw these conclusions from prompt output comparison. Heck, just compare a couple outcomes from ChatGPT and you should conclude the same thing, assuming rationality of course.
vineyardmike 3y ago
Bard is free, while GPT-4 is not, so it doesn’t seem like a totally fair comparison. Also what a wild comparison, because afaik chat gpt can’t make charts either.
- krona 3y ago
  > Bard is free, while GPT-4 is not, so it doesn’t seem like a totally fair comparison. Wait so if Google suddenly started charging for Bard, it would be instantly better?
  - KeplerBoy 3y ago
    It could be. Bard is clearly not the best model Google has and the only reason not to serve the better models is inference cost.
    - qeternity 3y ago
      Google is fighting for its life here. They are not worried about cost.
      - kccqzy 3y ago
        Google is one of the most cost conscious companies in tech when it comes to compute costs. Sure they are blind to other kinds of cost like reputation damage due to stupid leadership decisions. But in terms of their tech, they run their servers and their network to very high utilization, definitely exceeding competitors like Amazon.
        qeternity 3y ago
        You mention cost, and then proceed to describe efficiency. They are different things. They can spend billions giving away compute (cost) and be extremely efficient (high utilization) in doing so.
      - KeplerBoy 3y ago
mhh__ 3y ago
My favourite by miles was asking bard for some more mathematical examples after explaining quantum field theory quite well in words, to which it said "Ok here are some examples of mathematics: 2+2=4"
- mhh__ 3y ago
  https://bard.google.com/share/ff90b8239ebe https://bard.google.com/share/ff90b8239ebe found it
thorum 3y ago
You’re probably using the old Bard model. You can try the new one, bard-jan-24-gemini-pro, by clicking the Direct Chat tab on https://chat.lmsys.org https://chat.lmsys.org.
- random3 3y ago
  Thank you! Quick feedback - lmsys will introduce bias because it lacks rendering support for things like [ x_2 = \frac{1}{2} \left( 1.5 + \frac{2}{1.5} \right) = \frac{1}{2} \left( 1.5 + \frac{4}{3} \right) = \frac{1}{2} \left( \frac{9}{6} + \frac{8}{6} \right) = \frac{17}{12} \approx 1.4166666666666667 ]
lagt_t 3y ago
? https://i.imgur.com/b0fIGyS.png https://i.imgur.com/b0fIGyS.png
- random3 3y ago
  that makes it even worse for saying 2 times "here's your chart" and the third one appologizing for not having the capability, when in fact is has it? Here you go https://imgur.com/a/MT04viM https://imgur.com/a/MT04viM
  - lagt_t 3y ago
    That is indeed weird.
karmasimida 3y ago
I tried Bard today, it did improve surprisingly Disclaimers: I don’t work for Google

qeternity 3y ago

It’s based on users blind rating the output of two LLMs given the same prompt.

kozikow 3y ago
Where GPT really improved for me subjectively is long context - at least since launch. Such ranking can't compare it. Also I felt I keep getting more personalized results, i.e. models are somehow biased towards user. I heard they plan it, I don't know it's launched, but I feel it. And there's also fine-tuning in the other direction - my brain got used to ways of interacting with GPT. Same as with Google, I just somehow subconsciously know how to write prompts that get me what I want.
- nemonemo 3y ago
  Typically benchmarks have limited aspects they are measuring. I can imagine another suite of benchmarks with longer contexts, but in that case, it might be more difficult to do it in a blind comparison form. At the least, it would be quite costly to run such benchmarks.

bdhcuidbebe 3y ago

> It's just as frustrating as it was, compared to GPT-4. It's simply off the question and unable to realize it's off. If this aint anecdotal evidence, I dont know what is. You need to assume a whole lot to draw these conclusions from prompt output comparison. Heck, just compare a couple outcomes from ChatGPT and you should conclude the same thing, assuming rationality of course.

vineyardmike 3y ago

Bard is free, while GPT-4 is not, so it doesn’t seem like a totally fair comparison. Also what a wild comparison, because afaik chat gpt can’t make charts either.

krona 3y ago
> Bard is free, while GPT-4 is not, so it doesn’t seem like a totally fair comparison. Wait so if Google suddenly started charging for Bard, it would be instantly better?
- KeplerBoy 3y ago
  It could be. Bard is clearly not the best model Google has and the only reason not to serve the better models is inference cost.
  - qeternity 3y ago
    Google is fighting for its life here. They are not worried about cost.
    - kccqzy 3y ago
      Google is one of the most cost conscious companies in tech when it comes to compute costs. Sure they are blind to other kinds of cost like reputation damage due to stupid leadership decisions. But in terms of their tech, they run their servers and their network to very high utilization, definitely exceeding competitors like Amazon.
      - qeternity 3y ago
        You mention cost, and then proceed to describe efficiency. They are different things. They can spend billions giving away compute (cost) and be extremely efficient (high utilization) in doing so.
    - KeplerBoy 3y ago

mhh__ 3y ago

My favourite by miles was asking bard for some more mathematical examples after explaining quantum field theory quite well in words, to which it said "Ok here are some examples of mathematics: 2+2=4"

mhh__ 3y ago
https://bard.google.com/share/ff90b8239ebe https://bard.google.com/share/ff90b8239ebe found it

thorum 3y ago

You’re probably using the old Bard model. You can try the new one, bard-jan-24-gemini-pro, by clicking the Direct Chat tab on https://chat.lmsys.org https://chat.lmsys.org.

random3 3y ago
Thank you! Quick feedback - lmsys will introduce bias because it lacks rendering support for things like [ x_2 = \frac{1}{2} \left( 1.5 + \frac{2}{1.5} \right) = \frac{1}{2} \left( 1.5 + \frac{4}{3} \right) = \frac{1}{2} \left( \frac{9}{6} + \frac{8}{6} \right) = \frac{17}{12} \approx 1.4166666666666667 ]

lagt_t 3y ago

? https://i.imgur.com/b0fIGyS.png https://i.imgur.com/b0fIGyS.png

random3 3y ago
that makes it even worse for saying 2 times "here's your chart" and the third one appologizing for not having the capability, when in fact is has it? Here you go https://imgur.com/a/MT04viM https://imgur.com/a/MT04viM
- lagt_t 3y ago
  That is indeed weird.

karmasimida 3y ago

I tried Bard today, it did improve surprisingly Disclaimers: I don’t work for Google