6 ms·

Is Google actually a frontrunner in this race? I remember Bard was pretty unimpressive when it came out, is it improving fast enough to be competitive?

by jfim 3y ago

declaredapple 3y ago
Their currently available offerings are "meh", but they're definitely top 5 (Currently they're number 4). This leaderboard is the only one most people trust as it uses an ELO system with humans blindly comparing and rating them in the chatbot arena. https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... Current leaders are OpenAI, Anthropic, Mistral, Google, 01-ai, and then various Llama finetunes and Llama itself (Meta)
- Jackson__ 3y ago
  This leaderboard is so funny when you look at Anthropic's Claude. Every version After 1.0 gets a worse score. It seems their vision of "alignment" does not align with the users vision of a competent assistant even remotely.
  - declaredapple 3y ago
    Yeah it's mystifying to me that they because the #1 commercial OpenAI competitor and then decided to go all "Helpless and Harmless"
  - frognumber 3y ago
    I don't think that's their business model. I would never use Claude for personal use. However, my employer wants to make sure nothing reputation-harming happens. That's a lot more important than model quality; everything is light years ahead of where we were four years ago. For a lot of applications which face customers / citizens / students / ..., NEVER screwing up is a lot more important than quality. For my own use, I prefer interacting with soul, humanity, attitude, and edge. For my employer's use, it's different. I think there's space for both. As an investor, I'd be bullish on both Anthropic and something with no safety built in. I'd be a lot less bullish on what's in between.
    - declaredapple 3y ago
      I totally understand the need for an "on-rails" model. However I've found that using it as a personal assistant and for some automation tasks it suffers. The fact that it has only gotten worse as judged by humans doesn't inspire confidence for me. I could totally understand offering two options, but they're going so far it's actually unusable for many tasks. The famous case it refused to "kill a python process".
      - frognumber 3y ago
        Here's the thing about business models in this domain: You don't win by generality or working for as many tasks as possible. You win by being the best at something. I will pick the best tool for the job I'm doing, be that writing product descriptions, conversational agent, or tech support. Runner-up has a chance -- for example, by lowering margins, or simply by being subsidized by investors in hopes of moving into #1. However, the difference between "unusable" and "fourth-best" is negligible in terms of business returns. I won't pick your product. There isn't a snowball's chance of "safe AI" being #1, or even #5, for what you want to use it for. It might as well be unusable. It needs to be #1 in the niche it's targeting. (The above isn't universal; there are places where bundling many types of functionality has synergy; this just isn't one of them).
tracerbulletx 3y ago
Chat's cool and all but Google is out here discovering new useful bio-molecules and winning Go and StarCraft matches against the best players in the world still.
- dontupvoteme 3y ago
  well, that's deepmind, who are geniuses but still just part of a massive corpo.. Also shouldn't starcraft be solved for ages now? OpenAI the top teams in DoTA a long time ago, and i assumed the coordination problem would be a big issue. (I think their bots started winning 1v1 SF mid a long time before that...)
  - tracerbulletx 3y ago
    I believe this is going to be a long race and I think while OpenAI has solved important problems and gone to market first, they don't have anything to really be that "ahead" on. I think any one of these companies has the tools to pick a later moment to come to market without any serious risk of being too late.
huytersd 3y ago
Maybe it’s based on Gemini Ultra whose benchmarks seem to beat GPT4? It not out until later this quarter though so we’ll have to wait and see.
- declaredapple 3y ago
  You don't need to be #1 and beat GPT4 if we're going with "a frontrunner". Gemini Pro as available today already puts Google at #4 by company in the chatbot arena.
kromem 3y ago
Their internal models are allegedly competitive even if their public offerings are not. But given the amount of BS in this space these days, take with an accordingly sized grain of salt. I'll believe Google is competitive with the SotA when I see it myself.

Is Google actually a frontrunner in this race? I remember Bard was pretty unimpressive when it came out, is it improving fast enough to be competitive?

by jfim 3y ago

Is Google actually a frontrunner in this race? I remember Bard was pretty unimpressive when it came out, is it improving fast enough to be competitive?

declaredapple 3y ago

Their currently available offerings are "meh", but they're definitely top 5 (Currently they're number 4). This leaderboard is the only one most people trust as it uses an ELO system with humans blindly comparing and rating them in the chatbot arena. https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... Current leaders are OpenAI, Anthropic, Mistral, Google, 01-ai, and then various Llama finetunes and Llama itself (Meta)

Jackson__ 3y ago
This leaderboard is so funny when you look at Anthropic's Claude. Every version After 1.0 gets a worse score. It seems their vision of "alignment" does not align with the users vision of a competent assistant even remotely.
- declaredapple 3y ago
  Yeah it's mystifying to me that they because the #1 commercial OpenAI competitor and then decided to go all "Helpless and Harmless"
- frognumber 3y ago
  I don't think that's their business model. I would never use Claude for personal use. However, my employer wants to make sure nothing reputation-harming happens. That's a lot more important than model quality; everything is light years ahead of where we were four years ago. For a lot of applications which face customers / citizens / students / ..., NEVER screwing up is a lot more important than quality. For my own use, I prefer interacting with soul, humanity, attitude, and edge. For my employer's use, it's different. I think there's space for both. As an investor, I'd be bullish on both Anthropic and something with no safety built in. I'd be a lot less bullish on what's in between.
  - declaredapple 3y ago
    I totally understand the need for an "on-rails" model. However I've found that using it as a personal assistant and for some automation tasks it suffers. The fact that it has only gotten worse as judged by humans doesn't inspire confidence for me. I could totally understand offering two options, but they're going so far it's actually unusable for many tasks. The famous case it refused to "kill a python process".
    - frognumber 3y ago
      Here's the thing about business models in this domain: You don't win by generality or working for as many tasks as possible. You win by being the best at something. I will pick the best tool for the job I'm doing, be that writing product descriptions, conversational agent, or tech support. Runner-up has a chance -- for example, by lowering margins, or simply by being subsidized by investors in hopes of moving into #1. However, the difference between "unusable" and "fourth-best" is negligible in terms of business returns. I won't pick your product. There isn't a snowball's chance of "safe AI" being #1, or even #5, for what you want to use it for. It might as well be unusable. It needs to be #1 in the niche it's targeting. (The above isn't universal; there are places where bundling many types of functionality has synergy; this just isn't one of them).

tracerbulletx 3y ago

Chat's cool and all but Google is out here discovering new useful bio-molecules and winning Go and StarCraft matches against the best players in the world still.

dontupvoteme 3y ago
well, that's deepmind, who are geniuses but still just part of a massive corpo.. Also shouldn't starcraft be solved for ages now? OpenAI the top teams in DoTA a long time ago, and i assumed the coordination problem would be a big issue. (I think their bots started winning 1v1 SF mid a long time before that...)
- tracerbulletx 3y ago
  I believe this is going to be a long race and I think while OpenAI has solved important problems and gone to market first, they don't have anything to really be that "ahead" on. I think any one of these companies has the tools to pick a later moment to come to market without any serious risk of being too late.

huytersd 3y ago

Maybe it’s based on Gemini Ultra whose benchmarks seem to beat GPT4? It not out until later this quarter though so we’ll have to wait and see.

declaredapple 3y ago
You don't need to be #1 and beat GPT4 if we're going with "a frontrunner". Gemini Pro as available today already puts Google at #4 by company in the chatbot arena.

kromem 3y ago

Their internal models are allegedly competitive even if their public offerings are not. But given the amount of BS in this space these days, take with an accordingly sized grain of salt. I'll believe Google is competitive with the SotA when I see it myself.