6 ms·

> For Gemini Ultra, we’re currently completing extensive trust and safety checks, including red-teaming by trusted external parties, and further refining the m

> For Gemini Ultra, we’re currently completing extensive trust and safety checks, including red-teaming by trusted external parties, and further refining the model using fine-tuning and reinforcement learning from human feedback (RLHF) before making it broadly available. > As part of this process, we’ll make Gemini Ultra available to select customers, developers, partners and safety and responsibility experts for early experimentation and feedback before rolling it out to developers and enterprise customers early next year. Finally, some competition for GPT4 API!!! This is such good news.

logicchains 3y ago
>Finally, some competition for GPT4 API!!! This is such good news. Save your enthusiasm for after it launches; Google's got a habit of over-promising when it comes to AI.
- endisneigh 3y ago
  I’m curious which instances of overpromising you’re referring to.
  - logicchains 3y ago
    Like how much they hyped up Bard, which when released turned out to be barely competitive with GPT3.5. E.g. https://www.reuters.com/technology/google-ai-chatbot-bard-offers-inaccurate-information-company-ad-2023-02-08/ https://www.reuters.com/technology/google-ai-chatbot-bard-of...
    - endisneigh 3y ago
      I do not recall Bard being said to be better than any particular other model, but then having worse performance by some metric when released. Your link isn’t really an indication of an overpromise.
    - freedomben 3y ago
      I definitely think GPT is better than Bard, but Bard definitely did live up to the hype in a few ways. The two that blew my mind (and still do to some extent) are the blazing speed and the ability to pull information real time (no more pesky knowledge cutoff date). Bard also felt pretty comparable to 3.5 to me, better in some things and worse in others. Coding was definitely a bust with Bard.
      - vorticalbox 3y ago
        ChatGPT via plugins and tools can access real time data, the bot I built for slack at my work has the ability to load Web pages, search DDG etc.
    - danpalmer 3y ago
      Bard isn't a model, it's a product. Saying comparisons against "Bard" without specifying a particular point in time are like analyses of "ChatGPT" without specifying a model. There have been a number of releases adding more features, tool use, making it smarter, and crucially adding more languages. ChatGPT is not fine-tuned in different languages – it manages them but lacks cultural context. That's one place Bard is quite far ahead from what I've seen.
      - willsmith72 3y ago
        all that shows is that google screwed up their positioning, and openai got it right people don't see a difference between model and product, they think "gpt3 is ok", "gpt4 is great", "bard is like gpt3" it's not the consumer's fault when the business has a positioning mistake, the business has to try and win the consumer back
        danpalmer 3y ago
        Most people don't use LLMs. Of those that do most people just know they're using "ChatGPT". A slim minority care about the model. In my opinion, not focusing on the model, focusing on the product, and focusing on positioning for normal users (free, fast, fine tuned in many languages, "easy"), is a better product positioning.
        willsmith72 3y ago
        > In my opinion, not focusing on the model, focusing on the product, and focusing on positioning for normal users (free, fast, fine tuned in many languages, "easy"), is a better product positioning. Does google agree? doesn't the fact that they're so deliberately creating user-focused branding for different models (ultra, pro, nano) show they also see the value in the differentiation?
        danpalmer 3y ago
        I can't speak for Google, and must emphasise that these are personal opinions. However I'd say that this entire marketing push is mostly for the super-engaged early adopters, not targeted at the general public. Looking at the YouTube videos, the more they seem to be targeted towards a general audience the less they mention these specifics. So, I suspect that the Ultra/Pro/Nano branding will mostly be used on the advanced Bard product that they speak about in the launch blog post, and on the APIs available to developers.
  - kolinko 3y ago
    In terms of AI? Last year? A better question is what wasn't an overpromise?
- isoprophlex 3y ago
  Everything they published thus far in the generative AI space has been abysmal in quality compared to the competition. I'd be hella surprised if this reaches GPT-4 levels of quality...
  - jefftk 3y ago
    Manifold has this at 69%, so here's an opportunity to take some people's internet points: https://manifold.markets/brubsby/will-googles-gemini-beat-gpt4-in-te https://manifold.markets/brubsby/will-googles-gemini-beat-gp...
    - danpalmer 3y ago
      Looks like it's 75%? It briefly dropped to 68% in October/November, but has been fairly consistently around 75% for a while.
      - jefftk 3y ago
        It's very active today: 50+ trades in the last hour. When I checked it was 69%, but it's gone up and down since then. Click on the "trades" tab to see.
        danpalmer 3y ago
        Ah fair enough, this seems to be an awkward UI. Over time though 75% seems to be the representative recent position. It is up to 85% now though!
  - bklyn11201 3y ago
    I'm a GPT4 subscriber and a Google GSuite work subscriber. I've been using the latest Bard this morning to write and refine python code, and it's just as good if not slightly better than GPT4. I asked it to refine some obtuse code with lots of chaining, and it did an admirable job writing accurate comments and explaining the chained logic. It's ridiculously anecdotal of course, but I used Bard for all of 5 minutes last time they announced. This time seems different.
kolinko 3y ago
It won't be available to regular devs until Q2 next year probably (January for selected partners). So they are roughly a year behind OpenAI - and that is assuming their model is not overtrained to just pass the tests slightly better than GPT4
- logicchains 3y ago
  >So they are roughly a year behind OpenAI Possibly by that time GPT5 will already be out.
  - a_wild_dandan 3y ago
    Possibly, but I doubt it! I'd expect a response by OpenAI via GPT-4V improvements soon though.
  - rvz 3y ago
    I would not underestimate DeepMind with its access to Google's resources and private data which OpenAI lacks, even with Microsoft's help. They already caught up and surpassed GPT-4 and OpenAI's availability and APIs are very unstable and all that matters is that and the cost per token.
- n2d4 3y ago
  > and that is assuming their model is not overtrained to just pass the tests slightly better than GPT4 You are assuming GPT4 didn't do the exact same! Seriously, it's been like this for a while, with LLMs any benchmark other than human feedback is useless. I guess we'll see how Gemini performs when it's released next year and we get independent groups comparing them.
losvedir 3y ago
Heh, I read that and had the opposite conclusion. When I was reading the benchmarks and seeing how Gemini Ultra was outperforming GPT-4 I thought, "Finally, some competition for GPT4"! But when I got to that part, that's when I realized that it could potentially be caught in release hell and not actually see the light of day or significant use. Google, for better or worse, has more of a brand reputation to maintain and is more risk averse, so even if Gemini Ultra can, in theory, outperform GPT4, users might not get a chance to access it for a while.
- GreedClarifies 3y ago
  "Google, for better or worse, has more of a brand reputation to maintain" You think this is why Google is so far behind?
  - k_kelly 3y ago
    It absolutely is. Googlers here will know that there was an internal version of ChatGPT that got canned because of halucinations.
    - bcjordan 3y ago
      better than chatgpt? Or canned because hallucinations were even worse?
    - zarzavat 3y ago
      Isn’t that Bard?
  - aix1 3y ago
    It's by no means the whole story, but Google's very significant aversion to brand risk is definitely part of why they move slowly. (Speaking as someone who's worked on launching several somewhat risky technologies at Google.)
  - freedomben 3y ago
    Absolutely I do. Internally they have some incredible stuff, but the leadership is terrified of letting normies try it out because of the (real or perceived I don't know) damage to the brand that would happen if it said something racist or misogynist, etc.
kubb 3y ago
I’m afraid it won’t be nearly as good as GPT4, because of how lax Open AI can be with intellectual property. Google will not be able to train their model on Libgen or Opensubtitles, because they can’t afford the risk.
TechRemarker 3y ago
Well not exactly. Not coming out until later when presumably GPT4 will have grown as much as well. So far each time, Google has failed to catch up to OpenAI. Hopefully they do however eventually.
- xnx 3y ago
  > presumably GPT4 will have grown as much as well Most of the comments I see on Hacker News claim ChatGPT is getting worse at different things (though I don't believe those claims).
- Jeff_Brown 3y ago
  Even if they're perpetually a year behind, that's a strong competitive spur to keep OpenAI on the move.
ren_engineer 3y ago
won't be available for regular devs until probably Q2 next year, OpenAI will have probably released GPT5 or whatever new model by then. And GPT4 was done training in 2022, the fact Google is thumping their chest about being 2 years behind a much smaller company is kind of pathetic
- FirmwareBurner 3y ago
  That's actually not a bad achievement considering Google is now the new IBM.
deleted 3y ago
[deleted]

logicchains 3y ago

>Finally, some competition for GPT4 API!!! This is such good news. Save your enthusiasm for after it launches; Google's got a habit of over-promising when it comes to AI.

endisneigh 3y ago
I’m curious which instances of overpromising you’re referring to.
- logicchains 3y ago
  Like how much they hyped up Bard, which when released turned out to be barely competitive with GPT3.5. E.g. https://www.reuters.com/technology/google-ai-chatbot-bard-offers-inaccurate-information-company-ad-2023-02-08/ https://www.reuters.com/technology/google-ai-chatbot-bard-of...
  - endisneigh 3y ago
    I do not recall Bard being said to be better than any particular other model, but then having worse performance by some metric when released. Your link isn’t really an indication of an overpromise.
  - freedomben 3y ago
    I definitely think GPT is better than Bard, but Bard definitely did live up to the hype in a few ways. The two that blew my mind (and still do to some extent) are the blazing speed and the ability to pull information real time (no more pesky knowledge cutoff date). Bard also felt pretty comparable to 3.5 to me, better in some things and worse in others. Coding was definitely a bust with Bard.
    - vorticalbox 3y ago
      ChatGPT via plugins and tools can access real time data, the bot I built for slack at my work has the ability to load Web pages, search DDG etc.
  - danpalmer 3y ago
    Bard isn't a model, it's a product. Saying comparisons against "Bard" without specifying a particular point in time are like analyses of "ChatGPT" without specifying a model. There have been a number of releases adding more features, tool use, making it smarter, and crucially adding more languages. ChatGPT is not fine-tuned in different languages – it manages them but lacks cultural context. That's one place Bard is quite far ahead from what I've seen.
    - willsmith72 3y ago
      all that shows is that google screwed up their positioning, and openai got it right people don't see a difference between model and product, they think "gpt3 is ok", "gpt4 is great", "bard is like gpt3" it's not the consumer's fault when the business has a positioning mistake, the business has to try and win the consumer back
      - danpalmer 3y ago
        Most people don't use LLMs. Of those that do most people just know they're using "ChatGPT". A slim minority care about the model. In my opinion, not focusing on the model, focusing on the product, and focusing on positioning for normal users (free, fast, fine tuned in many languages, "easy"), is a better product positioning.
        willsmith72 3y ago
        > In my opinion, not focusing on the model, focusing on the product, and focusing on positioning for normal users (free, fast, fine tuned in many languages, "easy"), is a better product positioning. Does google agree? doesn't the fact that they're so deliberately creating user-focused branding for different models (ultra, pro, nano) show they also see the value in the differentiation?
        danpalmer 3y ago
        I can't speak for Google, and must emphasise that these are personal opinions. However I'd say that this entire marketing push is mostly for the super-engaged early adopters, not targeted at the general public. Looking at the YouTube videos, the more they seem to be targeted towards a general audience the less they mention these specifics. So, I suspect that the Ultra/Pro/Nano branding will mostly be used on the advanced Bard product that they speak about in the launch blog post, and on the APIs available to developers.
- kolinko 3y ago
  In terms of AI? Last year? A better question is what wasn't an overpromise?
isoprophlex 3y ago
Everything they published thus far in the generative AI space has been abysmal in quality compared to the competition. I'd be hella surprised if this reaches GPT-4 levels of quality...
- jefftk 3y ago
  Manifold has this at 69%, so here's an opportunity to take some people's internet points: https://manifold.markets/brubsby/will-googles-gemini-beat-gpt4-in-te https://manifold.markets/brubsby/will-googles-gemini-beat-gp...
  - danpalmer 3y ago
    Looks like it's 75%? It briefly dropped to 68% in October/November, but has been fairly consistently around 75% for a while.
    - jefftk 3y ago
      It's very active today: 50+ trades in the last hour. When I checked it was 69%, but it's gone up and down since then. Click on the "trades" tab to see.
      - danpalmer 3y ago
        Ah fair enough, this seems to be an awkward UI. Over time though 75% seems to be the representative recent position. It is up to 85% now though!
- bklyn11201 3y ago
  I'm a GPT4 subscriber and a Google GSuite work subscriber. I've been using the latest Bard this morning to write and refine python code, and it's just as good if not slightly better than GPT4. I asked it to refine some obtuse code with lots of chaining, and it did an admirable job writing accurate comments and explaining the chained logic. It's ridiculously anecdotal of course, but I used Bard for all of 5 minutes last time they announced. This time seems different.

kolinko 3y ago

It won't be available to regular devs until Q2 next year probably (January for selected partners). So they are roughly a year behind OpenAI - and that is assuming their model is not overtrained to just pass the tests slightly better than GPT4

logicchains 3y ago
>So they are roughly a year behind OpenAI Possibly by that time GPT5 will already be out.
- a_wild_dandan 3y ago
  Possibly, but I doubt it! I'd expect a response by OpenAI via GPT-4V improvements soon though.
- rvz 3y ago
  I would not underestimate DeepMind with its access to Google's resources and private data which OpenAI lacks, even with Microsoft's help. They already caught up and surpassed GPT-4 and OpenAI's availability and APIs are very unstable and all that matters is that and the cost per token.
n2d4 3y ago
> and that is assuming their model is not overtrained to just pass the tests slightly better than GPT4 You are assuming GPT4 didn't do the exact same! Seriously, it's been like this for a while, with LLMs any benchmark other than human feedback is useless. I guess we'll see how Gemini performs when it's released next year and we get independent groups comparing them.

losvedir 3y ago

Heh, I read that and had the opposite conclusion. When I was reading the benchmarks and seeing how Gemini Ultra was outperforming GPT-4 I thought, "Finally, some competition for GPT4"! But when I got to that part, that's when I realized that it could potentially be caught in release hell and not actually see the light of day or significant use. Google, for better or worse, has more of a brand reputation to maintain and is more risk averse, so even if Gemini Ultra can, in theory, outperform GPT4, users might not get a chance to access it for a while.

GreedClarifies 3y ago
"Google, for better or worse, has more of a brand reputation to maintain" You think this is why Google is so far behind?
- k_kelly 3y ago
  It absolutely is. Googlers here will know that there was an internal version of ChatGPT that got canned because of halucinations.
  - bcjordan 3y ago
    better than chatgpt? Or canned because hallucinations were even worse?
  - zarzavat 3y ago
    Isn’t that Bard?
- aix1 3y ago
  It's by no means the whole story, but Google's very significant aversion to brand risk is definitely part of why they move slowly. (Speaking as someone who's worked on launching several somewhat risky technologies at Google.)
- freedomben 3y ago
  Absolutely I do. Internally they have some incredible stuff, but the leadership is terrified of letting normies try it out because of the (real or perceived I don't know) damage to the brand that would happen if it said something racist or misogynist, etc.

kubb 3y ago

I’m afraid it won’t be nearly as good as GPT4, because of how lax Open AI can be with intellectual property. Google will not be able to train their model on Libgen or Opensubtitles, because they can’t afford the risk.

TechRemarker 3y ago

Well not exactly. Not coming out until later when presumably GPT4 will have grown as much as well. So far each time, Google has failed to catch up to OpenAI. Hopefully they do however eventually.

xnx 3y ago
> presumably GPT4 will have grown as much as well Most of the comments I see on Hacker News claim ChatGPT is getting worse at different things (though I don't believe those claims).
Jeff_Brown 3y ago
Even if they're perpetually a year behind, that's a strong competitive spur to keep OpenAI on the move.

ren_engineer 3y ago

won't be available for regular devs until probably Q2 next year, OpenAI will have probably released GPT5 or whatever new model by then. And GPT4 was done training in 2022, the fact Google is thumping their chest about being 2 years behind a much smaller company is kind of pathetic

FirmwareBurner 3y ago
That's actually not a bad achievement considering Google is now the new IBM.

deleted 3y ago

[deleted]