4 ms·

> Google's model is not as capable as llama-derived models, so I think they would actually benefit from this. Google's publically available model isn't as capa

> Google's model is not as capable as llama-derived models, so I think they would actually benefit from this. Google's publically available model isn't as capable. But they certainly have models that are far better already in house.

whimsicalism 3y ago
I have no idea how you are so certain of that. Meta is definitely ahead of Google in terms of NLP expertise and has been for a while. I suspect that Google released their best model at the time with Bard.
- dooraven 3y ago
  We still don't have access to Imagen last I checked, it's still in restricted access. We don't have access to SoundStorm or MusicLM https://imagen.research.google/ https://imagen.research.google/ https://google-research.github.io/seanet/soundstorm/examples/ https://google-research.github.io/seanet/soundstorm/examples... https://google-research.github.io/seanet/musiclm/examples/ https://google-research.github.io/seanet/musiclm/examples/ Why would it be surprising that they have better models for resarch that they don't want to give out yet?
  - whimsicalism 3y ago
    Because I work in NLP so I have a good sense of the different capabilities of different firms and for the Bard release, it would have made more sense for them to have a more limited release of a better model for PR reasons than what actually happened. The other things you are describing are just standard for research paper releases.
    - dooraven 3y ago
      > Bard release, it would have made more sense for them to have a more limited release of a better model for PR reasons than what actually happened. Yes I would agree with you if Google wasn't set on to full on panic mode by their investors about releasing something vs Open AI due to Chat GPT's buzz. Bard was just a "hey we can do this too" thing, it was released half assed, had next to no marketing or hype. Vertex AI is their real proper offering, and I want to see how PaLM 2 does in comparison.
      - whimsicalism 3y ago
        I can already tell you that PaLM is not anywhere near as good and PaLM-2 is at least not as good before RLHF. Not going to keep replying, believe what you want about Google's capabilities
        dooraven 3y ago
        ok now I am confused, as Meta themselves say Palm-2 is better than Llama 2? > Llama 2 70B results are on par or better than PaLM (540B) (Chowdhery et al., 2022) on almost all benchmarks. There is still a large gap in performance between Llama 2 70B and GPT-4 and PaLM-2-L. https://scontent.fsyd7-1.fna.fbcdn.net/v/t39.2365-6/10000000_6495670187160042_4742060979571156424_n.pdf https://scontent.fsyd7-1.fna.fbcdn.net/v/t39.2365-6/10000000... If Google's publically available model is better Llama 2 already then why is it so inconceivable that they'd have private models that are better than their public ones which are better than LLama already. Palm-2 isn't better than GPT-4 but the convo was about better than Llama models no?
        neonbjb 3y ago
        @dooraven - I also work in ML (including recently working at Google) and I agree with @whimsicalism. You seem to be under the mistaken belief that: 1. Google has competent high-level organization that effectively sets and pursues long term goals. 2. There is some advantage to developing a highly capable LLM but not releasing it. (2) could be the case if Google had built an extremely large model which was too expensive to deploy. Having been privy to what they had been working on up until mid-2022 and knowing how much work, compute and planning goes into extremely large models, this would very much surprise me. Note: I did not have much visibility into what deepmind was up to. Maybe they had something.
- flangola7 3y ago
  > I have no idea how you are so certain of that. Some among us work with it, or have friends or family who work with it. I imagine it is one of those.
WastingMyTime89 3y ago
Do they? Considering how much was at stack in term of PR when OpenAI released ChatGPT, I would be surprised that Google didn’t put out the best they could.
- freedomben 3y ago
  The other end of the PR stake was safety/alignment. If Google released a well functioning model, but it said some unsavory things or carried out requests that the public doesn't find agreeable, it could make Google look bad.
matt_holden 3y ago
Comments like this remind me of the old-timers from IBM saying "but wait, we invented the PC! and the cloud! and..." Gotta put products in the market, or it didn't happen...
- jefftk 3y ago
  It's fine not to give them public credit for in-house only things, but in this subthread we're speculating about whether Llama 2 would be useful to them, which does depend heavily on the quality of their internal models.
- foobiekr 3y ago
  bringing back PLOSTFU culture might not actually be a bad thing.
- cma 3y ago
  OpenAI seemingly downgraded ChatGPT 4 due to the expense of running it for pro customers (unless you run it through the API).

whimsicalism 3y ago

I have no idea how you are so certain of that. Meta is definitely ahead of Google in terms of NLP expertise and has been for a while. I suspect that Google released their best model at the time with Bard.

dooraven 3y ago
We still don't have access to Imagen last I checked, it's still in restricted access. We don't have access to SoundStorm or MusicLM https://imagen.research.google/ https://imagen.research.google/ https://google-research.github.io/seanet/soundstorm/examples/ https://google-research.github.io/seanet/soundstorm/examples... https://google-research.github.io/seanet/musiclm/examples/ https://google-research.github.io/seanet/musiclm/examples/ Why would it be surprising that they have better models for resarch that they don't want to give out yet?
- whimsicalism 3y ago
  Because I work in NLP so I have a good sense of the different capabilities of different firms and for the Bard release, it would have made more sense for them to have a more limited release of a better model for PR reasons than what actually happened. The other things you are describing are just standard for research paper releases.
  - dooraven 3y ago
    > Bard release, it would have made more sense for them to have a more limited release of a better model for PR reasons than what actually happened. Yes I would agree with you if Google wasn't set on to full on panic mode by their investors about releasing something vs Open AI due to Chat GPT's buzz. Bard was just a "hey we can do this too" thing, it was released half assed, had next to no marketing or hype. Vertex AI is their real proper offering, and I want to see how PaLM 2 does in comparison.
    - whimsicalism 3y ago
      I can already tell you that PaLM is not anywhere near as good and PaLM-2 is at least not as good before RLHF. Not going to keep replying, believe what you want about Google's capabilities
      - dooraven 3y ago
        ok now I am confused, as Meta themselves say Palm-2 is better than Llama 2? > Llama 2 70B results are on par or better than PaLM (540B) (Chowdhery et al., 2022) on almost all benchmarks. There is still a large gap in performance between Llama 2 70B and GPT-4 and PaLM-2-L. https://scontent.fsyd7-1.fna.fbcdn.net/v/t39.2365-6/10000000_6495670187160042_4742060979571156424_n.pdf https://scontent.fsyd7-1.fna.fbcdn.net/v/t39.2365-6/10000000... If Google's publically available model is better Llama 2 already then why is it so inconceivable that they'd have private models that are better than their public ones which are better than LLama already. Palm-2 isn't better than GPT-4 but the convo was about better than Llama models no?
      - neonbjb 3y ago
        @dooraven - I also work in ML (including recently working at Google) and I agree with @whimsicalism. You seem to be under the mistaken belief that: 1. Google has competent high-level organization that effectively sets and pursues long term goals. 2. There is some advantage to developing a highly capable LLM but not releasing it. (2) could be the case if Google had built an extremely large model which was too expensive to deploy. Having been privy to what they had been working on up until mid-2022 and knowing how much work, compute and planning goes into extremely large models, this would very much surprise me. Note: I did not have much visibility into what deepmind was up to. Maybe they had something.
flangola7 3y ago
> I have no idea how you are so certain of that. Some among us work with it, or have friends or family who work with it. I imagine it is one of those.

WastingMyTime89 3y ago

Do they? Considering how much was at stack in term of PR when OpenAI released ChatGPT, I would be surprised that Google didn’t put out the best they could.

freedomben 3y ago
The other end of the PR stake was safety/alignment. If Google released a well functioning model, but it said some unsavory things or carried out requests that the public doesn't find agreeable, it could make Google look bad.

matt_holden 3y ago

Comments like this remind me of the old-timers from IBM saying "but wait, we invented the PC! and the cloud! and..." Gotta put products in the market, or it didn't happen...

jefftk 3y ago
It's fine not to give them public credit for in-house only things, but in this subthread we're speculating about whether Llama 2 would be useful to them, which does depend heavily on the quality of their internal models.
foobiekr 3y ago
bringing back PLOSTFU culture might not actually be a bad thing.
cma 3y ago
OpenAI seemingly downgraded ChatGPT 4 due to the expense of running it for pro customers (unless you run it through the API).