5 ms·

In that sense, it's very similar to the GPT-4 Technical Report. The era of being "open" about LLMs or other "secret sauce" models in published papers may be ov

In that sense, it's very similar to the GPT-4 Technical Report. The era of being "open" about LLMs or other "secret sauce" models in published papers may be over, since these things have become existential threats to companies.

haldujai 3y ago
I wonder how special these architectures are compared to what's published. The "secret sauce" may just be getting 2 pages (~200) worth of engineers collaborating and either rolling out your own cloud service or spending $$$ at someone else's. Also not sure how much it matters other than academic interest of course. Realistically, there's only 4-5 (US) companies with the human resources and capital to roll something similar to these models out for what is most likely a complete write-off? They could claim whatever they wanted and it would be near impossible to validate.
- mr_toad 3y ago
  I think the secret sauce is just bucket loads of cash to spend on compute. And because of this I don’t buy that AI is an existential threat to Google at this point. If they were really worried they could spend a tiny portion of their ~280 billion dollars in revenue to train a bigger model.
  - haldujai 3y ago
    I assume this is just a PR/IR-driven project to stay the "Google is Dead" headlines hence the budget, especially considering an oversized chunk was spent on the scaling law, doesn't seem they were serious about building a GPT4-killer. I wasn't aware autoregressive LLMs were still considered an existential threat to Google. What's the threat supposed to be, ChatGPT is just going to keep eating Google search market share burning Microsoft capital on infra a la the Uber model or do they make money off of that at some point? Seems farfetched OpenAI can compete with Google's resources, vertical integration down to the TPU and access to significantly more training data.
    - sanxiyn 3y ago
      I agree that if training data is what matters, it is likely that no one can compete with Google with Google Books, which scanned 25 million volumes (source: http://www.nytimes.com/2015/10/29/arts/international/google-books-a-complex-and-controversial-experiment.html http://www.nytimes.com/2015/10/29/arts/international/google-...), which is approximately all the books. DeepMind's RETRO paper https://arxiv.org/abs/2112.04426 https://arxiv.org/abs/2112.04426 mentions a dataset called MassiveText, which includes 20 million books of 3T tokens. So we know Google is using Google Books, since there is simply no other source of 20 million books. Also as far as I know 3T tokens is more than publicly known to be used by anyone so far: Google could train on more data than anyone else, solely from Google Books, even without using its web crawl. Edit: it was 2005(!), so it is possible that many of you haven't heard of this. George Dyson, in Turing's Cathedral written in 2005 says: > My visit to Google? Despite the whimsical furniture and other toys, I felt I was entering a 14th-century cathedral: not in the 14th century but in the 12th century, while it was being built. Everyone was busy carving one stone here and another stone there, with some invisible architect getting everything to fit. The mood was playful, yet there was a palpable reverence in the air. "We are not scanning all those books to be read by people," explained one of my hosts after my talk. "We are scanning them to be read by an AI." https://www.edge.org/conversation/george_dyson-turings-cathedral https://www.edge.org/conversation/george_dyson-turings-cathe... Read the whole thing. It is not an accident Google got Google Books to train AI. That was the plan from the start.
      - fomine3 3y ago
        And YouTube. It's quite big datasets. Books is great for higher quality source.
  - cubefox 3y ago
    PaLM 2 is just an intermediate step. Their next big model is called "Gemini". Pichai mentioned it.
shrimpx 3y ago
Btw I've arrived at a different interpretation of the "Open" in OpenAI. It's open in the sense that the generic LLM is exposed via an API, allowing companies to build anything they want on top. Companies like Google have been working on language models (and AI more broadly) for years but have hid the generic intelligence of their models, exposing it only via improvements to their products. OpenAI bucked this trend and exposed an API to generic LLMs.
- cma 3y ago
  Sounds like the famous Facebook hoodie. "Open and connected" was one slogan on it. The API can be shut down at any time. https://venturebeat.com/social/facebook-insignia-hoodie/ https://venturebeat.com/social/facebook-insignia-hoodie/ In the end they just shat all over RSS etc.
  - shrimpx 3y ago
    This is an important point, although I think incentives don't align at all for OpenAI to close their APIs.
- magicalhippo 3y ago
  Guess they should rebrand as AvailableAI then...
  - EGreg 3y ago
    US healthcare should rebrand as “accessible”, even if it is not “affordable” by millions. With an interesting definition of accessible.
- ftxbro 3y ago
  > Btw I've arrived at a different interpretation of the "Open" in OpenAI. I don't understand why people have to keep trying to wrap their head around the word 'Open' in OpenAI. If you ever saw a commercial like a product has a 'great new taste' but then you tried it and it tasted bad, would you twist yourself into knots trying to understand how you went wrong in your interpretation of 'great'? No that's ridiculous. Same with 'Open' in 'OpenAI'. It's just some letters that form part of the name that they chose for themselves when they filled the form to incorporate their company.
flangola7 3y ago
> The era of being "open" about LLMs or other "secret sauce" models in published papers may be over, since these things have become existential threats FTFY

haldujai 3y ago

I wonder how special these architectures are compared to what's published. The "secret sauce" may just be getting 2 pages (~200) worth of engineers collaborating and either rolling out your own cloud service or spending $$$ at someone else's. Also not sure how much it matters other than academic interest of course. Realistically, there's only 4-5 (US) companies with the human resources and capital to roll something similar to these models out for what is most likely a complete write-off? They could claim whatever they wanted and it would be near impossible to validate.

mr_toad 3y ago
I think the secret sauce is just bucket loads of cash to spend on compute. And because of this I don’t buy that AI is an existential threat to Google at this point. If they were really worried they could spend a tiny portion of their ~280 billion dollars in revenue to train a bigger model.
- haldujai 3y ago
  I assume this is just a PR/IR-driven project to stay the "Google is Dead" headlines hence the budget, especially considering an oversized chunk was spent on the scaling law, doesn't seem they were serious about building a GPT4-killer. I wasn't aware autoregressive LLMs were still considered an existential threat to Google. What's the threat supposed to be, ChatGPT is just going to keep eating Google search market share burning Microsoft capital on infra a la the Uber model or do they make money off of that at some point? Seems farfetched OpenAI can compete with Google's resources, vertical integration down to the TPU and access to significantly more training data.
  - sanxiyn 3y ago
    I agree that if training data is what matters, it is likely that no one can compete with Google with Google Books, which scanned 25 million volumes (source: http://www.nytimes.com/2015/10/29/arts/international/google-books-a-complex-and-controversial-experiment.html http://www.nytimes.com/2015/10/29/arts/international/google-...), which is approximately all the books. DeepMind's RETRO paper https://arxiv.org/abs/2112.04426 https://arxiv.org/abs/2112.04426 mentions a dataset called MassiveText, which includes 20 million books of 3T tokens. So we know Google is using Google Books, since there is simply no other source of 20 million books. Also as far as I know 3T tokens is more than publicly known to be used by anyone so far: Google could train on more data than anyone else, solely from Google Books, even without using its web crawl. Edit: it was 2005(!), so it is possible that many of you haven't heard of this. George Dyson, in Turing's Cathedral written in 2005 says: > My visit to Google? Despite the whimsical furniture and other toys, I felt I was entering a 14th-century cathedral: not in the 14th century but in the 12th century, while it was being built. Everyone was busy carving one stone here and another stone there, with some invisible architect getting everything to fit. The mood was playful, yet there was a palpable reverence in the air. "We are not scanning all those books to be read by people," explained one of my hosts after my talk. "We are scanning them to be read by an AI." https://www.edge.org/conversation/george_dyson-turings-cathedral https://www.edge.org/conversation/george_dyson-turings-cathe... Read the whole thing. It is not an accident Google got Google Books to train AI. That was the plan from the start.
    - fomine3 3y ago
      And YouTube. It's quite big datasets. Books is great for higher quality source.
- cubefox 3y ago
  PaLM 2 is just an intermediate step. Their next big model is called "Gemini". Pichai mentioned it.

shrimpx 3y ago

Btw I've arrived at a different interpretation of the "Open" in OpenAI. It's open in the sense that the generic LLM is exposed via an API, allowing companies to build anything they want on top. Companies like Google have been working on language models (and AI more broadly) for years but have hid the generic intelligence of their models, exposing it only via improvements to their products. OpenAI bucked this trend and exposed an API to generic LLMs.

cma 3y ago
Sounds like the famous Facebook hoodie. "Open and connected" was one slogan on it. The API can be shut down at any time. https://venturebeat.com/social/facebook-insignia-hoodie/ https://venturebeat.com/social/facebook-insignia-hoodie/ In the end they just shat all over RSS etc.
- shrimpx 3y ago
  This is an important point, although I think incentives don't align at all for OpenAI to close their APIs.
magicalhippo 3y ago
Guess they should rebrand as AvailableAI then...
- EGreg 3y ago
  US healthcare should rebrand as “accessible”, even if it is not “affordable” by millions. With an interesting definition of accessible.
ftxbro 3y ago
> Btw I've arrived at a different interpretation of the "Open" in OpenAI. I don't understand why people have to keep trying to wrap their head around the word 'Open' in OpenAI. If you ever saw a commercial like a product has a 'great new taste' but then you tried it and it tasted bad, would you twist yourself into knots trying to understand how you went wrong in your interpretation of 'great'? No that's ridiculous. Same with 'Open' in 'OpenAI'. It's just some letters that form part of the name that they chose for themselves when they filled the form to incorporate their company.

flangola7 3y ago

> The era of being "open" about LLMs or other "secret sauce" models in published papers may be over, since these things have become existential threats FTFY