4 ms·

Most researchers aren't there to make a commercial product. They're there to publish well-cited research papers, talk at top conferences, and have access to far

Most researchers aren't there to make a commercial product. They're there to publish well-cited research papers, talk at top conferences, and have access to far more compute than anyone else can offer to have the biggest chance at state-of-the-art results. In terms of FLOPS, Google is rumoured to be by far the biggest, since they have lots of their own silicon, while most other people are limited by what NVidia can make or smallish amounts of custom silicon. If Google can't turn that into a commercial product, the researchers don't really care.

garblegarble 3y ago
>Most researchers aren't there to make a commercial product. They're there to publish well-cited research papers, talk at top conferences, and have access to far more compute than anyone else can offer. I think that's core to what I'm getting at - I don't see how Google continues to let them publish core research at this point given how much of a lead they've given their competitors (especially combined with their own inability to execute)
- leereeves 3y ago
  If Google didn't let them publish, the researchers would leave for sure.
- aix1 3y ago
  > I don't see how Google continues to let them publish core research at this point From what I hear from Google DeepMind researchers, the current policies are making it very difficult to publish NLP research. The barrier to disclosing research advances to the wider community appears much higher than it used to be.
visarga 3y ago
> and have access to far more compute than anyone else can offer to have the biggest chance at state of the art results What models do they have to show after 15 years of AI research? AlphaGo, AlphaFold, and that's about it. Search is mediocre, translation the same, not even their OCR APIs are very good. On pure research they invented the transformer in 2017 (Vaswani) and word embeddings in 2012 (Mikolov). But Microsoft invented residual connections (Kaiming He 2015) which underline transformers and modern CNNs. CNNs and LSTMs were invented longe before, by Yann LeCun (1989) and Jürgen Schmidhuber (1997). And yes, OpenAI also invented something of their own - the Adam optimiser (Kingma 2015), that trains 99% of the models today. Hinton invented Dropout (2014) and backpropagation (1986).
- londons_explore 3y ago
  They're good at research. They're bad at turning the research into a product. Specifically, most AI based products need to be trained on user data, and Google is too scared to make any real use of much user data in AI models for fear of getting sued into oblivion if, for example, gmail can be tricked into writing something in an email which reveals some other gmail users private data.
  - mrkramer 3y ago
    >They're good at research. They're bad at turning the research into a product. Tbh, Google Lens is one of the best computer vision products in the market. Generally speaking, it's hard to crate a product around something that is completely new.
- tempusalaria 3y ago
  OpenAI didn’t even exist when Adam was invented! The contact email on the paper is from 2017. OpenAI has made 1 major contribution - the auto regressive decoder. You could argue also their productisation of RL for LLMs has been highly influential even though they strictly didn’t invent it.
  - londons_explore 3y ago
    Plus all the things they haven't yet published... there must be something, because nobody has yet come close to the performance of GPT-4 8 months later, despite many people trying.
    - tempusalaria 3y ago
      The likely architecture of GPT-4 was invented at Google - MoE. OpenAI poached the team who developed it. OpenAI is ahead on the RL data side. To me that’s the likely biggest advantage they have in GPT-4.
      - danenania 3y ago

garblegarble 3y ago

>Most researchers aren't there to make a commercial product. They're there to publish well-cited research papers, talk at top conferences, and have access to far more compute than anyone else can offer. I think that's core to what I'm getting at - I don't see how Google continues to let them publish core research at this point given how much of a lead they've given their competitors (especially combined with their own inability to execute)

leereeves 3y ago
If Google didn't let them publish, the researchers would leave for sure.
aix1 3y ago
> I don't see how Google continues to let them publish core research at this point From what I hear from Google DeepMind researchers, the current policies are making it very difficult to publish NLP research. The barrier to disclosing research advances to the wider community appears much higher than it used to be.

visarga 3y ago

> and have access to far more compute than anyone else can offer to have the biggest chance at state of the art results What models do they have to show after 15 years of AI research? AlphaGo, AlphaFold, and that's about it. Search is mediocre, translation the same, not even their OCR APIs are very good. On pure research they invented the transformer in 2017 (Vaswani) and word embeddings in 2012 (Mikolov). But Microsoft invented residual connections (Kaiming He 2015) which underline transformers and modern CNNs. CNNs and LSTMs were invented longe before, by Yann LeCun (1989) and Jürgen Schmidhuber (1997). And yes, OpenAI also invented something of their own - the Adam optimiser (Kingma 2015), that trains 99% of the models today. Hinton invented Dropout (2014) and backpropagation (1986).

londons_explore 3y ago
They're good at research. They're bad at turning the research into a product. Specifically, most AI based products need to be trained on user data, and Google is too scared to make any real use of much user data in AI models for fear of getting sued into oblivion if, for example, gmail can be tricked into writing something in an email which reveals some other gmail users private data.
- mrkramer 3y ago
  >They're good at research. They're bad at turning the research into a product. Tbh, Google Lens is one of the best computer vision products in the market. Generally speaking, it's hard to crate a product around something that is completely new.
tempusalaria 3y ago
OpenAI didn’t even exist when Adam was invented! The contact email on the paper is from 2017. OpenAI has made 1 major contribution - the auto regressive decoder. You could argue also their productisation of RL for LLMs has been highly influential even though they strictly didn’t invent it.
- londons_explore 3y ago
  Plus all the things they haven't yet published... there must be something, because nobody has yet come close to the performance of GPT-4 8 months later, despite many people trying.
  - tempusalaria 3y ago
    The likely architecture of GPT-4 was invented at Google - MoE. OpenAI poached the team who developed it. OpenAI is ahead on the RL data side. To me that’s the likely biggest advantage they have in GPT-4.
    - danenania 3y ago