9 ms·

I can't understand the mindset of any AI researchers who stay at Google at this point. You're there (presumably) because you want to make an impact. You gave t

I can't understand the mindset of any AI researchers who stay at Google at this point. You're there (presumably) because you want to make an impact. You gave them a massive lead with a novel approach... and Google failed to capitalise on that so spectacularly that they're now investing billions into their competitor who's miles ahead of Google, using Google's approach, all to try and head off another competitor who's miles ahead of everybody also using Google's approach I hope that (counter to stories) they pay the researchers very well, because the best result I can see is Google shutting down what they can publish to try and stop this happening again.

ikekkdcjkfke 3y ago
It is also a testament to what happens when you open up powerful technology instead of hoarding
londons_explore 3y ago
Most researchers aren't there to make a commercial product. They're there to publish well-cited research papers, talk at top conferences, and have access to far more compute than anyone else can offer to have the biggest chance at state-of-the-art results. In terms of FLOPS, Google is rumoured to be by far the biggest, since they have lots of their own silicon, while most other people are limited by what NVidia can make or smallish amounts of custom silicon. If Google can't turn that into a commercial product, the researchers don't really care.
- garblegarble 3y ago
  >Most researchers aren't there to make a commercial product. They're there to publish well-cited research papers, talk at top conferences, and have access to far more compute than anyone else can offer. I think that's core to what I'm getting at - I don't see how Google continues to let them publish core research at this point given how much of a lead they've given their competitors (especially combined with their own inability to execute)
  - leereeves 3y ago
    If Google didn't let them publish, the researchers would leave for sure.
  - aix1 3y ago
    > I don't see how Google continues to let them publish core research at this point From what I hear from Google DeepMind researchers, the current policies are making it very difficult to publish NLP research. The barrier to disclosing research advances to the wider community appears much higher than it used to be.
- visarga 3y ago
  > and have access to far more compute than anyone else can offer to have the biggest chance at state of the art results What models do they have to show after 15 years of AI research? AlphaGo, AlphaFold, and that's about it. Search is mediocre, translation the same, not even their OCR APIs are very good. On pure research they invented the transformer in 2017 (Vaswani) and word embeddings in 2012 (Mikolov). But Microsoft invented residual connections (Kaiming He 2015) which underline transformers and modern CNNs. CNNs and LSTMs were invented longe before, by Yann LeCun (1989) and Jürgen Schmidhuber (1997). And yes, OpenAI also invented something of their own - the Adam optimiser (Kingma 2015), that trains 99% of the models today. Hinton invented Dropout (2014) and backpropagation (1986).
  - londons_explore 3y ago
    They're good at research. They're bad at turning the research into a product. Specifically, most AI based products need to be trained on user data, and Google is too scared to make any real use of much user data in AI models for fear of getting sued into oblivion if, for example, gmail can be tricked into writing something in an email which reveals some other gmail users private data.
    - mrkramer 3y ago
      >They're good at research. They're bad at turning the research into a product. Tbh, Google Lens is one of the best computer vision products in the market. Generally speaking, it's hard to crate a product around something that is completely new.
  - tempusalaria 3y ago
    OpenAI didn’t even exist when Adam was invented! The contact email on the paper is from 2017. OpenAI has made 1 major contribution - the auto regressive decoder. You could argue also their productisation of RL for LLMs has been highly influential even though they strictly didn’t invent it.
    - londons_explore 3y ago
visarga 3y ago
> I can't understand the mindset I can. None of this tech will maintain Google as the chokepoint of internet and make billions for them. You can download and run a LLaMA or Mistral but can't download a Google. On-topic information has been commoditised. Even OpenAI is in a bad spot. They owned the whole LLM mountain in 2020, now they only own the peaks. Almost the whole mountain has been conquered by open models. Their area of supremacy is shrinking by the day. The two factors at play here are 1. ability to run LLMs locally and 2. ability to improve local LLMs with data exfiltrated from SOTA LLMs. I foresee an era of increased privacy as a consequence of this. We will be filtering our browsers with LLMs to remove junk and override Google, Twitter and FB. Bad news for ad providers, they got to move into some other profitable line of business now. Nobody got time to read garbage and ads.
- lobochrome 3y ago
  Powerful vision! Thank you.
- dustingetz 3y ago
  is there a strongly written essay or paper arguing this viewpoint?
  - visarga 3y ago
    the original: "We Have No Moat, And Neither Does OpenAI" https://www.semianalysis.com/p/google-we-have-no-moat-and-neither https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
- esperent 3y ago
  > We will be filtering our browsers with LLMs to remove junk and override Google, Twitter and FB I cannot wait for the day where I can create a list of blocked terms "Musk, Trump, American politics, ..." and have an intelligent LLM filter so that it doesn't matter what site I visit, whether The Verge, or The Guardian, or Reddit, all articles or posts related to these terms will be gone. I think this has a chance to have a far higher effect than just ad tech. It's going to effect all media publishers. But I also expect them, led by Google and Facebook, to fight this tooth and nail with every dirty trick they can.
deleted 3y ago
[deleted]
lobochrome 3y ago
Google AI : Xerox PARC AI : GUI
schleck8 3y ago
Yeah, Bard is still the worst major LLM product
raverbashing 3y ago
Maybe Google doesn't see how it can turn AI into a low traction product that will only be used for renaming and turning some cosmetic improvements to justify someone's promotion Google truly has grown complacent and whoever has a good idea feel like they can't develop it there. But if you have an idea on how Google can lose money and reputation by retiring another product they're all ears
Frost1x 3y ago
>I can't understand the mindset of any AI researchers who stay at Google at this point. >You're there (presumably) because you want to make an impact. I can't speak to AI researchers specifically but I can speak to research more generally. Ideally you want to focus on somewhere you can make and impact, pursue the general line of research you find interesting/promising, AND have stability to at least live (or ideally enough pay to not care). I've known enough highly talented researchers who almost always have to sacrifice the "making a difference" and "pursuing their pathway" to large degree just to survive. Ultimately what people are paying for to be researched is what matters, be it some business R&D division's head, federal agency biased directions, of whatever philanthropic connections some organizations are able to extract decide you should be doing. It's not just medicore researchers, it's what one might consider "world class" top of their field researchers. Everyone at some point makes sacrifices to pay the bills but there's this neverending gaslighting that occurs about what researchers want to do and what areas they choose to impact or some nonsense like that. That happens in very few idealized cases. E.g., tenured professorship at highly endowed institutions where they've already invested enough time doing the stuff they don't want to game the system to get to a point they can finally pursue their own paths and so on. If you want to focus on an impact and perform research you need to be able to self-fund that and hope the area you work isn't capital intensive (AI specifically DNN, LLM, and highly data/compute intensive approaches is for the most part pretty capital intensive, hence one reason big tech pursued them to reduce competition--you can pursue paths theoretically but ultimately unless you find approaches that aren't as capital intensive it'll be awhile before you can experimentally test and iterate approaches). Researchers just hope to get remotely close to the area they want to work in but by in large have to chase the money and lines of research behind it if they hope to remain a researcher. Maybe Google is different but I doubt it. This happens in other industries as well with people in R&D near the tops of their field working at market leaders still caving. If by some miracle you happen to be near the top of your field and happen to guess or have some natural deep insight to the path forward where you're fortunate enough to make a huge breakthrough, you can often just spin it out yourself or take on some investment risk. Why work for Google if you have the practical path to say AGI in your hands? Try and do it on your own or find someone who doesn't own all your IP at the end of it. You work at big corp because there's ideally some stability balance with some semi-interesting research paths to you, probably not exactly what you want but as close as you can get.
toasted-subs 3y ago
Classic moment of business people not understanding how some of these technical things work.
deleted 3y ago
[deleted]
jeswin 3y ago
> You gave them a massive lead with a novel approach... From everything I've read/watched, the original researchers didn't know at that point their work was going to be so impactful. In hindsight, yeah. But otherwise no.
shawabawa3 3y ago
> I can't understand the mindset of any AI researchers who stay at Google at this point. Have you heard of money? But seriously, Google have talent because they pay insane salaries like all the other big tech companies
- hcks 3y ago
  Google doesn’t pay very well, actually
  - jorlow 3y ago
    They definitely pay a large number of people a large amount, but they're happy to pay less when they can.
    - riku_iki 3y ago
      they pay average market salaries to large number of mediocre people.
bartwr 3y ago
I have a simple answer for you - researchers measure impact differently. If they measured it primarily by shipping products, they would work as engineers, not research scientists. As a researcher you realize impact by researching and publishing (internally or externally) and are ok with the idea that it's up to others to pick it up and turn into cool products. Hopefully they work at the same company, but if they don't - it's not your problem to solve, but VPs and above. But yes, it can be frustrating if you're junior and take your work personally and attribute its success to its (and even worse, your own) value. After a few years of career, you learn to look at it from a distance. Google publishes amazing stuff, people are paid great salaries, they have a ton of fun doing it, work life balance is way better than at startups - what's not to like?
- user8872642489 3y ago
  [dead]
  - bartwr 3y ago
    This quote says nothing about product, but about continuing research. Most researchers don't care about products. Many don't even care about any practicality (just the scientific pursuit). How do I know? Well, I happen to work as Research Scientist. I worked at Google 5y. :) For the majority of my colleagues, "having to" support their work after it was published was a nuisance and they wanted to work on new cool stuff instead.
    - hnfong 3y ago
      I am, unironically, not sure whether your description is reflective of being a Research Scientist, or just being a Google engineer in general...
- lern_too_spel 3y ago
  > people are paid great salaries, they have a ton of fun doing it, work life balance is way better than at startups Compared to AI startups, neither of these is true.

ikekkdcjkfke 3y ago

It is also a testament to what happens when you open up powerful technology instead of hoarding

londons_explore 3y ago

Most researchers aren't there to make a commercial product. They're there to publish well-cited research papers, talk at top conferences, and have access to far more compute than anyone else can offer to have the biggest chance at state-of-the-art results. In terms of FLOPS, Google is rumoured to be by far the biggest, since they have lots of their own silicon, while most other people are limited by what NVidia can make or smallish amounts of custom silicon. If Google can't turn that into a commercial product, the researchers don't really care.

garblegarble 3y ago
>Most researchers aren't there to make a commercial product. They're there to publish well-cited research papers, talk at top conferences, and have access to far more compute than anyone else can offer. I think that's core to what I'm getting at - I don't see how Google continues to let them publish core research at this point given how much of a lead they've given their competitors (especially combined with their own inability to execute)
- leereeves 3y ago
  If Google didn't let them publish, the researchers would leave for sure.
- aix1 3y ago
  > I don't see how Google continues to let them publish core research at this point From what I hear from Google DeepMind researchers, the current policies are making it very difficult to publish NLP research. The barrier to disclosing research advances to the wider community appears much higher than it used to be.
visarga 3y ago
> and have access to far more compute than anyone else can offer to have the biggest chance at state of the art results What models do they have to show after 15 years of AI research? AlphaGo, AlphaFold, and that's about it. Search is mediocre, translation the same, not even their OCR APIs are very good. On pure research they invented the transformer in 2017 (Vaswani) and word embeddings in 2012 (Mikolov). But Microsoft invented residual connections (Kaiming He 2015) which underline transformers and modern CNNs. CNNs and LSTMs were invented longe before, by Yann LeCun (1989) and Jürgen Schmidhuber (1997). And yes, OpenAI also invented something of their own - the Adam optimiser (Kingma 2015), that trains 99% of the models today. Hinton invented Dropout (2014) and backpropagation (1986).
- londons_explore 3y ago
  They're good at research. They're bad at turning the research into a product. Specifically, most AI based products need to be trained on user data, and Google is too scared to make any real use of much user data in AI models for fear of getting sued into oblivion if, for example, gmail can be tricked into writing something in an email which reveals some other gmail users private data.
  - mrkramer 3y ago
    >They're good at research. They're bad at turning the research into a product. Tbh, Google Lens is one of the best computer vision products in the market. Generally speaking, it's hard to crate a product around something that is completely new.
- tempusalaria 3y ago
  OpenAI didn’t even exist when Adam was invented! The contact email on the paper is from 2017. OpenAI has made 1 major contribution - the auto regressive decoder. You could argue also their productisation of RL for LLMs has been highly influential even though they strictly didn’t invent it.
  - londons_explore 3y ago

visarga 3y ago

> I can't understand the mindset I can. None of this tech will maintain Google as the chokepoint of internet and make billions for them. You can download and run a LLaMA or Mistral but can't download a Google. On-topic information has been commoditised. Even OpenAI is in a bad spot. They owned the whole LLM mountain in 2020, now they only own the peaks. Almost the whole mountain has been conquered by open models. Their area of supremacy is shrinking by the day. The two factors at play here are 1. ability to run LLMs locally and 2. ability to improve local LLMs with data exfiltrated from SOTA LLMs. I foresee an era of increased privacy as a consequence of this. We will be filtering our browsers with LLMs to remove junk and override Google, Twitter and FB. Bad news for ad providers, they got to move into some other profitable line of business now. Nobody got time to read garbage and ads.

lobochrome 3y ago
Powerful vision! Thank you.
dustingetz 3y ago
is there a strongly written essay or paper arguing this viewpoint?
- visarga 3y ago
  the original: "We Have No Moat, And Neither Does OpenAI" https://www.semianalysis.com/p/google-we-have-no-moat-and-neither https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
esperent 3y ago
> We will be filtering our browsers with LLMs to remove junk and override Google, Twitter and FB I cannot wait for the day where I can create a list of blocked terms "Musk, Trump, American politics, ..." and have an intelligent LLM filter so that it doesn't matter what site I visit, whether The Verge, or The Guardian, or Reddit, all articles or posts related to these terms will be gone. I think this has a chance to have a far higher effect than just ad tech. It's going to effect all media publishers. But I also expect them, led by Google and Facebook, to fight this tooth and nail with every dirty trick they can.

deleted 3y ago

[deleted]

lobochrome 3y ago

Google AI : Xerox PARC AI : GUI

schleck8 3y ago

Yeah, Bard is still the worst major LLM product

raverbashing 3y ago

Maybe Google doesn't see how it can turn AI into a low traction product that will only be used for renaming and turning some cosmetic improvements to justify someone's promotion Google truly has grown complacent and whoever has a good idea feel like they can't develop it there. But if you have an idea on how Google can lose money and reputation by retiring another product they're all ears

Frost1x 3y ago

>I can't understand the mindset of any AI researchers who stay at Google at this point. >You're there (presumably) because you want to make an impact. I can't speak to AI researchers specifically but I can speak to research more generally. Ideally you want to focus on somewhere you can make and impact, pursue the general line of research you find interesting/promising, AND have stability to at least live (or ideally enough pay to not care). I've known enough highly talented researchers who almost always have to sacrifice the "making a difference" and "pursuing their pathway" to large degree just to survive. Ultimately what people are paying for to be researched is what matters, be it some business R&D division's head, federal agency biased directions, of whatever philanthropic connections some organizations are able to extract decide you should be doing. It's not just medicore researchers, it's what one might consider "world class" top of their field researchers. Everyone at some point makes sacrifices to pay the bills but there's this neverending gaslighting that occurs about what researchers want to do and what areas they choose to impact or some nonsense like that. That happens in very few idealized cases. E.g., tenured professorship at highly endowed institutions where they've already invested enough time doing the stuff they don't want to game the system to get to a point they can finally pursue their own paths and so on. If you want to focus on an impact and perform research you need to be able to self-fund that and hope the area you work isn't capital intensive (AI specifically DNN, LLM, and highly data/compute intensive approaches is for the most part pretty capital intensive, hence one reason big tech pursued them to reduce competition--you can pursue paths theoretically but ultimately unless you find approaches that aren't as capital intensive it'll be awhile before you can experimentally test and iterate approaches). Researchers just hope to get remotely close to the area they want to work in but by in large have to chase the money and lines of research behind it if they hope to remain a researcher. Maybe Google is different but I doubt it. This happens in other industries as well with people in R&D near the tops of their field working at market leaders still caving. If by some miracle you happen to be near the top of your field and happen to guess or have some natural deep insight to the path forward where you're fortunate enough to make a huge breakthrough, you can often just spin it out yourself or take on some investment risk. Why work for Google if you have the practical path to say AGI in your hands? Try and do it on your own or find someone who doesn't own all your IP at the end of it. You work at big corp because there's ideally some stability balance with some semi-interesting research paths to you, probably not exactly what you want but as close as you can get.

toasted-subs 3y ago

Classic moment of business people not understanding how some of these technical things work.

deleted 3y ago

[deleted]

jeswin 3y ago

> You gave them a massive lead with a novel approach... From everything I've read/watched, the original researchers didn't know at that point their work was going to be so impactful. In hindsight, yeah. But otherwise no.

shawabawa3 3y ago

> I can't understand the mindset of any AI researchers who stay at Google at this point. Have you heard of money? But seriously, Google have talent because they pay insane salaries like all the other big tech companies

hcks 3y ago
Google doesn’t pay very well, actually
- jorlow 3y ago
  They definitely pay a large number of people a large amount, but they're happy to pay less when they can.
  - riku_iki 3y ago
    they pay average market salaries to large number of mediocre people.

bartwr 3y ago

I have a simple answer for you - researchers measure impact differently. If they measured it primarily by shipping products, they would work as engineers, not research scientists. As a researcher you realize impact by researching and publishing (internally or externally) and are ok with the idea that it's up to others to pick it up and turn into cool products. Hopefully they work at the same company, but if they don't - it's not your problem to solve, but VPs and above. But yes, it can be frustrating if you're junior and take your work personally and attribute its success to its (and even worse, your own) value. After a few years of career, you learn to look at it from a distance. Google publishes amazing stuff, people are paid great salaries, they have a ton of fun doing it, work life balance is way better than at startups - what's not to like?

user8872642489 3y ago
[dead]
- bartwr 3y ago
  This quote says nothing about product, but about continuing research. Most researchers don't care about products. Many don't even care about any practicality (just the scientific pursuit). How do I know? Well, I happen to work as Research Scientist. I worked at Google 5y. :) For the majority of my colleagues, "having to" support their work after it was published was a nuisance and they wanted to work on new cool stuff instead.
  - hnfong 3y ago
    I am, unironically, not sure whether your description is reflective of being a Research Scientist, or just being a Google engineer in general...
lern_too_spel 3y ago
> people are paid great salaries, they have a ton of fun doing it, work life balance is way better than at startups Compared to AI startups, neither of these is true.