6 ms·

Transformers: The Google scientists who pioneered an AI revolution

macrolime 3y ago
https://archive.is/KHsKh https://archive.is/KHsKh
Simon_O_Rourke 3y ago
Strangely (or not) there's no reference made to their AI ethics team as being in any way contributing towards this "AI revolution". Maybe because that team was sacked or moved around a few years back - https://www.wired.com/story/google-timnit-gebru-ai-what-really-happened/ https://www.wired.com/story/google-timnit-gebru-ai-what-real...
- mellosouls 3y ago
  There's no comparison in importance. The "revolution" as described is technological and cultural, both of which are underpinned by transformers and represented by gpt3+ (particularly ChatGPT). The ethical questions have become more culturally prominent as a result of that revolution. It's also misleading to describe them as having been sacked - the political machinations that some of the ethics team involved themselves in to the point of employer conflict were heavily discussed here at the time. Eg. https://news.ycombinator.com/item?id=25292386 https://news.ycombinator.com/item?id=25292386 but plenty more threads I think.
  - choppaface 3y ago
    > It's also misleading to describe them as having been sacked Officially they were fired without cause. It’s important to compare how Google handled the employment of their Ethics team with that of Anthony Levandowski, who was allowed to instigate an accident on the freeway on the job, left with a over-$100m bonus, and has thus far thwarted jail time and most of Google’s attempts to recoup what they paid him. Levandowski had lots of approval directly from Larry Page, and Page was aware of the risks. In contrast, Jeff Dean never respected the Ethics team in a similar capacity.
    - mellosouls 3y ago
      I'm sure there's all sorts of comparisons you could make; the point remains the ethics team - whatever their merits as individuals and importance generally - were relatively unimportant compared to the engineers and scientists in this particular revolution, and describing them as having been sacked was misleading.
    - Simon_O_Rourke 3y ago
      Whatever many many bad things Anthony Levandowski may have done, he picked up his own check in Google. The guy developed street view for example. Their AI ethics team appears, at least to a Google outsider, to engage almost exclusively in rent seeking behavior. I'm surprised many of them continue to pick up a paycheck.
      - choppaface 3y ago
        Street View started before Levandowski ever joined Google, and his role in Street View was as an investor not an IC. https://en.m.wikipedia.org/wiki/Google_Street_View https://en.m.wikipedia.org/wiki/Google_Street_View If there’s any rent-seeking behavior, it’s Levandowski’s church of AI https://www.wired.com/story/anthony-levandowski-artificial-intelligence-religion/ https://www.wired.com/story/anthony-levandowski-artificial-i...
        Simon_O_Rourke 3y ago
        Perhaps the church of Al can throw a few bucks the way of the recently not-fired-but-no-longer-employed AI ethics crew, similar levels of grift going on in each.
  - Simon_O_Rourke 3y ago
    > the political machinations that some of the ethics team involved themselves in to the point of employer conflict were heavily discussed here at the time. It certainly seems like some of that ethics team appear, at best, as extremely bad faith actors in that drama.
    - choppaface 3y ago
      Well they were hired to study the bias and energy waste of large language models and that’s exactly what they found: https://www.technologyreview.com/2020/12/04/1013294/google-ai-ethics-research-paper-forced-out-timnit-gebru/ https://www.technologyreview.com/2020/12/04/1013294/google-a... The article notes how other Googlers have faced factual review in paper releases and what Jeff Dean did here was clearly something else. Google leadership moved against the Ethics team at a time of high uncertainty: COVID rates were spiking, the stock market had become extremely distorted, there had been a summer of massive protests, Bezos was leaving Amazon.. this was Google leadership being greedy. Since then “ethics of LLMs at Google” has pivoted to overt lobbying and attempts at controlling regulation: https://www.reuters.com/technology/openai-google-others-pledge-watermark-ai-content-safety-white-house-2023-07-21/ https://www.reuters.com/technology/openai-google-others-pled...
onetokeoverthe 3y ago
[dead]
jacknews 3y ago
"two Google research scientists, Ashish Vaswani and Jakob Uszkoreit, were in a hallway of the search giant’s Mountain View campus" "During their chat in the hallway, Uszkoreit and Vaswani were overheard by Noam Shazeer, a Google veteran..." etc Is this just thinly-veiled propaganda for the back-to-the-office movement?
7e 3y ago
Automated neural architecture search would have eventually found the transformer, and better architectures. Reading the paper, the names of the vectors (QKV) seem dressed up to imply that the authors know how exactly a transformer actually works, but I am not convinced. It just seems to me these folks where the first to stumble upon it. They didn’t invent self-attention, either. Right place, right time, tons of free TPU to experiment with.

https://archive.is/KHsKh https://archive.is/KHsKh

Strangely (or not) there's no reference made to their AI ethics team as being in any way contributing towards this "AI revolution". Maybe because that team was sacked or moved around a few years back - https://www.wired.com/story/google-timnit-gebru-ai-what-really-happened/ https://www.wired.com/story/google-timnit-gebru-ai-what-real...

mellosouls 3y ago
There's no comparison in importance. The "revolution" as described is technological and cultural, both of which are underpinned by transformers and represented by gpt3+ (particularly ChatGPT). The ethical questions have become more culturally prominent as a result of that revolution. It's also misleading to describe them as having been sacked - the political machinations that some of the ethics team involved themselves in to the point of employer conflict were heavily discussed here at the time. Eg. https://news.ycombinator.com/item?id=25292386 https://news.ycombinator.com/item?id=25292386 but plenty more threads I think.
- choppaface 3y ago
  > It's also misleading to describe them as having been sacked Officially they were fired without cause. It’s important to compare how Google handled the employment of their Ethics team with that of Anthony Levandowski, who was allowed to instigate an accident on the freeway on the job, left with a over-$100m bonus, and has thus far thwarted jail time and most of Google’s attempts to recoup what they paid him. Levandowski had lots of approval directly from Larry Page, and Page was aware of the risks. In contrast, Jeff Dean never respected the Ethics team in a similar capacity.
  - mellosouls 3y ago
    I'm sure there's all sorts of comparisons you could make; the point remains the ethics team - whatever their merits as individuals and importance generally - were relatively unimportant compared to the engineers and scientists in this particular revolution, and describing them as having been sacked was misleading.
  - Simon_O_Rourke 3y ago
    Whatever many many bad things Anthony Levandowski may have done, he picked up his own check in Google. The guy developed street view for example. Their AI ethics team appears, at least to a Google outsider, to engage almost exclusively in rent seeking behavior. I'm surprised many of them continue to pick up a paycheck.
    - choppaface 3y ago
      Street View started before Levandowski ever joined Google, and his role in Street View was as an investor not an IC. https://en.m.wikipedia.org/wiki/Google_Street_View https://en.m.wikipedia.org/wiki/Google_Street_View If there’s any rent-seeking behavior, it’s Levandowski’s church of AI https://www.wired.com/story/anthony-levandowski-artificial-intelligence-religion/ https://www.wired.com/story/anthony-levandowski-artificial-i...
      - Simon_O_Rourke 3y ago
        Perhaps the church of Al can throw a few bucks the way of the recently not-fired-but-no-longer-employed AI ethics crew, similar levels of grift going on in each.
- Simon_O_Rourke 3y ago
  > the political machinations that some of the ethics team involved themselves in to the point of employer conflict were heavily discussed here at the time. It certainly seems like some of that ethics team appear, at best, as extremely bad faith actors in that drama.
  - choppaface 3y ago
    Well they were hired to study the bias and energy waste of large language models and that’s exactly what they found: https://www.technologyreview.com/2020/12/04/1013294/google-ai-ethics-research-paper-forced-out-timnit-gebru/ https://www.technologyreview.com/2020/12/04/1013294/google-a... The article notes how other Googlers have faced factual review in paper releases and what Jeff Dean did here was clearly something else. Google leadership moved against the Ethics team at a time of high uncertainty: COVID rates were spiking, the stock market had become extremely distorted, there had been a summer of massive protests, Bezos was leaving Amazon.. this was Google leadership being greedy. Since then “ethics of LLMs at Google” has pivoted to overt lobbying and attempts at controlling regulation: https://www.reuters.com/technology/openai-google-others-pledge-watermark-ai-content-safety-white-house-2023-07-21/ https://www.reuters.com/technology/openai-google-others-pled...

onetokeoverthe 3y ago

[dead]

jacknews 3y ago

"two Google research scientists, Ashish Vaswani and Jakob Uszkoreit, were in a hallway of the search giant’s Mountain View campus" "During their chat in the hallway, Uszkoreit and Vaswani were overheard by Noam Shazeer, a Google veteran..." etc Is this just thinly-veiled propaganda for the back-to-the-office movement?

7e 3y ago

Automated neural architecture search would have eventually found the transformer, and better architectures. Reading the paper, the names of the vectors (QKV) seem dressed up to imply that the authors know how exactly a transformer actually works, but I am not convinced. It just seems to me these folks where the first to stumble upon it. They didn’t invent self-attention, either. Right place, right time, tons of free TPU to experiment with.