4 ms·

They invented the transformer architecture that powers GPT and other llms.

manquer 2y ago
Bell labs and Xerox PARC did great impactful work long after their parent companies were relevant and still do . The fact Google did the initial work on transformers but only OpenAI was able to productize is an indictment of their stagnation more than an achievement.
- gjvc 2y ago
  Bell Labs and Xerox PARC were great improvements on their successors.
- dageshi 2y ago
  Or they recognised the fundamental problem which is that LLM's will kill the motivation for people to upload new information to feed the LLM's with. If websites can't earn a living through ads because LLM's don't send them any traffic anymore, where do the LLM's ingest new information from? The web continues to deteriorate because of this and nobody has solved this problem.
  - pfannkuchen 2y ago
    I’m pretty sure they were just terrified of the PR impact of another “humans classified as gorillas” type incident.
    - mu53 2y ago
      And they ended up with "put elmers glue on pizza" type incident. AI is messy
  - lazide 2y ago
    It’s fundamental, IMO. LLMs have to have something ‘true’ (or at least statistically probable) to validate against for training. Taking the output from it and feeding it back into itself is, essentially, an Ouroboros. Or like locking a human in solitary. The more/longer it happens, the more deranged it’s going to get.
  - numpad0 2y ago
    I agree that they recognized the fundamental problem but I have different speculation as to what the problem is: the problem must be it's technologically too boring and obviously not profitable to them. It's culmination of decades of NLP researches, but it's also just East Asian predictive text percussively adapted for variable word length language. It must've been clear to whoever core people that it's fundamentally a sluggish demo or at most a cheap outright-sold commodity product. IMO they would be right unless someone finds a reason that LLMs can't ever be self hosted in the way Google Search can't be. They must have just saw by the way it is that LLM is at best a regularization engine for that magic box.
- kweingar 2y ago
  > only OpenAI was able to productize What do you mean? Anthropic and Google both have widely used products based on transformers.
  - mu53 2y ago
    OpenAI is the market leader by far with the most name recognition. Google was the last to market. Its initial release of Gemini was a total flop because of the meme "Use elmer's glue on pizza to keep the cheese on". It has finally become more consistent, and it manages to compete with other models though I never see anyone recommending Gemini first. All of these companies are in the red, but OpenAI has the most revenue.
    - 0xDEAFBEAD 2y ago
      This is a bit of a tangent, but I don't think OpenAI's brand is all that durable. You can see that Perplexity.AI has been gaining rapidly. At this point they have half as much search traffic as OpenAI: https://trends.google.com/trends/explore?date=today%205-y&geo=US&q=%2Fg%2F11txdn_sy7,%2Fg%2F11bxc656v6&hl=en-US https://trends.google.com/trends/explore?date=today%205-y&ge...
      - oefnak 2y ago
        You don't search for openai. You go to chat(gpt).com
        0xDEAFBEAD 2y ago
        Wouldn't that also apply to Perplexity.AI?
  - michaelt 2y ago
    As I recall, the timeline was: 2017, seven Google employees invent the transformer architecture and publish a paper. Google's investing heavily into ML, with their own custom 'TPU' chips and their 'Tensorflow' ML framework. 2019ish, Google has an internal chatbot they decide to do absolutely nothing with. Some idiot tells the press it's sentient, and they fire him. 2022, ChatGPT launches. It proves really powerful, a product loads of individuals and businesses are ready to pay for, and the value of the company skyrockets. 2023, none of the seven Transformer paper authors are at Google any more. Google rushes out Bard. Turns out they don't have a sentient super-intelligence after all. In fact it's badly received enough they end up needing to rebrand it a few months later. Classic tortoise-and-hare situation - Google spent 5 years napping, then had to sprint flat out just to take third place.
antupis 2y ago
Google has very research division but productising those inventions is the problem.
- cma 2y ago
  It was used in Google translate, and BERT was incorporated into search in 2019, though I don't think it was a clear win for search, I feel like I started having to add exact quotes to everything technical/programming around then.
  - jumping_frog 2y ago
    One thing I don't understand is google has so much metadata on search sessions to RLHF their search results. E.g. when I start a search session to solve a programming problem (before llms), I will continually search different terms to get to my solution webpage. Then stop. This session metadata and the path I took is highly significant data that can be used to help llms recognise what research itself looks like.
    - cma 2y ago
      Not RLHF, but my understanding was they heavily use that data and it was a big part of their moat, part of why competitors wanted to clone their results because they couldn't derive as good of quality from the web alone (Microsoft used the bing toolbar to clone them in the 2010s).
      - deleted 2y ago
        [deleted]

Bell labs and Xerox PARC did great impactful work long after their parent companies were relevant and still do . The fact Google did the initial work on transformers but only OpenAI was able to productize is an indictment of their stagnation more than an achievement.

gjvc 2y ago
Bell Labs and Xerox PARC were great improvements on their successors.
dageshi 2y ago
Or they recognised the fundamental problem which is that LLM's will kill the motivation for people to upload new information to feed the LLM's with. If websites can't earn a living through ads because LLM's don't send them any traffic anymore, where do the LLM's ingest new information from? The web continues to deteriorate because of this and nobody has solved this problem.
- pfannkuchen 2y ago
  I’m pretty sure they were just terrified of the PR impact of another “humans classified as gorillas” type incident.
  - mu53 2y ago
    And they ended up with "put elmers glue on pizza" type incident. AI is messy
- lazide 2y ago
  It’s fundamental, IMO. LLMs have to have something ‘true’ (or at least statistically probable) to validate against for training. Taking the output from it and feeding it back into itself is, essentially, an Ouroboros. Or like locking a human in solitary. The more/longer it happens, the more deranged it’s going to get.
- numpad0 2y ago
  I agree that they recognized the fundamental problem but I have different speculation as to what the problem is: the problem must be it's technologically too boring and obviously not profitable to them. It's culmination of decades of NLP researches, but it's also just East Asian predictive text percussively adapted for variable word length language. It must've been clear to whoever core people that it's fundamentally a sluggish demo or at most a cheap outright-sold commodity product. IMO they would be right unless someone finds a reason that LLMs can't ever be self hosted in the way Google Search can't be. They must have just saw by the way it is that LLM is at best a regularization engine for that magic box.
kweingar 2y ago
> only OpenAI was able to productize What do you mean? Anthropic and Google both have widely used products based on transformers.
- mu53 2y ago
  OpenAI is the market leader by far with the most name recognition. Google was the last to market. Its initial release of Gemini was a total flop because of the meme "Use elmer's glue on pizza to keep the cheese on". It has finally become more consistent, and it manages to compete with other models though I never see anyone recommending Gemini first. All of these companies are in the red, but OpenAI has the most revenue.
  - 0xDEAFBEAD 2y ago
    This is a bit of a tangent, but I don't think OpenAI's brand is all that durable. You can see that Perplexity.AI has been gaining rapidly. At this point they have half as much search traffic as OpenAI: https://trends.google.com/trends/explore?date=today%205-y&geo=US&q=%2Fg%2F11txdn_sy7,%2Fg%2F11bxc656v6&hl=en-US https://trends.google.com/trends/explore?date=today%205-y&ge...
    - oefnak 2y ago
      You don't search for openai. You go to chat(gpt).com
      - 0xDEAFBEAD 2y ago
        Wouldn't that also apply to Perplexity.AI?
- michaelt 2y ago
  As I recall, the timeline was: 2017, seven Google employees invent the transformer architecture and publish a paper. Google's investing heavily into ML, with their own custom 'TPU' chips and their 'Tensorflow' ML framework. 2019ish, Google has an internal chatbot they decide to do absolutely nothing with. Some idiot tells the press it's sentient, and they fire him. 2022, ChatGPT launches. It proves really powerful, a product loads of individuals and businesses are ready to pay for, and the value of the company skyrockets. 2023, none of the seven Transformer paper authors are at Google any more. Google rushes out Bard. Turns out they don't have a sentient super-intelligence after all. In fact it's badly received enough they end up needing to rebrand it a few months later. Classic tortoise-and-hare situation - Google spent 5 years napping, then had to sprint flat out just to take third place.

antupis 2y ago

Google has very research division but productising those inventions is the problem.

cma 2y ago
It was used in Google translate, and BERT was incorporated into search in 2019, though I don't think it was a clear win for search, I feel like I started having to add exact quotes to everything technical/programming around then.
- jumping_frog 2y ago
  One thing I don't understand is google has so much metadata on search sessions to RLHF their search results. E.g. when I start a search session to solve a programming problem (before llms), I will continually search different terms to get to my solution webpage. Then stop. This session metadata and the path I took is highly significant data that can be used to help llms recognise what research itself looks like.
  - cma 2y ago
    Not RLHF, but my understanding was they heavily use that data and it was a big part of their moat, part of why competitors wanted to clone their results because they couldn't derive as good of quality from the web alone (Microsoft used the bing toolbar to clone them in the 2010s).
    - deleted 2y ago
      [deleted]