5 ms·

How is it that Google invented the TPU and Google Research came up with the paper on LLM and NVDA and AI startup companies have captured ~100% of the value

beachy 3y ago
For historical precedent see Xerox Parc.
- readyplayernull 3y ago
  IBM, Intel, Apple's Newton.
  - chillfox 3y ago
    Kodak
    - Forgeties79 3y ago
      Man I remember my last semester of college taking a history of photography course that was only offered every 3-4 years by a pretty legendary professor. The the day before the first day of class (or super close), Eastman Kodak declared bankruptcy after what? 110 years? He scrapped his day 1 lecture and threw together a talk - with photos of course - about Kodak and how an intrepid engineer developed then the company foolishly hid the first digital camera because it would compete with their film line. Incredible lecturer for sure haha
  - klodolph 3y ago
    The story I like to tell for the Newton is that it was launched before the technology was ready yet. Like the Sega Game Gear. Old video phones. All those tablets that launched before the iPad. They’re good ideas, but they shipped a few years too early, and the technology to make them work well at a good price point wasn’t available until later. Like, the Sega Game Gear had a cool active matrix LCD screen, but it took six AA batteries and the batteries only lasted like four hours.
    - technofiend 3y ago
      The Palm Pilot V had a dockable cell phone modem, but the connectivity wasn't integrated into the OS. It worked but only as a demonstration. Then Palm released a model with integrated data, but the BlackBerry came out the same year. You can be first and still if someone comes along with a much more compelling product, that's the end of you. Google has a few years left as a search company, but their enshittification of results has doomed them to replacement by LLMs. They seem to have forgotten Google pushed out their predecessors by having the best search results. Targeted advertisements don't qualify.
    - Rinzler89 3y ago
      >and the batteries only lasted like four hours Still more than the OG Steam Deck today :)
      - sofixa 3y ago
        Vastly depends on the game played and the settings. In a plane (so airplane mode, with Bluetooth headset) I played Hitman Absolution for 3 hours and still had 50%+ of the battery left. It was on minimal brightness because it was dark and didn't need more, but still.
        Rinzler89 3y ago
        Yeah, no need to take a (semi)joke literally and go all technical to debunk it. Though without optimizations, battery life on the deck was lucky to hit 2h at first before valve brought in updates and people learned they had to cap resolution and FPS to increase battery life.
wstrange 3y ago
It's far too early to suggest Google will not capture value from AI. They have plenty of opportunity to integrate AI into their products.
- earthnail 3y ago
  Microsoft is sooooo far ahead in this game, it’s borderline ridiculous. Google really missed an opportunity to grab major market share from MS Office.
  - willvarfar 3y ago
    Yes, this! Google Docs is basically basic. But imagine if, years ago, Google had added built-in LLM-based auto-complete and refactoring and summation tools to documents and presentations etc, years ago... Oh to dream.
tw04 3y ago
Because Google can’t focus on a product for more than 18 months if it isn’t generating several billion in PROFIT. They are punch drunk on advertising.
- Culonavirus 3y ago
  They're like a hyperactive dog chasing its own tail. How many projects did they create only to shut them a bit later? All because there's always some nonsense to chase. Meanwhile the AI train has left the station without them and their search is now an ad infested hot piece of garbage. Don't even get me started on their customer/dev support or how aging things like Google Translate api got absolutely KILLED by GPT-4 like apis overnight. Google has stage 4 leadership incompetency and can't be helped. The only humane option is euthanasia.
neilv 3y ago
There's an old joke explanation about Xerox and PARC, about the difficulty of "pitching a 'paperless office' to a photocopier company". In Google's case, an example analogy would be pitching making something like ChatGPT widely available, when that would disrupt revenue from search engine paid placements, and from ads on sites that people wouldn't need to visit. (So maybe someone says, better to phase it in subtly, as needed for competitiveness, but in non-disruptive ways.) I doubt it's as simple as that, but would be funny if that was it.
- vineyardmike 3y ago
  The answer is far weirder - they had a chat bot, and no one even discussed it in the context of search replacements. They didn’t want to release it because they just didn’t think it should be a product. Only after OpenAI actually disrupted search did they start releasing Gemini/Bard which takes advantage of search.
  - ec109685 3y ago
    They were afraid to release it because of unaligned output and hallucinations. ChatGPT showed that people could still get value out of something that wasn’t perfect. E.g. they had this in their labs: https://www.theguardian.com/technology/2022/jun/12/google-engineer-ai-bot-sentient-blake-lemoine https://www.theguardian.com/technology/2022/jun/12/google-en... from July, 2022z
    - halflings 3y ago
      Agree re:hallucinations/safety issues, that was likely one of the main blockers. And here's the sad part: they had this back in 2019... see this paper released in Jan 2020: https://blog.research.google/2020/01/towards-conversational-agent-that-can.html https://blog.research.google/2020/01/towards-conversational-...
    - HarHarVeryFunny 3y ago
      LaMBDA was also briefly available for public testing, but then rapidly withdrawn due to unhinged responses. One advantage that OpenAI had over Google was having developed RLHF as a way to "align" the model's output to be more acceptable. Part of Google's dropping the ball at that time period (but catching up now with Gemini) may also have been just not knowing what to do with it. It certainly wasn't apparent pre-ChatGPT that there'd be any huge public demand for something like this, or that people would find so many uses for it in API form, and especially so with LaMBDA's behavioral issues.
romanovcode 3y ago
Pretty sure it is because if ChatGPT likes would update as frequently as google website index it would render search engines like google obsolete and thus make their revenue nonexistent.
snats 3y ago
to this day i am impressed that they have not figured out how to embed advertisments to bard outputs so that they can go free.
vineyardmike 3y ago
I think the TPU is simple. They do sell it (via cloud), but they focus on themselves first. When there was no shortage of compute, it was an also-ran in the ML hardware market. Now it’s trendy. ChatGPT v Google is a far crazier history. Not only did Google invent Transformers, not only did Google open-source PaLM and Bert, but they even built chat tuned LLM chat bots and let employees talk with it. This isn’t a case where they were avoiding for disruption or protecting search - they genuinely didn’t see its potential. Worse, they got so much negative publicity over it that they considered it an AI safety issue to release. If that guy hadn’t gone to the press and claimed LaMDA was sentient than they may have entirely open sourced it like PaLM. This would likely mean that GPT-3 was open sourced and maybe never chat tuned either. GPT-2 was freely available and OpenAI showed off GPT-3 freely as a parlor trick before ChatGPT came out. ChatGPT was originally the same - fun text generation as chat not a full product. TLDR - Tensors probably didn’t have a lot of value until NVidia because scarce and they actively invented the original ChatGPT and “AI Safety” concerns caused them to lock it down.
willvarfar 3y ago
OpenAI lured everyone away from Google with way higher pay. https://www.linkedin.com/posts/eolver_googles-defense-against-openai-talent-grab-activity-7155229581942095872-Gqgs https://www.linkedin.com/posts/eolver_googles-defense-agains...

beachy 3y ago

For historical precedent see Xerox Parc.

readyplayernull 3y ago
IBM, Intel, Apple's Newton.
- chillfox 3y ago
  Kodak
  - Forgeties79 3y ago
    Man I remember my last semester of college taking a history of photography course that was only offered every 3-4 years by a pretty legendary professor. The the day before the first day of class (or super close), Eastman Kodak declared bankruptcy after what? 110 years? He scrapped his day 1 lecture and threw together a talk - with photos of course - about Kodak and how an intrepid engineer developed then the company foolishly hid the first digital camera because it would compete with their film line. Incredible lecturer for sure haha
- klodolph 3y ago
  The story I like to tell for the Newton is that it was launched before the technology was ready yet. Like the Sega Game Gear. Old video phones. All those tablets that launched before the iPad. They’re good ideas, but they shipped a few years too early, and the technology to make them work well at a good price point wasn’t available until later. Like, the Sega Game Gear had a cool active matrix LCD screen, but it took six AA batteries and the batteries only lasted like four hours.
  - technofiend 3y ago
    The Palm Pilot V had a dockable cell phone modem, but the connectivity wasn't integrated into the OS. It worked but only as a demonstration. Then Palm released a model with integrated data, but the BlackBerry came out the same year. You can be first and still if someone comes along with a much more compelling product, that's the end of you. Google has a few years left as a search company, but their enshittification of results has doomed them to replacement by LLMs. They seem to have forgotten Google pushed out their predecessors by having the best search results. Targeted advertisements don't qualify.
  - Rinzler89 3y ago
    >and the batteries only lasted like four hours Still more than the OG Steam Deck today :)
    - sofixa 3y ago
      Vastly depends on the game played and the settings. In a plane (so airplane mode, with Bluetooth headset) I played Hitman Absolution for 3 hours and still had 50%+ of the battery left. It was on minimal brightness because it was dark and didn't need more, but still.
      - Rinzler89 3y ago
        Yeah, no need to take a (semi)joke literally and go all technical to debunk it. Though without optimizations, battery life on the deck was lucky to hit 2h at first before valve brought in updates and people learned they had to cap resolution and FPS to increase battery life.

wstrange 3y ago

It's far too early to suggest Google will not capture value from AI. They have plenty of opportunity to integrate AI into their products.

earthnail 3y ago
Microsoft is sooooo far ahead in this game, it’s borderline ridiculous. Google really missed an opportunity to grab major market share from MS Office.
- willvarfar 3y ago
  Yes, this! Google Docs is basically basic. But imagine if, years ago, Google had added built-in LLM-based auto-complete and refactoring and summation tools to documents and presentations etc, years ago... Oh to dream.

tw04 3y ago

Because Google can’t focus on a product for more than 18 months if it isn’t generating several billion in PROFIT. They are punch drunk on advertising.

Culonavirus 3y ago
They're like a hyperactive dog chasing its own tail. How many projects did they create only to shut them a bit later? All because there's always some nonsense to chase. Meanwhile the AI train has left the station without them and their search is now an ad infested hot piece of garbage. Don't even get me started on their customer/dev support or how aging things like Google Translate api got absolutely KILLED by GPT-4 like apis overnight. Google has stage 4 leadership incompetency and can't be helped. The only humane option is euthanasia.

neilv 3y ago

There's an old joke explanation about Xerox and PARC, about the difficulty of "pitching a 'paperless office' to a photocopier company". In Google's case, an example analogy would be pitching making something like ChatGPT widely available, when that would disrupt revenue from search engine paid placements, and from ads on sites that people wouldn't need to visit. (So maybe someone says, better to phase it in subtly, as needed for competitiveness, but in non-disruptive ways.) I doubt it's as simple as that, but would be funny if that was it.

vineyardmike 3y ago
The answer is far weirder - they had a chat bot, and no one even discussed it in the context of search replacements. They didn’t want to release it because they just didn’t think it should be a product. Only after OpenAI actually disrupted search did they start releasing Gemini/Bard which takes advantage of search.
- ec109685 3y ago
  They were afraid to release it because of unaligned output and hallucinations. ChatGPT showed that people could still get value out of something that wasn’t perfect. E.g. they had this in their labs: https://www.theguardian.com/technology/2022/jun/12/google-engineer-ai-bot-sentient-blake-lemoine https://www.theguardian.com/technology/2022/jun/12/google-en... from July, 2022z
  - halflings 3y ago
    Agree re:hallucinations/safety issues, that was likely one of the main blockers. And here's the sad part: they had this back in 2019... see this paper released in Jan 2020: https://blog.research.google/2020/01/towards-conversational-agent-that-can.html https://blog.research.google/2020/01/towards-conversational-...
  - HarHarVeryFunny 3y ago
    LaMBDA was also briefly available for public testing, but then rapidly withdrawn due to unhinged responses. One advantage that OpenAI had over Google was having developed RLHF as a way to "align" the model's output to be more acceptable. Part of Google's dropping the ball at that time period (but catching up now with Gemini) may also have been just not knowing what to do with it. It certainly wasn't apparent pre-ChatGPT that there'd be any huge public demand for something like this, or that people would find so many uses for it in API form, and especially so with LaMBDA's behavioral issues.

romanovcode 3y ago

Pretty sure it is because if ChatGPT likes would update as frequently as google website index it would render search engines like google obsolete and thus make their revenue nonexistent.

snats 3y ago

to this day i am impressed that they have not figured out how to embed advertisments to bard outputs so that they can go free.

vineyardmike 3y ago

I think the TPU is simple. They do sell it (via cloud), but they focus on themselves first. When there was no shortage of compute, it was an also-ran in the ML hardware market. Now it’s trendy. ChatGPT v Google is a far crazier history. Not only did Google invent Transformers, not only did Google open-source PaLM and Bert, but they even built chat tuned LLM chat bots and let employees talk with it. This isn’t a case where they were avoiding for disruption or protecting search - they genuinely didn’t see its potential. Worse, they got so much negative publicity over it that they considered it an AI safety issue to release. If that guy hadn’t gone to the press and claimed LaMDA was sentient than they may have entirely open sourced it like PaLM. This would likely mean that GPT-3 was open sourced and maybe never chat tuned either. GPT-2 was freely available and OpenAI showed off GPT-3 freely as a parlor trick before ChatGPT came out. ChatGPT was originally the same - fun text generation as chat not a full product. TLDR - Tensors probably didn’t have a lot of value until NVidia because scarce and they actively invented the original ChatGPT and “AI Safety” concerns caused them to lock it down.

willvarfar 3y ago

OpenAI lured everyone away from Google with way higher pay. https://www.linkedin.com/posts/eolver_googles-defense-against-openai-talent-grab-activity-7155229581942095872-Gqgs https://www.linkedin.com/posts/eolver_googles-defense-agains...