10 ms·

(Site was down.) Google already has many similar models. Most similar is maybe their LaMDA model. https://blog.google/technology/ai/lamda/ https://blog.google/

(Site was down.) Google already has many similar models. Most similar is maybe their LaMDA model. https://blog.google/technology/ai/lamda/ https://blog.google/technology/ai/lamda/ It's strange that the article does not mention that. I never really understood the argument that OpenAI has some technology that Google does not have. That's just not true. The opposite is much more true. And Google's LaMDA is even older than ChatGPT. Also, that technology is also not really so fancy. What matters is that you have enough computing power. When you have that, most people can replicate sth like ChatGPT. And in terms of computing power, it's hard to beat Google and their TPU clusters. I don't really have numbers, but I think Google also wins here.

anaganisk 4y ago
When you say its hard to beat Google and their TPU clusters, do you mean Azure has inferior capability? Isnt OpenAI limited by what they can do on Azure?
- albertzeyer 4y ago
  I don't have numbers, so I don't really know, but yes, I would assume Google (incl DeepMind) has more computing power than Microsoft + OpenAI.
  - dubcanada 4y ago
    Why? Both are huge super computer users, and both have similar setups. Both own a search engine, both run ads. I don't really see any reason to suggest Microsoft is less capable than Google. Unless I am missing something?
    - adammarples 4y ago
      Google has been at the forefront AI for years, decades maybe. Do azure even have tpus? It's reasonable to assume Google have close to the best ML compute resources out of probably everyone in the world.
      - visarga 4y ago
        Where do you think GPT-3 was trained and currently runs? In Azure. It doesn't need TPUs.
        adammarples 4y ago
        Gpt3 is an impressive model, I think it took 512 GPUs 10 days to train though. This alone doesn't mean that Microsoft has more computing power than Google
        hot_gril 4y ago
        It does mean that you don't need the most computing power to train a successful model.
    - albertzeyer 4y ago
      My assumption is based on this: For a long time, Microsoft's core business was Windows and (offline) Office. You don't need a super cluster for that. Google on the other side always had search as a service, available to the public, which needs more compute. So historically, Google needed more compute than Microsoft. Then, from experience, as I do research on neural networks, read papers in the field, Google/DeepMind were always more active in this field than Microsoft, so again, historically, they needed more computing power because of this. Microsoft has catched up, but still, I think Google/DeepMind is more active in the field. Then, Google has TPUs. I think this gives them computing power much more cost effectively than when you need to buy Nvidia GPUs. This is maybe the biggest argument. But in any case, as you say, both companies are big enough to easily train sth like ChatGPT, or also sth like GPT4, however big it might be. Their computing power is almost infinite (for today's standards), so it doesn't really matter who has more.
    - rrdharan 4y ago
      Historically Google far outspent Microsoft on their “super computers”, though in recent years MS has been closing the gap. https://www.platformonomics.com/2022/02/follow-the-capex-cloud-table-stakes-2021-retrospective/ https://www.platformonomics.com/2022/02/follow-the-capex-clo...
    - hot_gril 4y ago
      Google supposedly hires more of the ML experts and custom hardware experts. Whenever someone says that GCP is inferior to the competition, it's usually followed by "except in ML."
  - summerlight 4y ago
    I'm also interested in Azure's FPGA instance capacity which is a comparable counterpart to Google's TPU AFAIK. From what I know, Google has a large number of TPU units (not sure about the exact numbers, but I would assume this would exceed 5 digits?) and also has the luxury of sharing TPU instances with products that operate one of the largest models on the planet, namely search and ads. So Google has some advantages on flexibility of resource allocation if it really wants to prioritize AI stuffs.
Semaphor 4y ago
> (Site is down.) Up for me, but takes several seconds to load.
brookst 4y ago
Google has been slow because they can’t figure out how to release this without destroying their business model. It’s very hard to be commercially successful with an innovation when it attacks the cash cow side of your business. It can be done, but it creates a lot of internal conflict and incentives to wait, to gimp the innovation, or to try to bolt it on to the cash cow to mitigate the damage. I have no doubt Google could be a leader in the LLM space. . . But are they willing to destroy their own moat and take a serious revenue hit to do so?
- _visgean 4y ago
  > Google has been slow because they can’t figure out how to release this without destroying their business model. You can still display ads. My bet would be that they equally have not figured out how to make it reliable.
  - deleted 4y ago
    [deleted]
  - xiphias2 4y ago
    Embedding ads is much more natural than displaying them. I don't think Google will allow you to know what part is payed, just like how product search moved to ads-only.
  - nradov 4y ago
    Google search isn't reliable either. Most of the results I get are either irrelevant or at least partly wrong (occasionally outright misinformation). But it's clear that the content comes from third parties without endorsement by Google so advertisers and users mostly accept it. When Google deploys a large language model to directly synthesize content it will carry a greater reputational risk for them and their advertisers. What happens when an ad appears next to an AI answer that could be construed as racist or promoting medical misinformation? The legacy media already hates big tech and will spin it as a "gotcha" moment.
- twelve40 4y ago
  there are a bunch of unclear questions about production readiness of this as-is. where does chatgpt get its data? Does it just hijack the data from websites without referencing them? might work for a limited dataset (commercial or loose licences), but musk already said he won't let it use the Twitter data, for example. Google search is for mutual benefit to publishers that provide the data because it drives traffic. This thing doesn't drive anything, so if I as a publisher found my data in it, I would be getting ready to sue them. there are other questions like, how to you continually retrain as you crawl the bigger web? on a really large number of sources, how do you rank and filter out crap spam from the actual useful, trainable content? so way too early to kill regular search imo.
dilap 4y ago
I'm sort of confused -- so Google has had this amazing, breakthrough chat tech but just didn't feel like making it available? I asked ChatGPT and it had this to say: > is google's LaMDA available to the public? how does it compare to chatGPT in quality? > Google's LaMDA (Language Model for Dialogue Applications) is not currently available to the public. However, it has been used in a number of Google's products, such as Google Assistant and Google Meet's "smart compose" feature. It is not clear how it compares in quality to ChatGPT, as the models have different training data and architectures, and are used for different purposes. However, LaMDA is specifically designed for dialogue generation, while GPT-3 is a more general-purpose language model. Assuming ChatGPT isn't lying to me, if that's the best Google could do with the tech, at the very best they suffer from an absolutely devestating lack of ambition and creativity in their application of the tech. (Or perhaps they're just moving very slow.) Anyway, hopefully all this does motivate Google to do some impressive things.
- nprateem 4y ago
  Or perhaps it's the innovator's dilemma. Why would they voluntarily cannibalise their ad business by offering a chat interface that means users don't have to visit websites?
  - twelve40 4y ago
    in the other article here the deepmind guy mentioned they might release the bot once it's able to cite its sources which makes total sense
  - vintermann 4y ago
    I don't think it's any specific scenario like that they're afraid of. It's more that the changes this technology will lead to are extremely hard to predict, and Alphabet (and Alphabets many powerful friends) have far more to lose than to gain.
- dwringer 4y ago
  As I understand, a couple of the lead Google scientists from LaMDA went on to form character.ai which, for me, checks all the boxes that an LLM should. I haven't really seen many people try to compare it with ChatGPT, and I've never used the latter, but from using the former quite a bit I think it's very comparable to what people have done with ChatGPT.
onion2k 4y ago
I never really understood the argument that OpenAI has some technology that Google does not have. That's just not true. The opposite is much more true. And Google's LaMDA is even older than ChatGPT. Google has to be a long way off monetizing what they have though. If Google was in a position to roll out AI soon I can't imagine why they'd be getting rid of 12,000 people who they know can pass their hiring criteria. They would put some of those engineers to work integrating the AI code into Google's products. Unless Google's AI is so good it can integrate itself I suppose. The fact they're letting 12,000 people go shows they don't have profitable work for those people. That alone should tell us something about the position of Google's AI strategy right now.
- azeirah 4y ago
  Are the 12k people all software engineers?
- haldujai 4y ago
  Is OpenAI remotely close to being profitable? The MS deal suggests not so I don’t think it’s fair to compare to Google’s hesitancy. I think we’re conflating a proof of concept tool and marketing strategy with a viable commercial product, time will tell. Time will tell whether the pro subscriptions are a viable monetization strategy.
  - onion2k 4y ago
    Is OpenAI remotely close to being profitable? No, but HN is for talking about VC-backed startups. Profit is kind of optional. :)
- albertzeyer 4y ago
  Do we actually know a bit more on what people were let go, from what teams? Maybe the percentage of AI people in Google actually has grown relatively now? And Google is in fact using AI almost everywhere in production already. You have some sort of AI in almost every product. Also language models are everywhere, e.g. just prediction of typing on your phone, or in GMail, in speech recognition, and many other places. I think they just do not use the biggest models for those things but some more efficient models, which can partly even run offline on your device (e.g. for the typing prediction). Also in Google search, they use lots of AI, also neural networks. It's just that for LaMDA specifically, they don't have a good product yet.
tveita 4y ago
Google is an established brand. When ChatGPT makes up fake historical facts or has Opinions about Races, that's a cute quirk of the beta technology you were trying out. When a regular Google search does the same it is perceived differently. You can tell they've tried to be conservative with the AI-driven "People Also Ask" feature which quotes a snippet of source text, and I still see completely wrong answers every so often. They also depend on distinguishing "real" content from spam in order to have pages to link to and train on, and with GPT we may see the conclusive defeat of spam detection. There are detectors that claim to detect ChatGPT, but I suspect motivated adversaries can defeat those by training their own models.
- realusername 4y ago
  The real issue starts even before that, Google search is still a cash cow, integrating some GPT tech can threaten their own economic model, they are in a tricky situation here.
  - plastic3169 4y ago
    [dead]
  - klipt 4y ago
    No, the cash cow is ads, search is incidental. Showing a quick answer from AI is no worse for them than showing a quick answer from their knowledge graph (which they already do when available). What matters monetarily is the people searching "best 2023 SUV" and clicking on dealer ads.
    - visarga 4y ago
      Targeted, useful ads are so rare almost nobody has seen them. How can they push low quality ad inventory through a trusted AI? If they train the AI to trick people into bad purchases, it will be a disaster.
    - realusername 4y ago
      Search is exactly how they are displaying ads though. Language models have the possibility to replace a very large portion of the classic search with a different interface, maybe 40%. How they are going to evolve their current business model facing this new reality is unknown. Any tech initiative will be blocked by the executive side afraid of losing the cash cow.
machiaweliczny 4y ago
I think they are winning race to run these models cheaper (see drop in Embedding API)and that's the key here. I've heard that Google to run LaMDA for each query would cost 2x their revenue but if they gonna drop cost by 10x then it's "only" 20% of their revenue and that's probably possible with special hardware / optimizations. So whoever does it first will take over search space.
- visarga 4y ago
  Interesting, do you have a link? I'd like to read more.
  - mathteddybear 4y ago
    LaMDA consists of several fine-tunings of a single pre-trained model. Each of them generate candidate responses, the candidate responses are scored (using yet another fine-tuning of the model, iirc) and the best is selected. One expects that the version of LaMDA for internal and raters use does not have to be frugal.
- cma 4y ago
  > drop in Embedding API Any link to what you are talking about here?
  - machiaweliczny 4y ago
    See 99.8% price reduction - https://openai.com/blog/new-and-improved-embedding-model/ https://openai.com/blog/new-and-improved-embedding-model/
samvher 4y ago
I wonder when we will get to a point that network effects start to dominate in this space, rather than tech/research. It does seem like (a) ChatGPT being publicly available, (b) it learning from user interactions and (c) it having generated a significant level of hype could put it in a position where it pulls away from competition because there is a flow of user information coming in that might allow it to learn faster and better than competitors. Maybe this can't be compensated for just by scraping more data from the web. I'm not familiar enough with how they've implemented part (b) to be able to judge how effectively they're doing this though.
est 4y ago
sounds like IBM.
juujian 4y ago
I am pretty sure OpenAI primarily has really good marketing. (Or maybe the stronger fanboys). Of course they also have a good team. But the introduction of ChatGPT just went incredibly well, despite the bad things that could still be said about the technology.
- hot_gril 4y ago
  They have a publicly-accessible product with a userbase, that's their marketing. Google keeps saying they have something better, but nobody outside has ever seen it. > revealed that it plans to launch its chatbot in private beta sometime in 2023 Last year, kids in school were already writing essays with ChatGPT.
  - zalyh 4y ago
    This is the thing that people seems to miss. It's very accessible to the average Internet user. Kinda like TikTok and Instagram, most teens have tried it out. Sure, the other FAANGs might have far capable AI teams most of that output run in the background. Few are available to the general public that went on to became viral on social media.
    - deleted 4y ago
      [deleted]
    - hot_gril 4y ago
      Yeah, and besides going viral, ChatGPT is known to be good in real-world usage. Hardly anybody outside of Google knows how Lambda compares.
osigurdson 4y ago
I agree but it is interesting that Andrej Karpathy holds the view that Google is behind in these areas.
- bsaul 4y ago
  source ? (not to question your assertion, i'm genuinely curious to hear about Karpathy's POV)
  - osigurdson 4y ago
    He states this in his most recent interview with Lex Fridman. He doesn't go into much detail however. I recall him specifically stating that "maybe Bing will get another chance". My comment would be higher quality with a link to the exact time on the video but I don't have time to do that atm. Suggest watching the whole video however, it is very good.
- Der_Einzige 4y ago
  Karpathy is the reason why Tesla is now behind in a lot of their own AI areas. You're supposed to tell the idiot Elon that actually, sensor fusion does work, and maybe we shouldn't remove radar and parking sensors. But he didn't listen, and now the world is waking up to how shit Tesla's are. That's certainly part of why they're being heavily discounted right now. Just in time for the market to see the ROI on Elon's new bird shaped husk of a company. Google is not behind in these areas in any technical sense, Karpathy just doesn't know what he's talking about.
  - visarga 4y ago
    Ah yes, a rando on YC knows what he's talking about, but Karpathy who led the project doesn't.
    - osigurdson 4y ago
      Maybe the rando knows a lot - how do we know? The only thing I react to is rando's comment is overly negative.
px43 4y ago
Have you used LaMDA? Unsure if you've used the magical internal version, but the version that's available to the general public is terrible. I was so excited to get access to the AI Test Kitchen app, but even today, it strictly limits you to three narrow and useless scenarios, only lets you put in like 40 characters of input at a first, then subsequent inputs after the first are generated for you, and you can only select from a few available options (choose your own adventure style), and after 4 or so interactions, the history is wiped and it starts over. I'm sure Google COULD have something better than ChatGPT, but what they have today is so bad it's almost an insult to the users like me who excitedly signed up, and waited months on the waiting list to get access to it.
andrepd 4y ago
> Also, that technology is also not really so fancy. What matters is that you have enough computing power. And data
- hot_gril 4y ago
  Was going to say, the computing power isn't that meaningful compared to the data size and quality. Which Google probably has the lead on too.
  - visarga 4y ago
    Probably they do have the best data. But I am not holding my breath for them, while I can hardly wait to see the GPT-4 release. I've been trained to expect little/nothing new from Google in the last 5 years, in practical terms. I need to use OCR for document understanding - Amazon Textract beats Google. When I need to translate, DeepL is better. When I need text to speech, NaturalReaders is much better. Google's voice understanding is OK, but Whisper is just as good or better, and free. YouTube recommendations are not great, and not very flexible, I have had better suggestions from chatGPT, it is actually quite surprising. GCP ranks below Azure and AWS. And finally, search quality is bad - very often what you search is replaced with unrelated and useless results. Tell me where does Google's excellence shine? Computational photography? They have so many researchers and developers working on so many things, and yet their main products are mediocre.
    - hot_gril 4y ago
      I'm similarly pessimistic about Google. They can keep old successes running at least. Their one thing that bothers me extra for some reason is Waze. They boast so much about their self-driving technology and take shots at Tesla, but you know, where do I buy a Waze? Oh, it's not a product I can buy or even use, it's just a beta test run in two cities that's far away from being a real product, let alone profitable. Odds of Google (or Alphabet w/e) axing that project is pretty high, yet they're still arrogant. Edit: Hahaha I mean Waymo, not Waze
visarga 4y ago
Both have the technology, but Google has to protect their ad revenues, OpenAI has to impress everyone and get funding. That's why OpenAI released GPT-3 and chatGPT while Google is just starting to get serious. Google has a lot to lose from the paradigm shift in user-computer interaction that is coming. Eventually even regular people will run their local (smaller) AI model, so Google is reduced to a simple API call from the model.
dougmwne 4y ago
But it’s not just about compute. OpenAI did a lot of work to tame a wildcat. GTP-3 is a monster not fit for public consumption. ChatGTP is a kitten. Along the way OpenAI discovered that including code into the training corpus improves the general capabilities of LLMs. Google may have models with superior raw capability, but they do not have a safe and helpful product. They are clearly behind the ball and recent PR from Deepmind confirms they are feeling it.
GalahiSimtam 4y ago
ChatGPT improves over LaMDA in pretty-printing code and passing certain Turing-test-trick questions. It's an improvement not unlike a delivery bot that swaps packages recipients is an improvement over one that invents up non-existent recipients. What makes ChatGPT interesting is that perhaps it could end up a freemium service, whereas Google might remain closer to ad-supported business model(s).

anaganisk 4y ago

When you say its hard to beat Google and their TPU clusters, do you mean Azure has inferior capability? Isnt OpenAI limited by what they can do on Azure?

albertzeyer 4y ago
I don't have numbers, so I don't really know, but yes, I would assume Google (incl DeepMind) has more computing power than Microsoft + OpenAI.
- dubcanada 4y ago
  Why? Both are huge super computer users, and both have similar setups. Both own a search engine, both run ads. I don't really see any reason to suggest Microsoft is less capable than Google. Unless I am missing something?
  - adammarples 4y ago
    Google has been at the forefront AI for years, decades maybe. Do azure even have tpus? It's reasonable to assume Google have close to the best ML compute resources out of probably everyone in the world.
    - visarga 4y ago
      Where do you think GPT-3 was trained and currently runs? In Azure. It doesn't need TPUs.
      - adammarples 4y ago
        Gpt3 is an impressive model, I think it took 512 GPUs 10 days to train though. This alone doesn't mean that Microsoft has more computing power than Google
        hot_gril 4y ago
        It does mean that you don't need the most computing power to train a successful model.
  - albertzeyer 4y ago
    My assumption is based on this: For a long time, Microsoft's core business was Windows and (offline) Office. You don't need a super cluster for that. Google on the other side always had search as a service, available to the public, which needs more compute. So historically, Google needed more compute than Microsoft. Then, from experience, as I do research on neural networks, read papers in the field, Google/DeepMind were always more active in this field than Microsoft, so again, historically, they needed more computing power because of this. Microsoft has catched up, but still, I think Google/DeepMind is more active in the field. Then, Google has TPUs. I think this gives them computing power much more cost effectively than when you need to buy Nvidia GPUs. This is maybe the biggest argument. But in any case, as you say, both companies are big enough to easily train sth like ChatGPT, or also sth like GPT4, however big it might be. Their computing power is almost infinite (for today's standards), so it doesn't really matter who has more.
  - rrdharan 4y ago
    Historically Google far outspent Microsoft on their “super computers”, though in recent years MS has been closing the gap. https://www.platformonomics.com/2022/02/follow-the-capex-cloud-table-stakes-2021-retrospective/ https://www.platformonomics.com/2022/02/follow-the-capex-clo...
  - hot_gril 4y ago
    Google supposedly hires more of the ML experts and custom hardware experts. Whenever someone says that GCP is inferior to the competition, it's usually followed by "except in ML."
- summerlight 4y ago
  I'm also interested in Azure's FPGA instance capacity which is a comparable counterpart to Google's TPU AFAIK. From what I know, Google has a large number of TPU units (not sure about the exact numbers, but I would assume this would exceed 5 digits?) and also has the luxury of sharing TPU instances with products that operate one of the largest models on the planet, namely search and ads. So Google has some advantages on flexibility of resource allocation if it really wants to prioritize AI stuffs.

Semaphor 4y ago

> (Site is down.) Up for me, but takes several seconds to load.

brookst 4y ago

Google has been slow because they can’t figure out how to release this without destroying their business model. It’s very hard to be commercially successful with an innovation when it attacks the cash cow side of your business. It can be done, but it creates a lot of internal conflict and incentives to wait, to gimp the innovation, or to try to bolt it on to the cash cow to mitigate the damage. I have no doubt Google could be a leader in the LLM space. . . But are they willing to destroy their own moat and take a serious revenue hit to do so?

_visgean 4y ago
> Google has been slow because they can’t figure out how to release this without destroying their business model. You can still display ads. My bet would be that they equally have not figured out how to make it reliable.
- deleted 4y ago
  [deleted]
- xiphias2 4y ago
  Embedding ads is much more natural than displaying them. I don't think Google will allow you to know what part is payed, just like how product search moved to ads-only.
- nradov 4y ago
  Google search isn't reliable either. Most of the results I get are either irrelevant or at least partly wrong (occasionally outright misinformation). But it's clear that the content comes from third parties without endorsement by Google so advertisers and users mostly accept it. When Google deploys a large language model to directly synthesize content it will carry a greater reputational risk for them and their advertisers. What happens when an ad appears next to an AI answer that could be construed as racist or promoting medical misinformation? The legacy media already hates big tech and will spin it as a "gotcha" moment.
twelve40 4y ago
there are a bunch of unclear questions about production readiness of this as-is. where does chatgpt get its data? Does it just hijack the data from websites without referencing them? might work for a limited dataset (commercial or loose licences), but musk already said he won't let it use the Twitter data, for example. Google search is for mutual benefit to publishers that provide the data because it drives traffic. This thing doesn't drive anything, so if I as a publisher found my data in it, I would be getting ready to sue them. there are other questions like, how to you continually retrain as you crawl the bigger web? on a really large number of sources, how do you rank and filter out crap spam from the actual useful, trainable content? so way too early to kill regular search imo.

dilap 4y ago

I'm sort of confused -- so Google has had this amazing, breakthrough chat tech but just didn't feel like making it available? I asked ChatGPT and it had this to say: > is google's LaMDA available to the public? how does it compare to chatGPT in quality? > Google's LaMDA (Language Model for Dialogue Applications) is not currently available to the public. However, it has been used in a number of Google's products, such as Google Assistant and Google Meet's "smart compose" feature. It is not clear how it compares in quality to ChatGPT, as the models have different training data and architectures, and are used for different purposes. However, LaMDA is specifically designed for dialogue generation, while GPT-3 is a more general-purpose language model. Assuming ChatGPT isn't lying to me, if that's the best Google could do with the tech, at the very best they suffer from an absolutely devestating lack of ambition and creativity in their application of the tech. (Or perhaps they're just moving very slow.) Anyway, hopefully all this does motivate Google to do some impressive things.

nprateem 4y ago
Or perhaps it's the innovator's dilemma. Why would they voluntarily cannibalise their ad business by offering a chat interface that means users don't have to visit websites?
- twelve40 4y ago
  in the other article here the deepmind guy mentioned they might release the bot once it's able to cite its sources which makes total sense
- vintermann 4y ago
  I don't think it's any specific scenario like that they're afraid of. It's more that the changes this technology will lead to are extremely hard to predict, and Alphabet (and Alphabets many powerful friends) have far more to lose than to gain.
dwringer 4y ago
As I understand, a couple of the lead Google scientists from LaMDA went on to form character.ai which, for me, checks all the boxes that an LLM should. I haven't really seen many people try to compare it with ChatGPT, and I've never used the latter, but from using the former quite a bit I think it's very comparable to what people have done with ChatGPT.

onion2k 4y ago

I never really understood the argument that OpenAI has some technology that Google does not have. That's just not true. The opposite is much more true. And Google's LaMDA is even older than ChatGPT. Google has to be a long way off monetizing what they have though. If Google was in a position to roll out AI soon I can't imagine why they'd be getting rid of 12,000 people who they know can pass their hiring criteria. They would put some of those engineers to work integrating the AI code into Google's products. Unless Google's AI is so good it can integrate itself I suppose. The fact they're letting 12,000 people go shows they don't have profitable work for those people. That alone should tell us something about the position of Google's AI strategy right now.

azeirah 4y ago
Are the 12k people all software engineers?
haldujai 4y ago
Is OpenAI remotely close to being profitable? The MS deal suggests not so I don’t think it’s fair to compare to Google’s hesitancy. I think we’re conflating a proof of concept tool and marketing strategy with a viable commercial product, time will tell. Time will tell whether the pro subscriptions are a viable monetization strategy.
- onion2k 4y ago
  Is OpenAI remotely close to being profitable? No, but HN is for talking about VC-backed startups. Profit is kind of optional. :)
albertzeyer 4y ago
Do we actually know a bit more on what people were let go, from what teams? Maybe the percentage of AI people in Google actually has grown relatively now? And Google is in fact using AI almost everywhere in production already. You have some sort of AI in almost every product. Also language models are everywhere, e.g. just prediction of typing on your phone, or in GMail, in speech recognition, and many other places. I think they just do not use the biggest models for those things but some more efficient models, which can partly even run offline on your device (e.g. for the typing prediction). Also in Google search, they use lots of AI, also neural networks. It's just that for LaMDA specifically, they don't have a good product yet.

tveita 4y ago

Google is an established brand. When ChatGPT makes up fake historical facts or has Opinions about Races, that's a cute quirk of the beta technology you were trying out. When a regular Google search does the same it is perceived differently. You can tell they've tried to be conservative with the AI-driven "People Also Ask" feature which quotes a snippet of source text, and I still see completely wrong answers every so often. They also depend on distinguishing "real" content from spam in order to have pages to link to and train on, and with GPT we may see the conclusive defeat of spam detection. There are detectors that claim to detect ChatGPT, but I suspect motivated adversaries can defeat those by training their own models.

realusername 4y ago
The real issue starts even before that, Google search is still a cash cow, integrating some GPT tech can threaten their own economic model, they are in a tricky situation here.
- plastic3169 4y ago
  [dead]
- klipt 4y ago
  No, the cash cow is ads, search is incidental. Showing a quick answer from AI is no worse for them than showing a quick answer from their knowledge graph (which they already do when available). What matters monetarily is the people searching "best 2023 SUV" and clicking on dealer ads.
  - visarga 4y ago
    Targeted, useful ads are so rare almost nobody has seen them. How can they push low quality ad inventory through a trusted AI? If they train the AI to trick people into bad purchases, it will be a disaster.
  - realusername 4y ago
    Search is exactly how they are displaying ads though. Language models have the possibility to replace a very large portion of the classic search with a different interface, maybe 40%. How they are going to evolve their current business model facing this new reality is unknown. Any tech initiative will be blocked by the executive side afraid of losing the cash cow.

machiaweliczny 4y ago

I think they are winning race to run these models cheaper (see drop in Embedding API)and that's the key here. I've heard that Google to run LaMDA for each query would cost 2x their revenue but if they gonna drop cost by 10x then it's "only" 20% of their revenue and that's probably possible with special hardware / optimizations. So whoever does it first will take over search space.

visarga 4y ago
Interesting, do you have a link? I'd like to read more.
- mathteddybear 4y ago
  LaMDA consists of several fine-tunings of a single pre-trained model. Each of them generate candidate responses, the candidate responses are scored (using yet another fine-tuning of the model, iirc) and the best is selected. One expects that the version of LaMDA for internal and raters use does not have to be frugal.
cma 4y ago
> drop in Embedding API Any link to what you are talking about here?
- machiaweliczny 4y ago
  See 99.8% price reduction - https://openai.com/blog/new-and-improved-embedding-model/ https://openai.com/blog/new-and-improved-embedding-model/

samvher 4y ago

I wonder when we will get to a point that network effects start to dominate in this space, rather than tech/research. It does seem like (a) ChatGPT being publicly available, (b) it learning from user interactions and (c) it having generated a significant level of hype could put it in a position where it pulls away from competition because there is a flow of user information coming in that might allow it to learn faster and better than competitors. Maybe this can't be compensated for just by scraping more data from the web. I'm not familiar enough with how they've implemented part (b) to be able to judge how effectively they're doing this though.

est 4y ago

sounds like IBM.

juujian 4y ago

I am pretty sure OpenAI primarily has really good marketing. (Or maybe the stronger fanboys). Of course they also have a good team. But the introduction of ChatGPT just went incredibly well, despite the bad things that could still be said about the technology.

hot_gril 4y ago
They have a publicly-accessible product with a userbase, that's their marketing. Google keeps saying they have something better, but nobody outside has ever seen it. > revealed that it plans to launch its chatbot in private beta sometime in 2023 Last year, kids in school were already writing essays with ChatGPT.
- zalyh 4y ago
  This is the thing that people seems to miss. It's very accessible to the average Internet user. Kinda like TikTok and Instagram, most teens have tried it out. Sure, the other FAANGs might have far capable AI teams most of that output run in the background. Few are available to the general public that went on to became viral on social media.
  - deleted 4y ago
    [deleted]
  - hot_gril 4y ago
    Yeah, and besides going viral, ChatGPT is known to be good in real-world usage. Hardly anybody outside of Google knows how Lambda compares.

osigurdson 4y ago

I agree but it is interesting that Andrej Karpathy holds the view that Google is behind in these areas.

bsaul 4y ago
source ? (not to question your assertion, i'm genuinely curious to hear about Karpathy's POV)
- osigurdson 4y ago
  He states this in his most recent interview with Lex Fridman. He doesn't go into much detail however. I recall him specifically stating that "maybe Bing will get another chance". My comment would be higher quality with a link to the exact time on the video but I don't have time to do that atm. Suggest watching the whole video however, it is very good.
Der_Einzige 4y ago
Karpathy is the reason why Tesla is now behind in a lot of their own AI areas. You're supposed to tell the idiot Elon that actually, sensor fusion does work, and maybe we shouldn't remove radar and parking sensors. But he didn't listen, and now the world is waking up to how shit Tesla's are. That's certainly part of why they're being heavily discounted right now. Just in time for the market to see the ROI on Elon's new bird shaped husk of a company. Google is not behind in these areas in any technical sense, Karpathy just doesn't know what he's talking about.
- visarga 4y ago
  Ah yes, a rando on YC knows what he's talking about, but Karpathy who led the project doesn't.
  - osigurdson 4y ago
    Maybe the rando knows a lot - how do we know? The only thing I react to is rando's comment is overly negative.

px43 4y ago

Have you used LaMDA? Unsure if you've used the magical internal version, but the version that's available to the general public is terrible. I was so excited to get access to the AI Test Kitchen app, but even today, it strictly limits you to three narrow and useless scenarios, only lets you put in like 40 characters of input at a first, then subsequent inputs after the first are generated for you, and you can only select from a few available options (choose your own adventure style), and after 4 or so interactions, the history is wiped and it starts over. I'm sure Google COULD have something better than ChatGPT, but what they have today is so bad it's almost an insult to the users like me who excitedly signed up, and waited months on the waiting list to get access to it.

andrepd 4y ago

> Also, that technology is also not really so fancy. What matters is that you have enough computing power. And data

hot_gril 4y ago
Was going to say, the computing power isn't that meaningful compared to the data size and quality. Which Google probably has the lead on too.
- visarga 4y ago
  Probably they do have the best data. But I am not holding my breath for them, while I can hardly wait to see the GPT-4 release. I've been trained to expect little/nothing new from Google in the last 5 years, in practical terms. I need to use OCR for document understanding - Amazon Textract beats Google. When I need to translate, DeepL is better. When I need text to speech, NaturalReaders is much better. Google's voice understanding is OK, but Whisper is just as good or better, and free. YouTube recommendations are not great, and not very flexible, I have had better suggestions from chatGPT, it is actually quite surprising. GCP ranks below Azure and AWS. And finally, search quality is bad - very often what you search is replaced with unrelated and useless results. Tell me where does Google's excellence shine? Computational photography? They have so many researchers and developers working on so many things, and yet their main products are mediocre.
  - hot_gril 4y ago
    I'm similarly pessimistic about Google. They can keep old successes running at least. Their one thing that bothers me extra for some reason is Waze. They boast so much about their self-driving technology and take shots at Tesla, but you know, where do I buy a Waze? Oh, it's not a product I can buy or even use, it's just a beta test run in two cities that's far away from being a real product, let alone profitable. Odds of Google (or Alphabet w/e) axing that project is pretty high, yet they're still arrogant. Edit: Hahaha I mean Waymo, not Waze

visarga 4y ago

Both have the technology, but Google has to protect their ad revenues, OpenAI has to impress everyone and get funding. That's why OpenAI released GPT-3 and chatGPT while Google is just starting to get serious. Google has a lot to lose from the paradigm shift in user-computer interaction that is coming. Eventually even regular people will run their local (smaller) AI model, so Google is reduced to a simple API call from the model.

dougmwne 4y ago

But it’s not just about compute. OpenAI did a lot of work to tame a wildcat. GTP-3 is a monster not fit for public consumption. ChatGTP is a kitten. Along the way OpenAI discovered that including code into the training corpus improves the general capabilities of LLMs. Google may have models with superior raw capability, but they do not have a safe and helpful product. They are clearly behind the ball and recent PR from Deepmind confirms they are feeling it.

GalahiSimtam 4y ago

ChatGPT improves over LaMDA in pretty-printing code and passing certain Turing-test-trick questions. It's an improvement not unlike a delivery bot that swaps packages recipients is an improvement over one that invents up non-existent recipients. What makes ChatGPT interesting is that perhaps it could end up a freemium service, whereas Google might remain closer to ad-supported business model(s).