7 ms·
Ask HN: What prevents Google from having the leading LLM?
It’s weird and a bit surprising to me that Google hasn’t been able to release an LLM for at scale usage that surpasses GPT-4, though they seem like they wish they could. Gemini Ultra seems like it’ll surpass GPT-4 once released next year, though GPT-4.5 may take the lead back either before then or soon after.
What are some of the functional reasons for Google not having the leading LLM, and what are some of the more intangible reasons?
In theory, they have more money, more access to compute, and to data, they have many great researchers, and they have great distribution.
In practice, though, what has made the difference for OpenAI?
- PaulHoule 3y agoOpenAI doesn't have any legacy business to be threatened by it.
- xqb64 3y agoHow does branching out to artificial intelligence threaten their core business?
- PaulHoule 3y agoGoogle is a pay to play search engine. If somebody pays, their result goes to the top. Google will kill its own golden goose unless they can develop an AI that feeds out your question to a real time market and gives the answer that the highest bidder pays for. Trouble is that advertising in the search model seems sorta legitmate but an AI that answers whatever it is paid to answer seems thoroughly corrupt and wouldn't have any consumer or political acceptance. It's that simple. Google used to have "Don't be evil" as a motto, they haven't changed much, just deleted the first word.
- milar 3y agoIt threatens the status quo for product managers!
- smoldesu 3y agoIt doesn't, but competing with OpenAI does. Google likes their ducks in a row. Google Search and YouTube are prime examples; every interaction is subsidized with an advertisement. It's not hard to look at a series of screenshots from either service and circle the spot where your attention was monetized. With ChatGPT, that is not possible. Worse yet, the infrastructure costs of providing GPT-3 scale models for free is massive. It's being subsidized by API and premium subscription profits, which is even more baffling since only a few of the ChatGPT users will become paying customers with their current offerings. So, Google is apprehensive to release a competitor. It's not hard to imagine FAANG having "GPT-4 killers" in their labs, but engineering a way to make it all profitable is the hard part.
- shawndrost 3y agoThe basic story here is about Tay. Microsoft shut Tay down after 16 hours because people found ways to make Tay say "holocaust denier" kind of stuff. https://en.wikipedia.org/wiki/Tay_(chatbot) https://en.wikipedia.org/wiki/Tay_(chatbot) If Tay says out-of-Overton things, and Satya defends it for long enough, that can actually harm MS's legacy business. If Tay can't say out-of-Overton things, it's kind of hard to launch Tay. Launching Tay and taking it down within 16 hours is also not desirable.
- staticman2 3y agoGoogle reorganized their AI division in April 2023. Seems a little soon to say they won't have the leading LLM... assuming google is serious about the whole AI thing, I think you'd need to wait until, I don't know, 2028 or so to provisionally declare a long term winner?
- endisneigh 3y agoYou’re asking the wrong question. Who has the most profitable AI?
- we_love_idf 3y agoGoogle is an advertising company masquerading as tech company. This disguise has served them well and they have made a lot of PR stunts to give the impression that Google is the leader in AI. Fortunately OpenAI and MS showed everyone what true AI is. My prediction is Google will continue making PR stunts and some people will fall for it. Meanwhile OpenAI will get closer and closer to AGI. Whether that's good for humanity is a different discussion. But OpenAI's supremacy is beyond doubt at this point.
- deleted 3y ago[deleted]
- fswd 3y agothe management talent is preventing it
- cloudking 3y agoThis is most likely the reason, and the teams involved in launch approvals (legal, safety, privacy etc).
- gumby 3y agoI think your question is backwards. Why should Google have some especially preferential advantage over any other company? The answer to your question, as you asked, is straightforward: they do have a lot of smart people and lots of money and computing resources, but they have exhibited serious structural problems moving technology forward since the departure of Schmidt in 2011. It is painfully obvious that de facto they haven't had CEO leadership since then. People can and do develop fully functioning example systems, and of course demos, but they then peter out. Gianndrea was pusing them forward on the AI front but after he left it feels to me like the impetus was not replaced. But I don't think that's the real question. The real question is: what are the core functions needed to build leading LLMs, especially generative transformers. In these early days the key factor has been money for cycles. Personally I expect that advantage to diminish over the next few years -- IMHO it's one of those "with enough thrust you can get anything airborn" situations. There are a lot of smart opportunities to do more with less -- too much of the engineering is going into wrangling these hige systems but I see more effort going into wrangling what computation is done in the first place. Google, OpenAI et al are not preferentially positioned for such a transition. I could be wrong: after all the human brain has 80 giganodes with 10^5 fanout. But on the other hand it only runs at less than 100 Hz.
- dharmab 3y agoIndividual primate neurons fire as frequently as >600Hz [1] and they're not synced on the same "clock". It's a poor analogue overall. 1: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5067378/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5067378/
- quietbritishjim 3y agoI'm not sure if this is exactly what that paper is referring to, but a real neutron firing is not like an artificial one firing. An artificial one always fires on a fixed clock, as you say, but some numeric output says how much it is firing. With a real neuron, it either fires or it doesn't, there's no numeric intensity. But how often it fires says how intense the response is. So that does agree with your overall statement that they're not very good analogues... At the very least, their mechanism is quite different. But it does mean that the 600Hz figure is quite misleading, because it's only after a period somewhat longer than 1/600th of a second that you can start to understand the intensity of the output.
- cicce19 3y agoFurthermore, what is stopping Google/Alphabet from exercising their IP rights (i.e. They own a patent for the transformer architecture)? Sure it would be bad publicity, but it seems like they could prevent a ton of competition by simply enforcing that on a few large competitors.
- beardyw 3y agoI am not sure Google wants to be exciting. They seem to spend much of the time keeping their head down, getting on with stuff and making lots of money. It's a rare strategy nowadays, but not necessarily a bad one.
- tiffanyh 3y agoInevitably, any company that becomes very large - becomes more risk adverse for a variety of reason, including not wanting the regulatory spotlight on them.
- deleted 3y ago[deleted]
- whitten 3y agoI don’t know if Google has GPUs on the computers in their massive server farms. I expect they have the resources to make a large language model comparable to the best out there. I agree they didn’t come out with that as technology for sale since monetizing tech requires a special kind of genius. The old faithful process of advertising, selling, buying, and delivering, doesn’t require fancy intelligence so much as consistency and persistence
- Legend2440 3y agoThey actually have their own custom neural network accelerator called a TPU. They're one of the only companies not reliant on NVidia at the moment. Until OpenAI showed up, they were widely considered the leader in AI research.
- grobbyy 3y agoIt's a huge Google disadvantage. Everyone reliant on NVidia has an ecosystem. Google has internal systems which are expensive to maintain and generally worse than best-of-breed systems. Google can't leverage the progress everyone else makes. Google might even reach above average, but they'll still be behind market leaders. AMD and Intel are playing on this space too and it's hard for Google to beat all of them. Everyone else can just use the current winner.
- kromem 3y agoBlake Lemoine. Google had a knee jerk reaction after he released the transcripts and got a bunch of press coverage. If you read the transcripts, it was a much more capable text model closer to OpenAI's products than what they eventually released. Ironically, the same thing happened to OpenAI after licensing GPT-4 to Bing with the 'Sydney' issue. There keeps being early previews of "too human" behaviors from LLMs (to be expected as they are trained and evaluated by the ability to extend human thinking), which then prompts trying to scale back the model to what expectations around AI informed from legacy projections looks like (logical but not emotional or self-determining). It's kind of dumb, and holding back the industry at large. There's a host of applications for LLMs that are being artificially held back because of this trend, such as modeling user engagement with media. And now that synthetic data from SotA models is being used to train other models, it's even a compounding issue. The equivalent of the industry sanding against the grain rather than with it. But it started with Google who hasn't recovered from the setback.
- whalesalad 3y agoStartups can move fast and break things. Google cannot. I think they'll surpass OpenAI relatively quickly just based on the sheer volume of data that they possess for training, combined with the size of their GPU compute. It's just a matter of time. Gemini is already outperforming GPT-4 on paper, we just haven't been able to use it yet.
- pennomi 3y agoThe current mid-sized model they’ve released for Gemini is pretty garbage, and fails all the tests I personally use to separate the smart LLMs from the stupid ones. I can run several local models faster and better.
- surbas 3y agoWhich ones?
- guessbest 3y agoPeople don't like advertising. Google is an advertising company. The more these AI models get corrupted with advertising the more likely they are to take a lead. That the reason they don't dominate in cloud storage. Where's the competitive advantage in embedding advertising in people's personal and publicly shared documents?
- amadeuspagel 3y agoGoogle Drive dominates cloud storage[1]. [1]: https://www.datanyze.com/market-share/file-sharing--198/google-drive-market-share https://www.datanyze.com/market-share/file-sharing--198/goog...
- guessbest 3y agoI have a hard time believing that, but if that website says so, it must be true. I would wonder if they are including gmail users as well as all android users, but I don't know their methodology. If they include all android users, they should include all ios users for Apple Music and iTunes since Apple stores their music on Microsoft Azure, which last I checked is the same thing OneDrive uses.
- cyanydeez 3y agolawyers
- tsurba 3y agoIt could just be that improving the models does not parallelize that well, so you need to wait for the massive months long training runs to finish to see what worked and what to try next. OpenAI got started earlier going full on with scaling up the transformer architecture (even if Googlers came up with it first). Of course if you are smarter or can run more experiments simultaneously, you can catch up at some point. But it could still take a while even with just one year head start.
- xnx 3y agoReputational risk. As an established, global, trillion dollar company with 170,000+ employees, even more shareholders, and billions of users, Google is held to a much higher standard than any new company. Any of the minor slip-ups OpenAI has had with security or appropriateness of responses would result in front-page headlines and lawsuits for Google.
- aio2 3y agoI don't know much about this stuff, but the first idea tha came up is something similar to what Bing/Microsoft does, where with each search there is an AI section using Google's Bard or whatever
- xnx 3y agoGood thinking. This is pretty much what Google is doing now with SGE (Search Generative Experience): https://labs.google/sge/ https://labs.google/sge/ It's in limited opt-in roll-out now. This is probably go gather some feedback (on quality, usefulness, and impact to advertising) before launching it more broadly and also because there isn't enough compute power (even for Google) to run a LLM job for the billions of searches that they delivery daily.
- dekhn 3y agoI have access to SGE and had to file a report on low quality. Some results are OK (saves a trip to wikipedia) but many long-tail queries produce terrible results (and it's really slow: it can take 5-10 seconds to render a result, long after I've scoped the top 3 links). A great example of the sort of quality issue I had: if I search for [ <my name> string library c++ ] it says I'm the author of a popular string library (I am not). The result changes every few days, but is pretty persistent in showing me made-up results like that.
- vitorgrs 3y agoYeah, it's an awful use of LLM IMO... It's basically a summarization of Google search first page. The results lack depth.
- trelliscoded 3y agoPart of the reason CoPilot is so good is that they reportedly ingested a huge chunk of GitHub. Now that everyone's onto this scam where they're not getting paid to help train some model, there's legal challenges to the legality of this approach, and a move back towards self-hosted infrastructure. If there's an injunction and a ruling on that legality, it may be that GPT-4 will be the only model that will ever be trained on a codebase as large as GitHub.
- 4ndrewl 3y agoSame thing that prevents them from having the leading Cloud offering?
- diceduckmonk 3y agoHaving worked in GCP, it is not surprising at all to me that Google, despite having the resources, can’t productize an LLM. It feels like OP question stems from a belief that all there is to training a good model is throwing lots of compute at it. As with any product, there’s polishing and fine tuning you need to do, both before and after travel. Google can’t do that. You also have to accept imperfection and clever tricks to this end, a.k.a. the startup / hacker mentality, which Google is also not positioned to do. I think Meta has a good chance though
- natch 3y agoSome great answers on here. My vote is on they are stuck in their legacy businesses, but we could consider another possibility. Let's imagine they do have an ensemble model that is essentially a cluster of, just picking random numbers, 10 or 100 interacting GPT-6 level model instances all interacting as one combined "mind" that can think about anything you point it at. Would such a mind advise Google to reveal it to the world?
- peddling-brink 3y agoIf Google used the entire Internet to train a model, this would immediately be lambasted as a horrible overreach and they would be sued and regulators would use this as another data point to try and break them up. How long have their crawlers been tickling every part of the accessible Internet? How many chrome browsers could they use? If they acted unethically here, they could have many multiples of the data open ai can reach. Open ai can take on more risk. They didn’t already have a reputation as being anti privacy. I think the legal and reputational threat to Google would be far greater for the same actions.
- b20000 3y agothey can’t even fix google drive