6 ms·
I work at Alphabet and I recently went to an internal tech talk about deploying large language models like this at Google. As a disclaimer I'll first note that
by hncel 4y ago
I work at Alphabet and I recently went to an internal tech talk about deploying large language models like this at Google. As a disclaimer I'll first note that this is not my area of expertise, I just attended the tech talk because it sounded interesting.
Large language models like GPT are one of the biggest areas of active ML research at Google, and there's a ton of pretty obvious applications for how they can be used to answer queries, index information, etc. There is a huge budget at Google related to staffing people to work on these kinds of models and do the actual training, which is very expensive because it takes a ton of compute capacity to train these super huge language models. However what I gathered from the talk is the economics of actually using these kinds of language models in the biggest Google products (e.g. search, gmail) isn't quite there yet. It's one thing to put up a demo that interested nerds can play with, but it's quite another thing to try to integrate it deeply in a system that serves billions of requests a day when you take into account serving costs, added latency, and the fact that the average revenue on something like a Google search is close to infinitesimal already. I think I remember the presenter saying something like they'd want to reduce the costs by at least 10x before it would be feasible to integrate models like this in products like search. A 10x or even 100x improvement is obviously an attainable target in the next few years, so I think technology like this is coming in the next few years.
- ocdtrekkie 4y agoI'm thrilled we got to play with ChatGPT long before Google tried putting something like it directly into search. As neat as it is, it's also wrong, insistent that it is right when it's wrong, and frequently extremely biased to the nature of "Google probably shouldn't have fired Timnit Gebru for being right". Hopefully by the time Google gets this implemented based on cost and latency metrics, we'll have better controls to hold Google accountable for doing so.
- onlyrealcuzzo 4y ago> and the fact that the average revenue on something like a Google search is close to infinitesimal already Isn't Search ad revenue >$100Bn per year? Isn't that >$0.07 per search?
- hhda 4y agoFrom what I can tell, yearly Search ad revenue is in the neighborhood of $104 billion [0], and the number of yearly searches served by Google is somewhere in the neighborhood of 3.1 trillion [1]. This brings the revenue per search to somewhere between 3 and 3.5 cents. [0] https://abc.xyz/investor/static/pdf/20210203_alphabet_10K.pdf?cache=b44182d https://abc.xyz/investor/static/pdf/20210203_alphabet_10K.pd... (page 33) [1] https://www.oberlo.com/blog/google-search-statistics https://www.oberlo.com/blog/google-search-statistics
- spaceman_2020 4y agoThe problem for Google isn't just technical, it's organizational. The entire organization and all its products are built around ads. If a new product comes along that drastically reduces the number of pages a user views, what happens to the ad revenue? Right now, every click, every query is an impression. But if there's an all-knowing AI answering all my questions accurately, what incentive do I, as a user, have to search again, scroll through pages, and look around multiple pages? Google will have to adopt a radically new business model and there's organizational inertia in doing that.
- zackkatz 4y agoThey could still serve AdSense-style ads inside the generated content.
- spaceman_2020 4y agoYeah, but those ad impressions and clicks go down drastically. Is Google ready to live in a world with 50% lower revenue?
- jsnell 4y ago> Right now, every click, every query is an impression. The vast majority (maybe 95%?) of my searches don't show ads.
- jasonjmcghee 4y agoWow. Very different from my experience. If I'm not using Brave, often the first 5ish results are ads.
- vineyardmike 4y ago> The entire organization and all its products are built around ads. Citation? I assume ads are a big part of Google but I suspect it’s not organized around ads. Eg I assume the GCP teams don’t report to any ad teams. I bet Gmail team -which does show ads- is primarily managed to optimize for the paid enterprise customers and they just have an ads guy shove ad boxes in where they can. I bet no one at Nest reports to an ads team, and they’re organized around making money on a per-device basis instead. Is Google good at adopting new successful business models? Ask stadia. But I bet there’s plenty of organizational clarity that alternative revenue streams are important. Disclaimer: I don’t know the internal structure of these teams
- jeffbee 4y agoThe idea that LLMs will kill Google seems to me completely absurd. Name any other organization as well-positioned to create and exploit LLMs.
- jtode 4y agoAll the existing social platforms could also implement ActivityPub and have it working in a week. Name any other organizations that are as well-positioned to make the Fediverse a reality. They [don't] do it, because they have a business model. Same goes for Google. The problem for google is that apparently this other tool is already available, today, though the website is currently overloaded so I can't reach it. But if that site starts working for me, later today, why would I ever ask Google anything again?
- jeffbee 4y ago> All the existing social platforms could also implement ActivityPub and have it working in a week. Name any other organizations that are as well-positioned to make the Fediverse a reality. That's not a good analogy. There are architectural reasons why AP/fediverse will never work, no matter how hard anyone tries. It is not business reasons that prevent, say, Facebook from adopting ActivityPub. They are prevented from adopting it by critical thinking.
- jtode 4y agoBack in the 90s, when mcdonalds.com was owned by some yokel who thought to register it before anyone else, I used to say that they couldn't do capitalism on the internet, and look, they pulled it off! We only had to throw every shred of social cohesion out to make it happen, but hey, the attention economy is in full swing! Rubbish, lad. These platforms manage accounts in their millions within the garden every day, and you're telling me that they can't manage to open up a hole in the API to let someone add and read posts that way, rather than through their sadistic web interfaces? After everything they've already done? More to the point, ActivityPub is just the current popular prototype, the Bitcoin if you will, of the federated social space. We'll get it sorted just fine.
- thewarrior 4y agoI am already willing to pay 15-20 dollars a month for this. Google will fall behind as others give this out as a paid service. Don’t need ads for something this useful.
- goatlover 4y agoHow up to date are LLMs likely to be for search? chatGPT is behind by a year or more. How quickly can an LLM be updated with recent content? That would seem to favor Google for the latest stuff.
- random314 4y agoGoogles Ad revenue from US alone could be 100B. If there are 100M converting users, that's 1000$ per user. 200$ per month cannot get you got. Think more like 100$ per month
- thewarrior 4y agoWhy should serving only this model cost as much as all of Google ?
- s17n 4y agoGPT3 costs something like 5 cents a query. At 20 dollars a month, that would be 400 queries a month. I don't know about you but I'm pretty sure I do at least an order of magnitude more Google searches than that.
- thewarrior 4y agoI don’t think every query needs this
- hallqv 4y agoWay off. The pricing for gpt3 is $0.02 per 1000 tokens for the largest model. 1000 tokens is about 750 words.
- 4y ago
- cafed00d 4y ago> It's one thing to put up a demo that interested nerds can play with, but it's quite another thing to try to integrate it deeply in a system that serves billions of requests a day What’s funny is that Google itself started out as a “demo that interested nerds could play with” — searching your own name on the internet was so squarely a nerd move in the late 90s. Google’s disruption here does not lie in the “ChatGPT or LLMs wil kill Search” realm. Rather, the fact that there isn’t a small-scale, dozen users product leveraging LLMs coming from Google, oh idk maybe something like Quora or Google Answers, or maybe the “Google Flights experience with Assistant”
- likpok 4y agoWhen google started search was a big business with a number of competitors. The most popular, Yahoo, was one of the biggest sites! There was even a meta-search engine (dogpile, I think?) that would run searches against several of the top contenders, but personally I mostly used AltaVista until google killed them. Google did have a key product insight that you didn't need the "web portal" cruft -- just good search was enough.
- tomjen3 4y agoI agree with others that this would be something that was worth paying a monthly fee for, iff it was good.
- summerlight 4y agoThis is so true. Some folks in Ads also tried to explore using large language models (one example: LLM is going to be the ultimate solution for contextual targeting if it's properly done), but one of the major bottleneck is always its cost and latency. Even if you can afford cpu/gpu/tpu costs, you always have to play within a finite latency budget. Large language model often adds latency by order of seconds, not even milliseconds! This is simply not acceptable. I think Pathways is one approach to tackle this issue at scale by making the network sparsely activated so the computation cost can be somehow bounded based on difficulty of each query. This effectively gives Google knobs for the axis across computational cost and the result quality by limiting the size of network to be activated. If it turns out to work well, then we might be able to see it incorporated to Search in a foreseeable future.
- ignoramous 4y agoThat's the thing though, Google doesn't have to release this with Search or in Chrome. It could be a separate product that they can gate access to (charging say $5/mo for 'x' queries a day)? Or, API the model behind GCP? But: Outside of DeepMind, there's nothing comparable from them (in terms of utility AI).
- matesz 4y agoThis is the first thing which came to my mind as well, pay per use!
- summerlight 4y agoThis is the problem of Google, or almost every other big techs. Its infrastructures, products and businesses are designed to serve at least hundreds of millions of users. This works really well for established products but significantly elevates the launch bar for new products, even seemingly easy projects like "why not having this as a small experimental website?". I won't be surprised if someone in the research team actually tried to bring up a small demo site but immediately found a showstopper from product counsels or internal AI guidelines... Launching a full-fledged paid product is even harder, I guess you'll need to secure at least 3~40 headcounts just to integrate this into many subsystems inside Google. And this needs some senior executives driving the project since this is a cross-organization project between research and products. This creates a structural problem, in that they usually expect bigger impacts from these kind of projects to justify the cost. It's possible to pursue without involving top-down decision makers, but usually that kinds of project tends to fail to create consensus since everyone has different priority. So "a separate small, experimental product" is not going to work unless 1. the model becomes fully productionized, generally available inside the company so a single VP (or even director) can quickly build a prototype to demonstrate or 2. someone successfully proposes a convincing path to the major billion user product to draw senior executive's attention or 3. the research team decides to build their own product team from scratch and aggressively invest into the sub team.