5 ms·
> Google has a pretty extreme long-tail distribution (~15% of daily queries have never been seen before) Do you have a source for this? I'd be willing to bet
by zelon88 6y ago
> Google has a pretty extreme long-tail distribution (~15% of daily queries have never been seen before)
Do you have a source for this?
I'd be willing to bet that the ONLY reason why 15% of their daily queries "haven't been seen before" is because they add un-needed complexity like fingerprinting. You're making it seem like they've never seen a query for "cute animals" before when obviously they have. They choose to do a lot of extra leg work because of who you are.
So your claim that 15% of their queries have "never been seen before" is probably inaccurate. I'd be willing to bet that "15% of their queries are unique because of the user, location, or other external factor separate from the query itself."
They've seen your query before. They've just never seen you make this query from this device on this side of town before.
- tylerhou 6y agoIf you took into account user, location, etc. 15% seems too low. I almost never search for the exact same thing twice in the same location. 15% of the queries themselves are unique. https://blog.google/products/search/our-latest-quality-improvements-search/ https://blog.google/products/search/our-latest-quality-impro... https://www.google.com/search/howsearchworks/responses/ https://www.google.com/search/howsearchworks/responses/ I work for Google (and used to work on Search).
- ma2rten 6y agoI think you mean "15% seems too high". Any easy way to think about this is the following: even if search the entire internet you will almost never see the same sentence twice, assuming it's has a certain number of words. There is a combinatorial explosion in possible sentences to write. Search queries are essentially just sentences without stopwords.
- djhn 6y agoRemoving stop words is what old school users of IT systems do, because that's what we learned worked best at the time. Internet users who came online later, from GenZ to many boomers, will often just write conversational sentences and questions.
- IgorPartola 6y agoThe point is not that. It’s that when you search for “cute animals”, Google shouldn’t be storing that you searched for that, or even care. Your location is arguably potentially relevant but it could be coarse enough except when searching for directions to allow at least some caching.
- tylerhou 6y agoThis is right on the money — getting search results for queries that are too personalized to e.g. location means that you can't cache those search results (or if you did cache them, their entries would be useless).
- edmundsauto 6y agoHey Igor! Hate to be a bore, but I wanted to provide feedback that your comment may unintentionally come across as aggressive. OP has pretty relevant work experience that I know I’d love to hear more about, but there’s not really any room for them to respond. I know many folks IRL who work at big tech who have no interest in posting here because the community comes off as very unwelcoming. That’s a shame, because they have insight that would be great to hear. Regardless of anyone’s opinion of their employer. Apologies in advance if your intent was purely about the topic. I just thought I read something in your tone that might hinder discourse rather than encourage it. I wanted to point it out, in case it was unintentional.
- tylerhou 6y agoAgreed about the tone. The comment could have been less argumentative — instead of "that's not the point," they could have said "that's not the only reason." On the other hand, if I'm not responding, it's not because I find HN too abrasive — it's because I am afraid of leaking non-public information. That's why whenever I talk about Google, I try to cite a Google blog post or other authoritative source, or talk about my own personal experience; hence, "I rarely search for the same query twice."
- rStar 6y ago
- zelon88 6y agoI'd be interested in seeing how polluted that 15% of new queries is with people blasting malformed URLs or FQDNs into the omnibox of Chrome.
- Kiro 6y agoWhat's so unbelievable about 15%? I personally think it is way lower than I expected. We're clearly not googling in the same way.
- simias 6y agoI agree with you. Also in my experience less tech-savvy people tend to overcomplicate their queries instead of just entering the relevant keywords which I'm sure accounts for many uniques.
- jobigoud 6y agoI don't understand how you compute that estimate. I doubt you store the history of all searches ever? People don't need a google account to query the engine, others disable history, etc. Are you saying you still have all searches ever made ever? Because you would need this to say a query hasn't been made before wouldn't you?
- melq 6y agoYou'd only need to store the list of unique searches, but even if that's true and the 15% number is true, that must be a huge amount of data.
- freeone3000 6y agoWhy would you not store every search ever? It's only a few petabytes, and you can find out all sorts of useful info from it.
- simias 6y agoI don't know how they did it but I suspect that it wouldn't be very hard to model the distribution by sampling a few million queries and extrapolate from that.
- bagels 6y agohttps://blog.google/products/search/our-latest-quality-improvements-search/ https://blog.google/products/search/our-latest-quality-impro... "There are trillions of searches on Google every year. In fact, 15 percent of searches we see every day are new"
- zepto 6y agoIt would still be helpful to know what ‘new’ means. Does it mean literally the text string typed into the box by the user is new? Or does it mean the text string combined with a bunch of other inferred parameters we don’t know about is new?
- FalconSensei 6y ago> Does it mean literally the text string typed into the box by the user is new? I guess that could be the case. Many could be related to things that are on the news. Like, 'the cw powerpuff girls' for the new show that was announced. No one was searching for that until the announcement, probably
- zepto 6y agoRight - but without clarification from Google, we really don’t know what it means.
- jjk166 6y agoAt the lower bound, that's 150 Billion "new" searches per year. There are approximately 50,000 unique english words not including names or misspellings. If google searches were on average for 3 words, it would take 833 years at that rate to go through all the combinations. Alternatively, if we assume that google has already recorded 20 Trillion unique search queries (~ 1 Trillion new ones per year for 20 years), the odds that a query composed of 3 correctly spelled english words that are not names has been seen before is 1.6%. Even if we restrict queries to those using the most common 1000 words, there's a 50/50 chance of a query composed of 4 words being unique. Of course people do not just type random words into the search bar and some terms will be searched many thousands or even millions of times, but still if anything the fact that 85% of searches aren't unique seems surprising.
- wolfd 6y agohttps://blog.google/products/search/our-latest-quality-improvements-search/ https://blog.google/products/search/our-latest-quality-impro... They briefly mention the statistic in the last paragraph.
- Rastonbury 6y agoWell, you'd be wrong
- zelon88 6y agoMeh, it happens.
- rntksi 6y agoI think GP post has a point. I've noticed people use Google really differently from how I do. E.g. I would go search for "figure concave" while my brother would search a longer phrase. Also, speaking of people behaviour, it would not make sense to search everyday for "cute animals", but the volume of searches done for new things people discover as they get older would make more sense. I mean just look at search trends for things like "hydroxychloroquine" for example (and that's not to mention people who get it wrong, i.e. other factors for differing search queries too) Also, other languages can change the queries depending on how you phrase the sentence too. Add to that the people using other ways to search instead of just visiting google.com and I think you can get pretty close to 10%. If fingerprinting is the reason, 15% would be a figure too low I surmise. Would that be the case I think that would make probably 20-25% of searches rather than 15%. It could very well be that they do classify fingerprinted search differently only in some countries and not others? That would/might explain the 15% figure. I might be wrong and under-estimated fingerprinting techniques for Google. If they have really good fingerprinting techniques, that would reduce the estimate I have in mind to a better number (close to 15, maybe?)
- zelon88 6y agoSo consider your hydroxychloroquine example again this way; Nobody has ever searched for hydroxychloroquine before today. Today is the day the word is hypothetically invented. Today 2 million people will search for hydroxychloroquine. But only one of them was the first to do it. What I know about pop-culture and viral internet culture is telling me that 15% of 1 trillion searches being unique is shady math. So I am not fully convinced that the 15% claim is completely transparent.
- no_way 6y agoIt's a guess, but my thinking is that previously most people who searched term hydroxychloroquine were mainly scientists and other people related to that not your general population. Suddenly covid happens and now large numbers of people learn about this new drug they never heard before, they are gonna search, and I presume this, most wildly different things like: "how does it work?" "does it cause some disease?" "insert something political here about hydroxychloroquine" "did aliens make hydroxychloroquine?" and many more things I lack imagination to come up with and that's only about hydroxychloroquine. I doubt 15% number is about single word cases, but more about combination of words and that seems reasonable. Inventing new words daily seems unlikely, chaining them on the other hand seems plausible.
- wwwwewwww 6y agoIt's somewhat analogous to the claim that almost every spoken sentence had never been spoken before in the history of language.
- rfoo 6y ago> So your claim that 15% of their queries have "never been seen before" is probably inaccurate. I'm not sure, on my productive days maybe >50% of my Google searches are not very cachable. (for example, I just googled "htop namespace", "htop novel bytes", "htop pss", "htop nightly build ubuntu 14.04")