5 ms·

I work on AdSense at Google. We need to crawl all the pages that serve our ads so that we can show ads that are relevant. We actually store those crawl results

I work on AdSense at Google. We need to crawl all the pages that serve our ads so that we can show ads that are relevant. We actually store those crawl results in a different place than search to prevent exactly that problem. We could probably save a lot of money if we consolidated those indices, but we don't do that prevent biasing search results. As a result a large percentage of pages that show Google Ads are not in the search index. If Google was so evil, why would we purposefully send traffic to sites where we only get a percentage cut of the revenue? EDIT: I realize I needed to explain this statement a little bit more. If we show ads on google.com we get 100% of the revenue. If show ads on reversephonelookup.it they get majority of the revenue. There is a limited amount of advertiser demand. Instead of manipulating the organic search results, it would be more profitable for google to just show more ads on the search page or inflate the ad price or something.

WaltPurvis 8y ago
> If Google was so evil, why would we purposefully send traffic to sites where we only get a percentage cut of the revenue? Can you explain this? Not sure if it's your phrasing or what, but I'm not getting what you mean.
- deleted 8y ago
  [deleted]
rvnx 8y ago
Because if you show artificially irrelevant results too often (by biasing search results to prioritise AdSense pages), then users would stop using Google Search at some point. Eventually you would loose the revenues coming from the SERP, which are 10 times higher than AdSense :) Would you risk loosing 10, to get 1 ?
- ma2rten 8y ago
  Let's say the search index is size N and the AdSense index is size M. If we were to join them, we would save the storage space for pages that are in both indices. Also, the search index would gain all the sites that are in the AdSense index so search results would potentially improve. However, it would give an unfair advantage to google publishers.
rhizome 8y ago
If Google was so evil, why would we purposefully send traffic to sites where we only get a percentage cut of the revenue? I'm pretty sure you didn't mean to open this can of worms, did it have something else printed on its label? Phone number lookup sites are almost certainly a) easily detectable; and b) low traffic. If Google only gets a percentage cut of this, why index them for fractions of pennies per year?
- ma2rten 8y ago
  There may be cases where we drop low traffic sites from our index, which is separate from the search index. I don't how much of that is public information, so can't go into detail.
  - rhizome 8y ago
    Well but my point is why can't Google recognize these phone-number-lookup sites as the chaff that they are? Nothing should rank lower than them in any search that would return their pages. Said another way: it should be harder for them to clog the top of the SERPs (of course "them" could imply any number of topics).
- citrablue 8y ago
  It's because Google's constraint is number of engineer-attention-hours. This kind of thing probably just isn't a priority, when they have a billion other things they could work on. Keep in mind that Google is not a product company, they are a data company. They work on the biggest most impactful items, and don't have a lot of time for one-offs. Finally, it might not be clear at all that these sites should be removed from the index.
  - idlewords 8y ago
    Google has vast resources and could easily get this stuff right, but chooses not to. Their protests about not having the capacity can get pretty comical. Note also that the claim about "working on the things that have the most impact" is tautological.
trickstra 8y ago
because it's better to have 10% of all visits than 100% of only some visits. Google is not the only way people get to those pages. Yes, maybe phone registries, but since the policy cannot be split, we are talking about the whole internet.
JetSpiegel 8y ago
> As a result a large percentage of pages that show Google Ads are not in the search index. Those results are organic, at least they were not marked as ads. That's even worse, it means the regular crawler is preferring generated phone number sites over blogs. That's the real money waster right there. > We need to crawl all the pages that serve our ads so that we can show ads that are relevant. Why? I though Google's business model was tracking users to show relevant ads to them. You say it's more like DuckDuckGo, or the old magazine ads: film magazines get blockbuster trailers, gardening magazines get compost ads. Either the ads are personalized, so the surrounding content doesn't matter, or the ads are "static", so why track everyone all the time, then? > If we show ads on google.com we get 100% of the revenue. If show ads on reversephonelookup.it they get majority of the revenue. There is a limited amount of advertiser demand. Instead of manipulating the organic search results, it would be more profitable for google to just show more ads on the search page or inflate the ad price or something. As your sibling comments imply, this works like a protection racket. The profit is closing the market to webmasters that don't allow ads at all, or that use non-Google networks.

WaltPurvis 8y ago

> If Google was so evil, why would we purposefully send traffic to sites where we only get a percentage cut of the revenue? Can you explain this? Not sure if it's your phrasing or what, but I'm not getting what you mean.

deleted 8y ago
[deleted]

rvnx 8y ago

Because if you show artificially irrelevant results too often (by biasing search results to prioritise AdSense pages), then users would stop using Google Search at some point. Eventually you would loose the revenues coming from the SERP, which are 10 times higher than AdSense :) Would you risk loosing 10, to get 1 ?

ma2rten 8y ago
Let's say the search index is size N and the AdSense index is size M. If we were to join them, we would save the storage space for pages that are in both indices. Also, the search index would gain all the sites that are in the AdSense index so search results would potentially improve. However, it would give an unfair advantage to google publishers.

rhizome 8y ago

If Google was so evil, why would we purposefully send traffic to sites where we only get a percentage cut of the revenue? I'm pretty sure you didn't mean to open this can of worms, did it have something else printed on its label? Phone number lookup sites are almost certainly a) easily detectable; and b) low traffic. If Google only gets a percentage cut of this, why index them for fractions of pennies per year?

ma2rten 8y ago
There may be cases where we drop low traffic sites from our index, which is separate from the search index. I don't how much of that is public information, so can't go into detail.
- rhizome 8y ago
  Well but my point is why can't Google recognize these phone-number-lookup sites as the chaff that they are? Nothing should rank lower than them in any search that would return their pages. Said another way: it should be harder for them to clog the top of the SERPs (of course "them" could imply any number of topics).
citrablue 8y ago
It's because Google's constraint is number of engineer-attention-hours. This kind of thing probably just isn't a priority, when they have a billion other things they could work on. Keep in mind that Google is not a product company, they are a data company. They work on the biggest most impactful items, and don't have a lot of time for one-offs. Finally, it might not be clear at all that these sites should be removed from the index.
- idlewords 8y ago
  Google has vast resources and could easily get this stuff right, but chooses not to. Their protests about not having the capacity can get pretty comical. Note also that the claim about "working on the things that have the most impact" is tautological.

trickstra 8y ago

because it's better to have 10% of all visits than 100% of only some visits. Google is not the only way people get to those pages. Yes, maybe phone registries, but since the policy cannot be split, we are talking about the whole internet.

JetSpiegel 8y ago

> As a result a large percentage of pages that show Google Ads are not in the search index. Those results are organic, at least they were not marked as ads. That's even worse, it means the regular crawler is preferring generated phone number sites over blogs. That's the real money waster right there. > We need to crawl all the pages that serve our ads so that we can show ads that are relevant. Why? I though Google's business model was tracking users to show relevant ads to them. You say it's more like DuckDuckGo, or the old magazine ads: film magazines get blockbuster trailers, gardening magazines get compost ads. Either the ads are personalized, so the surrounding content doesn't matter, or the ads are "static", so why track everyone all the time, then? > If we show ads on google.com we get 100% of the revenue. If show ads on reversephonelookup.it they get majority of the revenue. There is a limited amount of advertiser demand. Instead of manipulating the organic search results, it would be more profitable for google to just show more ads on the search page or inflate the ad price or something. As your sibling comments imply, this works like a protection racket. The profit is closing the market to webmasters that don't allow ads at all, or that use non-Google networks.