8 ms·

> The spam site was checking for Googlebot IP addresses. If the visitor’s IP address matched as belonging to Google then the spam page displayed content to Goog

by amluto 3y ago

> The spam site was checking for Googlebot IP addresses. If the visitor’s IP address matched as belonging to Google then the spam page displayed content to Googlebot. > > All other visitors got a redirect to other domains that displayed sketchy content. Years ago, Google had an explicit policy that sites that showed different content to Googlebot than they showed to regular unauthenticated users were not allowed, and they got heavily penalized. This policy is long gone, but it would help here (assuming the automated tooling to enforce it was any good, and I assume to was). More recently, Google seems totally okay with sites that show content to Googlebot but go out of their way not to show that content to regular users.

altdataseller 3y ago
I think they did this because lots of publishers show paywalls to people but still want their content indexed by Google. In other words, they want their cake and eat it too!
- amadeuspagel 3y ago
  And of course many of these publishers are politically powerful, and are the trusted sources that google wants to promote over random blogs.
  - colordrops 3y ago
    [flagged]
    - ithkuil 3y ago
      Do you think there exists only one class of people with money and that they all have a unified narrative to feed us with?
      - colordrops 3y ago
        No, and my statement doesn't necessarily imply that.
        ithkuil 3y ago
        ok; the use of the singular threw me off. Another (related) question: do you think there are many points of view which aren't also subscribed (or exploited) by one or more moneyed classes?
        yukkuri 3y ago
        Just yell "NOT ALL RICH PEOPLE" and be done with it, bro.
        ithkuil 3y ago
        That's not what I'm saying. My intent is not to defend rich people. Yes obviously most of them don't spend their time controlling the media but instead spend time showing off on their yachts. My point is something else: I don't but the idea that there are two factions, the rich and the poor, and that all rich people have the same interests and thus are allies and that all poor people have the same interests and are allied (or so they should). Sure, this view is partially grounded in realty and that's why Marx did come up with it and it's why it stuck to this day as sensible to so many people. But I don't think it's true. I think it oversimplifies reality to the point that a spherical cow in comparison is anatomically accurate. But it's worse than just being wrong. It actively stifles conversation. Any attempt to have a nuanced conversation about these topics ultimately resolves in an accusation of "you're defending the rich, just admit it". That's what turns a idea into an ideology. Ideologies are ideas with built-in self-defense mechanisms.
        colordrops 3y ago
        I don't follow your question. What is your implication? Just come out and say it.
        ithkuil 3y ago
  - travoc 3y ago
    Well, they all show Google ads.
    - joe_the_user 3y ago
      You'd think they could make fine money as neutral brokers since everyone served their ads and for a long period they did make money as semi-neutral brokers. But since, IDK, 2019 they have become more and more garbage. This is broadly part of the concentration of wealth and power you see everywhere else but I don't know the specifics but you can see the result.
    - amadeuspagel 3y ago
      Not true for the NYT, which has its own ad system.
- franze 3y ago
  there is a special spec for google by that https://developers.google.com/search/docs/appearance/structured-data/paywalled-content https://developers.google.com/search/docs/appearance/structu... basically cloaking + json-ld markup
  - codetrotter 3y ago
    I wonder if Google trains its AI on paywalled data, that other scrapers don’t have access to but which those paywalled sites give full access for the Google bot to.
bhartzer 3y ago
That 'policy' is still actually in effect, I believe, in Google's webmaster guidelines. They just don't enforce it. Years ago (early 2000s) Google used to mostly crawl using Google-owned IPs, but they'd occasionally use Comcast or some other ISPs (partners) to crawl. If you were IP cloaking, you'd have to look out for those pesky non-Google IPs. I know, as I used to play that IP cloaking game back in the early 2000s, mostly using scripts from a service called "IP Delivery".
- bhartzer 3y ago
  You can actually get a manual action (penalty) from Google if you do IP cloaking/redirects. It's still mentioned prominently in Google's Webmaster Guidelines: https://support.google.com/webmasters/answer/9044175?hl=en#zippy=%2Ccloaking-andor-sneaky-redirects https://support.google.com/webmasters/answer/9044175?hl=en#z...
  - franze 3y ago
    And then there is Dynamic Rendering which OKed cloaking https://developers.google.com/search/docs/crawling-indexing/javascript/dynamic-rendering https://developers.google.com/search/docs/crawling-indexing/... and the there are AMP pages which is Google Enforced cloaking...
    - londons_explore 3y ago
      I think by now all search engines run JavaScript and index the rendered page...
      - gildas 3y ago
        As the founder of SEO4Ajax, I can assure you that this is far from the case. Googlebot, for example, still has great difficulty indexing dynamically generated JavaScript content on the client side.
      - charrondev 3y ago
        This isn’t about JavaScript vs no JavaScript. It’s about serving different pages based on User Agent.
victorbjorklund 3y ago
They still have that rule. Just not always easy to spot spammers getting around it.
- deleted 3y ago
  [deleted]
rented_mule 3y ago
About 10 years ago, I was working on a site that served several hundred million non-crawler hits a month. Many of our millions of pages had their content change multiple times a day. Because of the popularity and frequent changes, the crawlers hit us constantly... crawlers accounted for ~90% of our traffic - billions of hits per month. Bing was ~70% of the crawler traffic and Google was ~25% of it. We noticed it because Bing quickly became very aggressive about crawling, exposing some of our scaling limits as they doubled our already significant traffic in a few short months. I was working on the system that picked ads to show on our pages (we had our own internal ad system, doing targeting based on our own data). This was the most computationally intensive part of serving our pages and the ads were embedded directly in the HTML of the page. When we realized that 90% of our ad pick infrastructure was dedicated to feeding the crawlers, we immediately thought of turning ads off for them (we never billed advertisers for them anyway). But hiding the ads seemed to go directly against the spirit of Google's policy of showing their crawlers the same content. Among other things, we ended up disabling almost all targeting and showing crawlers random ads that roughly fit the page. This dropped our ad pick infra costs by nearly 80%, saving 6-figures a month. It also let us take a step back to decide where we could make long term investments in our infra rather than being overwhelmed with quick fixes to keep the crawlers fed. This kind of thing is what people are missing when they wonder why a company needs more than a few engineers - after all, someone could duplicate the core functionality of the product in 100 lines of code. At sufficient scale, it takes real engineering just to handle the traffic from the crawlers so they can send you more users. There are an untold number of other things like this that have to be handled at scale, but that are hard to imagine if you haven't worked at similar scale.
- bombcar 3y ago
  90% of traffic being crawlers seems (on the face) just absolutely batshit insane.
  - Retr0id 3y ago
    Adds a lot of weight to the dead internet theory.
flyinghamster 3y ago
> More recently, Google seems totally okay with sites that show content to Googlebot but go out of their way not to show that content to regular users. See also, pages behind Red Hat and Oracle tech support paywalls.
deleted 3y ago
[deleted]
xnx 3y ago
Why do you think that the rule is not in effect and that this is not an example of the constant cat and mouse game between Google and spammers?
fennecbutt 3y ago
The thing that annoys me most is that sites are allowed to use the http referrer from Google to see what you're searching for. That + spam sites spamming as many keywords as they can just mean whatever you search for 95% of the sites are spam after the first page. Idk why we've let the Internet get like this. There's gotta be a way to sign off on real/trusted content. That's certainly not ssl certs. Could probably crowd source the legitimacy rating of a site or something. That's another reason why people flock to the big names, reddit, youtube, etc. It's like McDonald's, people know that what they get this time will be exactly what they got before.

by amluto 3y ago

altdataseller 3y ago

I think they did this because lots of publishers show paywalls to people but still want their content indexed by Google. In other words, they want their cake and eat it too!

amadeuspagel 3y ago
And of course many of these publishers are politically powerful, and are the trusted sources that google wants to promote over random blogs.
- colordrops 3y ago
  [flagged]
  - ithkuil 3y ago
    Do you think there exists only one class of people with money and that they all have a unified narrative to feed us with?
    - colordrops 3y ago
      No, and my statement doesn't necessarily imply that.
      - ithkuil 3y ago
        ok; the use of the singular threw me off. Another (related) question: do you think there are many points of view which aren't also subscribed (or exploited) by one or more moneyed classes?
        yukkuri 3y ago
        Just yell "NOT ALL RICH PEOPLE" and be done with it, bro.
        ithkuil 3y ago
        That's not what I'm saying. My intent is not to defend rich people. Yes obviously most of them don't spend their time controlling the media but instead spend time showing off on their yachts. My point is something else: I don't but the idea that there are two factions, the rich and the poor, and that all rich people have the same interests and thus are allies and that all poor people have the same interests and are allied (or so they should). Sure, this view is partially grounded in realty and that's why Marx did come up with it and it's why it stuck to this day as sensible to so many people. But I don't think it's true. I think it oversimplifies reality to the point that a spherical cow in comparison is anatomically accurate. But it's worse than just being wrong. It actively stifles conversation. Any attempt to have a nuanced conversation about these topics ultimately resolves in an accusation of "you're defending the rich, just admit it". That's what turns a idea into an ideology. Ideologies are ideas with built-in self-defense mechanisms.
        colordrops 3y ago
        I don't follow your question. What is your implication? Just come out and say it.
        ithkuil 3y ago
- travoc 3y ago
  Well, they all show Google ads.
  - joe_the_user 3y ago
    You'd think they could make fine money as neutral brokers since everyone served their ads and for a long period they did make money as semi-neutral brokers. But since, IDK, 2019 they have become more and more garbage. This is broadly part of the concentration of wealth and power you see everywhere else but I don't know the specifics but you can see the result.
  - amadeuspagel 3y ago
    Not true for the NYT, which has its own ad system.
franze 3y ago
there is a special spec for google by that https://developers.google.com/search/docs/appearance/structured-data/paywalled-content https://developers.google.com/search/docs/appearance/structu... basically cloaking + json-ld markup
- codetrotter 3y ago
  I wonder if Google trains its AI on paywalled data, that other scrapers don’t have access to but which those paywalled sites give full access for the Google bot to.

bhartzer 3y ago

That 'policy' is still actually in effect, I believe, in Google's webmaster guidelines. They just don't enforce it. Years ago (early 2000s) Google used to mostly crawl using Google-owned IPs, but they'd occasionally use Comcast or some other ISPs (partners) to crawl. If you were IP cloaking, you'd have to look out for those pesky non-Google IPs. I know, as I used to play that IP cloaking game back in the early 2000s, mostly using scripts from a service called "IP Delivery".

bhartzer 3y ago
You can actually get a manual action (penalty) from Google if you do IP cloaking/redirects. It's still mentioned prominently in Google's Webmaster Guidelines: https://support.google.com/webmasters/answer/9044175?hl=en#zippy=%2Ccloaking-andor-sneaky-redirects https://support.google.com/webmasters/answer/9044175?hl=en#z...
- franze 3y ago
  And then there is Dynamic Rendering which OKed cloaking https://developers.google.com/search/docs/crawling-indexing/javascript/dynamic-rendering https://developers.google.com/search/docs/crawling-indexing/... and the there are AMP pages which is Google Enforced cloaking...
  - londons_explore 3y ago
    I think by now all search engines run JavaScript and index the rendered page...
    - gildas 3y ago
      As the founder of SEO4Ajax, I can assure you that this is far from the case. Googlebot, for example, still has great difficulty indexing dynamically generated JavaScript content on the client side.
    - charrondev 3y ago
      This isn’t about JavaScript vs no JavaScript. It’s about serving different pages based on User Agent.

victorbjorklund 3y ago

They still have that rule. Just not always easy to spot spammers getting around it.

deleted 3y ago
[deleted]

rented_mule 3y ago

About 10 years ago, I was working on a site that served several hundred million non-crawler hits a month. Many of our millions of pages had their content change multiple times a day. Because of the popularity and frequent changes, the crawlers hit us constantly... crawlers accounted for ~90% of our traffic - billions of hits per month. Bing was ~70% of the crawler traffic and Google was ~25% of it. We noticed it because Bing quickly became very aggressive about crawling, exposing some of our scaling limits as they doubled our already significant traffic in a few short months. I was working on the system that picked ads to show on our pages (we had our own internal ad system, doing targeting based on our own data). This was the most computationally intensive part of serving our pages and the ads were embedded directly in the HTML of the page. When we realized that 90% of our ad pick infrastructure was dedicated to feeding the crawlers, we immediately thought of turning ads off for them (we never billed advertisers for them anyway). But hiding the ads seemed to go directly against the spirit of Google's policy of showing their crawlers the same content. Among other things, we ended up disabling almost all targeting and showing crawlers random ads that roughly fit the page. This dropped our ad pick infra costs by nearly 80%, saving 6-figures a month. It also let us take a step back to decide where we could make long term investments in our infra rather than being overwhelmed with quick fixes to keep the crawlers fed. This kind of thing is what people are missing when they wonder why a company needs more than a few engineers - after all, someone could duplicate the core functionality of the product in 100 lines of code. At sufficient scale, it takes real engineering just to handle the traffic from the crawlers so they can send you more users. There are an untold number of other things like this that have to be handled at scale, but that are hard to imagine if you haven't worked at similar scale.

bombcar 3y ago
90% of traffic being crawlers seems (on the face) just absolutely batshit insane.
- Retr0id 3y ago
  Adds a lot of weight to the dead internet theory.

flyinghamster 3y ago

> More recently, Google seems totally okay with sites that show content to Googlebot but go out of their way not to show that content to regular users. See also, pages behind Red Hat and Oracle tech support paywalls.

deleted 3y ago

[deleted]

xnx 3y ago

Why do you think that the rule is not in effect and that this is not an example of the constant cat and mouse game between Google and spammers?

fennecbutt 3y ago

The thing that annoys me most is that sites are allowed to use the http referrer from Google to see what you're searching for. That + spam sites spamming as many keywords as they can just mean whatever you search for 95% of the sites are spam after the first page. Idk why we've let the Internet get like this. There's gotta be a way to sign off on real/trusted content. That's certainly not ssl certs. Could probably crowd source the legitimacy rating of a site or something. That's another reason why people flock to the big names, reddit, youtube, etc. It's like McDonald's, people know that what they get this time will be exactly what they got before.