7 ms·

Ah the good ol "whitelist the google bot IPs but pop-up paywalls for everyone else" trick. It'd be cool if google randomized their crawler's IPs and made their

Ah the good ol "whitelist the google bot IPs but pop-up paywalls for everyone else" trick. It'd be cool if google randomized their crawler's IPs and made their bot look like just another user, then we'd all get the same viewing experience.

jfoster 4y ago
My understanding is that Google have a policy against sites doing that. Does Google not enforce the policy sufficiently, or are there exceptions made?
- nostromo 4y ago
  Google’s policy shifted. You can show the crawler full text of an article and then put it behind a paywall for example.
  - MonkeyMalarky 4y ago
    Not only does that benefit the site owners, it benefits Google by creating a moat. Any new entrant to search is initially restricted to the same content as you or I.
    - orangeoxidation 4y ago
      Doesn't it make search worse though? I don't like my searches to end at paywalls and Google being able to do more of that doesn't seem an advantage.
klabb3 4y ago
Google should just make their cache accessible. Want to Paywall and be indexed, ie eat and have the cookie too? Don't be upset when your readers access it through the cache.
- jfoster 4y ago
  They used to.
- sen 4y ago
  That was one of my favorite features of Google as it let you see pages while the host server was down or if the content had been removed. Yet another actually-useful-for-users feature that got removed because profits.
  - kccqzy 4y ago
    I thought that was because many news sites pressured Google not to support it. The cache feature still works for certain sites like personal blogs.
    - klabb3 4y ago
      I don't get this mentality. First you go through hoops to serve the Google crawler with a full version of the article. Then you get pissed when Google presents it to the user. I think the publishers should go ahead and create their own paywall content search engine but I would like the option to have my search results actually be available when I click them.
  - ricardo81 4y ago
    It's not removed, just less visible. Webmasters also have the option of hinting that they don't want a page cached, which some use.
generalizations 4y ago
I think there's an extension that lets you fake your user agent, and one of the options is to present like you're the googlebot. Haven't tried it, but wouldn't that bypass that trick?
- Siira 4y ago
  No. The Google IPs come from known subnets.
  - danuker 4y ago
    Maybe some sites just check the User Agent, not the IP.
chrismarlow9 4y ago
Pretty old grey hat SEO tactic called cloaking. It has evolved from using basic js with obfuscation to redirect users to more sophisticated tactics where users are lured in via serps only to tell them they must sign in. This used to be a plague in the serps for technical error searches and Pinterest in the image searches. The problem with the random ip proposal is that it is a cat and mouse game. If I own 1000 sites on various topics and cross reference visits via IP, well now it's not very random anymore. At one point there was even a services (fantomas) that kept an updated database. This goes very deep in black hat SEO history with mosaic cloaking and other tactics. I used this as a teen in the earlier days of Google to make decent pocket change, and that was in the early 2000s. It goes even deeper when you get into parasite hosting tactics with paid backlinks and xrumer. What's funny is all of these old school black hat seos have moved on to social media. PVA social accounts combined with AI rewriting are where the magic is now and don't even get me started on repurposed content via YouTube. It's no longer economical to tackle organic serps. Heck I even have a bot that scans popular linked domains and monitors the expiration date on them for quick snatch with a nice grafana ui on it. And I've got quite a few nice domains with easy passive traffic and zero effort. All you do is restore the latest archive.org snapshots and setup a catch all email for leads. Dall e and gpt 3 are going to turn these into legit businesses. My point is if there is money to be made people will automate to the extreme. If you dig deep you'll realize an IP address doesn't mean much, hence recaptcha.
- stef25 4y ago
  > Heck I even have a bot that scans popular linked domains and monitors the expiration date on them for quick snatch with a nice grafana ui on it. And I've got quite a few nice domains with easy passive traffic and zero effort. All you do is restore the latest archive.org snapshots and setup a catch all email for leads. That's crazy. How does restoring an archive.org snapshot work, you just dump static html on the domain? Are those mostly just content sites?
  - chrismarlow9 4y ago
    There are a few out there ready to go. Here's one https://github.com/hartator/wayback-machine-downloader https://github.com/hartator/wayback-machine-downloader You just dump and sync to s3 and use terraform to provision the route53 and bucket setups. Yes they are mostly content sites. The hardest part is filtering adult domains assuming you don't want them. There are a staggering number of adult domains that expire every year and get huge traffic.

jfoster 4y ago

My understanding is that Google have a policy against sites doing that. Does Google not enforce the policy sufficiently, or are there exceptions made?

nostromo 4y ago
Google’s policy shifted. You can show the crawler full text of an article and then put it behind a paywall for example.
- MonkeyMalarky 4y ago
  Not only does that benefit the site owners, it benefits Google by creating a moat. Any new entrant to search is initially restricted to the same content as you or I.
  - orangeoxidation 4y ago
    Doesn't it make search worse though? I don't like my searches to end at paywalls and Google being able to do more of that doesn't seem an advantage.

klabb3 4y ago

Google should just make their cache accessible. Want to Paywall and be indexed, ie eat and have the cookie too? Don't be upset when your readers access it through the cache.

jfoster 4y ago
They used to.
sen 4y ago
That was one of my favorite features of Google as it let you see pages while the host server was down or if the content had been removed. Yet another actually-useful-for-users feature that got removed because profits.
- kccqzy 4y ago
  I thought that was because many news sites pressured Google not to support it. The cache feature still works for certain sites like personal blogs.
  - klabb3 4y ago
    I don't get this mentality. First you go through hoops to serve the Google crawler with a full version of the article. Then you get pissed when Google presents it to the user. I think the publishers should go ahead and create their own paywall content search engine but I would like the option to have my search results actually be available when I click them.
- ricardo81 4y ago
  It's not removed, just less visible. Webmasters also have the option of hinting that they don't want a page cached, which some use.

generalizations 4y ago

I think there's an extension that lets you fake your user agent, and one of the options is to present like you're the googlebot. Haven't tried it, but wouldn't that bypass that trick?

Siira 4y ago
No. The Google IPs come from known subnets.
- danuker 4y ago
  Maybe some sites just check the User Agent, not the IP.

chrismarlow9 4y ago

Pretty old grey hat SEO tactic called cloaking. It has evolved from using basic js with obfuscation to redirect users to more sophisticated tactics where users are lured in via serps only to tell them they must sign in. This used to be a plague in the serps for technical error searches and Pinterest in the image searches. The problem with the random ip proposal is that it is a cat and mouse game. If I own 1000 sites on various topics and cross reference visits via IP, well now it's not very random anymore. At one point there was even a services (fantomas) that kept an updated database. This goes very deep in black hat SEO history with mosaic cloaking and other tactics. I used this as a teen in the earlier days of Google to make decent pocket change, and that was in the early 2000s. It goes even deeper when you get into parasite hosting tactics with paid backlinks and xrumer. What's funny is all of these old school black hat seos have moved on to social media. PVA social accounts combined with AI rewriting are where the magic is now and don't even get me started on repurposed content via YouTube. It's no longer economical to tackle organic serps. Heck I even have a bot that scans popular linked domains and monitors the expiration date on them for quick snatch with a nice grafana ui on it. And I've got quite a few nice domains with easy passive traffic and zero effort. All you do is restore the latest archive.org snapshots and setup a catch all email for leads. Dall e and gpt 3 are going to turn these into legit businesses. My point is if there is money to be made people will automate to the extreme. If you dig deep you'll realize an IP address doesn't mean much, hence recaptcha.

stef25 4y ago
> Heck I even have a bot that scans popular linked domains and monitors the expiration date on them for quick snatch with a nice grafana ui on it. And I've got quite a few nice domains with easy passive traffic and zero effort. All you do is restore the latest archive.org snapshots and setup a catch all email for leads. That's crazy. How does restoring an archive.org snapshot work, you just dump static html on the domain? Are those mostly just content sites?
- chrismarlow9 4y ago
  There are a few out there ready to go. Here's one https://github.com/hartator/wayback-machine-downloader https://github.com/hartator/wayback-machine-downloader You just dump and sync to s3 and use terraform to provision the route53 and bucket setups. Yes they are mostly content sites. The hardest part is filtering adult domains assuming you don't want them. There are a staggering number of adult domains that expire every year and get huge traffic.