5 ms·

I did this to Google for a while only to have my domains listed as malicious. I did not offer any malicious material, just different content for search engines

I did this to Google for a while only to have my domains listed as malicious. I did not offer any malicious material, just different content for search engines was enough to flag my sites. They also did this to me when I gave google different IP addresses using a split DNS view. This was a while back so maybe they stopped this, I honestly don't know. Now I just give them and most bots a password prompt. Google and most bots can't speak HTTP/2.0 yet. Bing is the exception and I just trust user-agent for them. # all nginx virtual sites if ($server_protocol != HTTP/2.0) { return 302 https://auth.domain.tld$request_uri; } # in auth.domain.tld virtual site auth_delay 4s; location / { auth_basic "Wamp Wamp"; auth_basic_user_file /dev/shm/.p; }

morkalork 3y ago
Dumb question but how would they know the content is different unless they're also crawling incognito and comparing the results?
- LinuxBender 3y ago
  Google have numerous robots that do not say Googlebot in the user-agent. They look just like Android cell phones. That is how they spot malicious sites or sites that are trying to game SEO or what-not. They are not within published CIDR blocks for Google and appear to just use wireless networks.
  - muttled 3y ago
    I'm picturing Google Street View cars driving around with a box of Pixels in the back, connecting to open WiFi and trying sites and that's why Google can now narrow down your location from what SSIDs are available.
    - morkalork 3y ago
      Speaking of rolling around with box of android devices: https://www.theguardian.com/technology/2020/feb/03/berlin-artist-uses-99-phones-trick-google-maps-traffic-jam-alert https://www.theguardian.com/technology/2020/feb/03/berlin-ar... Also, I would've sworn that happened circa 2015 and not 2020. The passing of time for the last few years has such a muddled feeling.
- kevingadd 3y ago
  Keeping you honest with incognito crawling is something they have to do anyway, to catch various tricks and scams - malware served up to users, etc.
  - morkalork 3y ago
    So robots.txt is meaningless if they have to violate it to check for malicious content in blocked off pages anyways.
    - thenonsequitur 3y ago
      Well if you are blocking access to their crawler, I'd imagine they'd have no need to use an incognito crawler to check for malicious content. Why would they care if that content is not ending up in their index anyway? Presumably, the incognito crawlers are only used on sites that have already granted the regular crawler access. That's content that ends up in their index which they want to vet.

morkalork 3y ago

Dumb question but how would they know the content is different unless they're also crawling incognito and comparing the results?

LinuxBender 3y ago
Google have numerous robots that do not say Googlebot in the user-agent. They look just like Android cell phones. That is how they spot malicious sites or sites that are trying to game SEO or what-not. They are not within published CIDR blocks for Google and appear to just use wireless networks.
- muttled 3y ago
  I'm picturing Google Street View cars driving around with a box of Pixels in the back, connecting to open WiFi and trying sites and that's why Google can now narrow down your location from what SSIDs are available.
  - morkalork 3y ago
    Speaking of rolling around with box of android devices: https://www.theguardian.com/technology/2020/feb/03/berlin-artist-uses-99-phones-trick-google-maps-traffic-jam-alert https://www.theguardian.com/technology/2020/feb/03/berlin-ar... Also, I would've sworn that happened circa 2015 and not 2020. The passing of time for the last few years has such a muddled feeling.
kevingadd 3y ago
Keeping you honest with incognito crawling is something they have to do anyway, to catch various tricks and scams - malware served up to users, etc.
- morkalork 3y ago
  So robots.txt is meaningless if they have to violate it to check for malicious content in blocked off pages anyways.
  - thenonsequitur 3y ago
    Well if you are blocking access to their crawler, I'd imagine they'd have no need to use an incognito crawler to check for malicious content. Why would they care if that content is not ending up in their index anyway? Presumably, the incognito crawlers are only used on sites that have already granted the regular crawler access. That's content that ends up in their index which they want to vet.