4 ms·

Given that trick is spreading for several sites now, the trick won't last long. Google could for example generate secret unique user agents for the biggest play

by antpls 6y ago

Given that trick is spreading for several sites now, the trick won't last long. Google could for example generate secret unique user agents for the biggest players. Biggest players would then only allow requests from that secret unique UA.

txdv 6y ago
so much to google embracing the open web
smarx007 6y ago
I think Google shares IP range blocks so you could implement a check like "if(isGooglebot(user_agent) && isGooglebotIp(ip_addr))" in your system. Edit: ah no, they stopped https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/.... I don't think 2 DNS lookups are acceptable to block a GET request but I think it can be done out of band, i.e. isGooglebotIp function can fire away a Redis query and if nothing was found, to put the ip into the DNS verify queue. A few requests later, the bot will now get banned thanks to a new record in Redis.
- 1vuio0pswjnm7 6y ago
  No need to use a Googlebot UA string. Others will work. Such as Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0
miyuru 6y ago
no need a secret unique UA. just PTR the IP and check the host. This method is already used by stackoverflow to hide the sitemap. https://meta.stackexchange.com/a/37272/158100 https://meta.stackexchange.com/a/37272/158100
smileysteve 6y ago
Things that would be inadvisable while most of these companies are actively under suit for antitrust issues.
kobalsky 6y ago
it already happens, try to create your own search engine an index Amazon. captchas everywhere, their robots.txt is just for show.
Too 6y ago
This trick is almost older than the internet, so if someone cared they would block it already. Sending google.com as Referrer is another variant of it. Before stackoverflow days this was very useful for getting past the paywall on expertsexchange for example. I was under the impression that serving other content to google would greatly punish your pagerank and even pull you off the search results completely.

by antpls 6y ago

txdv 6y ago

so much to google embracing the open web

smarx007 6y ago

I think Google shares IP range blocks so you could implement a check like "if(isGooglebot(user_agent) && isGooglebotIp(ip_addr))" in your system. Edit: ah no, they stopped https://developers.google.com/search/docs/advanced/crawling/verifying-googlebot https://developers.google.com/search/docs/advanced/crawling/.... I don't think 2 DNS lookups are acceptable to block a GET request but I think it can be done out of band, i.e. isGooglebotIp function can fire away a Redis query and if nothing was found, to put the ip into the DNS verify queue. A few requests later, the bot will now get banned thanks to a new record in Redis.

1vuio0pswjnm7 6y ago
No need to use a Googlebot UA string. Others will work. Such as Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0

miyuru 6y ago

no need a secret unique UA. just PTR the IP and check the host. This method is already used by stackoverflow to hide the sitemap. https://meta.stackexchange.com/a/37272/158100 https://meta.stackexchange.com/a/37272/158100

smileysteve 6y ago

Things that would be inadvisable while most of these companies are actively under suit for antitrust issues.

kobalsky 6y ago

it already happens, try to create your own search engine an index Amazon. captchas everywhere, their robots.txt is just for show.

Too 6y ago

This trick is almost older than the internet, so if someone cared they would block it already. Sending google.com as Referrer is another variant of it. Before stackoverflow days this was very useful for getting past the paywall on expertsexchange for example. I was under the impression that serving other content to google would greatly punish your pagerank and even pull you off the search results completely.