4 ms·

"Bots that use headless Chrome will not likely be detected without JS and other tools." The bulk of bots seem to be running empty user agents which makes for a

"Bots that use headless Chrome will not likely be detected without JS and other tools." The bulk of bots seem to be running empty user agents which makes for a simple fix. I understand and agree that advanced bots (good or bad) will likely being impossible to detect. That's ok. I view it as "upping the ante". "Another method I have used to squash a handful of bots is to require HTTP/2.0." I'm at the quick, dirty, and effective stage so this is a solid recommendation. Thank you!

LinuxBender 4y ago
I'm at the quick, dirty, and effective stage so this is a solid recommendation. Thank you! You're welcome. One caveat I should have mentioned is that this will break Google's crawlers as they for some reason can not yet speak HTTP/2.0. Bing will be fine. Archive.ph will be fine. One possible work-around would be to allow HTTP/1.1 for Google's CIDR blocks used by their crawlers not their cloud. Some of Google's bots will identify themselves in the user-agent. Some will just look like an Android phone. They do this on purpose to see who is cloaking or changing content based on the visitor. Another work-around for this is to have your static/indexed content on a site that permits HTTP/1.1 and then enforce HTTP/2.0 on the authenticated portion of your site. This division can be useful later on to optimize for caching static content.
- shanebellone 4y ago
  I have the ability to make decisions before returning a response. The plan is to authorize desirable bots, default to 403 for unknown/unauthorized bots, and return content for the rest. This would allow me to serve Googlebot via 1.1 and default traffic to 2.0. I'm thinking that a parent request's time-on-page (or time between requests) would indicate whether to serve a 403 or content during a subsequent hit. There are a few false positive scenarios to deal with they seem manageable. Any thoughts on that particular approach?
  - LinuxBender 4y ago
    Time-between-requests would seem reasonable except that bots will be crawling your site and others at the same time. This adds a delay combined with whatever crawl delay you specify in your robots.txt for search engine bots so I suspect you may get some false positives. Perhaps you could operate this in a log-only passive mode for a while to log what would have been blocked? The same applies to malicious bots. They can be crawling hundreds of sites at the same time. There is no way to guarantee your site gets the same threads and resources by the bots. Some of the malicious bots are using pools of LTE modems so even their IP's will be spread across a large CIDR block. That said, if you specify a crawl delay of 2 seconds in robots.txt and the bots do not obey this then perhaps it may be worth blocking them. Allowing HTTP/1.1 for the bots that specify the google user-agent will not accommodate their bots that pretend to be Android phones and may result in Google flagging your site as potentially malicious. Some malicious bot tools will also use their user-agent in attempt to bypass authentication on some sites.
    - shanebellone 4y ago
      "Time-between-requests would seem reasonable except that bots ... I suspect you may get some false positives." Solid point. "Perhaps you could operate this in a log-only passive mode for a while to log what would have been blocked?" I can and will. This seems to be the safest way forward. "The same applies to malicious bots. They can be crawling hundreds of sites at the same time. There is no way to guarantee your site gets the same threads and resources by the bots. Some of them are using pools of LTE modems so even their IP's will be spread across a large CIDR block." This is very true. Best case scenario: you block all malicious bots... but then they evolve. I'm viewing this from a resource efficiency perspective. I want to limit the cost of automated hits. "That said, if you specify a crawl delay of 2 seconds in robots.txt and the bots do not obey this then perhaps it may be worth blocking them." Which makes this a great idea. It's simple and dichotomous. All desirable bots will respect this declaration. "Allowing HTTP/1.1 for the bots that specify the google user-agent will not accommodate their bots that pretend to be Android phones and may result in Google flagging your site as potentially malicious. Some malicious bot tools will also use their user-agent in attempt to bypass authentication on some sites." Ah, I just realized Googlebot and these "android bots" use 1.1. Wouldn't be a problem otherwise. I'll leave this idea alone for now. Never anger the SERP gods. :D Edit: It seems Googlebot does not accept the crawl-delay directive. Instead, it needs to be set in Search Console every 90 days. https://support.google.com/webmasters/answer/48620?hl=en https://support.google.com/webmasters/answer/48620?hl=en

LinuxBender 4y ago

I'm at the quick, dirty, and effective stage so this is a solid recommendation. Thank you! You're welcome. One caveat I should have mentioned is that this will break Google's crawlers as they for some reason can not yet speak HTTP/2.0. Bing will be fine. Archive.ph will be fine. One possible work-around would be to allow HTTP/1.1 for Google's CIDR blocks used by their crawlers not their cloud. Some of Google's bots will identify themselves in the user-agent. Some will just look like an Android phone. They do this on purpose to see who is cloaking or changing content based on the visitor. Another work-around for this is to have your static/indexed content on a site that permits HTTP/1.1 and then enforce HTTP/2.0 on the authenticated portion of your site. This division can be useful later on to optimize for caching static content.

shanebellone 4y ago
I have the ability to make decisions before returning a response. The plan is to authorize desirable bots, default to 403 for unknown/unauthorized bots, and return content for the rest. This would allow me to serve Googlebot via 1.1 and default traffic to 2.0. I'm thinking that a parent request's time-on-page (or time between requests) would indicate whether to serve a 403 or content during a subsequent hit. There are a few false positive scenarios to deal with they seem manageable. Any thoughts on that particular approach?
- LinuxBender 4y ago
  Time-between-requests would seem reasonable except that bots will be crawling your site and others at the same time. This adds a delay combined with whatever crawl delay you specify in your robots.txt for search engine bots so I suspect you may get some false positives. Perhaps you could operate this in a log-only passive mode for a while to log what would have been blocked? The same applies to malicious bots. They can be crawling hundreds of sites at the same time. There is no way to guarantee your site gets the same threads and resources by the bots. Some of the malicious bots are using pools of LTE modems so even their IP's will be spread across a large CIDR block. That said, if you specify a crawl delay of 2 seconds in robots.txt and the bots do not obey this then perhaps it may be worth blocking them. Allowing HTTP/1.1 for the bots that specify the google user-agent will not accommodate their bots that pretend to be Android phones and may result in Google flagging your site as potentially malicious. Some malicious bot tools will also use their user-agent in attempt to bypass authentication on some sites.
  - shanebellone 4y ago
    "Time-between-requests would seem reasonable except that bots ... I suspect you may get some false positives." Solid point. "Perhaps you could operate this in a log-only passive mode for a while to log what would have been blocked?" I can and will. This seems to be the safest way forward. "The same applies to malicious bots. They can be crawling hundreds of sites at the same time. There is no way to guarantee your site gets the same threads and resources by the bots. Some of them are using pools of LTE modems so even their IP's will be spread across a large CIDR block." This is very true. Best case scenario: you block all malicious bots... but then they evolve. I'm viewing this from a resource efficiency perspective. I want to limit the cost of automated hits. "That said, if you specify a crawl delay of 2 seconds in robots.txt and the bots do not obey this then perhaps it may be worth blocking them." Which makes this a great idea. It's simple and dichotomous. All desirable bots will respect this declaration. "Allowing HTTP/1.1 for the bots that specify the google user-agent will not accommodate their bots that pretend to be Android phones and may result in Google flagging your site as potentially malicious. Some malicious bot tools will also use their user-agent in attempt to bypass authentication on some sites." Ah, I just realized Googlebot and these "android bots" use 1.1. Wouldn't be a problem otherwise. I'll leave this idea alone for now. Never anger the SERP gods. :D Edit: It seems Googlebot does not accept the crawl-delay directive. Instead, it needs to be set in Search Console every 90 days. https://support.google.com/webmasters/answer/48620?hl=en https://support.google.com/webmasters/answer/48620?hl=en