3 ms·
I would not bother with looking at user-agent strings. Some bots can be spotted by oddly high packet TTL windows default is 128, Linux and Mac 64, missing TCP
by LinuxBender 4y ago
I would not bother with looking at user-agent strings. Some bots can be spotted by oddly high packet TTL windows default is 128, Linux and Mac 64, missing TCP options, inability to do HTTP/2.0. There are iptables modules to block some of them by their missing MSS or odd MSS values, high packet TTL, or simply enforce HTTP/2.0 especially if this is just a blog. This will block all search engines except for Bing if that matters. Maybe someone here from Google can chime in as to when they plan to support HTTP/2.0 on their crawlers.
Most bots are using really old libraries. Some of the newer bots are using farms of cellular cards but they can be spotted by the use of TCP Timestamps as that is enabled by default on Android. The downside of doing anything with this is that one would also block legit cell phones, so just give them a simple puzzle to solve. e.g. What is 1+4 or if you are certain they are a bot then ask How many roads must a man walk down?
- coredog64 4y agoWill I have access as a human if I answer anything but “42” to that last question?
- taskforcegemini 4y agoshouln't it be anything but 42, since that answer came from a machine?
- LinuxBender 4y agoWell not everyone knows that 42 is the answer. I guess you get full unfettered access.
- fsckboy 4y agothe answer to your question, my friend, is blowing in the wind.