4 ms·
Have you considered banning the problematic bots using robots.txt[0]? Then if they ignore that, you could block them more aggressively. My concern about gzip i
by Someone1234 9y ago
Have you considered banning the problematic bots using robots.txt[0]? Then if they ignore that, you could block them more aggressively.
My concern about gzip is that it could impact legacy clients or real people behind poorly configured proxies at places like businesses, colleges, and public hotspots.
[0] https://en.wikipedia.org/wiki/Robots_exclusion_standard https://en.wikipedia.org/wiki/Robots_exclusion_standard