3 ms·
One of the things I’ve done before among the other suggestions is to put a hidden link like /followmeifyouscraping.html in the landing page to get a bit of info
by BasiliusCarver 4y ago
One of the things I’ve done before among the other suggestions is to put a hidden link like /followmeifyouscraping.html in the landing page to get a bit of info about scraping volume and then you can use fail2ban filters to block if it’s visited if you want
- jiggawatts 4y agoI would add that link to robots.txt as an exclusion. That way well-behaved search engines won’t be affected, but naive scrapers get auto-banned.