3 ms·
I would add that link to robots.txt as an exclusion. That way well-behaved search engines won’t be affected, but naive scrapers get auto-banned.
by jiggawatts 4y ago
I would add that link to robots.txt as an exclusion.
That way well-behaved search engines won’t be affected, but naive scrapers get auto-banned.