3 ms·
What you gain? Nothing, but they and other AI companies have decided not to respect your robots.txt
by deaton 5mo ago
What you gain? Nothing, but they and other AI companies have decided not to respect your robots.txt
- loloquwowndueo 5mo agoThere are other ways to block robots from crawling our sites. I have a robots.txt but place no faith in it, it’s just there because it’s cheap and does stop some of the crawlers.