3 ms·
I've had this conversion recently with someone, as people start complaining about those AI companies having really aggressive crawlers hammering websites with 1
by iforgotpassword 2y ago
I've had this conversion recently with someone, as people start complaining about those AI companies having really aggressive crawlers hammering websites with 10+ requests a second per bot. I guess they all rush to have a slight competitive edge. Obviously, a lot of them ignore robots.txt...
So what if someone were to buy a bunch of domains, more of less mirror/proxy Wikipedia, and have a bunch of simple regexes that swap out dates, common names, places, and what not... Could you make an impact?