3 ms·
Interestingly I don't see any mention of your robots.txt, or if the scraper identified themselves in the UA string. If the person or organization running the sc
by subway 16y ago
Interestingly I don't see any mention of your robots.txt, or if the scraper identified themselves in the UA string. If the person or organization running the scraper is legit, they would probably like to know their software is misbehaving, so that they can fix the issue. Did you attempt to contact them before launching into a cat and mouse game?
- petercooper 16y agoYeah, definitely try contacting them first. Even if their software is being idiotic, they might be a fellow startup or someone who's a bit technically dim but has reasonable intentions. Of course, if you get the brush off, then you have carte blanche for raising hell ;-)