3 ms·
That's what robots.txt is for, not silly captchas. You put captchas for the people who ignore that file.
by SoftwareMaven 15y ago
That's what robots.txt is for, not silly captchas. You put captchas for the people who ignore that file.
- pud 15y agoNot all spiders honor robots.txt
- derwiki 15y agoI think the idea is that given a large corpus of filtered and unfiltered reviews, you might be able to reverse engineer signals in the algorithm and game the system. If that's your end goal, you and the software you write is likely to ignore robots.txt directives.