3 ms·
- I haven't used any existing tools, but built some easy blockers myself - Adding hidden-with-css honeypot links with URLs formatted with the same pattern as g
by raquo 13y ago
- I haven't used any existing tools, but built some easy blockers myself
- Adding hidden-with-css honeypot links with URLs formatted with the same pattern as good links
- Check user agents - many scrapers forget to change "phantomjs" to something else
- Maybe block ip ranges of known vps providers
- make an access_log DB, define some patterns (e.g. regularness / number of visits, skewed distribution of pages viewed, etc.) and look up offending IPs
- decide what to do with offenders - block outright, mangle content (e.g. Obnoxiously watermark images, hide prices, provide less detailed product descriptions, add your brand to titles/descriptions if these are probably your competitors scraping your catalog).