3 ms·
I worked at a company that had this problem and while I was not involved with the solution, I did sit near the guy that was and had extensive talk with him on h
by mullen 8y ago
I worked at a company that had this problem and while I was not involved with the solution, I did sit near the guy that was and had extensive talk with him on how to resolve the issue (Or solve the issue well enough).
* Require accounts with good robot account creation detection. Actually, not hard since there is a lot of canned coded for this.
* Subscribe to Tor and Proxy IP collection services and then block all of the Tor and Proxy addresses. Tor IP list is free from the Tor Project and Proxy IP's can be found through various services.
* Put in crawler detection code. Most scrappers are really simple and are really easy to detect. I think Apache even has a module to detect simple scrapers and autoblock them.
* Feed bad data to scrapers once they are detected. This was the thing that made scrapers go away. Once they know you can detect them and feed bad data, they know they are screwed and will give up. I did get a kick out of looking at the scrapers website and seeing "Penis won the 5K Penis run". Of course, you should really feed the scrapers bad data that looks believable.
* DMCA their website with their provider. The company I worked for did have several scraper sites deactivated.
* Ask Google to delist them. The company I worked for did have several scraper sites removed from Google.
* There are a number of services that will help you with all of this but your website is only 5K unique users a day and not really worth the cost.
- DEADBEEFC0FFEE 8y agoBanning Tor and proxies is pretty unfair to legitimate users who use those services.
- tehlike 8y agoWhile i agree, the op isnt doing a charity either, and op needs to keep their interest at a higher priority to be able to keep providing the service.