3 ms·
If someone wants to scrape your site he will do it, just find workarounds against your "protection". It is impossible to tell the difference between a real user
by XCSme 9y ago
If someone wants to scrape your site he will do it, just find workarounds against your "protection". It is impossible to tell the difference between a real user and an automated scrape request, you can only make their job a bit harder.
- heipei 9y agoTrue. Then again, the cost for the scraper can be raised significantly by changing your obfuscation / anti-scraping methods frequently. All of the sudden a scraper will need close monitoring to ensure his scripts / regexes are still working and he will likely need a person dedicated to implementing new workarounds as soon as the sites-to-be-scraped push out a new obfuscation method.
- afpx 9y agoIn which case, why not just provide a paid API? The content provider will then make extra revenue that would otherwise go to the endless arms race. As others have mentioned, there is nothing (that I know of) that can thwart a motivated and resourceful scraper.
- heipei 9y agoSome services have actually gone down that exact road. pastebin.com is one of them. But in other cases you simply don't want anyone to be able to extract your info automatically. A good example would be e-commerce sites which don't want anyone to be able to scrape their pricing information in large scale and real time.
- almost_usual 9y agoMaybe, anti scraping is already a multi million dollar industry with active development from startups and bigger names like Akamai. There are plenty of websites that want scrapers to disappear.