3 ms·
True. Then again, the cost for the scraper can be raised significantly by changing your obfuscation / anti-scraping methods frequently. All of the sudden a scra
by heipei 9y ago
True. Then again, the cost for the scraper can be raised significantly by changing your obfuscation / anti-scraping methods frequently. All of the sudden a scraper will need close monitoring to ensure his scripts / regexes are still working and he will likely need a person dedicated to implementing new workarounds as soon as the sites-to-be-scraped push out a new obfuscation method.
- afpx 9y agoIn which case, why not just provide a paid API? The content provider will then make extra revenue that would otherwise go to the endless arms race. As others have mentioned, there is nothing (that I know of) that can thwart a motivated and resourceful scraper.
- heipei 9y agoSome services have actually gone down that exact road. pastebin.com is one of them. But in other cases you simply don't want anyone to be able to extract your info automatically. A good example would be e-commerce sites which don't want anyone to be able to scrape their pricing information in large scale and real time.