4 ms·
One thing I've thought about but never had the chance to put into practice would be to randomize CSS classes and IDs. Most web scraping relies on these to ident
by mattm 9y ago
One thing I've thought about but never had the chance to put into practice would be to randomize CSS classes and IDs. Most web scraping relies on these to identify the content they are looking for.
Imagine if everyday they changed? It would make things a lot more difficult.
There would be disadvantages to actual users with this method like caching wouldn't work very well but maybe this alternative site could be displayed only to bots.
The crawler could get smart about it and only use xpaths like the 6th div on the page so maybe in the daily update you could throw in some random useless empty divs and spans in various locations.
It's a lot of work to setup but I think you would make scraping almost impossible.
- tomascot 9y agoIt works. But you have to create empty elements, too. That way you also target code that is using xpath. Also, you have to take into account regex.
- eli 9y agoThere are scraping tools that render a whole page with a headless browser and then look at what is actually there. Definitely possible. Think about it this way: you could render the whole page as a JPEG with no computer readable text, but someone could still ocr it.
- detaro 9y agoSome sites do that against adblockers, with mixed success.
- bdcravens 9y agoDepends on your content. If the content is dependable, but the DOM isn't, you can get pretty far with something like XPath's contains(). Calling .text on an element in many parsers will happily return all the child content. Worst case is you call .text on <body>