3 ms·
Stopping web scrapers with HTML character codes?
How robust is the idea to protect my website content from web scrapers by converting all my text into HTML character codes like this: <xmp>Hello World</xmp>
- v2interactive 13y agoI believe this could negatively affect SEO
- icu 13y agoThank you I didn't even consider this.
- mooism2 13y agoIf the scrapers are targeting your site specifically, then it's not even remotely robust.
- icu 13y agoYes this seems to be the case, thanks for your reply.
- pixeloution 13y agoIts not effective vs any scraping; how do you imagine scraping works? In the most basic case, a program makes a HTTP request, gets the source of your page, and processes it via a HTML parser. In some cases, the same HTML source will be run through a headless browser to parse it, in which case javascript trickery can't protect it either. The basic answer is: if an end user can see it in a browser, you can't hide it from a scraper.
- icu 13y agoI run scraping with Batch, CURL, SED, BCP and SQL Server 2012 or MySQL. My scripts basically download the source code into a text file, run a regex with SED to get content and then put the info into a DB with BCP. I don't take into account any HTML codes for text so such a technique would break my scraping technique.
- Lorenz-Kraft 13y agoYou can't stop scraping. The only thing you can do is detect them. If detected, there are several fun stuff to do. Like showing different content. BUT: Be aware that you might block a "legal" SEO scraper!