3 ms·
Anyone can legally complain if you copy their content. So they send a C&D letter and you remove whatever offended them, no harm done. If you want to prevent b
by snorkel 19y ago
Anyone can legally complain if you copy their content. So they send a C&D letter and you remove whatever offended them, no harm done.
If you want to prevent bots from scraping your content then take advantage of the fact that most bots don't do Javascript: in your server code render the content of each page with some simple encoding that makes text unreadable then add a piece of javascript to window.onload() thats decodes and displays the content.
- dpapathanasiou 19y agoSo they send a C&D letter and you remove whatever offended them, no harm done. Agreed. It's easier to ask forgiveness (or just stop scraping if they complain) than get permission.
- tocomment 19y agoI've always wondered. Are there any bots that can do javascript? Is there a js engine you can stick in your bot so it reads everything on a page just like a user would see it?
- lupin_sansei 19y agoYes. You you can either automate IE http://support.microsoft.com/kb/167658 http://support.microsoft.com/kb/167658 (or Firefox http://www.iol.ie/~locka/mozilla/mozilla.htm http://www.iol.ie/~locka/mozilla/mozilla.htm ) from your code and make a bot that way. Or use a "bot" like this which uses IE under the hood: http://search.cpan.org/dist/Win32-IE-Mechanize/lib/Win32/IE/Mechanize.pm http://search.cpan.org/dist/Win32-IE-Mechanize/lib/Win32/IE/Mechanize.pm or this one which uses Mozilla under the hood: http://search.cpan.org/~slanning/Mozilla-Mechanize-0.05/lib/Mozilla/Mechanize.pm http://search.cpan.org/~slanning/Mozilla-Mechanize-0.05/lib/Mozilla/Mechanize.pm