3 ms·

To generate the unique string, do SHA1(content + TheirIP + unique secret). The chance of collision is practically zero. Also, you can then conclusively prove th

by ay 16y ago

To generate the unique string, do SHA1(content + TheirIP + unique secret). The chance of collision is practically zero. Also, you can then conclusively prove that it was you who created this unique string - which may come in handy. (Talking about SHA1 because I used it myself in a lighter version - just a hash of my name - to check how the search engines work with the content of my little blog).

amccloud 16y ago
Not true. See http://en.wikipedia.org/wiki/Birthday_problem http://en.wikipedia.org/wiki/Birthday_problem
- wnoise 16y ago
  The birthday problem basically just cuts the number of bits in half. For SHA1 this changes from a chosen collision at absurd odds to a random collision at merely ridiculous odds.
a5seo 16y ago
I like it, but you need to do this in a way that you append a known/deterministic string so you can monitor THAT one in Google Alerts. I don't care whether the unique strings collide (although I love the idea of being able to tie the IP address of the crawler to the page on which the content was published for maximum evidence collection), I just want Google Alerts to find them reliably AND I don't want the spammer to catch on and be able to easily remove the unique strings before I catch him. The only way for him to find my little landmines (to strip them) would be to read through every piece of content he scraped from me.
- ay 16y ago
  By the way - thinking of all this - why would not you turn this scraper into a free advertisement for you ? If you can detect when it crawls your site, just insert randomly the backlinks to your site into the content you give out to them. This way you will want that they steal more from you :-) (how to put these backlinks in a way that would be difficult to remove - is another story. But if you use other links, e.g. tinyurl, in your material - then you could probably use that :-) Still fairly simple to remove but would require more work from them.)

by ay 16y ago

amccloud 16y ago

Not true. See http://en.wikipedia.org/wiki/Birthday_problem http://en.wikipedia.org/wiki/Birthday_problem

wnoise 16y ago
The birthday problem basically just cuts the number of bits in half. For SHA1 this changes from a chosen collision at absurd odds to a random collision at merely ridiculous odds.

a5seo 16y ago

I like it, but you need to do this in a way that you append a known/deterministic string so you can monitor THAT one in Google Alerts. I don't care whether the unique strings collide (although I love the idea of being able to tie the IP address of the crawler to the page on which the content was published for maximum evidence collection), I just want Google Alerts to find them reliably AND I don't want the spammer to catch on and be able to easily remove the unique strings before I catch him. The only way for him to find my little landmines (to strip them) would be to read through every piece of content he scraped from me.

ay 16y ago
By the way - thinking of all this - why would not you turn this scraper into a free advertisement for you ? If you can detect when it crawls your site, just insert randomly the backlinks to your site into the content you give out to them. This way you will want that they steal more from you :-) (how to put these backlinks in a way that would be difficult to remove - is another story. But if you use other links, e.g. tinyurl, in your material - then you could probably use that :-) Still fairly simple to remove but would require more work from them.)