4 ms·

A few ideas: 1. File a report with Amazon. 2. Block the IPs using iptables 3. File a DMCA complaint against them with Amazon 4. Send a cease and desist letter

A few ideas: 1. File a report with Amazon. 2. Block the IPs using iptables 3. File a DMCA complaint against them with Amazon 4. Send a cease and desist letter to them or offer to sell them data.

CWuestefeld 16y ago
I filed an abuse report with Amazon on Friday; I have received no acknowledgement. We blocked all the EC2 IP ranges (much as I hated to do so; and did you know how big a chunk of the Internet that is?). They responded by re-routing through anonymizers, especially TOR. We could play whack-a-mole with that, but it's a manpower-intensive arms race. File a DMCA complaint against them with Amazon Can you explain to me how DMCA comes into this? I think that, although there's nothing specific in our T&C (yet) disallowing this, the fact that it's arguably at a DoS level might push this into criminal territory.
- andrewthornton 16y ago
  This is no place for a DMCA. I think that your other actions are on track. At this point this seems like a technical issue, not a legal one.
- buro9 16y ago
  This is a good list of Tor endpoints that is proving to be useful for stopping spammers on my site: http://proxy.org/tor.shtml http://proxy.org/tor.shtml Is this a continuation of this: http://news.ycombinator.com/item?id=2085859 http://news.ycombinator.com/item?id=2085859 If so, did you try the iptables rate limiting as described? Or the Varnish rate limiting?
  - CWuestefeld 16y ago
    I hadn't seen that previous thread; no, this a complete different set of actors. Rate limiting is difficult, because our legitimate users are frequently Fortune 50 companies coming through a proxy. It's hard to differentiate that heuristically. Based on the fingerprints we've observed, we might be able to hard-code something, but if the bad guys change the factors we've picked out, it would become whack-a-mole again. Also, at this level, the fact that we've got multiple servers in a farm works against recognition. The kind of crawling they're doing isn't going to be helped by caching. They're looking for product pricing, and we have customized pricing for every customer (Fortune 500 customers want their own specific contracts), so we wouldn't be getting any cache hits.
    - buro9 16y ago
      If you have contracts in place for the customers you have, can't you whitelist and just allow access based on the IPs of those customers?
    - epc 16y ago
      Do you know or have a sense of who’s controlling the crawler? Much as I hate involving lawyers, that might be appropriate here. I take it you don't do any sort of authentication which you could use to differentiate your real customers from this actor? Do you have any large binary files you could serve up once you've identified the actor?

A few ideas: 1. File a report with Amazon. 2. Block the IPs using iptables 3. File a DMCA complaint against them with Amazon 4. Send a cease and desist letter to them or offer to sell them data.

CWuestefeld 16y ago

I filed an abuse report with Amazon on Friday; I have received no acknowledgement. We blocked all the EC2 IP ranges (much as I hated to do so; and did you know how big a chunk of the Internet that is?). They responded by re-routing through anonymizers, especially TOR. We could play whack-a-mole with that, but it's a manpower-intensive arms race. File a DMCA complaint against them with Amazon Can you explain to me how DMCA comes into this? I think that, although there's nothing specific in our T&C (yet) disallowing this, the fact that it's arguably at a DoS level might push this into criminal territory.

andrewthornton 16y ago
This is no place for a DMCA. I think that your other actions are on track. At this point this seems like a technical issue, not a legal one.
buro9 16y ago
This is a good list of Tor endpoints that is proving to be useful for stopping spammers on my site: http://proxy.org/tor.shtml http://proxy.org/tor.shtml Is this a continuation of this: http://news.ycombinator.com/item?id=2085859 http://news.ycombinator.com/item?id=2085859 If so, did you try the iptables rate limiting as described? Or the Varnish rate limiting?
- CWuestefeld 16y ago
  I hadn't seen that previous thread; no, this a complete different set of actors. Rate limiting is difficult, because our legitimate users are frequently Fortune 50 companies coming through a proxy. It's hard to differentiate that heuristically. Based on the fingerprints we've observed, we might be able to hard-code something, but if the bad guys change the factors we've picked out, it would become whack-a-mole again. Also, at this level, the fact that we've got multiple servers in a farm works against recognition. The kind of crawling they're doing isn't going to be helped by caching. They're looking for product pricing, and we have customized pricing for every customer (Fortune 500 customers want their own specific contracts), so we wouldn't be getting any cache hits.
  - buro9 16y ago
    If you have contracts in place for the customers you have, can't you whitelist and just allow access based on the IPs of those customers?
  - epc 16y ago
    Do you know or have a sense of who’s controlling the crawler? Much as I hate involving lawyers, that might be appropriate here. I take it you don't do any sort of authentication which you could use to differentiate your real customers from this actor? Do you have any large binary files you could serve up once you've identified the actor?