3 ms·
I hadn't seen that previous thread; no, this a complete different set of actors. Rate limiting is difficult, because our legitimate users are frequently Fortun
by CWuestefeld 16y ago
I hadn't seen that previous thread; no, this a complete different set of actors.
Rate limiting is difficult, because our legitimate users are frequently Fortune 50 companies coming through a proxy. It's hard to differentiate that heuristically. Based on the fingerprints we've observed, we might be able to hard-code something, but if the bad guys change the factors we've picked out, it would become whack-a-mole again. Also, at this level, the fact that we've got multiple servers in a farm works against recognition.
The kind of crawling they're doing isn't going to be helped by caching. They're looking for product pricing, and we have customized pricing for every customer (Fortune 500 customers want their own specific contracts), so we wouldn't be getting any cache hits.
- buro9 16y agoIf you have contracts in place for the customers you have, can't you whitelist and just allow access based on the IPs of those customers?
- epc 16y agoDo you know or have a sense of who’s controlling the crawler? Much as I hate involving lawyers, that might be appropriate here. I take it you don't do any sort of authentication which you could use to differentiate your real customers from this actor? Do you have any large binary files you could serve up once you've identified the actor?