2 ms·
Many organizations block IPs from places such as AWS, Azure, Tor and where it is trivial to get new addresses. It's a great tactic for webcrawling but if you e
by PaulHoule 2y ago
Many organizations block IPs from places such as AWS, Azure, Tor and where it is trivial to get new addresses.
It's a great tactic for webcrawling but if you ever get caught they'll block those netblocks and nobody will get away with it again. (Try editing Wikipedia from Tor and see what happens.) I've been on both sides of that one.
- bigdatta 2y agoAre there suggestions for work arounds? I'm guessing it's a bit of a whack-a-mole problem, but it seems like others have figured out a solution for server-side downloads.
- PaulHoule 2y agoDo your crawls from some place that isn't AWS. I bet there are many obscure data centers that aren't on their list. I was brainstorming for answers to the paywall problem (e.g. system has to see what's behind the paywall) and one answer is a browser extension that can access everything the user's web browser can access (that user subscribes); in many cases this could be made pretty transparent.
- cindycindy 2y ago[flagged]