3 ms·
Well, it's not a lot of links. Few hundred maybe. I bet that's not enough for them to start to notice. Hard to parse the page still, maybe.
by winrid 7y ago
Well, it's not a lot of links. Few hundred maybe. I bet that's not enough for them to start to notice. Hard to parse the page still, maybe.
- Havoc 7y ago>I bet that's not enough for them to start to notice. I got blocked on very first attempt to scrap from cloud. Literally request #1. ...with code that worked fine on residential IP five mins earlier (where I coded it). I'm sure it can be circumvented, but that experience led me to believe it's better to just save the time and pay for a res proxy (as sketchy as that concept is)
- winrid 7y agoWow that's impressive. One solution is to just have some random subset of your users do your scraping for you... Then send the data to the server. Insecure, but probably good enough for a page like this.