2 ms·
I had to solve nearly the exact same problem for the same reasons. I too ended up with Selenium. My favorite part was having a nice working system, then throw
by banana_giraffe 6y ago
I had to solve nearly the exact same problem for the same reasons. I too ended up with Selenium.
My favorite part was having a nice working system, then throwing it in the cloud and finding out a socking number of sites tell you to go away if you come at them from a cloud-based IP.
Shouldn't be surprising, but it was still annoying.
- Terretta 6y agoThere are a number of so-called “residential VPN” services with clients that also serve as the firm’s p2p VPN / proxy edge. Some can be subscribed to commercially to resolve precisely the above issue. Preferably, only give money to one that tells their users this is how it works.
- banana_giraffe 6y agoTrue. We went down that path for a while, but for our purposes, I was never happy with the ones I could find, since they were super vague how they got the IPs in the first place. Some of it felt like protecting commercial secrets, which is fine, but some of it felt like protecting questionable business practices. For our purposes, and the websites we needed to track, a traditional VPN was good enough.