3 ms·
It's overkill to use Tor for this. And I'm a little surprised that it works at all well, because Tor exits so commonly trigger CAPTCHAs. Better, I think, would
by mirimir 7y ago
It's overkill to use Tor for this. And I'm a little surprised that it works at all well, because Tor exits so commonly trigger CAPTCHAs.
Better, I think, would be using HTTPS proxies. But not free ones, which tend to get burned down pretty quickly. There are sites that lease private proxies, and guarantee that they work.
- judge2020 7y agoHTTP/S proxies probably wouldn't want you burning their IPs on web scraping though - it might be against the license/lease terms of the site.
- mirimir 7y agoOK, top DDG result (an ad) for "https proxies web scraping": > https://smartproxy.com/scraping/proxies https://smartproxy.com/scraping/proxies > Global Residential Proxy Network for Web Scraping. Never Get Blocked Again. High Reliability, Easy to Use. Choose Your Plan Now!
- 333c 7y agoWhat are the odds that this is essentially a botnet made of infected computers and IoT devices with unsuspecting owners?
- goatsi 7y agoInfecting computers and IoT devices takes too much effort these days. The latest method is to have the users brought to you - just publish a SDK and pay developers to stick it into their apps. Here's an example: https://luminati.io/sdk https://luminati.io/sdk
- FDSGSG 7y agoLuminati has not killed off the other providers in the market.
- viraptor 7y agoExtremely low. A temporary IP is cheaper, easier, and less legally risky to obtain compared to a hacked endpoint. Remember you're taking real, trackable money for that service. Iot devices also don't have an amazing throughout.
- nurettin 7y agoThere are such paid proxy services that provide you with a refreshable IP pool. Unfortunately, you can't control which IP they give you at each request (or when you can, your request rate drops significantly) so they do not work with websites which require you to keep a session alive.
- bshipp 7y agoI use crawlera, and it does have the capability to establish and maintain a session. I can't pick the precise initial IP but once the connection is made I can certainly tell it to keep using that IP. Sessions are the only way to use crawlera with libraries like cloudflare-scrape, which pin your authentication to a specific IP.
- nurettin 7y agoThat is interesting. $1000 tier is a considerable amount, but perhaps one can find a profitable line of products to justify the price. Of course, this is too expensive for initial development most of the time, so I'd go for an in-house network of machines initially.