28 ms·
This looks cool, how do you handle ip rotation? I'm scraping around 1TB a month and currently manage my own proxies. If you could offer something to replace tha
by nikon 10y ago
This looks cool, how do you handle ip rotation? I'm scraping around 1TB a month and currently manage my own proxies. If you could offer something to replace that it would certainly be something I'd be interested in.
- IanCal 10y agoI've used crawlera for a few years, but not at the same scale. Might be worth checking out, anyway: https://scrapinghub.com/crawlera/ https://scrapinghub.com/crawlera/
- novaleaf 10y agoRight now I do limited IP rotation (once per day) so if you need proxies you'd need to use an external proxy provider (you can specify the procxy to use via the PhantomJsCloud api) I'm toying with some built-in proxy options, but nothing public at the moment. As mentioned, Crawlera is pretty nifty, but unfortunately it doesn't work with PhantomJs on HTTPS sites.