4 ms·
This site could run on a $5/mo VM without issue. Proxies that charge for bandwidth can get expensive, sure, but that’s a separate issue. Waiting on a response f
by datalopers 4y ago
This site could run on a $5/mo VM without issue. Proxies that charge for bandwidth can get expensive, sure, but that’s a separate issue. Waiting on a response from Zillow/Proxies requires nearly zero cpu on your servers and it should be a background/async process. There’s only ~40k zip codes and only ~10k that remotely matter for even very small towns.
Find a shared proxy provider that doesn’t limit bandwidth. It’s usually about $0.50/mo per IP. Not that any of this really matters, you’ll still get blocked by Zillow. It’s not an IP limit l anyway, it’s a TLS fingerprinting or JS-based fingerprinting issue that is getting you blocked.
- tndibona 4y agoI'm using Puppeteer with Chromium to mimic a browser with the headers changed to look like a real user. The proxy service only handles the data transport. Running a browser cluster at scale costs me CPU, and memory (My peak yesterday on HN was 40-100 requests a second). Direct GET / POST requests to Zillow didn't work. I'm not sure $5 per month VM will cut it. Do you have the details on a provider that will give me a $5/m deal for this setup ?
- vgeek 4y agoCheck out residential proxy services. Some of them have apps users install, others may be more... nefarious. They're cheap and you can rotate through them. Some may have clients that are a pain to work with, some may be SOCKS access. https://www.webscrapingapi.com/top-residential-proxy-providers/ https://www.webscrapingapi.com/top-residential-proxy-provide... obviously SEO'd page, but it has useful info.