5 ms·

> they’re not able to keep up with the load Zillow can absolutely keep up with the load. Your proxies are being blocked by the WAFs that Zillow utilizes, usual

> they’re not able to keep up with the load Zillow can absolutely keep up with the load. Your proxies are being blocked by the WAFs that Zillow utilizes, usually PerimeterX. Also, based on your other comments, you have no idea how to architect a web application that can handle the trickle of HN traffic, and I'd wager any problems are more likely on your side.

linsomniac 4y ago
I assumed the "they're" referred to the "rotating proxies" not "zillow".
- tndibona 4y ago
  Is correct. The proxy service… it’s slower on the responses from the logs
tndibona 4y ago
I mean, I think I have some idea. The problem is more on the affordability side. I had to scale up the containers in my cluster to cope with the increase in 502s. I live with the fear of a high AWS bill. Then there is the question of the proxy rotators data limit being breached, which happened! I topped it up for now. Now I’d rather spend time asking Zillow to allow me to exist rather than trying to architect a more efficient proxy mechanism.
- datalopers 4y ago
  This site could run on a $5/mo VM without issue. Proxies that charge for bandwidth can get expensive, sure, but that’s a separate issue. Waiting on a response from Zillow/Proxies requires nearly zero cpu on your servers and it should be a background/async process. There’s only ~40k zip codes and only ~10k that remotely matter for even very small towns. Find a shared proxy provider that doesn’t limit bandwidth. It’s usually about $0.50/mo per IP. Not that any of this really matters, you’ll still get blocked by Zillow. It’s not an IP limit l anyway, it’s a TLS fingerprinting or JS-based fingerprinting issue that is getting you blocked.
  - tndibona 4y ago
    I'm using Puppeteer with Chromium to mimic a browser with the headers changed to look like a real user. The proxy service only handles the data transport. Running a browser cluster at scale costs me CPU, and memory (My peak yesterday on HN was 40-100 requests a second). Direct GET / POST requests to Zillow didn't work. I'm not sure $5 per month VM will cut it. Do you have the details on a provider that will give me a $5/m deal for this setup ?
    - vgeek 4y ago
      Check out residential proxy services. Some of them have apps users install, others may be more... nefarious. They're cheap and you can rotate through them. Some may have clients that are a pain to work with, some may be SOCKS access. https://www.webscrapingapi.com/top-residential-proxy-providers/ https://www.webscrapingapi.com/top-residential-proxy-provide... obviously SEO'd page, but it has useful info.

linsomniac 4y ago

I assumed the "they're" referred to the "rotating proxies" not "zillow".

tndibona 4y ago
Is correct. The proxy service… it’s slower on the responses from the logs

tndibona 4y ago

I mean, I think I have some idea. The problem is more on the affordability side. I had to scale up the containers in my cluster to cope with the increase in 502s. I live with the fear of a high AWS bill. Then there is the question of the proxy rotators data limit being breached, which happened! I topped it up for now. Now I’d rather spend time asking Zillow to allow me to exist rather than trying to architect a more efficient proxy mechanism.

datalopers 4y ago
This site could run on a $5/mo VM without issue. Proxies that charge for bandwidth can get expensive, sure, but that’s a separate issue. Waiting on a response from Zillow/Proxies requires nearly zero cpu on your servers and it should be a background/async process. There’s only ~40k zip codes and only ~10k that remotely matter for even very small towns. Find a shared proxy provider that doesn’t limit bandwidth. It’s usually about $0.50/mo per IP. Not that any of this really matters, you’ll still get blocked by Zillow. It’s not an IP limit l anyway, it’s a TLS fingerprinting or JS-based fingerprinting issue that is getting you blocked.
- tndibona 4y ago
  I'm using Puppeteer with Chromium to mimic a browser with the headers changed to look like a real user. The proxy service only handles the data transport. Running a browser cluster at scale costs me CPU, and memory (My peak yesterday on HN was 40-100 requests a second). Direct GET / POST requests to Zillow didn't work. I'm not sure $5 per month VM will cut it. Do you have the details on a provider that will give me a $5/m deal for this setup ?
  - vgeek 4y ago
    Check out residential proxy services. Some of them have apps users install, others may be more... nefarious. They're cheap and you can rotate through them. Some may have clients that are a pain to work with, some may be SOCKS access. https://www.webscrapingapi.com/top-residential-proxy-providers/ https://www.webscrapingapi.com/top-residential-proxy-provide... obviously SEO'd page, but it has useful info.