5 ms·

Already using redis for the cache. I’m using a rotating proxy to bypass Zillow captcha, they’re not able to keep up with the load

dylan604 4y ago
It'll be sad if they figure out who you and your bots are, and then deny you specifically access. Got backup plans for when you get zero results for every query because of that?
- tndibona 4y ago
  Yes lol. Realtor.com is also scrapable so that’s the immediate backup. Long term I have to speak FTP csv files from a bunch of archaic MLS integrations. If I need to go that route, I need money.
  - Rediscover 4y ago
    To me, that seems a much simpler route than screen-scraping (yes, despite problems parsing CSV format). I've been involved with screen-scraping of req's for <major airplane company> to produce spreadsheets for others in the same company, and I'll take FTPing the data->CSV almost any day. * Especially when the FAA is the eventual consumer *
    - tndibona 4y ago
      The bigger challenge is not the ftp integration per say. The fact that all the housing data is distributed across nearly 500 MLS sites that closely guard their data and make it available only to licensed realtors. I would expect each of these MLS sites to have different formats. So trying to strike up a conversation and allowing them to let me use their FTP site is the biggest hurdle. Zillow and the rest of the big guys already have their claws deep into the MLS sites and they probably want to closely guard their pipelines. Small guys like me don’t know where to start.
  - kQq9oHeAz6wLLS 4y ago
    I do integration work for healthcare; you want an interface engine to pull the data from the ftp site, then translate/convert it into your desired format. We do that all the time.
    - gooseyman 4y ago
      What’s your favorite tool for this? Just starting to get my feet wet in this world.
  - smcin 4y ago
    But I want to see Realtor.com vs Zillow numbers (vs MLSListings vs Redfin vs the others...). It's not like any of these are ground truth. How much are the discrepancies, where are they and why? Also, is zipcode the lowest level of granularity (presumably not)? Do any of these services have finer granularity?
- _tom_ 4y ago
  Or just get an injunction.
  - toomuchtodo 4y ago
    Seems like an uphill argument as long as the data isn't behind a login and you're not DDoSing the site. https://news.ycombinator.com/item?id=31075396 https://news.ycombinator.com/item?id=31075396
  - deleted 4y ago
    [deleted]
datalopers 4y ago
> they’re not able to keep up with the load Zillow can absolutely keep up with the load. Your proxies are being blocked by the WAFs that Zillow utilizes, usually PerimeterX. Also, based on your other comments, you have no idea how to architect a web application that can handle the trickle of HN traffic, and I'd wager any problems are more likely on your side.
- linsomniac 4y ago
  I assumed the "they're" referred to the "rotating proxies" not "zillow".
  - tndibona 4y ago
    Is correct. The proxy service… it’s slower on the responses from the logs
- tndibona 4y ago
  I mean, I think I have some idea. The problem is more on the affordability side. I had to scale up the containers in my cluster to cope with the increase in 502s. I live with the fear of a high AWS bill. Then there is the question of the proxy rotators data limit being breached, which happened! I topped it up for now. Now I’d rather spend time asking Zillow to allow me to exist rather than trying to architect a more efficient proxy mechanism.
  - datalopers 4y ago
    This site could run on a $5/mo VM without issue. Proxies that charge for bandwidth can get expensive, sure, but that’s a separate issue. Waiting on a response from Zillow/Proxies requires nearly zero cpu on your servers and it should be a background/async process. There’s only ~40k zip codes and only ~10k that remotely matter for even very small towns. Find a shared proxy provider that doesn’t limit bandwidth. It’s usually about $0.50/mo per IP. Not that any of this really matters, you’ll still get blocked by Zillow. It’s not an IP limit l anyway, it’s a TLS fingerprinting or JS-based fingerprinting issue that is getting you blocked.

dylan604 4y ago

It'll be sad if they figure out who you and your bots are, and then deny you specifically access. Got backup plans for when you get zero results for every query because of that?

tndibona 4y ago
Yes lol. Realtor.com is also scrapable so that’s the immediate backup. Long term I have to speak FTP csv files from a bunch of archaic MLS integrations. If I need to go that route, I need money.
- Rediscover 4y ago
  To me, that seems a much simpler route than screen-scraping (yes, despite problems parsing CSV format). I've been involved with screen-scraping of req's for <major airplane company> to produce spreadsheets for others in the same company, and I'll take FTPing the data->CSV almost any day. * Especially when the FAA is the eventual consumer *
  - tndibona 4y ago
    The bigger challenge is not the ftp integration per say. The fact that all the housing data is distributed across nearly 500 MLS sites that closely guard their data and make it available only to licensed realtors. I would expect each of these MLS sites to have different formats. So trying to strike up a conversation and allowing them to let me use their FTP site is the biggest hurdle. Zillow and the rest of the big guys already have their claws deep into the MLS sites and they probably want to closely guard their pipelines. Small guys like me don’t know where to start.
- kQq9oHeAz6wLLS 4y ago
  I do integration work for healthcare; you want an interface engine to pull the data from the ftp site, then translate/convert it into your desired format. We do that all the time.
  - gooseyman 4y ago
    What’s your favorite tool for this? Just starting to get my feet wet in this world.
- smcin 4y ago
  But I want to see Realtor.com vs Zillow numbers (vs MLSListings vs Redfin vs the others...). It's not like any of these are ground truth. How much are the discrepancies, where are they and why? Also, is zipcode the lowest level of granularity (presumably not)? Do any of these services have finer granularity?
_tom_ 4y ago
Or just get an injunction.
- toomuchtodo 4y ago
  Seems like an uphill argument as long as the data isn't behind a login and you're not DDoSing the site. https://news.ycombinator.com/item?id=31075396 https://news.ycombinator.com/item?id=31075396
- deleted 4y ago
  [deleted]

datalopers 4y ago

> they’re not able to keep up with the load Zillow can absolutely keep up with the load. Your proxies are being blocked by the WAFs that Zillow utilizes, usually PerimeterX. Also, based on your other comments, you have no idea how to architect a web application that can handle the trickle of HN traffic, and I'd wager any problems are more likely on your side.

linsomniac 4y ago
I assumed the "they're" referred to the "rotating proxies" not "zillow".
- tndibona 4y ago
  Is correct. The proxy service… it’s slower on the responses from the logs
tndibona 4y ago
I mean, I think I have some idea. The problem is more on the affordability side. I had to scale up the containers in my cluster to cope with the increase in 502s. I live with the fear of a high AWS bill. Then there is the question of the proxy rotators data limit being breached, which happened! I topped it up for now. Now I’d rather spend time asking Zillow to allow me to exist rather than trying to architect a more efficient proxy mechanism.
- datalopers 4y ago
  This site could run on a $5/mo VM without issue. Proxies that charge for bandwidth can get expensive, sure, but that’s a separate issue. Waiting on a response from Zillow/Proxies requires nearly zero cpu on your servers and it should be a background/async process. There’s only ~40k zip codes and only ~10k that remotely matter for even very small towns. Find a shared proxy provider that doesn’t limit bandwidth. It’s usually about $0.50/mo per IP. Not that any of this really matters, you’ll still get blocked by Zillow. It’s not an IP limit l anyway, it’s a TLS fingerprinting or JS-based fingerprinting issue that is getting you blocked.