14 ms·
Show HN: Property Trends Scraped from Zillow
- jeffbee 4y agoHow is this different from Redfin's sale-to-list data, and why do people use Zillow instead of Redfin when superficially at least Zillow appears to be wrong about everything?
- tndibona 4y agoI’m hoping the message and presentation makes the difference. It was hard for me to follow the multi coloured ratio lines on the graph.
- guimonz 4y agoUseful tool! One suggestion is that 30 days is not enough for sparsely populated areas. Maybe add the option for 3 months?
- tndibona 4y agoMy project will average out the data on recent "sale-price over-or-under the asking-price" for an entire zipcode. This should produce a report showing you how things are trending. I hope this will help people make an informed decision. Caveats : 1.This works for the USA only. 2.The report should take up to a minute to display as the data is being scraped from Zillow. 3.If the site does go down, it is due to the volume of traffic from reddit. Then I'm going to need to scale up my servers. I will try and stay on top of this. 4.If the no results are obtained, Zillow probably didn't have "sales" data for that zip code, or the connection to Zillow failed. I would just retry once again. I'm looking for feedback on these points: 1. How do I monatise without adverts? I would hate to display ads and kill the experience. 2. Do you want similar trends? Eg. I could also show "Average days in the market for properties", "Similar zip codes for your budget." etc 3. Would you be willing to register and voluntarily give up some data ? I would like to know where you are searching from, it will help me build a database of "Where are people from and where are they looking to move?" 4. Any thing that will help me improve is good.
- kQq9oHeAz6wLLS 4y ago> 1. How do I monatise without adverts? I would hate to display ads and kill the experience. Well, selling it to Zillow is probably a natural fit. Or some other realtor org, but you'd probably have to change the source of your data.
- linsomniac 4y agoCan you recommend a zip code to try, I tried several in Northern Colorado and got "We did not get any results". One way you could monetize would be to make deals with MLSes for integration into their services, either for realtors or for consumers. Can't tell how appropriate it would be for realtors, I assume it's more oriented towards consumers. If there's some way to get to see Northern Colorado there, I can send it around to some POI in the local MLS and see what they think. Again, you might not be looking for MLS feedback. Disclaimer: I run coloproperty.com and the MLS. :-)
- tndibona 4y agoTry 19130
- melenaboija 4y agoNot working for me
- tndibona 4y agoCould you try now? I made hacker news front page so my site is getting hit with traffic and the proxy service data limit maxed out. I’ve just topped up the account.
- melenaboija 4y agoIt works now
- tndibona 4y ago
- jschveibinz 4y agoI tried a couple of zip codes and it works pretty well. There were a couple of glitches in rendering the results, but I like it. It’s a useful tool.
- jareklupinski 4y agoeager to try, tried a couple places in NYC, got "Oops! We did not get any results" :(
- tndibona 4y agoTry 19130
- leobg 4y agoWhy not add this as placeholder value directly to your search input field? “Try 19130”. For those who come to the site without reading the comments here.
- tndibona 4y agoGood suggestion. Next iteration will surely have this
- shmoogy 4y agoNext next iteration maybe if no results show a similar one that's cached for you so there's no unavailable requests for a valid zip? And/or storing stale data with a last accessed date range. I wish you the best of luck getting the blessing of MLS, I had an idea for something similar and couldn't get anything useful. Also I think Zillow is litigious as part of their agreement to have access to the data (and proprietary reasons obviously)
- rnk 4y agoI think they got overloaded or too many queries on their backend. No results for several different zip codes in my big city.
- jareklupinski 4y agosame. I think today, OP learns about caching ;)
- linsomniac 4y agoCan't speak for Zillow, but some API endpoints don't allow you to cache results you obtain from them, license-wise.
- tndibona 4y agoAPI ? You’re funny, I’ve been begging Zillow for their API and no response. I’m scraping using html parsers
- tndibona 4y agoAlready using redis for the cache. I’m using a rotating proxy to bypass Zillow captcha, they’re not able to keep up with the load
- dylan604 4y agoIt'll be sad if they figure out who you and your bots are, and then deny you specifically access. Got backup plans for when you get zero results for every query because of that?
- tndibona 4y agoYes lol. Realtor.com is also scrapable so that’s the immediate backup. Long term I have to speak FTP csv files from a bunch of archaic MLS integrations. If I need to go that route, I need money.
- patothon 4y ago"over the asking price" is a useless metric, might be more interesting to elevate $/sqft.
- tndibona 4y agoNoted
- sellmethepen 4y agoI would disagree with this statement. It's really interesting in markets like the bay area from an individual consumer perspective
- patothon 4y agowhy though? genuine question. sometimes sellers over or under estimate their home. Over asking price doesn't indicate how a market is doing. It indicates other things: are sellers under estimating their home to start bidding wars? how delusional some sellers are on the price of their home? etc. These are interesting things to know, but as a buyer, I'm more interested in putting the right offer at the right time, and there's no better indicator than $/sqft for a specific location. Of course it varies (how many bedrooms? how many bathrooms? other cool amenities?) but this could be a subfilter of this product.
- carfacts 4y agoLooks similar to RealtorStats.org
- chologrande 4y agoDo you intend to make the dataset available? I've attempted to do something similar, but had difficulty, would love access to the dataset.
- tndibona 4y agoPlease reach out “info at pillr.io” . FYI, I’m a single person, not a big company.
- Panther34543 4y agoWhat is the legality of scraping those sites?
- tndibona 4y agoAfaik, US Supreme Court has ruled that it is not illegal to scrape public data for a similar case with LinkedIn. As for me being a pain for Zillows bandwidth - I’ve requested them umpteen times to allow me to pay for their API. They just simple ignore me. I’ve found a way around their captcha, in the long term, I have to obtain the data directly from the MLSes
- skanga 4y agoWould you be willing to share how you get around their captcha?
- wraptile 4y agoIt's perfectly legal! That being said, you still need to be resource-polite. A lot of people scrape Zillow through browser automation toolkits like Selenium, Puppeteer etc. because it's a JS heavy website and these tools are really bandwidth intensive. This could, in theory, get you in trouble for DDOS. Instead, since Zillow is using Next.js for their backend so, you can actually retrieve the dataset for any page just by parsing the nextjs cache. This can be done by selecting data in the <script id="__NEXT_DATA__"> node which requires minimal resources from both sides. e.g. in python: import json import httpx from parsel import Selector response = httpx.get("https://www.zillow.com/b/1625-e-13th-st-brooklyn-ny-5YGKWY/") script_data = Selector(text=response.text).css('#__NEXT_DATA__').get() script_data = json.loads(script_data) # all of the property data is here, for example building details: print(script_data['building']) I wrote a tutorial on this if you'd like to learn more: https://scrapfly.io/blog/how-to-scrape-zillow/#scraping-properties https://scrapfly.io/blog/how-to-scrape-zillow/#scraping-prop...
- skanga 4y agoOutstanding tip. Thank you VERY much.
- km3r 4y agoCould you add inputing multiple ZIP codes (or just a city)? The data for just one seems too noisy once you start playing with the slider.
- tndibona 4y agoGood suggestion. Noted for future work
- xnx 4y agoGood luck with this. Scraping may be legal, but redistributing the data probably isn't. I expect the real estate cartel protects their MLS data pretty aggressively, otherwise there would be dozens of sites like this already. https://apnews.com/article/technology-lifestyle-business-professional-services-residential-real-estate-6a2a3214a51b7ea899059b45afd93837 https://apnews.com/article/technology-lifestyle-business-pro...
- tndibona 4y agoThanks for your kind words
- tndibona 4y agoRead the whole article. How can I compete with a cartel? I’m doomed. Maybe take this idea as-is and switch countries to the British market is also an idea I’m thinking
- dclowd9901 4y agoHomeowners who are selling don't want their data locked up. This should be a class-action suit. I bet there are a dozen attorneys who'd take on this case. You may not make out personally, but a condition can be the opening of listing data. Its locked nature should be considered problematic at best for fair housing rules.
- Aromasin 4y agoThis sort of data is reasonably accessible in the UK, although perhaps not quite to the same degree as your app. Either way, I suspect the value-add is less than it would be in the US. Rightmove, the primary property app in the UK, gives you the previous price the properties were purchased for, although perhaps not analytics across an area per se.
- tndibona 4y agoDo you think region wide trends would be well received by the english public? I'm Exploring similar needs in other countries where data is a more readily available.
- tedd4u 4y agoRedfin has this and quite a lot more on their Market Insights feature. https://www.redfin.com/zipcode/<ZIP_CODE>/housing-market For example: https://www.redfin.com/zipcode/95050/housing-market https://www.redfin.com/zipcode/95050/housing-market
- tndibona 4y agoYep I’m aware of this but I’m trying to beat them with the simple UI and straight forward messaging.
- deleted 4y ago[deleted]
- confounded 4y agoThe question is, after this superb yak shaving, did you actually buy a house?
- datalopers 4y agoZillow publicizes a higher granularity (~90 MSAs and weekly) of this data at https://www.zillow.com/research/data/ https://www.zillow.com/research/data/ > Median List Price: The median price at which homes across various geographies were listed. > Median Sale Price: The median price at which homes across various geographies were sold. > Sale-to-List Ratio (mean/median): Ratio of sale vs. final list price. > Percent of Sales under/over List: Ratio of sales where Sale Price below/above the final list price; excludes homes sold for exactly the list price.
- tndibona 4y agoHey thanks for this. Im looking at this. But I’d still have to download spreadsheets and the data is aggregated by regions bigger than zipcodes I think. Im hoping my site offers a simple and straightforward experience
- bmelton 4y agoZillow's data packaging is undoubtedly better than ours (analytics isn't yet very core to our business) but if you'd like to see rent values statically associated to MSAs and ZIP codes on a map, we publish that at https://dwellsy.com/rentmaps https://dwellsy.com/rentmaps
- malshe 4y agoWow, this is really nicely made!
- tndibona 4y agoHello there. OP here! your site is similar. The UI is better than mine. Can we exchange notes?
- bmelton 4y agoGladly. My email is in my profile.
- breckenedge 4y agoThis could really use filters for # rooms and # baths.
- knicholes 4y ago"Generating the report for 84058, this will take a while!" contains two independent clauses. They should not be separated by a comma. They should be separated by a period. It should read "Generating the report for 84058. This will take a while!"
- tndibona 4y agoThanks. I’ll fix it.
- mixmastamyk 4y agoNeat, just tried it. I think the daily trend is too granular. Twice a month or maybe monthly would show a stronger trend. As I already knew, houses in this zip often selling $300k above list, some are pushing $800k. Yea inflation! cough
- tndibona 4y agoThanks for your words!
- eek2121 4y agoYep, site claims 107% over list price. It tracks for Nashville. Love this site.
- enonevets 4y agoIt seems the only filter is by square footage to narrow results. Are there plans to filter by more params?
- tndibona 4y agoI can do a lot more. But this whole thing is a house of cards because I’m relying on Zillow who can shut me down anytime. I need one of Zillow/ RedFin / MLS companies to agree to allow me to formally use their data. Then, I can start building the good filters and fancy dashboard. Getting their attention has been futile so far. Just cold email responses at best or no response at all.
- entwife 4y agoZillow published a sales price for my house that was 33% greater than the actual sale price. I don't trust Zillow's price data.
- zxcvbn4038 4y agoVery nice! Properties in the area I’m interested in are now selling for 7% below asking price whereas a year ago they were getting 30% above asking price. On a different site, several properties that I saw sell in the past few months have relisted. It’s obvious they overbought or were speculating and are trying to get out now. I can’t wait for the business study of how much wealth the Property Brothers end up destroying - should be much more interesting then the story of the guy who stopped putting olives on the airline salads.
- tndibona 4y agoI’m glad this project helped!
- BigBenBongs 4y ago
- victorbstan 4y agoCan you do by country and zip code? Would like to see Canada there.
- tndibona 4y agoThanks for the request. Could you give me the best sources in Canada similar to zillow? I can try and explore if I can scrape it.
- snapcaster 4y agoThis is super cool
- tndibona 4y ago(Saying this in the voice of Austin Powers) - Athankyu
- uptown 4y agoYou should add a column to show the closing price difference as a percentage of the asking price and include this in the summary value statement at the top of the page.
- tndibona 4y agoI have a toggle button to switch between "dollars" and "percentage". Does it not do what you're saying?
- jchonphoenix 4y agoJust an FYI, Zillow is litigious and your entire service goes against their ToS. They will sue if this takes off and doesn't help them to somehow.
- huitzitziltzin 4y agoContext for that: Zillow is not the original owner of the data. The data is owned by the several hundred regional MLS’s (note the watermarks in any picture you see on the site). Data use agreements with those organizations require that anyone using it make efforts to prevent its being scraped. Source: have signed a DUA with an MLS myself for academic research.
- tndibona 4y agoOP Here! Good insight. Do you have contacts with the MLS? I'd like to use them instead of relying on Zillow. How can I contact you ?
- tndibona 4y agoI have reached Hacker news front page which is a lifetime achievement for me. I just hope someone from Zillow takes notice and has a constructive conversation with me.
- deleted 4y ago[deleted]