5 ms·

Personal example. I bought a condo in the University District in Seattle back in 2017. It's a 92-year-old, 478 sq ft condo. I paid $279,000 (yikes, I know)...Zi

Personal example. I bought a condo in the University District in Seattle back in 2017. It's a 92-year-old, 478 sq ft condo. I paid $279,000 (yikes, I know)...Zillow's estimate? $500,000. I have no idea how they were so far off. Even right now: Zillow estimate: $479,100 - https://www.zillow.com/homedetails/905-NE-43rd-St-APT-212-Seattle-WA-98105/2100973940_zpid/ https://www.zillow.com/homedetails/905-NE-43rd-St-APT-212-Se... Redfin: $307,337 - https://www.redfin.com/WA/Seattle/901-NE-43rd-St-98105/unit-212/home/12537835 https://www.redfin.com/WA/Seattle/901-NE-43rd-St-98105/unit-... Keep in mind that Zillow's HQ is also in Seattle. While we may be protective of the data scientists on HN, this is one case where I agree, there's something wrong with their models. The "similar recently sold" condos that Zillow shows are indeed $400K, but those are at most 50 years old and more square footage. I think a simple logistic regression of square footage and age would give a better prediction than whatever model they are using.

aparsons 5y ago
I think part of it has to do with black box models being easier to build incrementally. I’ve seen many teams start off with 3-5 features, which gets you to some threshold for launch (say 90% precision and recall) and get badly burnt on the other 10% post-launch
benrapscallion 5y ago
From personal experience, two major contributors to the Zestimate are: 1. The town/city’s assessment of the home’s value for tax purposes 2. Recent sale prices of homes nearby, mainly as the per square foot price.
- TheRealDunkirk 5y ago
  Another thing I don't understand. How do you develop a model with VOLUMES of real world data to train it on, and not just check how well it predicted pricing every month, both in aggregate (trending prices) and on individual listings that sold that month? This seems like a very small, closed loop to me. You could tweak the heck out of that algorithm in short order based on WHAT ACTUALLY HAPPENED.
  - nuclearnice3 5y ago
    They publish some accuracy data. I think this compares the model to WHAT ACTUALLY HAPPENED https://www.zillow.com/z/zestimate/ https://www.zillow.com/z/zestimate/ Based on this I suspect they do check the model based on WHAT ACTUALLY HAPPENED. what makes you think they don't check the model on WHAT ACTUALLY HAPPENED?
    - TheRealDunkirk 5y ago
      > what makes you think they don't check the model on WHAT ACTUALLY HAPPENED? As I said, it seems like a pretty tight feedback loop to check, and then tweak the model. So it's confusing to me how they could be far enough off to ruin the business plan. I'll put it back to you: What am I missing? How could they revise the model based on many months of real-world data, and yet get it wrong enough to nearly destroy the business?
      - deleted 5y ago
        [deleted]
      - nuclearnice3 5y ago
        Good question. I don’t know. There are a lot of possible interacting causes. Let’s itemize a few. So there is not checking. This seems unlikely given, as you point out, it’s data right at hand and, as I point out, they publish checks. In general, I’m suspicious of WOW PEOPLE ARE STUPID explanations. There is checking and mistranslating model accuracy into business outcomes. The model is 90% accurate. Bid for 1000 houses. Maybe mainly people in the wrong by overpricing segment take you up on it. See the vast literature on adverse selection and winner’s curse. There is tweaking the model because you want the a certain outcome. > Over the last few weeks, it’s become clear that Zillow made a lot of mistakes. It tweaked its algorithm to be more aggressive with offers, winning bidding wars just as the market was starting to cool and overpaying for properties [1] The CEO has commented that the volatility of the prediction was a problem. He mentioned that the unit economics swung from over plus 5% to minus 7%. So the model might be a accurate on average but too noisy for their purpose. [2] Point is it’s not at all obvious that the problem here was accuracy on predictions of house prices. [1] https://www.curbed.com/2021/11/zillows-i-buying-algorithm-house-flipping.html https://www.curbed.com/2021/11/zillows-i-buying-algorithm-ho... [2] https://www.google.com/amp/s/seekingalpha.com/amp/article/4464694-zillow-group-inc-z-ceo-rich-barton-on-q3-2021-results-earnings-call-transcript https://www.google.com/amp/s/seekingalpha.com/amp/article/44...
- annoyingnoob 5y ago
  My neighborhood is a mix of single family homes, condos, and apartments. I live in a single family home. Zillow regularly shows nearby condos as comps - well the condos have always sold for less than the single family homes. The condos also have a high HOA that does not show up in the price. Zillow does not know that there are different kinds of buyers for condos vs single family homes, even when they are close by and a similar size. My neighbor's house is the mirror image of my own, same size, same layout, same size lot, right next door. Zillow shows my home currently at $226K more than the one next door that is the same thing. Zillow's Zestimate of my home is more than a recent appraisal - which is odd because until 2021 Zillow had a really low Zestimate for my home, like the house next door. No idea why Zillow decided on a much higher value recently for me but not for my neighbor. And I'll have to say that lenders do consider the Zestimate as part of a loan, especially in a case where needing an appraisal might not be needed - if it on the bubble then the Zestimate might be the difference of whether you get an appraisal or not.
ianai 5y ago
Right. I read their current actions as the result of a liquidity crunch. Something forced them to realize a 25% reduction in labor and other cuts. They also completely ended this home buying program. They’re hoping they’ve cut enough that when they sell their properties they have enough funds to keep them afloat. If not, chapter 11 filings are not far behind.
- FFRefresh 5y ago
  Chapter 11 filings? Look at their balance sheet, they have over $3B in cash. They will be staying afloat comfortably.
  - ianai 5y ago
    Just some quick figures. They purchased 15k homes in Q2 alone. Debt on 15,000 homes with average purchase price of 350k (as a ballpark figure) comes to 5.25B. There’s a lot of information we don’t have but them laying off huge percentages of staff is a scary indicator. Edit-also note their market cap is about 17B.
    - FFRefresh 5y ago
      Can you point me to the source that says they bought 15k homes in Q2 alone? Zillow says they bought 3,805 homes in Q2: https://s24.q4cdn.com/723050407/files/doc_financials/2021/q2/Zillow-Q2'21-Shareholder-Letter_Exhibit-99.3.pdf https://s24.q4cdn.com/723050407/files/doc_financials/2021/q2...
      - ianai 5y ago
        This might be where google got that figure. Again, as I state above, I was using rough figures to get an idea of the magnitude of the funds they were devoting to their flipping/analysis unit. It looks like they were doing thousands of units instead of tens of thousands with discrepancies among the sources I'm finding in google: https://www.zillow.com/research/zillow-ibuyer-report-q2-2021-30011/ https://www.zillow.com/research/zillow-ibuyer-report-q2-2021... What's not changed is the fact they were leveraging a significant portion of their company's market cap into this segment and it resulted, so far, in them laying off a quarter of their labor and completely terminating the program - amongst other things. This is not good news for their business and prospects of continuing to be an ongoing concern.
manquer 5y ago
Curious why didn't you flip it to Zillow ? If their prices are still not revised even after your buy wouldn't that be a good spread to exploit?
linuxdude314 5y ago
I think you underestimate the market and shortage in supply. I’d be surprised if anyone cared about your condo being older.
rp1 5y ago
A 92-year old house probably is worth more than a 50 year old house, unless you like doing asbestos abatement.
chasely 5y ago
I think (hope?!) that the Zestimate and the model predictions in their iBuying business are completely separate. The Zestimate is essentially a top-of-funnel strategy to get home owners and potential buyers into their normal business platform. For my home, the Zestimate is about 8% higher than Redfin's.

aparsons 5y ago

I think part of it has to do with black box models being easier to build incrementally. I’ve seen many teams start off with 3-5 features, which gets you to some threshold for launch (say 90% precision and recall) and get badly burnt on the other 10% post-launch

benrapscallion 5y ago

From personal experience, two major contributors to the Zestimate are: 1. The town/city’s assessment of the home’s value for tax purposes 2. Recent sale prices of homes nearby, mainly as the per square foot price.

TheRealDunkirk 5y ago
Another thing I don't understand. How do you develop a model with VOLUMES of real world data to train it on, and not just check how well it predicted pricing every month, both in aggregate (trending prices) and on individual listings that sold that month? This seems like a very small, closed loop to me. You could tweak the heck out of that algorithm in short order based on WHAT ACTUALLY HAPPENED.
- nuclearnice3 5y ago
  They publish some accuracy data. I think this compares the model to WHAT ACTUALLY HAPPENED https://www.zillow.com/z/zestimate/ https://www.zillow.com/z/zestimate/ Based on this I suspect they do check the model based on WHAT ACTUALLY HAPPENED. what makes you think they don't check the model on WHAT ACTUALLY HAPPENED?
  - TheRealDunkirk 5y ago
    > what makes you think they don't check the model on WHAT ACTUALLY HAPPENED? As I said, it seems like a pretty tight feedback loop to check, and then tweak the model. So it's confusing to me how they could be far enough off to ruin the business plan. I'll put it back to you: What am I missing? How could they revise the model based on many months of real-world data, and yet get it wrong enough to nearly destroy the business?
    - deleted 5y ago
      [deleted]
    - nuclearnice3 5y ago
      Good question. I don’t know. There are a lot of possible interacting causes. Let’s itemize a few. So there is not checking. This seems unlikely given, as you point out, it’s data right at hand and, as I point out, they publish checks. In general, I’m suspicious of WOW PEOPLE ARE STUPID explanations. There is checking and mistranslating model accuracy into business outcomes. The model is 90% accurate. Bid for 1000 houses. Maybe mainly people in the wrong by overpricing segment take you up on it. See the vast literature on adverse selection and winner’s curse. There is tweaking the model because you want the a certain outcome. > Over the last few weeks, it’s become clear that Zillow made a lot of mistakes. It tweaked its algorithm to be more aggressive with offers, winning bidding wars just as the market was starting to cool and overpaying for properties [1] The CEO has commented that the volatility of the prediction was a problem. He mentioned that the unit economics swung from over plus 5% to minus 7%. So the model might be a accurate on average but too noisy for their purpose. [2] Point is it’s not at all obvious that the problem here was accuracy on predictions of house prices. [1] https://www.curbed.com/2021/11/zillows-i-buying-algorithm-house-flipping.html https://www.curbed.com/2021/11/zillows-i-buying-algorithm-ho... [2] https://www.google.com/amp/s/seekingalpha.com/amp/article/4464694-zillow-group-inc-z-ceo-rich-barton-on-q3-2021-results-earnings-call-transcript https://www.google.com/amp/s/seekingalpha.com/amp/article/44...
annoyingnoob 5y ago
My neighborhood is a mix of single family homes, condos, and apartments. I live in a single family home. Zillow regularly shows nearby condos as comps - well the condos have always sold for less than the single family homes. The condos also have a high HOA that does not show up in the price. Zillow does not know that there are different kinds of buyers for condos vs single family homes, even when they are close by and a similar size. My neighbor's house is the mirror image of my own, same size, same layout, same size lot, right next door. Zillow shows my home currently at $226K more than the one next door that is the same thing. Zillow's Zestimate of my home is more than a recent appraisal - which is odd because until 2021 Zillow had a really low Zestimate for my home, like the house next door. No idea why Zillow decided on a much higher value recently for me but not for my neighbor. And I'll have to say that lenders do consider the Zestimate as part of a loan, especially in a case where needing an appraisal might not be needed - if it on the bubble then the Zestimate might be the difference of whether you get an appraisal or not.

ianai 5y ago

Right. I read their current actions as the result of a liquidity crunch. Something forced them to realize a 25% reduction in labor and other cuts. They also completely ended this home buying program. They’re hoping they’ve cut enough that when they sell their properties they have enough funds to keep them afloat. If not, chapter 11 filings are not far behind.

FFRefresh 5y ago
Chapter 11 filings? Look at their balance sheet, they have over $3B in cash. They will be staying afloat comfortably.
- ianai 5y ago
  Just some quick figures. They purchased 15k homes in Q2 alone. Debt on 15,000 homes with average purchase price of 350k (as a ballpark figure) comes to 5.25B. There’s a lot of information we don’t have but them laying off huge percentages of staff is a scary indicator. Edit-also note their market cap is about 17B.
  - FFRefresh 5y ago
    Can you point me to the source that says they bought 15k homes in Q2 alone? Zillow says they bought 3,805 homes in Q2: https://s24.q4cdn.com/723050407/files/doc_financials/2021/q2/Zillow-Q2'21-Shareholder-Letter_Exhibit-99.3.pdf https://s24.q4cdn.com/723050407/files/doc_financials/2021/q2...
    - ianai 5y ago
      This might be where google got that figure. Again, as I state above, I was using rough figures to get an idea of the magnitude of the funds they were devoting to their flipping/analysis unit. It looks like they were doing thousands of units instead of tens of thousands with discrepancies among the sources I'm finding in google: https://www.zillow.com/research/zillow-ibuyer-report-q2-2021-30011/ https://www.zillow.com/research/zillow-ibuyer-report-q2-2021... What's not changed is the fact they were leveraging a significant portion of their company's market cap into this segment and it resulted, so far, in them laying off a quarter of their labor and completely terminating the program - amongst other things. This is not good news for their business and prospects of continuing to be an ongoing concern.

manquer 5y ago

Curious why didn't you flip it to Zillow ? If their prices are still not revised even after your buy wouldn't that be a good spread to exploit?

linuxdude314 5y ago

I think you underestimate the market and shortage in supply. I’d be surprised if anyone cared about your condo being older.

rp1 5y ago

A 92-year old house probably is worth more than a 50 year old house, unless you like doing asbestos abatement.

chasely 5y ago

I think (hope?!) that the Zestimate and the model predictions in their iBuying business are completely separate. The Zestimate is essentially a top-of-funnel strategy to get home owners and potential buyers into their normal business platform. For my home, the Zestimate is about 8% higher than Redfin's.