6 ms·

This post strikes me as empty bluster. It basically says "the data scientists didn't consider the possibility that the model might fail to predict the real worl

This post strikes me as empty bluster. It basically says "the data scientists didn't consider the possibility that the model might fail to predict the real world." Which is like looking at the Boeing 737 MAX debacle and saying "they didn't consider the possibility that the plane might not be stable". This is most likely both untrue and lacking insight. What almost certainly happened was they validated their model extensively, but overlooked the angle relevant to this use case. Or overexcited executives failed to heed the caveats of the nerds. > When I advise Data Scientists to understand Topology and Differential Geometry, this is why. Oh really? I've taken grad-level courses in those fields. It's nice general education but has nothing to do with this situation. Knowing about the Lucas critique from economics would be much more relevant. My pet theory -- completely unvalidated but at least potentially insightful -- is that they failed to put in adequate margin for adverse selection: homeowners whose homes are worth less than they seem on paper were much more likely to take the Zillow offer. Adverse selection turns prediction variance, which you might naively expect to "average out" when using a well-calibrated model at scale, into a bias towards underperformance. Another way to put it: asymmetric information. Be very careful betting against homeowners who probably know much more than you do. Or read this comment, which puts it better than I did: https://news.ycombinator.com/item?id=29140104 https://news.ycombinator.com/item?id=29140104

88913527 5y ago
What if the model said there were only $100m in homes that were viable, but executives wanted $1 billion in purchases? They could have had an alright model, but the volume wouldn't sustain the purchasing quantities leadership wanted. So, they relax the model to accept lower quality results. It's another form of executives attempting to squeeze blood from a stone. It just doesn't work, but it's a common strategy. They'll ask simple questions like "Why can't you just do X?", you advise them not to, and they ignore you.
- kshacker 5y ago
  Basically the equivalent of 2008 no docs no income market for housing loans.
- pc86 5y ago
  Do you have any evidence that this is what happened?
deleted 5y ago
[deleted]
colincooke 5y ago
I totally agree, this is more of a case of simple over-confidence and exessive risk taking than lack of mathematical understanding. Regardless of the quality of the model you are using there are still countless other factors that may impact your prediction quality. To me, the only way to mitigate this is to take it slow and plan ways for failure correction. Zillow is an example of greed and impatiance winning over common sense and risk-mitigation. If anything this could have been because of _too much_ higher level statistics and math, as those kinds of things can often shroud underlying problems.
ethbr0 5y ago
> What almost certainly happened was they validated their model extensively, but overlooked the angle relevant to this use case. To me, this is a systemic organizational problem in ML, rather than a model problem. Or essentially, pre-Agile development all over again. Stop me if this sequence sounds familiar: (1) attempt to capture all the details in a nuanced domain that's foreign to you, from people who are experts but not expert developers, (2) develop code from that spec, (3) deliver code to the experts as a black box that they can't opine on, (4) cross fingers and run in production. Which isn't to say this is the way everyone does it, but is to say it will be the way the majority does it, just as the majority of development is still waterfall under the hood. But the mismatch is a mismatch in understanding, not in correctness. The experts can't offer meaningful feedback to the technological black box, and developers don't have enough time to spec a lifetime of subject matter experience.
- ItsMonkk 5y ago
  Relevant xkcd: Here to Help[0]. If you don't have the minimum skills on the team, you're going to fail in ways that you don't see coming, the unknown unknowns. To spend billions when a former Zillow employee says that they have no finance people on the team, is just, drastically, wow. I would much sooner believe he is mistaken then believe that is what actually happened. [0]: https://xkcd.com/1831 https://xkcd.com/1831
- RandomLensman 5y ago
  Not sure how they could really validate their models given that "market making" in houses is probably really thin - if at all. Finding alpha in financial markets is just tough (real assets or not), and real estate also needs a lot of capital, which makes "waiting it out" tough
  - JackFr 5y ago
    I can make a market by quoting a bid and ask. To run a market making business I have to know how to hedge my net position and manage my cost of carry. I find it crazy hard to believe that they just went long a few billion in houses just based on a model of price and failed to hedge vs. the secular market and didn’t model time to sale. But if you believe the popular press maybe that’s what they did.
curiousgal 5y ago
> This post strikes me as empty bluster. That's because it is. A common characteristic of almost all blogs run by consultants. The only exception that jumps to mind is John Cook's [0]. 0.https://www.johndcook.com/blog/ https://www.johndcook.com/blog/
- hyperbovine 5y ago
  I think John cook has hit on a brilliant marketing strategy : write mostly about basic, upper-division probability and statistical concepts (e.g., the tail behavior of the gamma distribution, or a list of conjugate distributions), interspersed with the occasional non-controversial polemic (“don’t invert that matrix”). To the uninformed—including his potential clients—he looks like some sort of genius, and for the rest of us, it’s a great reference.
bko 5y ago
I think they failed to consider adverse selection. For instance, if I'm selling my home, I will check Zillow instant offer and try the traditional market. The Zillow offer will be my floor plus some value for convenience. So they will likely be overbidding. The other thing is that running a website and being a data aggregator is an entirely different business than speculating on houses. Matt Levine pointed this out last week [0], but its not making markets if the turnover takes 6 months. [0] https://www.bloomberg.com/opinion/articles/2021-11-03/zillow-is-done-trading-houses https://www.bloomberg.com/opinion/articles/2021-11-03/zillow...
deleted 5y ago
[deleted]
paganel 5y ago
> Knowing about the Lucas critique from economics would be much more relevant “The Market for Lemons: Quality Uncertainty and the Market Mechanism” by George Akerlof is also a nice related resource [1] > widely-cited[1] 1970 paper by economist George Akerlof which examines how the quality of goods traded in a market can degrade in the presence of information asymmetry between buyers and sellers, leaving only "lemons" behind. [1] https://wikipedia.org/wiki/The_Market_for_Lemons https://wikipedia.org/wiki/The_Market_for_Lemons
- rdtwo 5y ago
  This was ebay now is Amazon
lordnacho 5y ago
I'm a quant trader and I agree. There's a big difference between being able to predict what will happen, and having a sensible strategy for how to behave. One notable place this happens is backtesting. Particularly for passive strategies, you don't actually know from historical data what would have happened if you had been in the market at a given time. Would your prices move the market? Maybe. Would you get filled disproportionately when it's bad to get filled? Fair chance of it. So don't act like your presence would not be felt. Another angle is risk management. You may have high confidence in your model, but you should have a plan for what to do if things don't go as expected. It seems crazy to me that it could get so far out of hand that a quarter of the company gets sacked. Why not set a budget and dip your toes? Then you can watch out for the adverse selection issues as you grow the portfolio and make adjustments. Finally, it's very normal for quant strategies to look good when they're first proposed, then look a bit worse when traded in real life, and perhaps then degrade over time. Another reason to take it easy when sizing the bets. You're right about Lucas critique. It's fundamental that models affect the thing they're modelling.
- treis 5y ago
  The big difference here is that the counterparty is deciding which orders get filled. Zillow could be right 99% of the time, but that 1% of the time where they're overpaying are the transactions that for sure will go through. And indeed if you go to /r/realestate anecdotally that is what happened. People would get offers from Zillow/Opendoor etc and talk to agents to see what they would list it at. If Zillow was higher than what the agents said they went with Zillow. If not they'd list it and sell it the old fashion way.
  - lordnacho 5y ago
    Yeah, it's remarkable. You'd think that "I'll buy your house immediately" should be at a worse price than the guy who insists on the old school inspection and due diligence. In fact it might not be a bad strategy to simply start with that... see if there's desperate people needing the money fast, and use your corporate heft to both sweep up the deal (brand) and patiently dump it back into the ordinary market, pocketing the spread as a liquidity premium. One thing I remember my first boss saying is that it's hard to get a good trade on. Attractive deals have several interested parties. If you're getting a lot of trades, they might not be good ones.
KarlKemp 5y ago
> overexcited executives failed to heed the caveats of the nerd Yes, the nerds are so well-known to consider the risks of what they create. If only they’d thought of the risk that someone could use their work in the real world, even though it came with specific instructions not to do that. Maybe they wouldn’t create possibly-destructive ML models then. Alas, it remains a mystery what a private company might do after paying a lot of money for something. Now where was I? Ah, right! Predicting criminality from baby photos! I love how DARPA sponsors basic research with absolutely no expectation of ever using it. Although I‘d better name it README.MD, just to make sure they do not forget to read it.
- civilized 5y ago
  Technology has dual uses. Kitchen knives can be used to kill people, so should they never have been invented? You can take this line of reasoning very, very far. Some people think we never should have left our hunter-gatherer lifestyle. To these people, the agricultural revolution was a disaster and things have only gotten worse from there. It's a tremendously ignorant viewpoint IMO, but it's where you land if you believe technology with the potential for bad use shouldn't exist in general.
- solveit 5y ago
  Yes, at every startup I was in, the stats nerds were in fact well-known to tell the nontechnical people and even the devs that what they were doing wasn't going to work, or rather that we didn't have good evidence that it would work. This is such a well-known feature of stats nerds that it's a meme.
FFRefresh 5y ago
As someone who doesn't work at Zillow but follows the space a little closer, don't buy into the superficial explanation/meme going around that they failed because of ML. My advice would be to read the earnings call transcripts over the last couple of years. What Zillow has communicated in the last earnings call is fairly contradictory to what they've been saying in previous calls. Some bullet points on my understanding of what went wrong: -They were paying way too much for too many homes. I believe this is ultimately due to internal incentives and ambitious goal setting. IIRC, they mentioned they had to alter their model output to be able to give themselves the permission to keep buying at high volumes. They lacked fiscal discipline here. -The holding times on these homes were too long, which increased their costs. My impression is that this was due to lack of operational experience with renovating and selling homes. Zillow was new to this, and they were trying to scale up very quickly, independent of whether they were efficient. -The core business model of Zillow historically has been as an advertising site/lead gen site. This business model is not in their DNA and their operational expertise. The negative margin pressure from this business in tougher times can really stress the overall business at times. Look at the other iBuyers in the space - they are still in the space. They just aren't as naive as Zillow was. Zillow's behavior here is really confusing and speaks to poor operations/strategy/discipline. Housing prices have been mostly been only going up in this market and Zillow was losing their shirt.
- civilized 5y ago
  Well, their strategy of blaming the nerds will at least ensure that they never have the opportunity to bet big on ML again. No one competent will want to work for them now.
  - wussboy 5y ago
    I wish this was true but I suspect it is not. If Facebook can still have excellent engineers because they pay a ton of money, then so can Zillow.
    - ivanyahoocom 5y ago
      Zillow has always paid low compared to the market. They never got the “top nerds” and this was really a CEO problem. I would never bet on that company on any decently complex technology like predictive models. It took them years to make their model decent, something that a machine learning tutorial does better - just a linear regression on square feet and location. They were literally comparing my house to ones on the other side of the city - comparing a downtown property with outskirts.
JumpCrisscross 5y ago
Their CEO also seems to have fundamentally misunderstood what he was doing. He described their activity as market making. That’s wrong. Market makers attempt to buy and sell instantaneously without changing the asset. Zillow had no intention of simultaneous purchase and sale. They also modified the asset. They used a market making model glued to a real estate speculating one to run a house-flipping operation at scale and from a distance. Instead of focussing on sellers who would sell at a discount for immediacy, they paid over market. There was no effort put into pre-selling or hedging. (The latter would have been so possible as a security against their entire portfolio.)
- rdtwo 5y ago
  A market making model would actually be interesting. Setup a potential seller with a buyer beforehand. Then when the seller buys the new home both transactions occur together and you get 2 sales instead of one
omegalulw 5y ago
> is that they failed to put in adequate margin for adverse selection. Adverse selection turns prediction variance, which you might naively expect to "average out" when using a well-calibrated model at scale, into a bias towards underperformance. Presumably, they trusted that their model does a more accurate assessment of price and future potential than a person themselves. If you had an excellent model, why _not_ would it balance out? My theory is that most people would consult various sources and professionals-who-assess-homes (whatever they are called) and this the likelihood of Zillow having a better assessment than the average person is just low.
- fuzzfactor 5y ago
  Trained by all kinds of nominally relevant data. Makes nominal decisions with at best somewhat nominal ability.

88913527 5y ago

What if the model said there were only $100m in homes that were viable, but executives wanted $1 billion in purchases? They could have had an alright model, but the volume wouldn't sustain the purchasing quantities leadership wanted. So, they relax the model to accept lower quality results. It's another form of executives attempting to squeeze blood from a stone. It just doesn't work, but it's a common strategy. They'll ask simple questions like "Why can't you just do X?", you advise them not to, and they ignore you.

kshacker 5y ago
Basically the equivalent of 2008 no docs no income market for housing loans.
pc86 5y ago
Do you have any evidence that this is what happened?

deleted 5y ago

[deleted]

colincooke 5y ago

I totally agree, this is more of a case of simple over-confidence and exessive risk taking than lack of mathematical understanding. Regardless of the quality of the model you are using there are still countless other factors that may impact your prediction quality. To me, the only way to mitigate this is to take it slow and plan ways for failure correction. Zillow is an example of greed and impatiance winning over common sense and risk-mitigation. If anything this could have been because of _too much_ higher level statistics and math, as those kinds of things can often shroud underlying problems.

ethbr0 5y ago

> What almost certainly happened was they validated their model extensively, but overlooked the angle relevant to this use case. To me, this is a systemic organizational problem in ML, rather than a model problem. Or essentially, pre-Agile development all over again. Stop me if this sequence sounds familiar: (1) attempt to capture all the details in a nuanced domain that's foreign to you, from people who are experts but not expert developers, (2) develop code from that spec, (3) deliver code to the experts as a black box that they can't opine on, (4) cross fingers and run in production. Which isn't to say this is the way everyone does it, but is to say it will be the way the majority does it, just as the majority of development is still waterfall under the hood. But the mismatch is a mismatch in understanding, not in correctness. The experts can't offer meaningful feedback to the technological black box, and developers don't have enough time to spec a lifetime of subject matter experience.

ItsMonkk 5y ago
Relevant xkcd: Here to Help[0]. If you don't have the minimum skills on the team, you're going to fail in ways that you don't see coming, the unknown unknowns. To spend billions when a former Zillow employee says that they have no finance people on the team, is just, drastically, wow. I would much sooner believe he is mistaken then believe that is what actually happened. [0]: https://xkcd.com/1831 https://xkcd.com/1831
RandomLensman 5y ago
Not sure how they could really validate their models given that "market making" in houses is probably really thin - if at all. Finding alpha in financial markets is just tough (real assets or not), and real estate also needs a lot of capital, which makes "waiting it out" tough
- JackFr 5y ago
  I can make a market by quoting a bid and ask. To run a market making business I have to know how to hedge my net position and manage my cost of carry. I find it crazy hard to believe that they just went long a few billion in houses just based on a model of price and failed to hedge vs. the secular market and didn’t model time to sale. But if you believe the popular press maybe that’s what they did.

curiousgal 5y ago

> This post strikes me as empty bluster. That's because it is. A common characteristic of almost all blogs run by consultants. The only exception that jumps to mind is John Cook's [0]. 0.https://www.johndcook.com/blog/ https://www.johndcook.com/blog/

hyperbovine 5y ago
I think John cook has hit on a brilliant marketing strategy : write mostly about basic, upper-division probability and statistical concepts (e.g., the tail behavior of the gamma distribution, or a list of conjugate distributions), interspersed with the occasional non-controversial polemic (“don’t invert that matrix”). To the uninformed—including his potential clients—he looks like some sort of genius, and for the rest of us, it’s a great reference.

bko 5y ago

I think they failed to consider adverse selection. For instance, if I'm selling my home, I will check Zillow instant offer and try the traditional market. The Zillow offer will be my floor plus some value for convenience. So they will likely be overbidding. The other thing is that running a website and being a data aggregator is an entirely different business than speculating on houses. Matt Levine pointed this out last week [0], but its not making markets if the turnover takes 6 months. [0] https://www.bloomberg.com/opinion/articles/2021-11-03/zillow-is-done-trading-houses https://www.bloomberg.com/opinion/articles/2021-11-03/zillow...

deleted 5y ago

[deleted]

paganel 5y ago

> Knowing about the Lucas critique from economics would be much more relevant “The Market for Lemons: Quality Uncertainty and the Market Mechanism” by George Akerlof is also a nice related resource [1] > widely-cited[1] 1970 paper by economist George Akerlof which examines how the quality of goods traded in a market can degrade in the presence of information asymmetry between buyers and sellers, leaving only "lemons" behind. [1] https://wikipedia.org/wiki/The_Market_for_Lemons https://wikipedia.org/wiki/The_Market_for_Lemons

rdtwo 5y ago
This was ebay now is Amazon

lordnacho 5y ago

I'm a quant trader and I agree. There's a big difference between being able to predict what will happen, and having a sensible strategy for how to behave. One notable place this happens is backtesting. Particularly for passive strategies, you don't actually know from historical data what would have happened if you had been in the market at a given time. Would your prices move the market? Maybe. Would you get filled disproportionately when it's bad to get filled? Fair chance of it. So don't act like your presence would not be felt. Another angle is risk management. You may have high confidence in your model, but you should have a plan for what to do if things don't go as expected. It seems crazy to me that it could get so far out of hand that a quarter of the company gets sacked. Why not set a budget and dip your toes? Then you can watch out for the adverse selection issues as you grow the portfolio and make adjustments. Finally, it's very normal for quant strategies to look good when they're first proposed, then look a bit worse when traded in real life, and perhaps then degrade over time. Another reason to take it easy when sizing the bets. You're right about Lucas critique. It's fundamental that models affect the thing they're modelling.

treis 5y ago
The big difference here is that the counterparty is deciding which orders get filled. Zillow could be right 99% of the time, but that 1% of the time where they're overpaying are the transactions that for sure will go through. And indeed if you go to /r/realestate anecdotally that is what happened. People would get offers from Zillow/Opendoor etc and talk to agents to see what they would list it at. If Zillow was higher than what the agents said they went with Zillow. If not they'd list it and sell it the old fashion way.
- lordnacho 5y ago
  Yeah, it's remarkable. You'd think that "I'll buy your house immediately" should be at a worse price than the guy who insists on the old school inspection and due diligence. In fact it might not be a bad strategy to simply start with that... see if there's desperate people needing the money fast, and use your corporate heft to both sweep up the deal (brand) and patiently dump it back into the ordinary market, pocketing the spread as a liquidity premium. One thing I remember my first boss saying is that it's hard to get a good trade on. Attractive deals have several interested parties. If you're getting a lot of trades, they might not be good ones.

KarlKemp 5y ago

> overexcited executives failed to heed the caveats of the nerd Yes, the nerds are so well-known to consider the risks of what they create. If only they’d thought of the risk that someone could use their work in the real world, even though it came with specific instructions not to do that. Maybe they wouldn’t create possibly-destructive ML models then. Alas, it remains a mystery what a private company might do after paying a lot of money for something. Now where was I? Ah, right! Predicting criminality from baby photos! I love how DARPA sponsors basic research with absolutely no expectation of ever using it. Although I‘d better name it README.MD, just to make sure they do not forget to read it.

civilized 5y ago
Technology has dual uses. Kitchen knives can be used to kill people, so should they never have been invented? You can take this line of reasoning very, very far. Some people think we never should have left our hunter-gatherer lifestyle. To these people, the agricultural revolution was a disaster and things have only gotten worse from there. It's a tremendously ignorant viewpoint IMO, but it's where you land if you believe technology with the potential for bad use shouldn't exist in general.
solveit 5y ago
Yes, at every startup I was in, the stats nerds were in fact well-known to tell the nontechnical people and even the devs that what they were doing wasn't going to work, or rather that we didn't have good evidence that it would work. This is such a well-known feature of stats nerds that it's a meme.

FFRefresh 5y ago

As someone who doesn't work at Zillow but follows the space a little closer, don't buy into the superficial explanation/meme going around that they failed because of ML. My advice would be to read the earnings call transcripts over the last couple of years. What Zillow has communicated in the last earnings call is fairly contradictory to what they've been saying in previous calls. Some bullet points on my understanding of what went wrong: -They were paying way too much for too many homes. I believe this is ultimately due to internal incentives and ambitious goal setting. IIRC, they mentioned they had to alter their model output to be able to give themselves the permission to keep buying at high volumes. They lacked fiscal discipline here. -The holding times on these homes were too long, which increased their costs. My impression is that this was due to lack of operational experience with renovating and selling homes. Zillow was new to this, and they were trying to scale up very quickly, independent of whether they were efficient. -The core business model of Zillow historically has been as an advertising site/lead gen site. This business model is not in their DNA and their operational expertise. The negative margin pressure from this business in tougher times can really stress the overall business at times. Look at the other iBuyers in the space - they are still in the space. They just aren't as naive as Zillow was. Zillow's behavior here is really confusing and speaks to poor operations/strategy/discipline. Housing prices have been mostly been only going up in this market and Zillow was losing their shirt.

civilized 5y ago
Well, their strategy of blaming the nerds will at least ensure that they never have the opportunity to bet big on ML again. No one competent will want to work for them now.
- wussboy 5y ago
  I wish this was true but I suspect it is not. If Facebook can still have excellent engineers because they pay a ton of money, then so can Zillow.
  - ivanyahoocom 5y ago
    Zillow has always paid low compared to the market. They never got the “top nerds” and this was really a CEO problem. I would never bet on that company on any decently complex technology like predictive models. It took them years to make their model decent, something that a machine learning tutorial does better - just a linear regression on square feet and location. They were literally comparing my house to ones on the other side of the city - comparing a downtown property with outskirts.

JumpCrisscross 5y ago

Their CEO also seems to have fundamentally misunderstood what he was doing. He described their activity as market making. That’s wrong. Market makers attempt to buy and sell instantaneously without changing the asset. Zillow had no intention of simultaneous purchase and sale. They also modified the asset. They used a market making model glued to a real estate speculating one to run a house-flipping operation at scale and from a distance. Instead of focussing on sellers who would sell at a discount for immediacy, they paid over market. There was no effort put into pre-selling or hedging. (The latter would have been so possible as a security against their entire portfolio.)

rdtwo 5y ago
A market making model would actually be interesting. Setup a potential seller with a buyer beforehand. Then when the seller buys the new home both transactions occur together and you get 2 sales instead of one

omegalulw 5y ago

> is that they failed to put in adequate margin for adverse selection. Adverse selection turns prediction variance, which you might naively expect to "average out" when using a well-calibrated model at scale, into a bias towards underperformance. Presumably, they trusted that their model does a more accurate assessment of price and future potential than a person themselves. If you had an excellent model, why _not_ would it balance out? My theory is that most people would consult various sources and professionals-who-assess-homes (whatever they are called) and this the likelihood of Zillow having a better assessment than the average person is just low.

fuzzfactor 5y ago
Trained by all kinds of nominally relevant data. Makes nominal decisions with at best somewhat nominal ability.