6 ms·

I agree with the OP that "it's put up or shut up time for Data Science teams. When the business depends on our work for revenue growth, success is expected. Fai

by cs702 5y ago

I agree with the OP that "it's put up or shut up time for Data Science teams. When the business depends on our work for revenue growth, success is expected. Failure is fatal." Zillow's CEO blamed the Data Science team for the failure of a mission-critical business unit and fired 25% of all employees. The stock lost a third of its market cap in a few days. When revenue growth depends on an ML/AI system, failure is indeed fatal. However, I disagree with the OP's diagnosis. The initiative failed because to a close approximation the only homeowners who would routinely accept Zillow's offers were those whose homes could not fetch anywhere near as much via traditional sales channels. Everyone else would rather sell their home via conventional listing. Zillow gave homeowners a free put option: Whenever Zillow's offer was above market, the homeowner would take it; otherwise the homeowner could freely ignore it. The incentives were strongly stacked against Zillow from day one. Zillow unwittingly launched what I would describe as a seller's market for lemons. Naturally, Zillow ended up buying... a lot of lemons.

curiousgal 5y ago
https://en.m.wikipedia.org/wiki/The_Market_for_Lemons https://en.m.wikipedia.org/wiki/The_Market_for_Lemons
BryantD 5y ago
I think your basic point is good -- a free put option is always going to be a problem. However, can you unpack "Everyone else would rather sell their home..."? At the core there are two factors at play in this decision. One is financial; anyone with the luxury of time will compare a realtor's comps with an ibuyer's offer. The second is certainty; an ibuyer offer is guaranteed, and can be executed without having to show a house over the course of 1 to X months. It seems to me that certainty has value to some sellers in some situations. Right now, the certainty value of the traditional route is pretty good because the market is on fire. That is not always the case everywhere.
- cs702 5y ago
  It depends mainly on how much more money the homeowners believe they can fetch by listing the home versus selling it right away on the spot. For example, if you're the homeowner and you think you can get, say, a few extra $100K's by listing your home instead of selling it right away on the spot, you would probably take your changes and list it. Even if you're under pressure to sell quickly, you might still try listing the home for a couple of months to see what happens. Only if you think your home is a lemon will you consider selling to Zillow. That's why Zillow ended up buying all those lemons.
  - imajoredinecon 5y ago
    > Only if you think your home is a lemon will you consider selling to Zillow. What if Zillow just makes an offer that’s high enough that you think it’s unlikely you’d do better if you went through the hassle of looking for other buyers?
    - JoshCole 5y ago
      Lets say P(h) is the price that a home fetches after being listed, O(h) is the price that an owner expects to get from selling the home, and Z(h) is the price that Zillow expects to get if they sell the home. An owner will sell to Zillow when O(h) < Z(h). They will not sell when O(h) > Z(h). Zillow's outcome would then be (P(h) - Z(h)) * (Z(h) >= O(h)). The extremely important thing to note is that this is an asymmetric information game. Therefore O(h) < Z(h) is going to be self-selected such that O(h) < Z(h) because P(h) is closer to O(h) then Z(h) is to P(h). If Z(h) was actually able to be a reliable and better estimate of O(h) then it would mean that P(h) wasn't determined by information available to O(h) that was not also available to P(h). The market would already offer this value to O(h). So O(h) value is going to be reflective of Z(h). Its only if the non-public information changes the value that O(h) != Z(h).
      - acjohnson55 5y ago
        Like the previous posts said, the advantages of selling to Zillow are price certainty and immediacy, so your model isn't quite right. But I agree with the adverse selection problem you highlighted.
essdas 5y ago
Exactly. And to add to what you say, adversarial problems in data science are much harder than the regular ones which data scientists deal with and the hubris that I often see with data scientists won't cut it here. You need a different mindset to tackle these problems and they must be handled incrementally with the real world in the loop.
mattmcknight 5y ago
I made the same point on twitter a few days ago. https://twitter.com/mattmcknight/status/1456096267768147971 https://twitter.com/mattmcknight/status/1456096267768147971 Even CarMax and Carvana with commodity products in the traditional lemon market have to work around this problem by adding an inspection to confirm the data- and they have a large spread between their buying price and their selling price.
- ethbr0 5y ago
  > large spread between their buying price and their selling price This is the market I was amazed Zillow didn't get into: the "WE BUY UR HOUSE 4 CASH" market. I'm guessing the volume is too low (few people sell their house for stupid low offers) and advertising costs are too high (those people probably aren't obsessively checking Zillow for their home's value every day). But still, it seems like easy money, that someone far less knowledgeable and talented than them is currently scooping up, that would better insulate them against market turns.
  - JumpCrisscross 5y ago
    > the market I was amazed Zillow didn't get into: the "WE BUY UR HOUSE 4 CASH" market They could have bootstrapped by offering a product to those cash4home operations.
- benjaminjackman 5y ago
  And don't forget flipping a car is a lot faster and easier than flipping a house at much lower capital. This makes each cycle of trial and error a lot cheaper and a lot more frequent. Also there is something to be said for a market where the customer brings the product to your location for inspection, and whose product is considerable faster to inspect more rigorously than a house. Also the products within that market are much more uniform and share several unifying characteristic across even different makes and model so its easier to learn a good and accurate pricing function. That gives them a lot more cycles to improve their algorithms (both human and machine based). Plus when CarMax were starting out they likely did so with a bevy of folks already experienced and profitable at buying and selling used cars at a profit who had the systematic knowledge of how to do so in their heads. It then was simply a matter of building up customer goodwill by focusing on another metric customers cared about, taking a perhaps slight loss to avoid the risk of being considerably ripped off by a savvy local dealer and saving oneself the anxiety induced by high pressure sales tactics. Having said that I think Zillow could still be quite successful in this endeavor however from my experience they didn't deploy their trading algorithm in a prudent way, they massively over-traded an untested by the real world model, they trusted their paper trading (backtesting) way to much, and they didnt focus on the nuts and bolts of building a successful market making system: making actual money over and over at small size and ramping up size as your empirically demonstrated pnl grows. Also putting trading aside they didn't have to be a flipper, they would probably naturally be more fit as an exchange+ or intermediary that provides exchange type backing and reduces the friction in buyers finding sellers, taking on risk by only making an offer when they have a bona fide buyer for fixed or % fee, but glossing over all the hassle of two people who dont know each other making a large trade. Offering services like if you dont like the house you just bought through us we will help your sell it at a cost proportional to renting no questions asked.
ethbr0 5y ago
The broader point is... Zillow's alpha should not be valuation, for all the reasons you listed. It should be geospatial trends. There's a fixed margin on any given home, caused by the inefficiency of real estate markets. But many actors (the seller, other agents) have an inkling of what that margin and the numbers on each side of the sale are. What they do not have as good of a bead on, and Zillow is best placed to know, is high vs low neighborhood boundaries, and how they're moving, across time. It seems like there would be more (and more reliable) margin in betting on the boundary movement, that the seller isn't aware of yet, than the absolute value of a particular home. And yes, they are presumably capturing this in their price difference, but it seems like it should be a core foundation.
imajoredinecon 5y ago
> Zillow gave homeowners a free put option: Whenever Zillow's offer was above market, the homeowner would take it; otherwise the homeowner could freely ignore it. Sorry for the possibly dumb question, but doesn’t this just describe buying things in any financial market? Your offer will only be accepted in cases where nobody else is willing to pay more.
- DwnVoteHoneyPot 5y ago
  Yeah, it’s not correct to call it a put option. A put option has a time component where you are guaranteed a contract price while the market price fluctuates. Zillow is an example just bad bidding (due to bad valuation model)
- dragonwriter 5y ago
  > Sorry for the possibly dumb question, but doesn’t this just describe buying things in any financial market? Your offer will only be accepted in cases where nobody else is willing to pay more. This isn't a financial (securities and derivatives) market, nor is that behavior specific to financial markets. OTOH, in financial markets, you are dealing with masses of interchangeable items (e.g., every share of IBM is the same); whereas with real estate every individual item is different. In financial markets, you typically have (some) other participants order data for the same asset as a pricing signal. It's a lot easier, when you are willing to give an offer on any item in the market, to be way off in real estate than financial markets without knowing it till you try to flip the assset.
- 88913527 5y ago
  In a financial market (e.g: stocks) 1 share owned an individual investor is just as good as 1 share owned by an institutional investor: they have identical representation. The financialization of housing glosses over the fact the some houses will have foundational issues, need new roofs in a year, etc but you can't see that when you treat all houses as if they were shares, which seems like Zillow did, or by the nature of the asset class, sellers with low-quality product dumped for high prices. Maybe Zillow went wrong because they deployed too much capital too quickly. With the opportunity to be selective, they could have found homes with 'good bones' that were under-priced. But if the executives mandated X billions of purchases in Y months, you move downstream to progressively less good deals, due to your own insatiable demand, when housing stock is already extremely low. That's not the AI/ML team's fault. That's you simply taking what you can get.
PKop 5y ago
>those whose homes could not fetch anywhere near as much via traditional sales channels Isn't this just another way of saying the models overstated the value of the homes?
- kshacker 5y ago
  On an average they may not have. Three separate million dollar houses get offered 900k, 900k and 1.2M. The point is that the guy offered 1.2m knows he will not get the 1.2 anywhere else and takes the offer. The individual house is overvalued but you don't find out until after, let's say, 7000 of those sales.
encoderer 5y ago
It wasn’t a free put, the problem here is that it was a put but it was not priced correctly.

by cs702 5y ago

curiousgal 5y ago

https://en.m.wikipedia.org/wiki/The_Market_for_Lemons https://en.m.wikipedia.org/wiki/The_Market_for_Lemons

BryantD 5y ago

I think your basic point is good -- a free put option is always going to be a problem. However, can you unpack "Everyone else would rather sell their home..."? At the core there are two factors at play in this decision. One is financial; anyone with the luxury of time will compare a realtor's comps with an ibuyer's offer. The second is certainty; an ibuyer offer is guaranteed, and can be executed without having to show a house over the course of 1 to X months. It seems to me that certainty has value to some sellers in some situations. Right now, the certainty value of the traditional route is pretty good because the market is on fire. That is not always the case everywhere.

cs702 5y ago
It depends mainly on how much more money the homeowners believe they can fetch by listing the home versus selling it right away on the spot. For example, if you're the homeowner and you think you can get, say, a few extra $100K's by listing your home instead of selling it right away on the spot, you would probably take your changes and list it. Even if you're under pressure to sell quickly, you might still try listing the home for a couple of months to see what happens. Only if you think your home is a lemon will you consider selling to Zillow. That's why Zillow ended up buying all those lemons.
- imajoredinecon 5y ago
  > Only if you think your home is a lemon will you consider selling to Zillow. What if Zillow just makes an offer that’s high enough that you think it’s unlikely you’d do better if you went through the hassle of looking for other buyers?
  - JoshCole 5y ago
    Lets say P(h) is the price that a home fetches after being listed, O(h) is the price that an owner expects to get from selling the home, and Z(h) is the price that Zillow expects to get if they sell the home. An owner will sell to Zillow when O(h) < Z(h). They will not sell when O(h) > Z(h). Zillow's outcome would then be (P(h) - Z(h)) * (Z(h) >= O(h)). The extremely important thing to note is that this is an asymmetric information game. Therefore O(h) < Z(h) is going to be self-selected such that O(h) < Z(h) because P(h) is closer to O(h) then Z(h) is to P(h). If Z(h) was actually able to be a reliable and better estimate of O(h) then it would mean that P(h) wasn't determined by information available to O(h) that was not also available to P(h). The market would already offer this value to O(h). So O(h) value is going to be reflective of Z(h). Its only if the non-public information changes the value that O(h) != Z(h).
    - acjohnson55 5y ago
      Like the previous posts said, the advantages of selling to Zillow are price certainty and immediacy, so your model isn't quite right. But I agree with the adverse selection problem you highlighted.

essdas 5y ago

Exactly. And to add to what you say, adversarial problems in data science are much harder than the regular ones which data scientists deal with and the hubris that I often see with data scientists won't cut it here. You need a different mindset to tackle these problems and they must be handled incrementally with the real world in the loop.

mattmcknight 5y ago

I made the same point on twitter a few days ago. https://twitter.com/mattmcknight/status/1456096267768147971 https://twitter.com/mattmcknight/status/1456096267768147971 Even CarMax and Carvana with commodity products in the traditional lemon market have to work around this problem by adding an inspection to confirm the data- and they have a large spread between their buying price and their selling price.

ethbr0 5y ago
> large spread between their buying price and their selling price This is the market I was amazed Zillow didn't get into: the "WE BUY UR HOUSE 4 CASH" market. I'm guessing the volume is too low (few people sell their house for stupid low offers) and advertising costs are too high (those people probably aren't obsessively checking Zillow for their home's value every day). But still, it seems like easy money, that someone far less knowledgeable and talented than them is currently scooping up, that would better insulate them against market turns.
- JumpCrisscross 5y ago
  > the market I was amazed Zillow didn't get into: the "WE BUY UR HOUSE 4 CASH" market They could have bootstrapped by offering a product to those cash4home operations.
benjaminjackman 5y ago
And don't forget flipping a car is a lot faster and easier than flipping a house at much lower capital. This makes each cycle of trial and error a lot cheaper and a lot more frequent. Also there is something to be said for a market where the customer brings the product to your location for inspection, and whose product is considerable faster to inspect more rigorously than a house. Also the products within that market are much more uniform and share several unifying characteristic across even different makes and model so its easier to learn a good and accurate pricing function. That gives them a lot more cycles to improve their algorithms (both human and machine based). Plus when CarMax were starting out they likely did so with a bevy of folks already experienced and profitable at buying and selling used cars at a profit who had the systematic knowledge of how to do so in their heads. It then was simply a matter of building up customer goodwill by focusing on another metric customers cared about, taking a perhaps slight loss to avoid the risk of being considerably ripped off by a savvy local dealer and saving oneself the anxiety induced by high pressure sales tactics. Having said that I think Zillow could still be quite successful in this endeavor however from my experience they didn't deploy their trading algorithm in a prudent way, they massively over-traded an untested by the real world model, they trusted their paper trading (backtesting) way to much, and they didnt focus on the nuts and bolts of building a successful market making system: making actual money over and over at small size and ramping up size as your empirically demonstrated pnl grows. Also putting trading aside they didn't have to be a flipper, they would probably naturally be more fit as an exchange+ or intermediary that provides exchange type backing and reduces the friction in buyers finding sellers, taking on risk by only making an offer when they have a bona fide buyer for fixed or % fee, but glossing over all the hassle of two people who dont know each other making a large trade. Offering services like if you dont like the house you just bought through us we will help your sell it at a cost proportional to renting no questions asked.

ethbr0 5y ago

The broader point is... Zillow's alpha should not be valuation, for all the reasons you listed. It should be geospatial trends. There's a fixed margin on any given home, caused by the inefficiency of real estate markets. But many actors (the seller, other agents) have an inkling of what that margin and the numbers on each side of the sale are. What they do not have as good of a bead on, and Zillow is best placed to know, is high vs low neighborhood boundaries, and how they're moving, across time. It seems like there would be more (and more reliable) margin in betting on the boundary movement, that the seller isn't aware of yet, than the absolute value of a particular home. And yes, they are presumably capturing this in their price difference, but it seems like it should be a core foundation.

imajoredinecon 5y ago

> Zillow gave homeowners a free put option: Whenever Zillow's offer was above market, the homeowner would take it; otherwise the homeowner could freely ignore it. Sorry for the possibly dumb question, but doesn’t this just describe buying things in any financial market? Your offer will only be accepted in cases where nobody else is willing to pay more.

DwnVoteHoneyPot 5y ago
Yeah, it’s not correct to call it a put option. A put option has a time component where you are guaranteed a contract price while the market price fluctuates. Zillow is an example just bad bidding (due to bad valuation model)
dragonwriter 5y ago
> Sorry for the possibly dumb question, but doesn’t this just describe buying things in any financial market? Your offer will only be accepted in cases where nobody else is willing to pay more. This isn't a financial (securities and derivatives) market, nor is that behavior specific to financial markets. OTOH, in financial markets, you are dealing with masses of interchangeable items (e.g., every share of IBM is the same); whereas with real estate every individual item is different. In financial markets, you typically have (some) other participants order data for the same asset as a pricing signal. It's a lot easier, when you are willing to give an offer on any item in the market, to be way off in real estate than financial markets without knowing it till you try to flip the assset.
88913527 5y ago
In a financial market (e.g: stocks) 1 share owned an individual investor is just as good as 1 share owned by an institutional investor: they have identical representation. The financialization of housing glosses over the fact the some houses will have foundational issues, need new roofs in a year, etc but you can't see that when you treat all houses as if they were shares, which seems like Zillow did, or by the nature of the asset class, sellers with low-quality product dumped for high prices. Maybe Zillow went wrong because they deployed too much capital too quickly. With the opportunity to be selective, they could have found homes with 'good bones' that were under-priced. But if the executives mandated X billions of purchases in Y months, you move downstream to progressively less good deals, due to your own insatiable demand, when housing stock is already extremely low. That's not the AI/ML team's fault. That's you simply taking what you can get.

PKop 5y ago

>those whose homes could not fetch anywhere near as much via traditional sales channels Isn't this just another way of saying the models overstated the value of the homes?

kshacker 5y ago
On an average they may not have. Three separate million dollar houses get offered 900k, 900k and 1.2M. The point is that the guy offered 1.2m knows he will not get the 1.2 anywhere else and takes the offer. The individual house is overvalued but you don't find out until after, let's say, 7000 of those sales.

encoderer 5y ago

It wasn’t a free put, the problem here is that it was a put but it was not priced correctly.