49 ms·

Zillow lost money because they weren't willing to lose money

526 points by mjmayank 5y ago

zatkin 5y ago
What does 'bootstrap' mean in the context of this article?
- ec109685 5y ago
  In order to actually understand true risk (to create a profitable model), you’ll actually have to experiment and lose money in order to bootstrap your own ML model. Taking data acquired elsewhere and hoping it can make your own model instantly profitable isn’t possible.
xibalba 5y ago
The article offers no new or inside information, just more armchair quarterbacking. I'm surprised that it is getting traction on HN. I think it says more about the zeitgeist than it does about the (lack of) insightful-ness of the content.
- gwern 5y ago
  Yeah, it just repeats the Narrative in a giant post hoc. "Zillow uses ML models in some way; Zillow failed; QED, ML models are dangerous." Except the reporting by Bloomberg and insiders is that Zillow failed because they overrode the models predicting lower prices and bought like drunken sailors, and it's just a story of yet another marketmaker being run over by the market. So sad, too bad, largely irrelevant to the tech world, yet, it looks like it's entering the mythology of ML up there with Cambridge Analytica or the tank story - unkillable by mere facts or tardy reporting.
- adjkant 5y ago
  As someone who upvoted but didn't care much for the content, I think it's worth mentioning that sometimes/often I upvote for the currently occurring conversation to get more eyes, not for the link. I haven't seen too many specifics on the Zillow collapse and I've learned a nice deal through many of the comments here, most not having much at all to do with the article.
1cvmask 5y ago
The essence of the article is that they underestimated how flawed their algorithms are and how hard it is to build a good lasting algorithm in a dynamic world. Many seasoned wall street algorithms have suffered many times over 5 decades, and when they fail we call them black swan events.
- mjdesa 5y ago
  That’s not what I read in that article at all. What I read was that their data and methodology was flawed, and they weren’t willing to pay the price to fix it.
  - human 5y ago
    Their methodology might have been flawed. The author is speculating. He uses Zillow to explain how datasets – especially the ones with money tied-in – can’t be trusted blindly. Building a high-quality dataset is an expensive endeavour.
  - seoaeu 5y ago
    Zillow thought they already had enough data and accurate enough models to buy and sell houses profitably. The last two quarters proved they didn’t. In the first quarter they were puzzled by making too much money and in the second they lost a whole bunch The author is arguing that they should have pivoted from “we already have models” to “we’re intentionally gambling hundreds of millions of dollars so we can build good models over the next few years”. That might be a good strategy for a startup with loads of VC money and no other products, but it makes less sense for a more established company to risk going under on that bet
- throwawayboise 5y ago
  And wall street algorithms should be easier because securities are fungible. One share of AAPL is the same as another. Houses are not like that. Real estate is local, local, local. Every house has a hundred unique attributes that each potential buyer will value differently.
danielvaughn 5y ago
I think the article makes an interesting point about this being the first of many, but I disagree with the initial tone of the article. It seemed to paint Zillow as being afraid of loss. On the contrary, I viewed Zillow as demonstrating good common sense and an ability to make hard decisions. To me it shows that they aren't committing the sunken cost fallacy, and are willing to cut an entire 25% of the company and take massive losses so they can redirect themselves towards better objectives.
- quickthrowman 5y ago
  I agree, I think they realized it wouldn’t work and made a hard decision to save the company. Zillow realized the only time their ask was hit is when it was at a premium to the actual market price. If they used competitive offers, they’d never have the winning bid. In a hot market where you’re offering a premium, you’re going to have owners of lower quality properties accepting your offer, while owners of higher quality properties have more offers to select from. Zillow got left holding a bag of lemons and decided to get out before buying the whole lemon grove.
  - marcinzm 5y ago
    >If they used competitive offers, they’d never have the winning bid. Why do you assume that, seems like a cash buyout would be a great deal for many sellers if it was at the appropriate price. Issue is I think that Zillow's information was less granular than what the buyers/sellers had. Let's say Zillow priced two houses near each other at 1million each. However one was close to a busy road so would only sell for $900k while the other could sell for $1.1. Zillow made the right average offer of $1million to both but the buyers/sellers actually had more information. So the 1.1m seller didn't take Zillow's offer while the 900k seller did. Now Zillow was out $100k essentially not counting fees.
    - rpvnwnkl 5y ago
      They are out 200k. They bought for 100 too much and will have to sell for a 100 less than planned.
black_13 5y ago
That it was a bad idea?
pid-1 5y ago
https://www.youtube.com/watch?v=ajGX7odA87k&t=833s https://www.youtube.com/watch?v=ajGX7odA87k&t=833s
marcinzm 5y ago
I think a key point that is missed is the feedback cycle time. Real time bidding advertising has I believe a number of the listed concerns however the feedback time is maybe hours at most and might be milliseconds. So the risk is in general a lot smaller and worst case you just lose some of the money you spent that day/week. With long term assets you could lose months worth of investments before your feedback loop fully kicks in.
droopyEyelids 5y ago
In the original Foundation books by Asimov, the conceit of "Psychohistory" was similar to the concept of machine learning for pricing: The future can be predicted _if people aren't aware of the prediction to change their behavior in relation to it_ This is similar to 'adverse selection' in real life & in Zillow's model. The article makes a nod to this, but seems to imply that if you train your model on that adverse selection, you can come out ahead after paying to learn about it. To me that kind of misses the point. Adverse Selection isn't a static feature of the landscape you can identify and avoid, it is people understanding what you understand, adapting, and responding. Train your model with adversaries trying to beat it, then you'll maybe counter the specific first round strategies they use, and they'll learn new ones and beat your new model with their 2nd round strategies. It's a continuous game. Your requirement to gather a corpus of training data will keep you in the 2nd turn of a game where the wins are biased to whoever has the 1st move.
- JKCalhoun 5y ago
  I'm reminded always of the Hunt brothers that tried (and failed) to corner the silver market in the 70's/80's: https://en.wikipedia.org/wiki/Silver_Thursday https://en.wikipedia.org/wiki/Silver_Thursday
  - JanisL 5y ago
    Basically the COMEX changed the rules explicitly to disadvantage the Hunt Brothers. The changes made to margin requirements is what made the difference here. I don't think anyone could claim that the silver market is an entirely free market, I remember last year a press release where the COMEX said they weren't sure how much they actually had in their vaults in eligible and registered, with a plus/minus 50% figure being given on their estimates. I can't think of any other major market where someone would come out and say they didn't know how much inventory they had and that their best estimate could be 50% off. And the participants in the silver market are still rather ridiculous to this day: https://www.reuters.com/business/finance/jpmorgan-pay-60-mln-settle-precious-metals-spoofing-lawsuit-2021-11-19/ https://www.reuters.com/business/finance/jpmorgan-pay-60-mln...
JKCalhoun 5y ago
> At a high level, the story of Zillow Offers is a story of our industry at its best. Not in my book. All I see is the price of real estate being driven up by corporate greed and the individual home-buyer being shut out of the market. Is it wrong of me to hate "flippers" (be they corporate or private)? Pure capitalists will tell me that every property sold went to the highest bidder — in the case of a flipper winning they were willing (able) to risk the capital to hopefully turn a profit on the flip. I suspect if you dig deeper you might find sales going to flippers because they had 100% cash offers, because they are better at "the game". I see no reason to punish prospective first-time home owners in this sort of market. But I don't know what the answer is either.
- chii 5y ago
  flippers make the real estate market more liquid, in the same way high-frequency trading does for stocks. Flippers take the risk of the market falling while they're flipping - that's the price they pay for their profits.
  - JKCalhoun 5y ago
    You'll have to educate me then: is more liquid better for the buyers or sellers?
    - loeg 5y ago
      Liquidity is good for both buyers and sellers.
    - javert 5y ago
      Both. In an illiquid market, it takes a long time to find buyers or sellers. You are incentivized to overprice (if selling) or underprice (if buying) and wait a long time to see if someone will match you. A liquid housing market means people can buy or sell the house at the "right" price without waiting many months or years. As a seller, would you rather wait a year to make a bit more money? That wouldn't be good. That would be crappy.
deleted 5y ago
[deleted]
skohan 5y ago
> They thought they needed to build a machine learning model when they really needed to build an entirely new organization, one that possessed the technical and cultural mindset necessary to succeed in this space. I totally agree. It's not impossible to imagine their model working: why couldn't you serve as a market-maker for homes at a large scale, especially with the unique insights Zillow could have based on their datasets. However I think where the hubris lay is in how they thought they could leapfrog all the way to an automated solution before building a competency as a house-flipping company. From what I understand, where they failed was partly in building a rich enough model to properly account for the less easily quantifiable elements which ultimately account for a property's value. I.e. the price per square foot might make a property look like a steal, while something like a sewer main nearby, or problematic neighbor could radically change the value proposition to anyone standing at the site. That's a non-trivial problem to solve for even the best ML and it's not clear how you would automate this. If you ask me, instead of focusing on building an automated price discovery system, they should have started by trying to build a quality home-flipping organization, and figuring out how to super-charge manual work using their datasets. Over time you might find ways to optimize the process and increase the level of automation to scale output relative to head-count.
- lotsofpulp 5y ago
  > why couldn't you serve as a market-maker for homes at a large scale, especially with the unique insights Zillow could have based on their datasets. Why would Zillow have unique insights? With the exception of Texas, I thought real estate sales information is public information in the US.
  - ctvo 5y ago
    Can you not imagine how useful it is to know user data e.g. what neighborhoods receive the most clicks, what type of homes generate the most favorites, how long people view one listing vs. another, … that is unrelated to public MLS data?
    - lotsofpulp 5y ago
NoblePublius 5y ago
Buy low, sell high. You need “data science” to do this? $VNQ is up 33% since 2016. Do you realize how dumb and bad you have to be to lose money on real estate in this time? I imagine randomly picking homes off the MLS would have yielded better returns in the last five years than whatever Tableau-powered nonsense the biz ops analysts at Zillow used. The entire iBuying concept is a farce, completely divorced from basic fundamental analysis.
mgraczyk 5y ago
I really liked this quote, which is also true of machine learning organizations at large tech companies: The most valuable data is not social data, ... but your own data because every dataset that you’re looking at internally describes your own process, including your bugs, ... building models from your own data is the only way to build a really successful system. This is one thing that a lot of outsiders do not understand. Facebook/Google's data is basically worthless to anybody but Facebook/Google. The data has value because it is derived from their own processes, which in this case are the requests and context of each product surface.
- ethbr0 5y ago
  It makes sense when you ask the question another way: "What is the likelihood that a preexisting assemblage of data contains all the nuances for my specific process?" Some domains are intricately mapped in available data (e.g. equity pricing), but most, and especially most physical, are not (e.g. freight transportation).
- deleted 5y ago
  [deleted]
- robbedpeter 5y ago
  Yeah, I'm gonna say that romanticizing mass surveillance is a bit much. Cambridge Analytica, the five eyes countries, Clearview - all these are using Facebook and Google's data to great effect. Facebook and Google's data are not their own. That data is comprised of private lives, stripped bare pixel by pixel, bit by bit, and it's offensive to frame it as if they're doing something alchemical and special with it. Google's search dominance came from something special, creating the right algorithm and seizing the first mover advantage, but the relentless and ruthless invasion of privacy is a rent seeking race to the bottom. All of the ills of the internet and political turmoil in the west from algorithmic amplification are the brainchilren of Facebook and Google. It turns out that "tailoring search results" and "targeted advertisement" are excuses for something that can cost far more than a society might want to pay.
  - mgraczyk 5y ago
JackFr 5y ago
While the point the article makes is true — it costs money to acquire the real world data, the comparison to credit underwriting is misguided. Underwriting credit is fundamentally different than predicting house prices. In particular when you’re auto-underwriting credit it’s not typically an origination-for-sale model. So the value of the loan is the present value of the future payments, less the future value of defaults, less the cost of acquiring the customer. Historically those things can be modeled pretty accurately and the aspects that can’t be modeled accurately can often be hedged or eliminated by the law of large numbers. The innovation of the new ML underwriting with respect to accuracy is at the margins. The real disruption is the speed and cost. (Disclosure: I worked at a SMB fin tech and we reran multiple credit models for a million customers and past customers every night.) If Zillow were getting into the rental business, in some ways it might have been easier for them. But they needed to model where they could sell an illiquid asset which is a much harder and much less well understood problem. And yes with enough capital to plow through and the appropriate risk attitude they could likely have gotten the handle on what their pipeline was really going to look like. But it’s hardly the same problem as credit underwriting.
_xnmw 5y ago
Good riddance. If large-scale house flipping took off, we might actually end up in a scenario where housing was treated as a speculative asset, with empty houses getting flipped between investors looking to make a quick buck, further lowering the supply of actual places to live (because housing units remain empty while being flipped), driving up the cost for families who just want a place to live. Oh wait...
- h2odragon 5y ago
  My wife did some work for the Census last year. Our extremely rural neighborhood has lots of unused housing, some for a decade+. That work got her out to see some of the places not visible from the roads, and increased our awareness of the scale of the problem. At a guess, in our county, 20%+ of the housing is idle, owned by out-of-state companies, some of whom pay property taxes and some dont. The county isn't auctioning off because of tax default anymore, no one was buying these places at $100. Many of these places are complete teardowns now; some actually no longer exist, having burned or apparently been scrapped. The tax assessments on those have not been adjusted, for the few i checked. I think the housing market is so fucked no one really grasps the scale of the problem.
  - jason-phillips 5y ago
    > I think the housing market is so fucked no one really grasps the scale of the problem. I don't think I agree with this assessment. I live in a very rural area two hours northwest of Austin, literally in the middle of nowhere. I've studied the local economy and understand how things work here. I think the characteristics you've identified in the rural housing supply are not unusual and also not as serious in a practical sense as you seem to be indicating. For example, in San Saba, Texas, 20-30% of the households are under the federal poverty threshold. The median household income in the town of San Saba is about $32K/yr. People just don't have any excess cash so the maintenance on dwellings is neglected. That means folks become extremely thrifty and resourceful patching what needs to be patched, very cheaply, if not for free. Some dwellings simply aren't maintained and one day won't be there anymore. Families live on small budgets, don't require much and generally just "get by". The municipal and county governments have very small budgets but extremely resourceful staff who accomplish a lot with very little. Everyone comes together as a community when needed (see: February 2021 freeze event) and it all works very efficiently, actually. To someone who is not from here and who doesn't understand that dynamic, they might see those properties as you described and believe a tragedy was unfolding. But that doesn't reflect reality on the ground vis-a-vis my neighbors.
ridaj 5y ago
This is a good take, but > A machine learning organization thinks of risk entirely differently than an automated risk underwriting organization. It's possible and maybe even advisable to use machine learning in the automated risk underwriting business, but it is a different setup / set of objectives. As the author notes, IMO the adversarial and antifraud aspect of risk underwriting turns it less into a straight-up estimation problem and much more into a game theory type of problem. ML models can assist in evaluating risk, but you do indeed have to be preocuppied by your risk as a party to the transaction in the first place, and not just trying to predict prices as a third party observer (which by itself is pretty riskless).
goatherders 5y ago
This is really well written. Thanks for sharing.
deleted 5y ago
[deleted]
lifeisstillgood 5y ago
>>> you should expect to lose 50% of your capital allocated towards underwriting. How ?
flerchin 5y ago
> One of the things that happens for a brand-new launched credit card: done right, you lose about 50% of the dollar volume in the first several months What does this mean? 50% of the money is held as debt? Or 50% of the money is lost to fraud?
- Petabits 5y ago
  Getting people to initially sign up through bonuses causes a lot of money to be shed, and are thus not profitable until people renew (without the bonus) the second year. I remember seeing the CEO of Chase saying he was excited that they lost billions in the new sapphire card because it meant they had so many members
abiro 5y ago
I think the title is highly misleading. The main point here is that Zillow simply had no idea what it takes to be a market maker and their pool was picked off by savvy traders. Good tweetstorms with technical explanations on how that happened: https://twitter.com/macrocephalopod/status/1455887352371597312 https://twitter.com/macrocephalopod/status/14558873523715973... https://twitter.com/0xdoug/status/1456032851477028870?s=21 https://twitter.com/0xdoug/status/1456032851477028870?s=21
- treis 5y ago
  I'll second that this article is just wrong. Zillow burned plenty of money in their Offers business. The problem is that all that spending revealed that they performed poorly in a questionable market segment. Ultimately they were really bad as flippers. More often than not paying more than market price for the homes they bought. I think the root problem is that this was a panic move. They saw Open Door's success and thought they had no choice but to try and replicate it. But its a questionable business move for Zillow and ultimately they couldn't make it work
- MisterBastahrd 5y ago
  Zillow offered to buy my home at 30% more than everyone else in the market for cash, without an inspection, and I wasn't even looking to sell it at the time.
Petabits 5y ago
Would it be too dystopian if governments sectioned off certain neighborhoods and set price caps per sqft? This would make it so speculative investors are unable to build capital in houses, thus leaving homes for actual people. I'm not super familiar with land grant homes, but the prospect of seemingly fixed price homes seems to prevent investors from buying in.
tinyhouse 5y ago
If you have a good business with high margins, why not grow that business instead of starting a new low margins business of flipping houses?
- jdross 5y ago
  Because they were having a lot of trouble growing that business. See their earnings reports before they entered iBuying
rossdavidh 5y ago
While no doubt Zillow made many of these mistakes, I think the reality is more sobering that the author of the article realizes. The more grim possibility, is that Zillow got out of the house buying business, not because they weren't good enough at it, but because they _were_ good enough at it to realize that it was at the top. If buyers want more now for their house, than it can be sold for in a few months time (which is necessary for renovations and other prep for sale), then there is no ML (and no non-ML) method to make money. Either you overpay and lose money, or you don't overpay and you don't buy any houses. In that situation, the only smart play, is to get out of the market. Zillow is, no doubt, not perfect. But they have a lot of knowledge of the housing market, and they thought it was time to get out entirely. I think the author of the article either isn't able, or doesn't want, to consider that Zillow might have been exactly correct in doing so.
- dsizzle 5y ago
  But they lost money last quarter while the market was still rising. Seems there was some problem with their prediction process.
  - sudosysgen 5y ago
    Their plan wasn't to flip homes in a month or two as far as I'm aware so that's expected if they get out.
  - rossdavidh 5y ago
    Rumor is that they had to put their thumb on the scales (i.e. tweak the model) to get enough sellers to sell to them. In other words, if paying what their model actually thought was the right price, not many people sold to them. Instead of saying "our division's whole business model won't work, you should fire us", they tried to cut the margin too close, resulting in losses which got the CEO's attention to the problem. This kind of thing is difficult to confirm from the outside, of course. But that they adjusted the model to pay more towards the end is pretty widely known.
    - 5y ago
PaulHoule 5y ago
I can't agree with the article or many of the comments on it. (A) Both Wall Street and Machine Learning Modelers struggle with tail risk. Hedge funds measure performance against https://en.wikipedia.org/wiki/Sharpe_ratio https://en.wikipedia.org/wiki/Sharpe_ratio which assumes risk is (i) normally distributed and (ii) a source of reward. For most people, however, risk looks like Theranos or the Fukushima accident or the Challenger distaster. It's unbelievable that a machine learning model trained to predict house prices based on experience would be accurate in the face of events like the COVID-19 pandemic or what will happen when the Fed raises interest rates. You can model risks like that, but to the extent that you're working from experience you are working from a database from the 1929 Crash, South Sea Bubble, etc. (B) Mark Levine wrote a good article about how you'd exploit such a predictive model. If you consistently gave people low offers, a few people would accept them. You would get a high rate of return but could invest little capital. To invest more capital you have to make more offers that get accepted, that is, give better prices. Your rate of return goes down and if there is shrinkage from errors, accidents, etc. you could get a negative return. It's that "tendency towards a declining rate of profit" that Marx warned about. (C) The analogy with stock market market makers doesn't sound good when you consider the differing timescales. Market makers are isolated from some risk because of the length of their holdings. Yet, they make profits by exploiting the stochastics of a stationary market (e.g. if you don't like the price at time t1, you will usually get a better price at t2) but they lose money when markets move definitively in one direction or another. That kind of trader heads for the bathroom when things go South and in the interest of being orderly markets impose sanctions on market makers who do the natural thing and press the "STOP & UNWIND ALL POSITIONS" button when it gets tough. In the case of Zillow I see holding times that go on for weeks or months and all kinds of real world risk like planning to do certain renovations but having to delay the work because out of 20 things you need from Home Depot they only have 16 of them.
leot 5y ago
Real estate is one of the few markets where non-experts can make money, where it’s not a hyper-liquid winner-take-all game. Coupled with this is the fact that housing is a necessity and owning a home leads people to invest in their communities more than if they were renting, I think it’s a good thing if Zillow (and OpenDoor, etc.) fail at pushing everyday people out of the business of real estate investing. Here’s hoping we see some regulation—the illiquidity of the home buying market is not a problem that needs to be solved.
- jdross 5y ago
  Opendoor doesn't compete with real estate investors, they compete with realtors and mortgage brokers. Opendoor's primary benefit is to enable people to move when they otherwise could not easily do so, creating more liquidity and matching supply and demand (often number of bedrooms in house to number of bedrooms now needed). The challenge with moving is that most people need to sell their current house before they can afford (or even know what they can afford) to buy their next home. Opendoor lets a family buy that next home with its cash, then list their current home on the market or sell it to the company so they avoid the double mortgage or double move (home->rental->home)
  - robocat 5y ago
    Does Opendoor avoid some of the standard x% realtor fees on either or both of the transactions? Reduced fees could easily make a huge difference to expected profitability. In contrast, “Zillow Seeks to Sell 7,000 Homes for $2.8 Billion” so Zillow lost more than a few percentage points.
ezconnect 5y ago
They lost money because they were gaming their own system for their own profit.
dboreham 5y ago
They build an AI that perfectly emulated Wall St masters of the universe.
csours 5y ago
Your data is not neutral, it is opinionated. Who is asking the question? What do they use the data for? What questions are they not asking?
jedberg 5y ago
Zillow's mistake is that they thought their AI could replace human buyers instead of augment them. Most AIs today are for augmentation, not replacement. Vehicle autopilots are a perfect example. The ones that are commercially available aren't capable of replacing the human, they just augment the human's abilities.
wiradikusuma 5y ago
Does anyone know where Zillow get its dataset from? I reckon it's essentially sale price? Can a "hobbyist" investor do the same?
- deleted 5y ago
  [deleted]
- andromeda-brain 5y ago
  There's a lot of information that is only available to MLS members. Zillow used to not have access to this information, but they slowly brokered deals with MLSes around the country to get it. One example: The MLS in Austin, TX recently banned publicly sharing a home's sold price. https://www.zillow.com/austin-tx-78701/sold/ https://www.zillow.com/austin-tx-78701/sold/
  - MisterBastahrd 5y ago
    Yeah, each MLS org has its own data set which is a giant pain in the ass for the newspapers who are publishing listings for the MLSes in their areas. I don't know if they ever standardized it, but I know that one of the first tasks I had as a new dev for a newspaper back in the early 00s was to build a tool to take the data and normalize it into a single CSV.
jdross 5y ago
I think some reasons Zillow lost were that their pricing and risk processes were terribly underdeveloped in order to scale fast, their models were obviously inaccurate, and they didn't understand the difference between an acquisition cohort and resale cohort, and specifically how much the tail sales of an acquisition cohort determines profitability.
abernard1 5y ago
> Because when you have a hammer, everything tends to look like a nail and when you have TensorFlow, everything tends to look like an ML problem. And if you have billions of dollars in cheap capital, everything looks like an investment problem. Which is ultimately the suggestion of this article: "Why aren't you more like Wall Street?" The implications are exactly the opposite of Zillow being an innovative company. If they require billions of dollars in deep pockets (nbd) and a restructuring of their org to be more like old-school operators, all signs point to existing players as more fundamentally correct about the strategy required to succeed in the space.
dr_dshiv 5y ago
Is it fair to call this the result of “AI thinking?” Meaning that urge to automate away human involvement, because —after all—-if people are involved in analyzing data and decision making, then clearly the AI isn’t finished let.
MisterBastahrd 5y ago
SMEs are smarter than developers in their space. Always has been that way. Always will be that way. AI is great for when you need to tame a firehose and make millisecond decisions. But there's a 90 year old in Omaha who is better than the best AI.
cbsmith 5y ago
The amount of Monday morning quarterbacking of Zillow is just staggering.
gcanyon 5y ago
If the assumption is that you're going to lose half your money up front, then my plan would be to make sure "my money" is as little as possible: learn based on smaller bets. It sounds like Zillow built the Sea Dragon first, when they should have started with the Redstone and moved toward the Saturn V. If Zillow thought they had all the data they needed, there would have been little harm starting with $100 million in properties -- if the loss there ended up being $5 million, they would have known immediately something was up and that they had work to do.
wly_cdgr 5y ago
There's something really funny about white collar office worker businessmen talking about how it takes balls of steel to do what they do. Ok bro, sure. Trackballs of steel maybe
jollybean 5y ago
All of this reads like a Dickensian nightmare, where corporations have bought up all the water and air. This is ridiculous, we need much better regulation on this stuff. I wonder if higher property taxes would help a bit? If you own a 'home' then you're going to be paying for the water, school, electricity infrastructure whether you use electricity, water, or not. Of course, that would be gamed hard and would have to be strongly regulated as well. But that, and vacant property taxes, limits on some other things, and some other adjustments might help.
mistrial9 5y ago
I wonder why so few here question the basic assumption of injecting from above, machine-learning models to extract profit, into a vital part of the reproductive cycle of human families.
dcposch 5y ago
Framing it as machine learning undersells the problem. It's a hybrid model trading in an adversarial, real-dollar environment. The leverage comes from having a small human team trade big volume, much more than they could possibly trade directly, by augmenting their human abilities with automation and a model. Or seen from the other side, it's a model with human oversight. Any system like that is high risk, high reward. All the successful ones started out by losing a lot of money. Paypal lost an incredible amount to fraud before they started breaking even. OpenDoor lost an incredible amount to mispricing, and took on a ton of balance sheet risk, before their business really started working. "To live, you must be willing to die" - poker legend Amir Vahedi
- damanamathos 5y ago
  I think Opendoor still does poorly in new markets, but then it improves as they work out the quirks of the local market and start asking the right questions and collecting the right information. A big part of Opendoor is creating the right apps and processes to collect this information to feed their models. The machine learning part is important, but can give the false impression it's just about data scientists crunching numbers at head office, when in reality there's a huge real-world operational machine that's driving it.
Dowwie 5y ago
Feels analogous to the history of the collateralized debt obligation debacle where the models used to value CDOs were trained on data that no longer resembled reality. At least Zillow can live to fight another day, where as Stan O'Neal put all of Merrill Lynch's chips in with one of the biggest make-or-break gambles in the history of finance and the market turned against it, rendering Merrill to a fatally wounded company bailed out by Bank of America.
Animats 5y ago
Well, maybe they just exited because we're going into a recession and it's a good time to get out of house-flipping.
nickkell 5y ago
I love this guy’s movies. Finding out he writes so articulately to boot? Wow
- 88913527 5y ago
  You're thinking of Steve Buscemi. This article is authored by Steven Buccini, a Software Engineer and former political candidate.
m3kw9 5y ago
Buying assets using models, I’ve seen that in stocks, but people usually don’t go all in with how hard it is to predict the economy
vasilipupkin 5y ago
“You cannot bootstrap off an existing dataset. Full stop. These datasets can contain implicit assumptions or associations that you are not aware of. This is the original sin of many a algorithmic risk underwriting startup” False. You can definitely bootstrap and adjust the model as you either gather more data yourself or get more outside data. You can also build confidence intervals around the model predictions and decide how you want to proceed based on that. There is lots you can do with that initial model.
rajacombinator 5y ago
Good luck getting a public company to incentivize the people capable of pulling this off…
a-dub 5y ago
"data is fungible?" i think the author does not understand what fungibility means. if something is fungible, that means that any unit of it is exactly the same as any other unit of it. fungible data would be random data, which wouldn't help you predict or model anything. i think zillow also did not understand what fungibility means. real estate is not fungible. in fact, it's anti-fungible- that's why there are huge diligence processes that exist around most real estate transactions. maybe floors in an office building may be fungible, but residences are definitely not- with all their quirks, customizations and problems. this whole argument that they failed because the ceo of zillow didn't have big balls is pretty putrid. pair this with the word salad of misused words and twisting of quotes and i'd say this is probably one of the worst pieces i've ever seen posted here. i feel worse for having given it any time at all. the simple fact is that the ceo of zillow didn't know what they were doing, had a team that (supposedly) applied facebook's infrastructure scaling prediction library to house prices and then attempted to apply a market making mindset to the real estate market at scale. not only is this probably something nobody should try to do, considering we contribute so much in tax dollars to first time homebuying incentives as it's recognized that the housing market is where j q public can start to build wealth, it's also probably something that nobody could do (well, at scale, with machines) given that housing is not fungible. sure, financial instruments are fungible, maybe even late model cars, but definitely not houses. doesn't take a scientist to spot that.
ok123456 5y ago
This is the same folly as Long Term Capital Management. You're not going to be able to reliably model asset prices at the resolution and accuracy needed to front run the market for a long period of time. This case was worse because the "Zestimate" directly created a feedback loop that moved the underlying asset prices higher.
sam0x17 5y ago
Let's not forget there was also a huge public outcry on HN and in other places when it came out that they were buying and selling real estate. So if it was socially detrimental to the company's image, and they didn't have the corporate will to collect enough data for it to become profitable, it's a no-brainer that they would get rid of it. I see this as a victory for us calling out companies for immoral behavior.
- cowpig 5y ago
  Yeah, reading this article I couldn't help but notice that there would be a massive conflict of interest in Zillow entering the real estate market, and would probably create externalities. Seems like a win and strange that the author is so derisive of the company for doing what is probably in all of our best interests. It's telling that the article opens with a clip of Alec Baldwin talking about needing "brass balls" from Glengarry Glen Ross, seemingly oblivious to the fact that it's a dark comedy mocking cutthroat sales culture.
Reuzel 5y ago
Zillow lost money, because they were hit really hard during pandemic. This article does not mention that. Instead, the rest of the article deals with Linkedin-wisdom and hard platitudes, such that it is not possible to build a good model on someone else's data (as if Zillow even was). Data scientists remarking on the Zillow fold, are like psychiatrists or engineers remarking on non-clients and bridges build by others. They know nothing about the business, about the constraints, about how the estimates are consumed. They end up silly, but without good information coming from Zillow, we assign value to their analysis, purely on Twitter-soundbite-ability and internet-authority.
- starik36 5y ago
  How exactly were they hit hard during pandemic? Their main income stream is the funnel of services they provide (or get a cut of) through lead generation (agents, title, insurance, etc...). During the time of the pandemic, the house prices rose and so did the volume. If anything, they made out like a bandit.
civilized 5y ago
Reminds me of Boeing. You could replace Zillow with Boeing here and it would apply perfectly.
aruanavekar 5y ago
Realtor and Mortgage Lending industry is very slow in tech adoption and adaptation of digital strategy. Unless there is a lift across the industry on the buy-sell-marketplace together, such mishaps will occur. This industry will fail if injected with viral nature of social media algorithms.
speby 5y ago
If Zillow had "figured it all out" on solving the magic pricing problem, they could have put the entire appraisal industry out of business. Well, guess what, they didn't and they didn't even come close.
- speby 5y ago
  No "machine learning" model can successfully peer into the inside of the home, and in the walls, or in the plumbing, to get an accurate sense of the worth of the home because no machine learning model would have that information. It isn't available until you actually go and look, with real human eyes, into the house. No pricing model will ever get this right.

526 points by mjmayank 5y ago

zatkin 5y ago

What does 'bootstrap' mean in the context of this article?

ec109685 5y ago
In order to actually understand true risk (to create a profitable model), you’ll actually have to experiment and lose money in order to bootstrap your own ML model. Taking data acquired elsewhere and hoping it can make your own model instantly profitable isn’t possible.

xibalba 5y ago

The article offers no new or inside information, just more armchair quarterbacking. I'm surprised that it is getting traction on HN. I think it says more about the zeitgeist than it does about the (lack of) insightful-ness of the content.

gwern 5y ago
Yeah, it just repeats the Narrative in a giant post hoc. "Zillow uses ML models in some way; Zillow failed; QED, ML models are dangerous." Except the reporting by Bloomberg and insiders is that Zillow failed because they overrode the models predicting lower prices and bought like drunken sailors, and it's just a story of yet another marketmaker being run over by the market. So sad, too bad, largely irrelevant to the tech world, yet, it looks like it's entering the mythology of ML up there with Cambridge Analytica or the tank story - unkillable by mere facts or tardy reporting.
adjkant 5y ago
As someone who upvoted but didn't care much for the content, I think it's worth mentioning that sometimes/often I upvote for the currently occurring conversation to get more eyes, not for the link. I haven't seen too many specifics on the Zillow collapse and I've learned a nice deal through many of the comments here, most not having much at all to do with the article.

1cvmask 5y ago

The essence of the article is that they underestimated how flawed their algorithms are and how hard it is to build a good lasting algorithm in a dynamic world. Many seasoned wall street algorithms have suffered many times over 5 decades, and when they fail we call them black swan events.

mjdesa 5y ago
That’s not what I read in that article at all. What I read was that their data and methodology was flawed, and they weren’t willing to pay the price to fix it.
- human 5y ago
  Their methodology might have been flawed. The author is speculating. He uses Zillow to explain how datasets – especially the ones with money tied-in – can’t be trusted blindly. Building a high-quality dataset is an expensive endeavour.
- seoaeu 5y ago
  Zillow thought they already had enough data and accurate enough models to buy and sell houses profitably. The last two quarters proved they didn’t. In the first quarter they were puzzled by making too much money and in the second they lost a whole bunch The author is arguing that they should have pivoted from “we already have models” to “we’re intentionally gambling hundreds of millions of dollars so we can build good models over the next few years”. That might be a good strategy for a startup with loads of VC money and no other products, but it makes less sense for a more established company to risk going under on that bet
throwawayboise 5y ago
And wall street algorithms should be easier because securities are fungible. One share of AAPL is the same as another. Houses are not like that. Real estate is local, local, local. Every house has a hundred unique attributes that each potential buyer will value differently.

danielvaughn 5y ago

I think the article makes an interesting point about this being the first of many, but I disagree with the initial tone of the article. It seemed to paint Zillow as being afraid of loss. On the contrary, I viewed Zillow as demonstrating good common sense and an ability to make hard decisions. To me it shows that they aren't committing the sunken cost fallacy, and are willing to cut an entire 25% of the company and take massive losses so they can redirect themselves towards better objectives.

quickthrowman 5y ago
I agree, I think they realized it wouldn’t work and made a hard decision to save the company. Zillow realized the only time their ask was hit is when it was at a premium to the actual market price. If they used competitive offers, they’d never have the winning bid. In a hot market where you’re offering a premium, you’re going to have owners of lower quality properties accepting your offer, while owners of higher quality properties have more offers to select from. Zillow got left holding a bag of lemons and decided to get out before buying the whole lemon grove.
- marcinzm 5y ago
  >If they used competitive offers, they’d never have the winning bid. Why do you assume that, seems like a cash buyout would be a great deal for many sellers if it was at the appropriate price. Issue is I think that Zillow's information was less granular than what the buyers/sellers had. Let's say Zillow priced two houses near each other at 1million each. However one was close to a busy road so would only sell for $900k while the other could sell for $1.1. Zillow made the right average offer of $1million to both but the buyers/sellers actually had more information. So the 1.1m seller didn't take Zillow's offer while the 900k seller did. Now Zillow was out $100k essentially not counting fees.
  - rpvnwnkl 5y ago
    They are out 200k. They bought for 100 too much and will have to sell for a 100 less than planned.

black_13 5y ago

That it was a bad idea?

pid-1 5y ago

https://www.youtube.com/watch?v=ajGX7odA87k&t=833s https://www.youtube.com/watch?v=ajGX7odA87k&t=833s

marcinzm 5y ago

I think a key point that is missed is the feedback cycle time. Real time bidding advertising has I believe a number of the listed concerns however the feedback time is maybe hours at most and might be milliseconds. So the risk is in general a lot smaller and worst case you just lose some of the money you spent that day/week. With long term assets you could lose months worth of investments before your feedback loop fully kicks in.

droopyEyelids 5y ago

In the original Foundation books by Asimov, the conceit of "Psychohistory" was similar to the concept of machine learning for pricing: The future can be predicted _if people aren't aware of the prediction to change their behavior in relation to it_ This is similar to 'adverse selection' in real life & in Zillow's model. The article makes a nod to this, but seems to imply that if you train your model on that adverse selection, you can come out ahead after paying to learn about it. To me that kind of misses the point. Adverse Selection isn't a static feature of the landscape you can identify and avoid, it is people understanding what you understand, adapting, and responding. Train your model with adversaries trying to beat it, then you'll maybe counter the specific first round strategies they use, and they'll learn new ones and beat your new model with their 2nd round strategies. It's a continuous game. Your requirement to gather a corpus of training data will keep you in the 2nd turn of a game where the wins are biased to whoever has the 1st move.

JKCalhoun 5y ago
I'm reminded always of the Hunt brothers that tried (and failed) to corner the silver market in the 70's/80's: https://en.wikipedia.org/wiki/Silver_Thursday https://en.wikipedia.org/wiki/Silver_Thursday
- JanisL 5y ago
  Basically the COMEX changed the rules explicitly to disadvantage the Hunt Brothers. The changes made to margin requirements is what made the difference here. I don't think anyone could claim that the silver market is an entirely free market, I remember last year a press release where the COMEX said they weren't sure how much they actually had in their vaults in eligible and registered, with a plus/minus 50% figure being given on their estimates. I can't think of any other major market where someone would come out and say they didn't know how much inventory they had and that their best estimate could be 50% off. And the participants in the silver market are still rather ridiculous to this day: https://www.reuters.com/business/finance/jpmorgan-pay-60-mln-settle-precious-metals-spoofing-lawsuit-2021-11-19/ https://www.reuters.com/business/finance/jpmorgan-pay-60-mln...

JKCalhoun 5y ago

> At a high level, the story of Zillow Offers is a story of our industry at its best. Not in my book. All I see is the price of real estate being driven up by corporate greed and the individual home-buyer being shut out of the market. Is it wrong of me to hate "flippers" (be they corporate or private)? Pure capitalists will tell me that every property sold went to the highest bidder — in the case of a flipper winning they were willing (able) to risk the capital to hopefully turn a profit on the flip. I suspect if you dig deeper you might find sales going to flippers because they had 100% cash offers, because they are better at "the game". I see no reason to punish prospective first-time home owners in this sort of market. But I don't know what the answer is either.

chii 5y ago
flippers make the real estate market more liquid, in the same way high-frequency trading does for stocks. Flippers take the risk of the market falling while they're flipping - that's the price they pay for their profits.
- JKCalhoun 5y ago
  You'll have to educate me then: is more liquid better for the buyers or sellers?
  - loeg 5y ago
    Liquidity is good for both buyers and sellers.
  - javert 5y ago
    Both. In an illiquid market, it takes a long time to find buyers or sellers. You are incentivized to overprice (if selling) or underprice (if buying) and wait a long time to see if someone will match you. A liquid housing market means people can buy or sell the house at the "right" price without waiting many months or years. As a seller, would you rather wait a year to make a bit more money? That wouldn't be good. That would be crappy.

deleted 5y ago

[deleted]

skohan 5y ago

> They thought they needed to build a machine learning model when they really needed to build an entirely new organization, one that possessed the technical and cultural mindset necessary to succeed in this space. I totally agree. It's not impossible to imagine their model working: why couldn't you serve as a market-maker for homes at a large scale, especially with the unique insights Zillow could have based on their datasets. However I think where the hubris lay is in how they thought they could leapfrog all the way to an automated solution before building a competency as a house-flipping company. From what I understand, where they failed was partly in building a rich enough model to properly account for the less easily quantifiable elements which ultimately account for a property's value. I.e. the price per square foot might make a property look like a steal, while something like a sewer main nearby, or problematic neighbor could radically change the value proposition to anyone standing at the site. That's a non-trivial problem to solve for even the best ML and it's not clear how you would automate this. If you ask me, instead of focusing on building an automated price discovery system, they should have started by trying to build a quality home-flipping organization, and figuring out how to super-charge manual work using their datasets. Over time you might find ways to optimize the process and increase the level of automation to scale output relative to head-count.

lotsofpulp 5y ago
> why couldn't you serve as a market-maker for homes at a large scale, especially with the unique insights Zillow could have based on their datasets. Why would Zillow have unique insights? With the exception of Texas, I thought real estate sales information is public information in the US.
- ctvo 5y ago
  Can you not imagine how useful it is to know user data e.g. what neighborhoods receive the most clicks, what type of homes generate the most favorites, how long people view one listing vs. another, … that is unrelated to public MLS data?
  - lotsofpulp 5y ago

NoblePublius 5y ago

Buy low, sell high. You need “data science” to do this? $VNQ is up 33% since 2016. Do you realize how dumb and bad you have to be to lose money on real estate in this time? I imagine randomly picking homes off the MLS would have yielded better returns in the last five years than whatever Tableau-powered nonsense the biz ops analysts at Zillow used. The entire iBuying concept is a farce, completely divorced from basic fundamental analysis.

mgraczyk 5y ago

I really liked this quote, which is also true of machine learning organizations at large tech companies: The most valuable data is not social data, ... but your own data because every dataset that you’re looking at internally describes your own process, including your bugs, ... building models from your own data is the only way to build a really successful system. This is one thing that a lot of outsiders do not understand. Facebook/Google's data is basically worthless to anybody but Facebook/Google. The data has value because it is derived from their own processes, which in this case are the requests and context of each product surface.

ethbr0 5y ago
It makes sense when you ask the question another way: "What is the likelihood that a preexisting assemblage of data contains all the nuances for my specific process?" Some domains are intricately mapped in available data (e.g. equity pricing), but most, and especially most physical, are not (e.g. freight transportation).
deleted 5y ago
[deleted]
robbedpeter 5y ago
Yeah, I'm gonna say that romanticizing mass surveillance is a bit much. Cambridge Analytica, the five eyes countries, Clearview - all these are using Facebook and Google's data to great effect. Facebook and Google's data are not their own. That data is comprised of private lives, stripped bare pixel by pixel, bit by bit, and it's offensive to frame it as if they're doing something alchemical and special with it. Google's search dominance came from something special, creating the right algorithm and seizing the first mover advantage, but the relentless and ruthless invasion of privacy is a rent seeking race to the bottom. All of the ills of the internet and political turmoil in the west from algorithmic amplification are the brainchilren of Facebook and Google. It turns out that "tailoring search results" and "targeted advertisement" are excuses for something that can cost far more than a society might want to pay.
- mgraczyk 5y ago

JackFr 5y ago

While the point the article makes is true — it costs money to acquire the real world data, the comparison to credit underwriting is misguided. Underwriting credit is fundamentally different than predicting house prices. In particular when you’re auto-underwriting credit it’s not typically an origination-for-sale model. So the value of the loan is the present value of the future payments, less the future value of defaults, less the cost of acquiring the customer. Historically those things can be modeled pretty accurately and the aspects that can’t be modeled accurately can often be hedged or eliminated by the law of large numbers. The innovation of the new ML underwriting with respect to accuracy is at the margins. The real disruption is the speed and cost. (Disclosure: I worked at a SMB fin tech and we reran multiple credit models for a million customers and past customers every night.) If Zillow were getting into the rental business, in some ways it might have been easier for them. But they needed to model where they could sell an illiquid asset which is a much harder and much less well understood problem. And yes with enough capital to plow through and the appropriate risk attitude they could likely have gotten the handle on what their pipeline was really going to look like. But it’s hardly the same problem as credit underwriting.

_xnmw 5y ago

Good riddance. If large-scale house flipping took off, we might actually end up in a scenario where housing was treated as a speculative asset, with empty houses getting flipped between investors looking to make a quick buck, further lowering the supply of actual places to live (because housing units remain empty while being flipped), driving up the cost for families who just want a place to live. Oh wait...

h2odragon 5y ago
My wife did some work for the Census last year. Our extremely rural neighborhood has lots of unused housing, some for a decade+. That work got her out to see some of the places not visible from the roads, and increased our awareness of the scale of the problem. At a guess, in our county, 20%+ of the housing is idle, owned by out-of-state companies, some of whom pay property taxes and some dont. The county isn't auctioning off because of tax default anymore, no one was buying these places at $100. Many of these places are complete teardowns now; some actually no longer exist, having burned or apparently been scrapped. The tax assessments on those have not been adjusted, for the few i checked. I think the housing market is so fucked no one really grasps the scale of the problem.
- jason-phillips 5y ago
  > I think the housing market is so fucked no one really grasps the scale of the problem. I don't think I agree with this assessment. I live in a very rural area two hours northwest of Austin, literally in the middle of nowhere. I've studied the local economy and understand how things work here. I think the characteristics you've identified in the rural housing supply are not unusual and also not as serious in a practical sense as you seem to be indicating. For example, in San Saba, Texas, 20-30% of the households are under the federal poverty threshold. The median household income in the town of San Saba is about $32K/yr. People just don't have any excess cash so the maintenance on dwellings is neglected. That means folks become extremely thrifty and resourceful patching what needs to be patched, very cheaply, if not for free. Some dwellings simply aren't maintained and one day won't be there anymore. Families live on small budgets, don't require much and generally just "get by". The municipal and county governments have very small budgets but extremely resourceful staff who accomplish a lot with very little. Everyone comes together as a community when needed (see: February 2021 freeze event) and it all works very efficiently, actually. To someone who is not from here and who doesn't understand that dynamic, they might see those properties as you described and believe a tragedy was unfolding. But that doesn't reflect reality on the ground vis-a-vis my neighbors.

ridaj 5y ago

This is a good take, but > A machine learning organization thinks of risk entirely differently than an automated risk underwriting organization. It's possible and maybe even advisable to use machine learning in the automated risk underwriting business, but it is a different setup / set of objectives. As the author notes, IMO the adversarial and antifraud aspect of risk underwriting turns it less into a straight-up estimation problem and much more into a game theory type of problem. ML models can assist in evaluating risk, but you do indeed have to be preocuppied by your risk as a party to the transaction in the first place, and not just trying to predict prices as a third party observer (which by itself is pretty riskless).

goatherders 5y ago

This is really well written. Thanks for sharing.

deleted 5y ago

[deleted]

lifeisstillgood 5y ago

>>> you should expect to lose 50% of your capital allocated towards underwriting. How ?

flerchin 5y ago

> One of the things that happens for a brand-new launched credit card: done right, you lose about 50% of the dollar volume in the first several months What does this mean? 50% of the money is held as debt? Or 50% of the money is lost to fraud?

Petabits 5y ago
Getting people to initially sign up through bonuses causes a lot of money to be shed, and are thus not profitable until people renew (without the bonus) the second year. I remember seeing the CEO of Chase saying he was excited that they lost billions in the new sapphire card because it meant they had so many members

abiro 5y ago

I think the title is highly misleading. The main point here is that Zillow simply had no idea what it takes to be a market maker and their pool was picked off by savvy traders. Good tweetstorms with technical explanations on how that happened: https://twitter.com/macrocephalopod/status/1455887352371597312 https://twitter.com/macrocephalopod/status/14558873523715973... https://twitter.com/0xdoug/status/1456032851477028870?s=21 https://twitter.com/0xdoug/status/1456032851477028870?s=21

treis 5y ago
I'll second that this article is just wrong. Zillow burned plenty of money in their Offers business. The problem is that all that spending revealed that they performed poorly in a questionable market segment. Ultimately they were really bad as flippers. More often than not paying more than market price for the homes they bought. I think the root problem is that this was a panic move. They saw Open Door's success and thought they had no choice but to try and replicate it. But its a questionable business move for Zillow and ultimately they couldn't make it work
MisterBastahrd 5y ago
Zillow offered to buy my home at 30% more than everyone else in the market for cash, without an inspection, and I wasn't even looking to sell it at the time.

Petabits 5y ago

Would it be too dystopian if governments sectioned off certain neighborhoods and set price caps per sqft? This would make it so speculative investors are unable to build capital in houses, thus leaving homes for actual people. I'm not super familiar with land grant homes, but the prospect of seemingly fixed price homes seems to prevent investors from buying in.

tinyhouse 5y ago

If you have a good business with high margins, why not grow that business instead of starting a new low margins business of flipping houses?

jdross 5y ago
Because they were having a lot of trouble growing that business. See their earnings reports before they entered iBuying

rossdavidh 5y ago

While no doubt Zillow made many of these mistakes, I think the reality is more sobering that the author of the article realizes. The more grim possibility, is that Zillow got out of the house buying business, not because they weren't good enough at it, but because they _were_ good enough at it to realize that it was at the top. If buyers want more now for their house, than it can be sold for in a few months time (which is necessary for renovations and other prep for sale), then there is no ML (and no non-ML) method to make money. Either you overpay and lose money, or you don't overpay and you don't buy any houses. In that situation, the only smart play, is to get out of the market. Zillow is, no doubt, not perfect. But they have a lot of knowledge of the housing market, and they thought it was time to get out entirely. I think the author of the article either isn't able, or doesn't want, to consider that Zillow might have been exactly correct in doing so.

dsizzle 5y ago
But they lost money last quarter while the market was still rising. Seems there was some problem with their prediction process.
- sudosysgen 5y ago
  Their plan wasn't to flip homes in a month or two as far as I'm aware so that's expected if they get out.
- rossdavidh 5y ago
  Rumor is that they had to put their thumb on the scales (i.e. tweak the model) to get enough sellers to sell to them. In other words, if paying what their model actually thought was the right price, not many people sold to them. Instead of saying "our division's whole business model won't work, you should fire us", they tried to cut the margin too close, resulting in losses which got the CEO's attention to the problem. This kind of thing is difficult to confirm from the outside, of course. But that they adjusted the model to pay more towards the end is pretty widely known.
  - 5y ago

PaulHoule 5y ago

I can't agree with the article or many of the comments on it. (A) Both Wall Street and Machine Learning Modelers struggle with tail risk. Hedge funds measure performance against https://en.wikipedia.org/wiki/Sharpe_ratio https://en.wikipedia.org/wiki/Sharpe_ratio which assumes risk is (i) normally distributed and (ii) a source of reward. For most people, however, risk looks like Theranos or the Fukushima accident or the Challenger distaster. It's unbelievable that a machine learning model trained to predict house prices based on experience would be accurate in the face of events like the COVID-19 pandemic or what will happen when the Fed raises interest rates. You can model risks like that, but to the extent that you're working from experience you are working from a database from the 1929 Crash, South Sea Bubble, etc. (B) Mark Levine wrote a good article about how you'd exploit such a predictive model. If you consistently gave people low offers, a few people would accept them. You would get a high rate of return but could invest little capital. To invest more capital you have to make more offers that get accepted, that is, give better prices. Your rate of return goes down and if there is shrinkage from errors, accidents, etc. you could get a negative return. It's that "tendency towards a declining rate of profit" that Marx warned about. (C) The analogy with stock market market makers doesn't sound good when you consider the differing timescales. Market makers are isolated from some risk because of the length of their holdings. Yet, they make profits by exploiting the stochastics of a stationary market (e.g. if you don't like the price at time t1, you will usually get a better price at t2) but they lose money when markets move definitively in one direction or another. That kind of trader heads for the bathroom when things go South and in the interest of being orderly markets impose sanctions on market makers who do the natural thing and press the "STOP & UNWIND ALL POSITIONS" button when it gets tough. In the case of Zillow I see holding times that go on for weeks or months and all kinds of real world risk like planning to do certain renovations but having to delay the work because out of 20 things you need from Home Depot they only have 16 of them.

leot 5y ago

Real estate is one of the few markets where non-experts can make money, where it’s not a hyper-liquid winner-take-all game. Coupled with this is the fact that housing is a necessity and owning a home leads people to invest in their communities more than if they were renting, I think it’s a good thing if Zillow (and OpenDoor, etc.) fail at pushing everyday people out of the business of real estate investing. Here’s hoping we see some regulation—the illiquidity of the home buying market is not a problem that needs to be solved.

jdross 5y ago
Opendoor doesn't compete with real estate investors, they compete with realtors and mortgage brokers. Opendoor's primary benefit is to enable people to move when they otherwise could not easily do so, creating more liquidity and matching supply and demand (often number of bedrooms in house to number of bedrooms now needed). The challenge with moving is that most people need to sell their current house before they can afford (or even know what they can afford) to buy their next home. Opendoor lets a family buy that next home with its cash, then list their current home on the market or sell it to the company so they avoid the double mortgage or double move (home->rental->home)
- robocat 5y ago
  Does Opendoor avoid some of the standard x% realtor fees on either or both of the transactions? Reduced fees could easily make a huge difference to expected profitability. In contrast, “Zillow Seeks to Sell 7,000 Homes for $2.8 Billion” so Zillow lost more than a few percentage points.

ezconnect 5y ago

They lost money because they were gaming their own system for their own profit.

dboreham 5y ago

They build an AI that perfectly emulated Wall St masters of the universe.

csours 5y ago

Your data is not neutral, it is opinionated. Who is asking the question? What do they use the data for? What questions are they not asking?

jedberg 5y ago

Zillow's mistake is that they thought their AI could replace human buyers instead of augment them. Most AIs today are for augmentation, not replacement. Vehicle autopilots are a perfect example. The ones that are commercially available aren't capable of replacing the human, they just augment the human's abilities.

wiradikusuma 5y ago

Does anyone know where Zillow get its dataset from? I reckon it's essentially sale price? Can a "hobbyist" investor do the same?

deleted 5y ago
[deleted]
andromeda-brain 5y ago
There's a lot of information that is only available to MLS members. Zillow used to not have access to this information, but they slowly brokered deals with MLSes around the country to get it. One example: The MLS in Austin, TX recently banned publicly sharing a home's sold price. https://www.zillow.com/austin-tx-78701/sold/ https://www.zillow.com/austin-tx-78701/sold/
- MisterBastahrd 5y ago
  Yeah, each MLS org has its own data set which is a giant pain in the ass for the newspapers who are publishing listings for the MLSes in their areas. I don't know if they ever standardized it, but I know that one of the first tasks I had as a new dev for a newspaper back in the early 00s was to build a tool to take the data and normalize it into a single CSV.

jdross 5y ago

I think some reasons Zillow lost were that their pricing and risk processes were terribly underdeveloped in order to scale fast, their models were obviously inaccurate, and they didn't understand the difference between an acquisition cohort and resale cohort, and specifically how much the tail sales of an acquisition cohort determines profitability.

abernard1 5y ago

> Because when you have a hammer, everything tends to look like a nail and when you have TensorFlow, everything tends to look like an ML problem. And if you have billions of dollars in cheap capital, everything looks like an investment problem. Which is ultimately the suggestion of this article: "Why aren't you more like Wall Street?" The implications are exactly the opposite of Zillow being an innovative company. If they require billions of dollars in deep pockets (nbd) and a restructuring of their org to be more like old-school operators, all signs point to existing players as more fundamentally correct about the strategy required to succeed in the space.

dr_dshiv 5y ago

Is it fair to call this the result of “AI thinking?” Meaning that urge to automate away human involvement, because —after all—-if people are involved in analyzing data and decision making, then clearly the AI isn’t finished let.

MisterBastahrd 5y ago

SMEs are smarter than developers in their space. Always has been that way. Always will be that way. AI is great for when you need to tame a firehose and make millisecond decisions. But there's a 90 year old in Omaha who is better than the best AI.

cbsmith 5y ago

The amount of Monday morning quarterbacking of Zillow is just staggering.

gcanyon 5y ago

If the assumption is that you're going to lose half your money up front, then my plan would be to make sure "my money" is as little as possible: learn based on smaller bets. It sounds like Zillow built the Sea Dragon first, when they should have started with the Redstone and moved toward the Saturn V. If Zillow thought they had all the data they needed, there would have been little harm starting with $100 million in properties -- if the loss there ended up being $5 million, they would have known immediately something was up and that they had work to do.

wly_cdgr 5y ago

There's something really funny about white collar office worker businessmen talking about how it takes balls of steel to do what they do. Ok bro, sure. Trackballs of steel maybe

jollybean 5y ago

All of this reads like a Dickensian nightmare, where corporations have bought up all the water and air. This is ridiculous, we need much better regulation on this stuff. I wonder if higher property taxes would help a bit? If you own a 'home' then you're going to be paying for the water, school, electricity infrastructure whether you use electricity, water, or not. Of course, that would be gamed hard and would have to be strongly regulated as well. But that, and vacant property taxes, limits on some other things, and some other adjustments might help.

mistrial9 5y ago

I wonder why so few here question the basic assumption of injecting from above, machine-learning models to extract profit, into a vital part of the reproductive cycle of human families.

dcposch 5y ago

Framing it as machine learning undersells the problem. It's a hybrid model trading in an adversarial, real-dollar environment. The leverage comes from having a small human team trade big volume, much more than they could possibly trade directly, by augmenting their human abilities with automation and a model. Or seen from the other side, it's a model with human oversight. Any system like that is high risk, high reward. All the successful ones started out by losing a lot of money. Paypal lost an incredible amount to fraud before they started breaking even. OpenDoor lost an incredible amount to mispricing, and took on a ton of balance sheet risk, before their business really started working. "To live, you must be willing to die" - poker legend Amir Vahedi

damanamathos 5y ago
I think Opendoor still does poorly in new markets, but then it improves as they work out the quirks of the local market and start asking the right questions and collecting the right information. A big part of Opendoor is creating the right apps and processes to collect this information to feed their models. The machine learning part is important, but can give the false impression it's just about data scientists crunching numbers at head office, when in reality there's a huge real-world operational machine that's driving it.

Dowwie 5y ago

Feels analogous to the history of the collateralized debt obligation debacle where the models used to value CDOs were trained on data that no longer resembled reality. At least Zillow can live to fight another day, where as Stan O'Neal put all of Merrill Lynch's chips in with one of the biggest make-or-break gambles in the history of finance and the market turned against it, rendering Merrill to a fatally wounded company bailed out by Bank of America.

Animats 5y ago

Well, maybe they just exited because we're going into a recession and it's a good time to get out of house-flipping.

nickkell 5y ago

I love this guy’s movies. Finding out he writes so articulately to boot? Wow

88913527 5y ago
You're thinking of Steve Buscemi. This article is authored by Steven Buccini, a Software Engineer and former political candidate.

m3kw9 5y ago

Buying assets using models, I’ve seen that in stocks, but people usually don’t go all in with how hard it is to predict the economy

vasilipupkin 5y ago

“You cannot bootstrap off an existing dataset. Full stop. These datasets can contain implicit assumptions or associations that you are not aware of. This is the original sin of many a algorithmic risk underwriting startup” False. You can definitely bootstrap and adjust the model as you either gather more data yourself or get more outside data. You can also build confidence intervals around the model predictions and decide how you want to proceed based on that. There is lots you can do with that initial model.

rajacombinator 5y ago

Good luck getting a public company to incentivize the people capable of pulling this off…

a-dub 5y ago

"data is fungible?" i think the author does not understand what fungibility means. if something is fungible, that means that any unit of it is exactly the same as any other unit of it. fungible data would be random data, which wouldn't help you predict or model anything. i think zillow also did not understand what fungibility means. real estate is not fungible. in fact, it's anti-fungible- that's why there are huge diligence processes that exist around most real estate transactions. maybe floors in an office building may be fungible, but residences are definitely not- with all their quirks, customizations and problems. this whole argument that they failed because the ceo of zillow didn't have big balls is pretty putrid. pair this with the word salad of misused words and twisting of quotes and i'd say this is probably one of the worst pieces i've ever seen posted here. i feel worse for having given it any time at all. the simple fact is that the ceo of zillow didn't know what they were doing, had a team that (supposedly) applied facebook's infrastructure scaling prediction library to house prices and then attempted to apply a market making mindset to the real estate market at scale. not only is this probably something nobody should try to do, considering we contribute so much in tax dollars to first time homebuying incentives as it's recognized that the housing market is where j q public can start to build wealth, it's also probably something that nobody could do (well, at scale, with machines) given that housing is not fungible. sure, financial instruments are fungible, maybe even late model cars, but definitely not houses. doesn't take a scientist to spot that.

ok123456 5y ago

This is the same folly as Long Term Capital Management. You're not going to be able to reliably model asset prices at the resolution and accuracy needed to front run the market for a long period of time. This case was worse because the "Zestimate" directly created a feedback loop that moved the underlying asset prices higher.

sam0x17 5y ago

Let's not forget there was also a huge public outcry on HN and in other places when it came out that they were buying and selling real estate. So if it was socially detrimental to the company's image, and they didn't have the corporate will to collect enough data for it to become profitable, it's a no-brainer that they would get rid of it. I see this as a victory for us calling out companies for immoral behavior.

cowpig 5y ago
Yeah, reading this article I couldn't help but notice that there would be a massive conflict of interest in Zillow entering the real estate market, and would probably create externalities. Seems like a win and strange that the author is so derisive of the company for doing what is probably in all of our best interests. It's telling that the article opens with a clip of Alec Baldwin talking about needing "brass balls" from Glengarry Glen Ross, seemingly oblivious to the fact that it's a dark comedy mocking cutthroat sales culture.

Reuzel 5y ago

Zillow lost money, because they were hit really hard during pandemic. This article does not mention that. Instead, the rest of the article deals with Linkedin-wisdom and hard platitudes, such that it is not possible to build a good model on someone else's data (as if Zillow even was). Data scientists remarking on the Zillow fold, are like psychiatrists or engineers remarking on non-clients and bridges build by others. They know nothing about the business, about the constraints, about how the estimates are consumed. They end up silly, but without good information coming from Zillow, we assign value to their analysis, purely on Twitter-soundbite-ability and internet-authority.

starik36 5y ago
How exactly were they hit hard during pandemic? Their main income stream is the funnel of services they provide (or get a cut of) through lead generation (agents, title, insurance, etc...). During the time of the pandemic, the house prices rose and so did the volume. If anything, they made out like a bandit.

civilized 5y ago

Reminds me of Boeing. You could replace Zillow with Boeing here and it would apply perfectly.

aruanavekar 5y ago

Realtor and Mortgage Lending industry is very slow in tech adoption and adaptation of digital strategy. Unless there is a lift across the industry on the buy-sell-marketplace together, such mishaps will occur. This industry will fail if injected with viral nature of social media algorithms.

speby 5y ago

If Zillow had "figured it all out" on solving the magic pricing problem, they could have put the entire appraisal industry out of business. Well, guess what, they didn't and they didn't even come close.

speby 5y ago
No "machine learning" model can successfully peer into the inside of the home, and in the walls, or in the plumbing, to get an accurate sense of the worth of the home because no machine learning model would have that information. It isn't available until you actually go and look, with real human eyes, into the house. No pricing model will ever get this right.