12 ms·

> They thought they needed to build a machine learning model when they really needed to build an entirely new organization, one that possessed the technical and

• lotsofpulp 5y ago
  > why couldn't you serve as a market-maker for homes at a large scale, especially with the unique insights Zillow could have based on their datasets. Why would Zillow have unique insights? With the exception of Texas, I thought real estate sales information is public information in the US.
  • ctvo 5y ago
    Can you not imagine how useful it is to know user data e.g. what neighborhoods receive the most clicks, what type of homes generate the most favorites, how long people view one listing vs. another, … that is unrelated to public MLS data?
    • lotsofpulp 5y ago
      Maybe, but I was under the impression that Redfin/Trulia/Realtor.com would have the same information. Also, unless Zillow started imposing confidentiality agreements on their bids, then competing buyers would just have to bid $1 more without their dataset, right?
      • ctvo 5y ago
        Zillow is the largest aggregator. They own Trulia. I can squint and see the thought process here by Zillow, though execution, as evident, did not go as planned.
        • lotsofpulp 5y ago
          I somehow missed they had bought Trulia way back when.
    • indymike 5y ago
      > Can you not imagine how useful it is to know user data e.g. what neighborhoods receive the most clicks, what type of homes generate the most favorites, how long people view one listing vs. another, … that is unrelated to public MLS data? Click data is much less valuable that the recent sale price data available in MLS. Using 90s style dwell time and click counts would likely yeild a lot of very noisy data. False positives from people's browser reopening with 15 tabs looking at different houses. False positives from social and paid advertising boosting a particular home or neighborhood's numbers. False positives from enterprising real estate entrepreneurs doing everything they can to get the clicks up in areas they own property to drive up prices. Meanwhile, the recent sale prices tell you much more, with certainty and are very expensive to manipulate.
      • sokoloff 5y ago
        Surely Zillow was also using that data.
        • 5y ago
  • iandanforth 5y ago
    One example, User search behavior should contain several leading indicators.
  • skohan 5y ago
    My understanding is that they believed they had an advantage in terms of buyer intent information. Everyone can see who buys what, but Zillow has access to more information about how people shop for homes, and the events leading up to the actual sale.
  • sokoloff 5y ago
    I have given Zillow a lot of non-public information over years of searching. How many people search on bedrooms but not bathrooms? When people search on both, what’s the pattern they use? If we highlight prices and BRs on the map does that give more clicks than just prices? How important are photos (times 50 different questions there)? How strong a signal is repeat views spaced over time? Saving a house to favorites? Sending a link to a friend? Clicking on comps in the neighborhood? Which comps do people zero in on (as evidenced by spending more time on the page)? How strong a signal is sending a message to the real estate agent on the listing? What areas of the country are seeing an uptick in search traffic? How long between claiming a house as an owner on the site, updating the information, and listing it for sale? They are sitting on a (well-earned) treasure trove of data and it’s not unreasonable to think they could use that to be better informed than another buyer without that information. Where they seem to have failed is in not augmenting that advantageous data with regular old boots-on-the-ground observations.
    • silvestrov 5y ago
      I think the opposite: there is not much valuable data, it is just noise. It is very difficult to go from what users browse to what they actually buy. People very often say one thing, then do something completely different. And sometimes they browse stuff just to make sure that their current decision is correct, so they will look at a lot of items they're not going to buy. (oh, and everybody and their mother knows photos are important. No need for ML to find that out)
      • sokoloff 5y ago
        Do you think that A/B or multi-variate testing works in general?
        • silvestrov 5y ago
          In some cases A/B testing works very well and in other cases not at all. So it is easy to test UI changes, but difficult to find out why people do what the do.
  • deleted 5y ago
    [deleted]
• johnebgd 5y ago
  People forget that tech is able to automate workflows. You don’t often yield success when you attempt to automate and invent the workflows in parallel.
  • spywaregorilla 5y ago
    I would say the opposite is true. Dying companies are stuck in their own routines because they're trying to automate their poorly designed processes that require humans at multiple steps. Smart companies are designing newer, better processes that are enabled by tech. Starting from scratch can be a huge advantage.
    • javajosh 5y ago
      This is a statement I would have agreed with wholeheartedly 20 years ago, and that I disagree with wholeheartedly now.
      • spywaregorilla 5y ago
        I'd be curious to learn why. I've seen the pain of companies tricked into thinking robotic process automation to do their horrendous excel workflows is a good idea. I've seen the benefit of a decent python data engineer with a small AWS budget. The techier folks definitely have a different set of problems but the speed at which hings get done is night and day. Companies with old school work patterns (which, in my personal experience, means dusty old banks) are terminally entrenched in their ways.
        • mattcwilson 5y ago
          I think you’re both right. Taking some hopelessly byzantine, spreadsheet-driven process and “automating” it by building a Rube Goldberg scripting framework around it is the kind of totally stupid automation that doesn’t work. Actually getting down to surface level and understanding fundamentally what each of those humans is accomplishing via those spreadsheets, extracting that all the way back out to a domain model and process flow diagram, and then selectively replacing process steps, whole cloth, with tech designed to be an actual subservice with SLA targets, is the right way to do it. Throwing the spreadsheets and/or humans out altogether and starting “from scratch” is so exceedingly and needlessly risky from an information loss and hubris point that, well, good luck, but you’re nearly certain to fail.
• draw_down 5y ago
  That sounds hard. Why don’t we just rub some machine learning on it?
• JumpCrisscross 5y ago
  > a market-maker for homes at a large scale…a house-flipping company These are different things. Archetypal market making involves simultaneously buying and selling an asset. Flipping involves buying, improving and later selling. One might be able to deal with the heterogeneity of houses by operating at scale. (Zillow attempted this.) One might also deal with the delay between buying and selling by hedging. (Zillow never seems to have thought about this.) But the improvement function makes what Zillow attempted fundamentally separate from market making. They weren’t paid to provide liquidity. If anything, they paid a premium for scale and immediacy. They were a real estate operation masquerading as a tech outfit. WeWork in different stripes.
  • wpietri 5y ago
    > Archetypal market making involves simultaneously buying and selling an asset Does it? I worked for a few years for a market maker, and that's not what we did. Simultaneous buying and selling is what the arb guys did. We'd buy and sell with generally short hold times. Which makes sense to me given that the exchange has market makers to provide liquidity. If something can be simultaneously bought and sold, then the market-maker is unnecessary.
    • JumpCrisscross 5y ago
      > that's not what we did Archetypal, not predominant. > Simultaneous buying and selling is what the arb guys did. We'd buy and sell with generally short hold times The ideal market maker is arbitraging (and eliminating the arbitrage-able inefficiency). That’s why humans were replaced by faster-trading machines everywhere they could be. In most cases, the arbitrage is synthetic or approximate, e.g. hedging an options or swaps book. But a fundamental separation between speculating and marketing making is the latter does not take a view on the assets per se, and should not be betting on their future price movement. No market maker always achieves the ideal. But they tend towards it. Zillow didn’t have that tendency. In fact, they erected fundamental obstacles between themselves and that ideal.
      • wpietri 5y ago
• martincmartin 5y ago
  [W]hy couldn't you serve as a market-maker for homes at a large scale, especially with the unique insights Zillow could have based on their datasets. Indeed, I believe this is what OpenDoor does. From The Economist article [1], "They [OpenDoor] charge a fee for the services they provide: buying and selling homes immediately, with zero fuss. The quick in-and-out makes them more like marketmakers than property investors, who buy to hold. ... "A former Zillow employee told Business Insider that management had been hellbent on catching up with Opendoor, the front-runner. In order to compete, the employee alleged, the company pushed to offer generous deals to potential clients. It called this “Project Ketchup”. Now it has its own fake blood on its hands." [1] https://www.economist.com/finance-and-economics/2021/11/13/a-whodunnit-on-zillow https://www.economist.com/finance-and-economics/2021/11/13/a...
• pge 5y ago
  Isn't is also true that the original pricing algorithm was built for a very different purpose? It was useful for getting a ball park estimate of value, but it was hardly accurate in the underwriting sense (for the reasons you point out). The hubris of assuming that those prices were so accurate that Zillow was willing to buy at them sight unseen is mind blowing, particularly when one takes into account the adverse selection (if Zillow's estimate is above what I can get from in-person bidders, I am more likely to take it than if the error is in the other direction).
  • lordnacho 5y ago
    Yes, you might discover that your average price is accurate, which is just fine for a reporting site. But beneath that there could be some structure, for instance there might be blobs of houses that your model makes too cheap vs reality, and blobs that are too expensive. If those are identifiable, eg via some sort of local knowledge, you might find that people will sell you houses that you've marked too high, but you can't buy the ones that you've marked too low.
    • spoonjim 5y ago
      Totally. There’s a surly creep who lives on our street and the houses next to his are worth less because of that. But Zillow would never know that.
      • hamburglar 5y ago
        This “annoying neighbor” problem keeps getting cited as an example of something that affects housing prices that the automatic algorithms don’t know about, but does it really exist? What is the mechanism by which the potential buyers find out about the annoying neighbor? I’m highly skeptical. I’ve had annoying neighbors before and there is zero chance that any potential buyers knew about them when I sold. Are people interviewing all the neighbors before making a house purchase? Are those neighbors not incentivized to withhold any negative information about the neighborhood because it affects their own property values?
• SteveGerencser 5y ago
  Or a house full of cats. I had a 'cat lady' friend who struggled to sell her home because she had 13 cats. 13 'indoor' cats. Even at a great price the house would not sell. Enter the wonderful folks at Zillow that bought her house based purely on the numbers. Last I heard they still hadn't been able to move that house at any price.
  • zionic 5y ago
    Toxoplasmosis is a scary thing.
    • halfmatthalfcat 5y ago
      Not exactly. From what I read it’s almost completely benign in most humans.
      • willcipriano 5y ago
        Certainly it isn't a issue after they cats no longer live there and it's been cleaned to a reasonable standard. If it knocks 20 grand off the price of the house it's worth spending 2k to have everything deep cleaned.
        • mint2 5y ago
          Cat pee permanently stains flooring and is also extremely hard to get the smell out. 2k will not be nearly enough if there’s extensive cat damage. Wood floors turn black with it and must be replaced.
          • tartoran 5y ago
            Yes, even after extensive renovations cat pee smell can persist and some people are bothered by that smell. Im one of those people but I do like cats and wouldn’t mind having cats if they wondered around the neighborhood rather than be inside only.
• EMM_386 5y ago
  > I.e. the price per square foot might make a property look like a steal, while something like a sewer main nearby, or problematic neighbor This is the real problem. Even if they have the historical data for that exact house/unit, it won't help them in cases such as: * That nice view of the woods out the window is now blocked by a massive radio antenna that was just built there * The river running through the back yard is now heavily polluted by something up-stream * The new neighbor across the street is a huge nuisance and says they will never move * The house just had a mass-murder event in it Just because something is now cheaper than "comps" at price/sq ft and other metrics doesn't mean it's comparable.
  • ricardobayes 5y ago
    anectodal evidence, but yes, an apartment I lived in was 30k cheaper than everything in the house, and the estimate. So I was really happy to have caught a bargain. Then after moving in it turned out the neighbors were unbearable. The father had developed a DIY habit during covid and he would regularly put together furniture at 11PM. Then they had two monsters for kids. Would literally jump around uncontrollably for hours, and for some reason, through some defect or lack in the sound proofing, the sounds were even amplified. Every time they went down to the playground the kids were literally screaming at the top of their lungs the whole way down. I nearly lost my sanity and fortunately could sell it at at 2k loss in a year.
    • lisper 5y ago
      You could sue your seller for failure to disclose.
      • desmosxxx 5y ago
        "i never really noticed"
        • lisper 5y ago
          Yes, you'd have to be prepare to counter that with e.g. testimony from neighbors, or friends who heard the previous owners complain. It's not a slam-dunk, but it is an actionable tort.
• it_does_follow 5y ago
  > a machine learning model Not to mention that generally ML models are not useful for assessing risk. ML nearly always focuses almost exclusively on some point estimate rather than a distribution of what you believe about a value. The former case is all about expectation and the latter about variance. Correctly modeling variance is far more essential to risk modeling than expectation alone. I recall talking to a startup that was attempting to model credit risk by building a binary classier for defaulting, and trying to figure out a way to use this to score people for credit (obviously they chose to ignore the fact that there is a huge industry with decades of experience in assessing consumer credit risk). They focused exclusively on finding more advanced models to get better AUC without even realizing that that's not important. I mentioned that the most simplistic credit score model should at least model P(default|info) and then set the interest rate to - P(default|X)/(P(default|X)-1) to break even and they couldn't comprehend this basic reasoning. It was doubly hilarious since their population's base default rate was such that the solution to this equation was higher than the legal limit they could charge for interest. In the early part of the current startup/tech boom there was a focus on "disruption", the idea that new ideas could easily dominate old ways of doing things. But for many industries, such as credit/lending and real estate, you should at least understand the basic principles of how these "old ways" work before trying to disrupt them.
  • JoeyBananas 5y ago
    > Not to mention that generally ML models are not useful for assessing risk. ML nearly always focuses almost exclusively on some point estimate rather than a distribution of what you believe about a value. It is actually quite a common practice to design neural networks that output probability distributions.
    • it_does_follow 5y ago
      That distribution is still a point estimate for a multinomial, not truly the distribution of your certainty in that estimate itself. This is essentially a generalization of logistic regression, which will of course give the probability of a binary outcome, but in order to understand the variance of your prediction itself you need to take into account the uncertainty around your parameters themselves. This can be done for neural networks, through either bootsrap resampling of the training data or more formal bayesian neural networks, both of these are fairly computationally intensive and not typically done in practice.
• poulsbohemian 5y ago
  Yes!! You nailed it! I spent a long career in software and now work in real estate, and you are spot on that they are not a company who understands real estate well enough to be buying and selling it. There are plenty of bad real estate agents in the world, but the amount of know-how and connections that good ones have is exactly the encapsulated in the examples you gave - local, specific knowledge that a national/international player isn’t going to have and isn’t going to be able to scale without a whole lot of human investment… gee wiz, kinda like real estate firms. I wish I had the data that I assume they have internally, because watching their actions I’m not convinced they understand what questions would actually be interesting to explore with ml.
• winternett 5y ago
  It's a pretty interesting discussion as I sold my house just this year, and was frequently watching Zillow trends and information. Originally the estimate on Zillow said my house was 20% over the value I actually sold my house for just last month. I listed with a traditional realtor for a 5% commission, because when I looked up the service and other fees for Zillow sales, I found they included around 20% of cost for buying homes and closing within generally 10 days. As I listed my house, and as I reduced price on it for it to gain attention, I noticed the zillow estimate also went down to always stay below my listed price. I believe the estimate that both Zillow and Redfin display prominently were purely based on what my list price was changed to last, not on any meaningful algorithm, which can be very harmful to sellers and buyers, because it makes the process a bit deceptive by nature. Luckily Zillow also displays the price history on homes, which apparently cannot be "gamed" as much as the "zestimate" can be. Another thing I noticed was that the view stats on my listing that zillow regularly provided changed, even after days passed, that was very concerning because stats of that kind aren't supposed to change... They indicate real interest in a property, that guide decisions for sellers to reduce price, and they also indicate what is truly a "hot home". No matter what, there is always the "human factor" that can corrupt or even destroy any company, where realtors can game the process to maximize their own sales profit or positions, or where appraisers can inflate an estimate as a favor for a personal friend, even despite laws against doing so. In a bad economy, the lengths people will go to to suit their advantage are wild. This type of issue can never be properly addressed by any algorithm, and that's why trusting technology too much can so easily lead to failure in any setting. Ultimately I am glad I did not sell to Zillow, because of all of the potential for hidden costs and because they manipulate the process even when you don't use their service, but I am not feeling sorry for them as a company... I felt the impact of their presence in the market whether I involved them or not, and that's a big problem when it comes to preserving the value of traditional investment and stable investment in a house that should be properly addressed by regulation.
  • breischl 5y ago
    >as I reduced price on it for it to gain attention, I noticed the zillow estimate also went down to always stay below my listed price. I believe the estimate that both Zillow and Redfin display prominently were purely based on what my list price was changed to last, not on any meaningful algorithm The fact that a house is for sale at a given price, but has not sold after some time, is a strong signal that it's overpriced. The longer it's been sitting, the stronger that signal is. They'd be crazy not to include that data in the Zestimate. Now, if it's extremely fast, eg they adjust the price down within a day or so, then it seems a little ridiculous. OTOH the Zestimate has always been a rough indicator at best.
• opportune 5y ago
  I don’t think it’s just that they had a poor model, but the combination of that and adverse selection. If you pledge to purchase at the Zestimate then people who reasonably think they can get more than the Zestimate on the open market don’t have an incentive to sell their house to Zillow (besides convenience). But people who think the Zestimate is an over estimate will of course sell to Zillow. So instead of a normal distribution of actual value:estimated value you end up with a skew towards the end where the estimate is over the actual value. Trading housing is very different from normal market making because houses are not fungible commodities like most securities are. For most entities trading securities at low frequency it does not really matter whether a market maker skims off a few pennies on their trade; it’s worth it for the liquidity. Houses are less liquid (because they are non fungible) so the liquidity is more valuable, but the price improvement routing around a MM can also be many percentage points of a trade because there are not only so many factors affecting their valuation, but also just chance and random noise (bidding war, a particular buyer falling in love with the property, not-price-conscious buyers).
  • emodendroket 5y ago
    Well, that makes sense, but wouldn't it sink Opendoor too then? They are doing fine, as far as I know.
    • icelancer 5y ago
      Rabois talks a lot of noise about how well they're doing, but Opendoor is a miniscule fraction of the size of Zillow, and it's hard to know exactly how good Opendoor is doing anyway. We'll see as the years roll on with a lot of economic uncertainty coming up.
  • perl4ever 5y ago
    >adverse selection According to Matt Levine's recent column, while you might think that, it wasn't what sunk them in practice. Bidding low in fact worked; it just was inherently limited in scale, which is why they switched to bidding higher. Unfortunately, being wrong in the other direction is very bad. "I know, I know, the traders are saying: “No, this is stupid, your algorithms will not be 100% precise, some of your ‘lowball’ bids will in fact be too high, and those will be the ones that sellers accept. You’ll get adverse selection and end up losing money.” But that was not Zillow’s actual experience in the first quarter! The actual experience is presumably that some people accidentally got too-high bids, realized they were good and accepted them, but mostly Zillow sent too-low bids to everyone, and some people, for whatever irrational reason — market ignorance or financial necessity or laziness or whatever — accepted the too-low bids. The general point is that there is no reason at all to think that the people on the other side of these trades from Zillow are generally better informed than Zillow is. Sure they know more about their houses than Zillow does, but Zillow knows more about the market, and has more money" "If you systematically bid too low, you will not do many trades, but you will make a lot of money on each trade. If you systematically bid too high, you will lose money on each trade, and also you will do a whole ton of trades. This is much worse!"
• cardosof 5y ago
  Would investors pour their money if they were more conservative and said something along the lines of "look, this is a very complex subject driven by and for humans, we should hire a bunch of non-technical people with relevant industry experience and try to make some bucks of profit before going full scale engineering and AI"?
  • skohan 5y ago
    Theoretically investors should reward a realistic and well-reasoned business plan, and punish hand-wavy science fiction. The fact that this is largely not the case (cough metaverse cough) is probably an indicator about how frothy the market currently is.
• elliekelly 5y ago
  > However I think where the hubris lay is in how they thought they could leapfrog all the way to an automated solution before building a competency as a house-flipping company. In my mind this is the problem with consultants who try to automate processes. It’s really difficult (maybe even impossible?) to successfully write a program to make a computer do $thing if you don’t understand the intricacies of how to do $thing manually.
• technobabbler 5y ago
  Orrr maybe someone in the org could've practiced some basic morality and compassion and refrained from further contributing to the housing shortage. Just because you can make money being a sociopath doesn't mean you should...
• Lhiw 5y ago
  > and it's not clear how you would automate this. The amount of things you encounter in the real world is ultimately limited. Any half decent valuer will already have a literal book on these types of things predefined. Automating it isnt really the problem, properly surveying the grounds and the area are. I'd also posit going this way is the wrong approach, they should be using metrics like time on site and ratio of views online vs views on site. These things work as proxies and are enough to apply as a modifier to more usual pricing models.
• spoonjim 5y ago
  When we were looking for a house we rejected many because of the “wrong sort” of neighbor, ascertained entirely (and possibly erroneously) without meeting them. I doubt Zillow can model that with public data.
• initplus 5y ago
  Even if Zillow had an algorithm that was 100% accurate at predicting current house prices, the housing market is just incompatible with market making. A market maker isn’t exposed to changes in the price, they clip the ticket on providing liquidity regardless of price direction. Zillow may have been able to successfully speculate on house prices with an accurate model, but they would not be a market maker. Houses trade slowly, so would sit on Zillows books for a long time (days/months). Market makers on the stock market can have assets sit on the books for under a second. Houses are not fungible, which extenuates the slow trade problem.
• Robotbeat 5y ago
  One huge difficult-to-quantify risk is the public opinion risk of being a very high profile company that flips houses. If it had succeeded to actually push up housing prices considerably, the whole company could be destroyed in the court of public opinion and therefore probably would be destroyed legally, regulatorally, and legislatively as well.
• deleted 5y ago
  [deleted]
• dmurray 5y ago
  Very much this! I can count on one hand the number of successful businesses I've seen that use machine learning to do some magic inferences humans could never do. I've seen hundreds where humans are pretty good at doing their job, but machines can do it faster, more consistently and cheaper.