13 ms·

Zillow just gave us a look at machine learning's future

• lvl100 5y ago
  Most data scientists are not smart. They are basically pushing a few buttons. This is true even for “Ivy” trained PhDs. There are may be a handful of people who can do truly profound work today. The rest are only capable of replication ie mouse clicks here and there.
  • ianai 5y ago
    Further, they should be telling people a model is only as good as the data it has as input. If the data only sees upward movement in valuations then it will only predict upwards. A model can’t tell you anything about a situation where the underlying factors are different from its data. Same thing happened with mortgage backed securities in 2007/2008. The math was very complex and still couldn’t see around a corner.
    • mariodiana 5y ago
      The bottom line is that the domain here is under the umbrella of the social sciences. The phenomena the models describe is historical, and how closely they will match what happens from that point on relies on social stability. In the grand scheme of things, social stability is ephemeral. It isn't rocket science. I'm not making a joke. Newton published his models in the late 17th century. The models hold no better or worse today than they did then. If you had rockets in 1687, they would work the same way then that they do now. But, social science is nothing like hard science. I worry on some fundamental level, even people who will claim that they "of course" know this in fact forget this. There is hubris here.
• 1024core 5y ago
  ML is a sharp tool, just like a knife is. In the wrong hands, it can cut you up and leave you bleeding. Moral of the story is: handle sharp tools with care; know how to use them properly.
  • mathgenius 5y ago
    Modern ML algorithms are so powerful they will soak up any entropy that leaks into the data whatsoever. Much of the time spent developing these models is checking that this isn't happening. I compare it to high voltage electricity: a 100kV line needs some serious insulation around it.
• ricardobayes 5y ago
  What if zillow's defective pricing model is single-handedly responsible for the housing price bubble in the US? I have not seen prices increase at all elsewhere in the world.
  • hasbot 5y ago
    I think they've been a contributor. Having looked at many houses in various markets over the last few years, I feel their estimates are high but sellers take them as gospel.
• jleyank 5y ago
  Pharma has been using QSAR/QSPR and machine learning, along with various other statistical models for 50+ years. There have been repeated over-predictions as each new technology has come down the line. Yet the principles, like in this article, hold - have a good training set, have a test set that isn't just an echo of the training set and then finish with a validation set that is sufficiently different to help expose edge failures if not outright failures. Remember Egan's rule of 1 - the number of times a modeler can fail before their predictions are ignored...
• angarg12 5y ago
  My takeaway from the Zillow debacle has nothing to do with OPs approach. Instead I look at it from the Antifragile framework. It looks like this business model had a lot of ways to go wrong and very few ways to go right. In other words, it was fragile. It doesn't even sound like the Black Swan of Covid was needed to make this fail. In other words, if a system is fragile, you can bet on it breaking sooner or later.
• jwilber 5y ago
  “Pull back the covers on large scale deep learning models and this is one reason overfitting creates miraculous accuracy. Large scale models have a primitive understanding of the deformations leading to a degree of generalization. The more the model learns about how the inference space changes over time, the better it generalizes.“ Pretentious mention of topology followed by super broad, almost completely incorrect characterization of the double descent phenomena, color me surprised.
• jollybean 5y ago
  Predicting home prices is like predicting which way the cows will stampede. It's radical hubris and lack of self awareness on their part. What this 'exposes' is that both leaders, and Data Science leaders will probably overstate their value, and the rank and file will just go along with it, because, well, it's a job.
• dexter89_kp3 5y ago
  So... the failure may not be due to ML, but due to people :) See this blind thread: https://www.teamblind.com/post/Did-Zillow-fail-miserably-due-to-forcing-application-of-Machine-Learning-o8fCCZCD https://www.teamblind.com/post/Did-Zillow-fail-miserably-due...
• h2odragon 5y ago
  > The Data Science team didn’t go to the lengths necessary to support their models. That was their failing. However, the business is ultimately responsible for betting the farm without demanding a real world track record to support those models. "Well, sure, them folks were (wrong | lying), but its really your fault for listening to them! It's not like making that call was their actual job or anything, ... oh." I'd say the lesson here is that horseshit is still horseshit even after it has been extensively processed by the finest machines and learning we can buy. Just imagine being the poor sod there who got a model to say "Bail the fak OUT of this market" 3 months ago: think anyone would have listened?
  • ianai 5y ago
    I think it’s long been known that if the market actors don’t want to heed results of analysis then they simply won’t.
    • h2odragon 5y ago
      see also: https://everything2.com/title/SNAFU+Principle https://everything2.com/title/SNAFU+Principle
  • JasonFruit 5y ago
    Maybe they would have listened if the model successfully used historical data to predict when people should have bailed in the past, in addition to now. That is, if it was a properly validated model. I am not an expert, but this seems like a reasonable way to see if a model works.
  • ashtonkem 5y ago
    One of the harder lessons of life is that occasionally people get wedged into ideological positions where they want comforting lies that confirm their existing beliefs. These situations often are very dangerous for companies, because it creates incredible pressure on lower level people to produce dubious models, and on middle managers to accept dubiously supported models. As far as who is culpable for those situations … it’s not clear to me.
• civilized 5y ago
  This post strikes me as empty bluster. It basically says "the data scientists didn't consider the possibility that the model might fail to predict the real world." Which is like looking at the Boeing 737 MAX debacle and saying "they didn't consider the possibility that the plane might not be stable". This is most likely both untrue and lacking insight. What almost certainly happened was they validated their model extensively, but overlooked the angle relevant to this use case. Or overexcited executives failed to heed the caveats of the nerds. > When I advise Data Scientists to understand Topology and Differential Geometry, this is why. Oh really? I've taken grad-level courses in those fields. It's nice general education but has nothing to do with this situation. Knowing about the Lucas critique from economics would be much more relevant. My pet theory -- completely unvalidated but at least potentially insightful -- is that they failed to put in adequate margin for adverse selection: homeowners whose homes are worth less than they seem on paper were much more likely to take the Zillow offer. Adverse selection turns prediction variance, which you might naively expect to "average out" when using a well-calibrated model at scale, into a bias towards underperformance. Another way to put it: asymmetric information. Be very careful betting against homeowners who probably know much more than you do. Or read this comment, which puts it better than I did: https://news.ycombinator.com/item?id=29140104 https://news.ycombinator.com/item?id=29140104
  • 88913527 5y ago
    What if the model said there were only $100m in homes that were viable, but executives wanted $1 billion in purchases? They could have had an alright model, but the volume wouldn't sustain the purchasing quantities leadership wanted. So, they relax the model to accept lower quality results. It's another form of executives attempting to squeeze blood from a stone. It just doesn't work, but it's a common strategy. They'll ask simple questions like "Why can't you just do X?", you advise them not to, and they ignore you.
    • kshacker 5y ago
      Basically the equivalent of 2008 no docs no income market for housing loans.
    • 5y ago
• sAbakumoff 5y ago
  I hope that it gave us look at what will happen with all these real-estate-bubble-inducing a*holes.
  • brink 5y ago
    I too am tired of the greed.
  • ianai 5y ago
    It might and it might not. Many things have rolled down the pike that in normal markets would cause reevaluation and correction downwards. But so far they’ve all been pushed past. My professors used to say “when the last stalwart holdout is sold on the premise of the irrational factors holding the market up the correction is near.” (Or something like that.) I seem to recall it being Enron filing bankruptcy that really kicked off the Great Recession. Oh further, anytime you have your market appraiser becoming a market participant their valuation becomes highly unreliable. This is another thing that’s just not done if you want to have a reliable indicator of an assets value and sell it knowing whether you’re getting a deal you’ll be comfortable with long term or not.
  • ashtonkem 5y ago
    It’s really not clear to me if we’re in a bubble. A lot of these houses sold to investors are aiming to generate rent, not be resold at a higher price. Zillow appears to be relatively alone among institutional investors in aiming to flip homes and not rent them. If corporate land lords are extracting enough rent to justify the higher than normal prices, is that really a bubble? It’s socially pernicious, yes, but that doesn’t sound like an asset bubble to me.
  • bpodgursky 5y ago
    The bubble was caused by fiscal, tax, and homebuilding policies. Zillow wasn't even renting the the houses it bought, just flipping. Blame the government; Zillow had nothing to do with the housing price blowup.
• deleted 5y ago
  [deleted]
• rossdavidh 5y ago
  Or, more simply, this business only works if prices are going up, and home prices aren't going up. Zillow saw the house price bust coming, and got out.
• gz5 5y ago
  Seems there are 2 parts: 1. The business decision to go into the 'flipping homes' business. 2. The capabilities of the ML/AI to help optimize that business decision. The post does a nice job outlining that #2 will have great variance across businesses, use cases, time, etc. The biz decision is maybe more interesting and instructive. A wrong biz decision can doom the ML/AI out of the blocks - the ML/AI is often a tool to do a job, but there is a business decision to do the job to begin with.
• endisneigh 5y ago
  Is it even possible to predict things whose predictions influence the events that are supposed to be predicted? Suppose you did have a perfect way to appraise houses by some arbitrary criteria. If the houses are "perfectly" appraised, that would change the values of the houses themselves, no? e.g. magical system predicts 3b/2bath house should cost 500,000. People know the magical system's price is the perfect one. someone who has more money to burn can acknowledge this and still pay even more money, knowing most will not want to "overpay" for a house. house sells for 550,000. prediction is wrong in the end anyway. schrodinger's predictor - it's accurate right until it's observed.
  • raverbashing 5y ago
    Yes. This is one of the fundamental teachings of economics. The ideal price is not knowable
    • paganel 5y ago
      Nitpicking here, but it is my understanding that there isn't even a consensus about the existence (or not) of an "ideal price" in economics. Just to make it clear, I'm not an economics major nor do I work in the field, this opinion is just based on my amateurish readings.
  • JasonFruit 5y ago
    Predictions like this work on aggregates, not individual cases. There are also sellers who are in a hurry to move and will settle for $450,000. Magical system is wrong in both cases but right in the aggregate.
  • rq1 5y ago
    Yes it is possible. You can tackle it through different technics like game theory. Cf. also the “market impact” in HFT, for a practical example.
  • sidr 5y ago
    They don't need to predict the actual sale price of every given property perfectly to make money. They need to be able to say that with 99% probability, we will find at least one offer for this house >= 500,000. The errant buyer with loads of cash is free to come in and offer more. As long as they turn a sufficient profit at the 500,000 price point they should be set. In competitive markets, for people familiar with local dynamics, this is very much possible in a short-term time window. The question is doing this at scale in a fully automated way for a much longer time horizon. Zillow tried and failed.
• tguvot 5y ago
  There was a post on Blind a few days ago from somebody who works in Zillow, stating that management disregarded models and decided to go instead with "intuition"
  • programbreeding 5y ago
    What is Blind? I tried searching various things but just keep getting results for accessibility.
    • fullshark 5y ago
      https://www.teamblind.com/ https://www.teamblind.com/ Workplace chat app, you use your work email to sign in and chatter anonymously with your coworkers and others in the industry. You are ID'ed only by the company you work for.
    • who_me_qmark 5y ago
      https://www.teamblind.com/ https://www.teamblind.com/
    • l8rpeace 5y ago
      https://www.teamblind.com/ https://www.teamblind.com/
    • deleted 5y ago
      [deleted]
    • deleted 5y ago
      [deleted]
  • preetamjinka 5y ago
    This seems to be the thread. I’m not sure how to link to the specific reply. https://www.teamblind.com/post/Zillowers-what-happened-fXvbZGKn https://www.teamblind.com/post/Zillowers-what-happened-fXvbZ...
  • chasely 5y ago
    That post tracks with some of my experiences working in forecasting. Senior leadership loves the DS team until they are told something may not work how they want or reaching their quarterly KPI (which is often tied to their bonus) is infeasible/a bad idea. This is a generalization, and has happened a couple times in my career. I feel lucky now to be at an organization where this is no longer the case.
• MrPowers 5y ago
  It's difficult reading articles without any Headings! Data mining is common in finance. Finance PhDs are constantly identifying new "factors" (e.g. value, momentum, price/book) based on historical data that don't pan out as time goes on. See here for some academic research: https://www.researchaffiliates.com/publications/articles/710-alices-adventures-in-factorland-three-blunders-that-plague-factor-investing https://www.researchaffiliates.com/publications/articles/710... Robert Shiller has does extensive research on how psychology impacts real estate prices. Monetary policy isn't standard right now. Machine learning isn't always a good tool for making predictions. Look at Shiller's interviews and surveys for predicting home prices. He doesn't use machine learning or advanced analytics.
• aabajian 5y ago
  Personal example. I bought a condo in the University District in Seattle back in 2017. It's a 92-year-old, 478 sq ft condo. I paid $279,000 (yikes, I know)...Zillow's estimate? $500,000. I have no idea how they were so far off. Even right now: Zillow estimate: $479,100 - https://www.zillow.com/homedetails/905-NE-43rd-St-APT-212-Seattle-WA-98105/2100973940_zpid/ https://www.zillow.com/homedetails/905-NE-43rd-St-APT-212-Se... Redfin: $307,337 - https://www.redfin.com/WA/Seattle/901-NE-43rd-St-98105/unit-212/home/12537835 https://www.redfin.com/WA/Seattle/901-NE-43rd-St-98105/unit-... Keep in mind that Zillow's HQ is also in Seattle. While we may be protective of the data scientists on HN, this is one case where I agree, there's something wrong with their models. The "similar recently sold" condos that Zillow shows are indeed $400K, but those are at most 50 years old and more square footage. I think a simple logistic regression of square footage and age would give a better prediction than whatever model they are using.
  • aparsons 5y ago
    I think part of it has to do with black box models being easier to build incrementally. I’ve seen many teams start off with 3-5 features, which gets you to some threshold for launch (say 90% precision and recall) and get badly burnt on the other 10% post-launch
  • benrapscallion 5y ago
    From personal experience, two major contributors to the Zestimate are: 1. The town/city’s assessment of the home’s value for tax purposes 2. Recent sale prices of homes nearby, mainly as the per square foot price.
    • TheRealDunkirk 5y ago
      Another thing I don't understand. How do you develop a model with VOLUMES of real world data to train it on, and not just check how well it predicted pricing every month, both in aggregate (trending prices) and on individual listings that sold that month? This seems like a very small, closed loop to me. You could tweak the heck out of that algorithm in short order based on WHAT ACTUALLY HAPPENED.
• bananaquant 5y ago
  One thing that I feel this article misses is the notion of domain expertise. Zillow has entered the house flipping market that has been already populated by human real estate agents and investors, who already do their work of pricing houses and striking deals quite well. To compete, a successful entrant needs to excel in at least one thing and be not much worse in all the others. In practice, that is a tall order.
  • jollybean 5y ago
    Real Estate agents 'price' by looking at rough comparable down the street, and padding it up a little bit. That's it.
    • bananaquant 5y ago
      Yes, but there is more than pricing that goes into a successful operation.
• tomp 5y ago
  A fascinating thread about Zillow, comparing it to quant trading https://mobile.twitter.com/0xdoug/status/1456032851477028870 https://mobile.twitter.com/0xdoug/status/1456032851477028870 TL;DR: not taking into account adverse selection - other people's prices & trades aren't random/Gaussian, but adversarial (they're trying to profit at your loss) and better (they probably have better information than you)
  • jjgreen 5y ago
    From that (fabulous) thread: Behind every order is a thinking, breathing shark. A shark with mouths to feed and a mortgage to pay. A shark who's likely smarter than you. Definitely more ruthless. You're swimming in shark-infested waters, treating apex predators like specks of geometric Brownian dust
• TheRealDunkirk 5y ago
  > "The CEO points directly to the failure of their Data Science team as the root cause. Their investors aren’t buying it and I don’t either. Zillow’s Data Science team is excellent. The business knew the risks and ignored expert advice. The initial rollout should have been far more limited and cautious. > The Data Science team didn’t go to the lengths necessary to support their models. That was their failing. However, the business is ultimately responsible for betting the farm without demanding a real world track record to support those models." I'm confused. If the "team" was "excellent" (and he says this multiple times), why didn't they do the extra leg work? It feels like he has a dog in this hunt. Did they use him for external consulting or something?
  • aparsons 5y ago
    This author doesn’t have a clue about anything. After the first paragraph, this might as well have been written by GPT-3. Sounds like a consultant trying to cash in on the news of the week to me.
• _3u10 5y ago
  Zillow blinked.
• rq1 5y ago
  Did they consider that they just hired the people who would tell them what they wanted to hear? It would have been fortune cookies, it would have been the same. > When I advise Data Scientists to understand Topology and Differential Geometry, this is why. This is ridiculous, it has strictly nothing to do with that. The author thinks he’s smarter than everyone else.
  • aparsons 5y ago
    I’d really like to see the author write an explanation as to why. I completed by PhD in “AI” (as it was then called) in 2001. Very little DNN work then, but I haven’t a clue where the connection lies.
  • throwawaysleep 5y ago
    > Did they consider that they just hired the people who would tell them what they wanted to hear? Management everywhere thinks they are getting good info from those below them when everywhere I have worked, they get a selective job preserving subset.
• tomrod 5y ago
  This reflects something I say often as a data scientist: the best data science implementations augment humans. Domain understanding is key. Integration of the domain into ML models are a key part of generalizability -- far beyond train/validate/test, which are subject to data collection biases and temporal trends. Zillow had a high, general equilibrium impact on a large market. We saw microcosms of this behavior in the "good old days" of Amazon book-selling when competitors would mark up a books price to one penny more than another, resulting in outrageously priced books valued far below the offering price. No behavioral governors were in place, review was inadequate, and they become part of Data Science lore we tell ourselves as our single threaded scripts running Python payloads warm our homes and melt our s'mores whilst our GPU weeps. Data Science as a field is distinct from ML in that it uses the tools of ML to augment and enhance digital services. Despite the exuberance of boot camps to market prediction and neural nets as the end-all-be-all solution for hard business problems, value in data science adoption isn't solely in prediction, but in validation. Causal modeling and inference are key in this space, but also simply quantitative understanding of statistics and algorithms. As an example, you need someone in the room who understands multi-armed bandit sometimes. But, more importantly, you'll always need someone in the room who understands how multi-armed bandit would improve a business' value proposition.
• tsywke44 5y ago
  Sounds more like classic large organization things. I bet there were some employees who saw the estimates were untrustable and the entire thing was heading towards failure, but good luck changing the course of a ship controlled by hundreds of people.
• cs702 5y ago
  I agree with the OP that "it's put up or shut up time for Data Science teams. When the business depends on our work for revenue growth, success is expected. Failure is fatal." Zillow's CEO blamed the Data Science team for the failure of a mission-critical business unit and fired 25% of all employees. The stock lost a third of its market cap in a few days. When revenue growth depends on an ML/AI system, failure is indeed fatal. However, I disagree with the OP's diagnosis. The initiative failed because to a close approximation the only homeowners who would routinely accept Zillow's offers were those whose homes could not fetch anywhere near as much via traditional sales channels. Everyone else would rather sell their home via conventional listing. Zillow gave homeowners a free put option: Whenever Zillow's offer was above market, the homeowner would take it; otherwise the homeowner could freely ignore it. The incentives were strongly stacked against Zillow from day one. Zillow unwittingly launched what I would describe as a seller's market for lemons. Naturally, Zillow ended up buying... a lot of lemons.
  • curiousgal 5y ago
    https://en.m.wikipedia.org/wiki/The_Market_for_Lemons https://en.m.wikipedia.org/wiki/The_Market_for_Lemons
  • BryantD 5y ago
    I think your basic point is good -- a free put option is always going to be a problem. However, can you unpack "Everyone else would rather sell their home..."? At the core there are two factors at play in this decision. One is financial; anyone with the luxury of time will compare a realtor's comps with an ibuyer's offer. The second is certainty; an ibuyer offer is guaranteed, and can be executed without having to show a house over the course of 1 to X months. It seems to me that certainty has value to some sellers in some situations. Right now, the certainty value of the traditional route is pretty good because the market is on fire. That is not always the case everywhere.
    • cs702 5y ago
      It depends mainly on how much more money the homeowners believe they can fetch by listing the home versus selling it right away on the spot. For example, if you're the homeowner and you think you can get, say, a few extra $100K's by listing your home instead of selling it right away on the spot, you would probably take your changes and list it. Even if you're under pressure to sell quickly, you might still try listing the home for a couple of months to see what happens. Only if you think your home is a lemon will you consider selling to Zillow. That's why Zillow ended up buying all those lemons.
• encoderer 5y ago
  > Zillow believed their inference quality was high enough to support a new business model This isn’t why Zillow started Offers and the zestimate did not play a big part of it. Some same people but those were different teams.
• bitL 5y ago
  "A stock trading company uses inaccurate stock prediction models for technical analysis, loses money, downsizes." How is this different? They were playing roulette anyway...
• version_five 5y ago
  Predicting the future is not a use case for machine learning. And failure of financial models happens all the time - Long Term Captial Management is probably the best example of a big one. This has nothing to do with ML/Data Science, other than maybe some people thought it could do something it cant. Also, the article reads a bit too much like a stream of conscious sprinkled with buzzwords and causal references that the author knows better than Zillow.
• aparsons 5y ago
  I hate to use this language on HN - but this post is a crock of shit. This is just an opportunistic post by the author that cleverly evades defining anything in solid terms, providing even one concrete example, and is a pot of handwavey BS. It started off encouragingly, and two paragraphs in, the smoke alarms in my head were blaring. The author doesn’t even have a clue (or for some mysterious reason didn’t mention) about what went wrong for Zillow. Zillow’s massive screw up was that their key business assumption was a self fulfilling prophecy. If a model learns that “price to pay today is price to pay yesterday in this area * 1.05”, and Zillow is buying multiple homes in a neighborhood spread over time, you can see that the price is headed to the moon (and the buyer who puts their faith in the model to the gutter) in no time. In my parents’ town, they offered a neighbor a price, and came back 3 months later with an offer $70k higher. I have a strong suspicion what was supposed to be a price prediction model ended up being a price discovery agent due to the lack of a “visited” set. Oops!
  • version_five 5y ago
    I agree with your characterization. I've been in meetings with people like this that will throw out irrelevant stuff (topology, distributions of distributions!?) as if it's an insight, and there is a certain kind of "leader" that loves that stuff, to the frustration of everyone who actually has some background in the area. I assume this is where he's coming from
• dpweb 5y ago
  This is not data science, its speculative trading. So, a data science approach without the massive paranoia and appropriate hedges that must accompany any trading endeavor, is a problem. But I wonder of the future of mass corporate home buying, which I would assume Airbnb would do like Amazon, create a huge platform and then enter that market to compete with your customers.
• helsinkiandrew 5y ago
  This is BS. The issue had little todo with AI, they could have used professional real estate valuers and had the same problem. This was caused by a faulty business model - houses aren’t shares or bonds that can be flipped instantly. The feedback for their pricing model wasn’t known for three to six months after buying the houses, by which time they’d lost billions.
• nknealk 5y ago
  There’s another consideration here. The entire buying operation was debt financed which created incentives for risky behavior and left very little margin for error. Not only does your sales price need to cover the cost of the house and any improvements, it must also cover the interest expense (123 million YTD according to their latest 10Q, vs 2.67B in sales of houses or about 4.6% yield relative to sales). And that interest expense is there regardless of how many houses you sell. If Zillow predicted housing prices to go up and it merely stayed flat instead, they can’t just hold the inventory for longer in hopes that the market might go up a few months from now. They’ve got that interest expense to cover. The whole operation implodes the moment they made a bad prediction because of the debt financing. Yes, their models were wrong. But if they had a different capital structure maybe they’d be able to hold the inventory for longer to try to turn a profit and rebound instead of collapse.
• fghorow 5y ago
  The stability (or lack thereof, actually) of distributions is a fundamental problem for any statistical model of complex systems. The author stated that more succinctly than I personally have heard anywhere else. Even if the rest of the piece might be BS (as many others here are arguing) pay attention to that aspect, since it is reality for most complex systems...
• whatever1 5y ago
  Machine learning was never supposed to be good at modeling constraints (hard constraints, equilibrium constraints). Applying it to real applications with real constraints was a mistake. Google will not collapse if they serve me one recommendation that is out of bounds (a monkey photo instead of a black person). I will ignore it (as we all did for years, before the journalists took on this). Google’s business model does not strongly correlate with the accuracy of their ML algos. I have yet to see 1 successful company whose business decisions rely on the recommendations of a machine learning framework. The explanation is trivial. ML works, most of the times. And that is the problem.
• marcinzm 5y ago
  I suspect something like this happened: - The team behind the zestimate kept touting how accurate it is. Since they needed to keep improving it to keep their jobs they kept cherry-picking the best results to showcase. Edge cases were silently not mentioned and failures were spoken of as little as possible. The further up the management chain the result went the nicer they looked as each layer tweaked them to be even more impressive sounding. None of it mattered much so who cares about a few white lies? - The other executives seeing these great results decided to create a new business model based on top of it. The DS team and their management was not consulted about this until many wheels were already moving. - The zestimate team and their management chain would get fired (or otherwise punished) if they suddenly showed all the edge cases and potential failures they never mentioned before. So they collectively kept quiet, hoped they could fix the edge cases and hoped it would all go well. Messengers tended to get shot in this situation and no one wanted be that messenger. edit: In my experience skewed institutional incentives are more often the source of issues than technical mistakes or low level employees being blind to things. You don't get promoted in most companies by telling the CEO they're wrong so people don't.
  • indymike 5y ago
    If it went down like this, then senior management is where the problem lies. Every CEO should know that expecting an AI to understand systems that humans can study for a lifetime and still get wrong is irrationally optimistic. Firing 25% of a team for a self-inflicted failure (by senior management) is another signal there is a deep management problem.
  • encoderer 5y ago
    This is not even close to the reality. Zestimate is not the tail that wags the dog. It’s wild how much people get wrong in their assumptions here.
• 1024core 5y ago
  Also, I take issue with this: > their CEO put a lot of blame on the Data Science team. That is the height of hypocrisy on the CEO's part. If this move had worked, who would have taken all the glory and the $$$$$? The CEO! That MF should take the blame too, since the buck stops with him. He didn't know how to use their predictions properly; stop blaming them.
• sonabinu 5y ago
  IMHO the Data Science team at any organization should work closely with a strategy team who uses data to ask hard questions with tools like scenario planning. It’s hard work, and when things are going well it feels like that team is wasting time but when things are hard, they have already thought through of ways to pivot and how to move forward.
• m3kw9 5y ago
  It’s no surprise AI cannot understand and predict economic outcomes, yet Zillow tried
• esics6A 5y ago
  ML/AI will go through the typical AI hype-cycle that it has always gone through for several decades in a row. Remain skeptical as always because some things just can't be modeled easily in nature or the real world. People have tried before for over a hundred years to use math to predict markets like the stock or housing market and they all failed. If you don't have any real maths other than human judgement based formulas like discounted free cash flows to value assets at a point in time then you're going to have massive problems with the validity of the model. Then you're just taking something that's up to human judgement with all of the errors and extending to machines at a vast scale. The end result will almost always be catastrophic at that scale, the risk is just orders of magnitude higher. Statistics are useless because markets like this are driven by irrational human emotions that's never predictable. An obscure historical event can send market's tumbling for reasons that nobody really understands and can never understand. Sorry to break it to the Data Scientists on this board who are likely reacting poorly to this article. We're a century away from having ML/AI that can ever predict these real world scenarios. By that time Data Scientists won't even have jobs because the ML/AI will work directly for business executives and not require salaries other than a few engineers to keep it running.
• deleted 5y ago
  [deleted]
• adrianmonk 5y ago
  Back in the 1980s, people used to say things like, "It's in the computer, so it can't be wrong." The computer says your account balance is $86.23, so it's $86.23, end of story. Can't argue with the computer, the all-seeing, all-knowing, infallible source of information. Such statements sound totally crazy now, but it was a pretty widespread misconception. Computerization made such big improvements in availability and accuracy of information that, for a time, people simply overestimated what computers could do. Maybe we're seeing some of the same thing now with machine learning. (This isn't a comment on whether attitudes about machine learning were the cause of Zillow's troubles. I'm not a Zillow insider, plus the situation is probably complex.)
• vkk8 5y ago
  My take on all of this is that someone in too high position at Zillow forgot to read their Taleb books.
• jaclaz 5y ago
  I don't know anything about the US housing buy/sell market, but I have some little experience in my country (Italy). Appraisal of houses is extremely difficult because there are both objective data (like size, quality of construction) and many very subjective (at least here while you can draw some rough lines distinguishing neighborhood/areas in a city, sometimes a house just around the corner has much more - or much less - value) and there are what I personally call "financial capability of the masses" (in a given area/city). Say that you have a house that is appraised by several people to have a value around 400,000 Euro (or dollars, anyway "units"). What a young couple (both working) can afford is 350,000 Euro max, let's say 100,000 Euro cash and 250,000 loan. So you have out of 100 people - willing to buy at the moment - 98 of these young couples and only 2 other people (these latter that can afford the 400,000 Euro or more). Your house suddenly is not worth anymore 400,000 Euro because most of the houses (similar to yours and valued anything between 360,000 and 420,000) after - say - six months that are unsold will lower the asking price, and your asked 400,000 will appear way too much and you will be forced to lower it. The other way round, if your house is valued 300,000, there are good chances that you can sell it for 330/340/350,000, because there are 98 people out of 100 looking for exactly that price range in the area, and as soon as one or a few ones are sold in the neighborhood for 340,000, all similar houses will go from 300 to 350,000. And this tends to be cyclic, at a certain point that particular area/neighborhood becomes (for whatever reasons) trendy and prices skyrocket, then a couple years later (again for whatever reasons) it goes down to the bottom. Everything is very ephemeral and fluctuating, maybe what Zillow attempted to do (and seemingly failed) was more to become a sort of monopolistic entity capable to "impose" prices?
• vadym909 5y ago
  This is exactly the problem with AI for recruiting too. Humans are smart and will evolve their strategy to counter Machine learning strategies.
• rewq4321 5y ago
  "the Zillow failure was largely in part due to humans 'overriding' ML outputs in order to meet their quotas" https://twitter.com/sh_reya/status/1457094567111528455 https://twitter.com/sh_reya/status/1457094567111528455
  • reducesuffering 5y ago
    Yep, that’s not the only thread from Zillow folks saying the same things. That their output has still been accurate and was overridden by management’s gut feelings. If true, quite atrocious that the CEO is blaming the data science, either by maliciousness or ignorance of how the company is operating.
• shmerl 5y ago
  They were driving prices up by speculating on buying houses? Feels like it's great their data science team failed. More such failures should happen.
• xyzelement 5y ago
  I used to work at a trading systems vendor and once in a while, someone would ask "couldn't we use ML to tell our clients what to buy and how much to pay?" My response was that "if we could do that, we'd never give it to our customers and just trade ourselves." So I always take things like the Zillow estimate with a grain of salt because "if Zillow really believed this number, they'd use it to buy houses rather than put it on their website." So it was interesting when Zillow actually started to essentially "trade on their numbers" while also sharing them with the public. That was a very bullish sign that the company deeply believed the quality of its signals. That can be totally normal. Every investment (and in fact, every project - which is a form of investment) happens because people believe that it's a good bet which is different than guaranteed. Some of the most money-making hedge funds are right in only like 51% of their bets, but that's enough! The thing that somehow went wrong is simply risk management. It seems like nobody with power at Zillow asked the question "how do we make sure this doesn't ruin the company if it's wrong?" UNLESS the scenario we're seeing now is their predicted acceptable worst case scenario. It's possible that they simply said "this is a juicy bet we want to make. If it goes wrong, we'll loose X millions of dollars and have to fire 25% of our team. Are we willing to take this bet?"
• jgalt212 5y ago
  Any bond trader can tell you that overall your model can have zero bias, and a small error term, but your bids that get hit and offers that get lifted are the ones that are "off". This could have been solved with a "human in the loop". Of course, that may have broken the economics of the entire operation.
• bognition 5y ago
  It seems like zillow used an algorithm to decide which houses to buy and sent in people to execute the purchase. I suspect many of the purchasers knew they were offering too much or they were ignorant to the realities of real estate. What would have made more sense would have been to use an algorithm to identify homes and then flag those homes to highly skilled agents to vet and buy. Additionally you’d want some incentive to keep the agents motivated to buy the right homes.
  • rp1 5y ago
    Did Zillow even have buying agents? I imagine a lawyer could have handled the entire transaction.
    • bognition 5y ago
      That’s kind of my point, hitting someone to go execute a contract will have different outcomes than hiring someone to evaluate if a contract is a good idea
      • rp1 5y ago
        The whole point of their operation was to buy real estate based on an algorithm. Hiring an agent to evaluate and execute each sale was the opposite of what they were trying to do and likely wouldn’t scale to the billions of dollars they were looking to spend. Obviously it didn’t work out for them though.
• deleted 5y ago
  [deleted]
• shinzui 5y ago
  The article premise is completely wrong. Zestimates were never meant to be accurate. In fact, they were deliberately inflated to drive traffic to Zillow.
• woeirua 5y ago
  This post is mostly full of hot air. Not sure how much experience this person really has in data science as they contradict themselves multiple times. I think the execs at Zillow bear almost all of the responsibility for this. The harsh reality is that anyone with a brain could see that they were going to face a serious adversarial selection issue (only homeowners with crappy homes would sell to them). They tried to put into place guardrails to prevent that but they systematically failed to prevent that . And I think they failed because they fatally underestimated how corrupt the real estate industry is. The inspectors and appraisers that were supposed to protect Zillow from overpaying failed to do their job systematically. I say this because I’m seeing homes from Zillow and Opendoor with major structural issues. No sane homebuyer would have paid what Zillow did for these homes. The only possible explanations are either: Zillow did not do their due diligence, or the inspectors/appraisers didn’t do their jobs. If it’s the former than Zillow is a cautionary tale. If it’s the latter then we’re definitely in a bubble again and the appraisers are going to be the target of more legislation down the road.
• JoshTko 5y ago
  This is a poor use case for ML. You'd only have a couple hundred examples for a neighborhood with maybe two dozen variables. The model will also miss key variables that that there aren't datasets for but would be apparent upon a visit. ML only beats humans in very, very narrow scenarios. https://youtu.be/j0z4FweCy4M?t=9319 https://youtu.be/j0z4FweCy4M?t=9319
• Barrin92 5y ago
  I think it probably would have helped to start with some basic economics. The housing market is large, diverse and very efficient. Trying to throw some data science voodoo at it and thinking you can somehow outsmart the market with some magical ML despite the fact that all the data you're crunching is already incorporated in the price by the existing participants sounds stupid to begin with.
• htrp 5y ago
  From the blind thread > Our back tested model error as well as true error rates (models only) were considerably smaller than what was realized. Largely because leadership wanted to be aggressive on acquisitions and incentivized all teams to err on the side of growth, which manifested in system overrides. Without proper oversight and controls these lead to compounding and unchecked error Sounds less like blaming the data science team and more like the entire organization didn't want to face the music.
• hungrigekatze 5y ago
  Agreeing with other commentors here who don't buy into the superficial 'silly Zillow's ML folks didn't consider the possiblity that the model might fail to predict the real world' narrative. Below I'll outline what Zillow learned from the 'iBuying failure' from my perspective having worked in real estate tech. As someone who worked as a senior Data Scientist at one of the Silicon Valley companies involved in iBuying some years ago (5+ years ago) I see the recent Zillow iBuying spree as a mechanism to test how much market pressure needed to be applied to historically not-so-competitive residential real estate markets to induce 'FOMO' / social contagion behaviors of large-ticket items AND as a way to produce a dataset on the actual dollar amount that (residential) property sellers would need to abandon the 'safe', 'we've always done it this way' process of selling a piece of real estate through a real estate agent / broker. The upside (for real estate tech companies) in removing the middle[wo]man - the real estate agent - in a residential home transaction is that the real estate platform can now control both sides of information asymmetry in the real estate transaction. They can also start offering (like Zillow does) mortage services and other ancillary financial services, allowing them to earn millions of dollars in fees by capturing the home-buying financial services markets. In my work at the $real_estate_tech_company I mainly developed lead-generation data products which were used by $real_estate_tech_company market to get less-desirable single family homes to be bought up by high-net-worth indiviudals who invest in real estate. The end goal of this process was to get the foreclosure and pre-foreclosure single family homes (SFHs) off of the bank's ledgers and leave someone else holding the (debt) bag. HNW individuals would buy up the assets, the banks would have someone with sufficient collateral now in possession of the single family home, and the HNW individual could rent out the home to less-likely-to-default-than-the-original-homeowners family / renter. According to what I read about Zillow's iBuying model, Zillow focused on buying up assets (SFHs) in markets with strong, diversified economies, i.e., economies that are less sensitive to economic downswings. (Blackstone is doing the same thing and is also buying up trailer parks / mobile home communities near tech hubs.) As someone who builds data products for a living the datapoints that Zillow was able to gather are, in my opinion: - A hard number, in USD, of the amount of money needed to get humans to abandon the process of selling their home in the traditional way: through a real estate agent, broker, etc. Because Zillow has home buyer and seller data they now know what that switching cost trigger is, in USD, for homeowners with an income of x, a mortgage of y, and a debt-to-income ratio of z. Anecdotally, from reading posts on Twitter and other sites from people who sold their SFH's to Zillow in the iBuyer program it appears that in economically depressed regions of the United States (the Midwest, rural places within 1-2 hours of a medium-sized city, etc.) that 'cash-in-hand' amount that the iBuyer program offered homeowners to sell their houses to Zillow is only $15,000 - $30,000 over list price per property. People who sold their houses to Zillow via the iBuyer program were talking about how they could 'pay off their new car and have a bit left over to buy new appliances in their new house'. These sums are rounding errors to Zillow's business mode even when multiplied by the thousands of properties that Zillow bought. But to the home sellers, $30,000 or $50,000 is written about as thought it is some life-changing sum of money. I say this not to mock, demean nor poke fun at the homeowners, having grown up in one of these economically-depressed regions of the United States. A good portion of these homeowners are selling their modest homes and taking on risky levels of debt in a real estate bubble. I hope that I'm wrong about the risk that they're incurring - all the while celebrating getting $30,000 cash-in-hand from Zillow - but I don't think that I am misreading the situation. - Hard numbers on how much (or how little) a given housing market's supply need to be (artificially) constrained before home prices skyrocket. Real estate markets in different regions behave differently: rural Iowa's market is nothing like Santa Barbara's market which is different from Boston's market. Until Zillow undertook large-scale coordinated (artifical) reduction in supply *at a time of unprecendented _physical_ mobility of workers due to remote work status during covid19* we really had no way to model which markets would be more resistent to large upticks in housing prices, which markets would see meteoric growth quickly and in a sustained fashion ('pent-up demand'). So if I were running Zillow's iBuying experiment 'failure' as a data scientist I would be delighted in the new data points gleaned from the 'failed experiment', namely: - What's the exact dollar amount that causes single family homeowners to abandon the 'sticky' process of selling their home through a human (broker, real estate agent)? Answer: it's pretty damn low for most folks: less than $100,000 over the Zestimate price or price that their real estate agent quoted them. And home sellers talked about that being some huge windfall enabling them to pay off a new car, or buy all new appliances in their new home (that they probably overpaid for). - To what degree do I have to (artifically) constrain the housing supply in different regions to induce a 'feeding frenzy' / FOMO / social-contagion-like behavior? By manipulating public perception of the real estate market in their area can I induce irrational / deleterious individual behaviors that then spread to others in their geographic area and social circles? To me Zillow's iBuying experiment mirrors what Facebook allowed researchers to do in the mid-2010s when they manipulated content in user's feeds to see if they could induce positive or negative emotional states: https://www.theguardian.com/technology/2014/jun/29/facebook-users-emotions-news-feeds https://www.theguardian.com/technology/2014/jun/29/facebook-... Until recently there has never been a way to leverage mechanisms of social contagion in the nation-wide housing market for the middle class. Five plus years ago when working at the real estate tech company I would have loved to get my hands on a dataset of linked behavioral data like this, especially a dataset that had reduced geographic buying pressure (due to remote work) as the dataset would have revolutionalized supply- and demand-side real estate data product development.
• wdr1 5y ago
  It was pretty obvious to me these algos are busted when an agent mispriced a house in our neighborhood at $600,000,000 (instead of just $6M) and the "estimate" was something like $604,082,000
• SergeAx 5y ago
  I don't understand one thing. Predicting a price of a stock or derivative for next 24 hours should be much easier task. We have a tremendous amount of perfect quality data to train our models on and to infere from. Having those prediction accurate enough will be an Eldorado. And still no one solved that problem. Why Zillow ever thought they gonna solve much more complex problem with much higher risk using much worse data?
  • SergeAx 5y ago
    On the other hand, what if everybody solved the problem of stock price prediction the way Zillow made it with housing market, and a steady growth we are seeing now is just a byproduct of those algos? When will we have a Zillow moment there?
• culebron21 5y ago
  Couple of years ago, there's been an article posted here: How to Recognize AI Snake Oil. https://readhacker.news/c/4cSun https://readhacker.news/c/4cSun It looks like this was the case with Zillow. Most of commenters here insist ML and data science have nothing to do with this failure, and put all blame on the management. I don't by it's not failure of ML. In the last several years I consistently see Tech leads oversell the possibilities of ML and theorizing that ML could solve many problems with deep learning. The result is always modest. Error margins for real world metrics that involve location data are around 20-30%, which is somewhat better than analysts predictions in Excel, but not enough to rely on them. I saw very competent ML engineers try to improve a model, take a huge amount of satellite imagery, tracking data, etc., train their neural networks for couple of months, and only improve error margin by 1 percent point! (e.g. 25% -> 24%)