14 ms·

Zillow, Prophet, time series, and prices

213 points by zorked 5y ago

trhoad 5y ago
Imagine being this angry about a job listing mentioning a Python library...
- extr 5y ago
  He's not angry. He's responding to people who are angry he flippantly made fun of the job posting.
  - trhoad 5y ago
    > The requirement that people come to your company knowing how to use piss easy baby tools is an extremely dumb and lazy hiring practice. Sounds angry to me!
    - brodouevencode 5y ago
      Nah he's mocking the crap out of them.
    - rmah 5y ago
      He sounded at least a bit put out to me too
  - windsignaling 5y ago
    Worse. He sounds "fake angry" and spent what looks like an inordinate amount of time writing a crappy article to express his fake outrage. His understanding of time series seems OK at best.
evergrande 5y ago
Is it possible that Zillow artificially inflated the housing market? They bought up bunch of houses while having the most popular pricing tool? That's a strong incentive to predict ever increasing prices. Plus it whipped up a frenzy and site traffic. Dangerous incentives all around. You can see how this could be self reinforcing with systems that all want the numbers to go up.
- brodouevencode 5y ago
  Zillow is just an aggregate typically MLS data. Sure it can probably have some influence via it's Zestimate product, but that's not what failed. Anecdotally 1/4 of the folks in my neighborhood currently have a for sale sign in their yard because of the price increases. I'm not sure where this upward pressure is coming from, but I fear another 2008.
  - gpapilion 5y ago
    The economy itself has been good the last 12 months in coastal cities. The upward pressure is coming from those who have money to spend upgrading the life. I don’t fear another 2008 since the mortgage market hasn’t gone as crazy. I do think we’ll see a correction, but it’s more likely to be a slow down and increase in time to sell homes.
  - poulsbohemian 5y ago
    > Anecdotally 1/4 of the folks in my neighborhood currently have a for sale sign in their yard because of the price increases. If I might offer some thoughts as a real estate agent... so we are seeing some of this in my market as well. In particular, we're seeing houses that are sitting on market longer than they have the rest of the past two years. This is causing many people in my market to prognosticate that the market is slowing down / we're in a bubble / this is the top, etc. Instead the problem is that there are sellers who see dollar signs and are listing their homes much higher than the market can actually bear. It's not that there aren't buyers or that the cost of capital has increased, they have just gotten greedy as seller. So - I don't know what market you are in, but my guess would be that many of those opportunistic sellers will be disappointed and that the market will self correct - not that it will crash, but simply self-correct.
    - brodouevencode 5y ago
      That makes sense. The ones I do know that are selling - they're not tied to this area by family/jobs/other commitments and they probably see an easy cash grab because of that. One family was in their house for 9 months then sold it for $80k, or ~25%, more than what they paid for it (according to Zillow). Honestly not a bad payday but how long can something like that last?
rossdavidh 5y ago
While there are some decent points in this post, it ignores another possibility for why Zillow Offers got shut down. The first explanation, when it was just being paused, was that there was a shortage of labor and materials to do the renovating. Probably true, but not the real reason. It is entirely possible that the second explanation, that it was too hard to model accurately the price 6 months out, is also true but not the real reason. The real reason might be something the Zillow CEO doesn't want to say out loud. Like, we are near a top in house prices, and it looks like it might fall a lot, or for a long time, or both, before we get back to reasonable house prices that have a chance of going up again. Zillow does not want to say "real estate is about to bust, as an investment strategy", because they are still tied to the real estate market. But, flawed as their analysis might have been, they might have been able to see enough to know that no matter how good their algorithm could be made, they're not going to be able to buy low and sell high if the prices keep falling. Now, maybe Zillow is wrong again, and housing is not in for a rough ride in the very near future. But maybe they're not wrong, and shutting down Zillow Offers (rather than fixing it) will look like a quite prudent move in a year's time.
- sokoloff 5y ago
  Rising interest rates will be a drag on the market. It’s clear those will eventually come, but maybe not for five years. Pundits have called 23 (or more) of the last 2 housing market drops.
- dsizzle 5y ago
  The post seems more like an explanation for how Zillow they lost a bunch of money going long housing in a real estate boom than why Zillow is shutting down Zillow Offers.
- anonAndOn 5y ago
  The only time I've seen house flipping work at scale is 2010-2012 when those with huge piles of cash could buy foreclosures in a few heavily affected metros, do nothing for 3-6 months then put them back on the market for a tidy profit. After that brief window closed, you either had to become a landlord for a couple years or employ a crew of handymen on staff to fix all the surprises that come with being a homeowner. You're either a landlord or a builder if you want to actually make money outside of a recession.
tqi 5y ago
This was pretty stupid straw-man. I doubt anyone who has used Prophet is under any illusions as to what it is doing under the hood (after all, the only input is a single time series dataframe...). Companies use it to do quarterly/yearly goal setting and anomaly detection, not deploying it to production to power product features. This whole thing has the tone and substance of a giant "i am very smart" faff. The author also seems annoyed that people get paid 200k for such little "technical" skill, to which I would ask why he cares?
- lainga 5y ago
  >to which I would ask why he cares? Do you think he didn't think enough about the reasons he stated in TFA for why he cares? Or they're not good reasons? He says, from the employer's perspective, "$200k a year can attract people who actually know what they’re doing. Maybe a math or econ PhD. Maybe a Microsoft Excel pros with 10 years of finance sector experience"; from the prospective employee's, "The aggregate effect of this practice being widespread is that talented people with unusual backgrounds get gatekept out of good paying jobs that they’d be exceptional at. ... The job post excludes people not based on aptitude, but based on whether they have previous experience and familiarity with a tool they could be introduced to and then master in under 15 minutes."
  - tqi 5y ago
    I think reason he gives about gatekeeping is only weakly related to the bulk of the article. The point is that job listings shouldn't be arbitrarily specific is an obvious one that most already agree with, and was a thin excuse to show off his deep knowledge of modeling jargon.
- zwaps 5y ago
  If you read the article further it morphs from "you gotta have a better time series tool bro" to an article about "if you don't know what adverse selection is, no amount of highly-paid machine learning engineers will help you". This, I think, it a more valid point (and it was not only made by the author). Or, let's say, it is at least a fun implication: When companies like Amazon and Google hire hundreds of Econ PhDs to design platforms, market ops and auctions, it IS amusing to imagine that Zillow thought they could just "Machine Learn" their way through it and they fall flat on their noses.
vmception 5y ago
BTFD. Zillow issued $4bn corporate bonds that they only have to pay 2% on annually while they pay it back over 10 years. This is way below their annual revenue which was already $2.7bn annually. The bonds were for the expansion of Zillow Offers, but based on their prior revenue. And now, they are trying to sell a bundle of homes for $2.8bn all at once “at a loss”, with no information about the percentage of loss. -5%? -25%? Does anyone else see that it doesn't matter even if it was 100% loss which it clearly isnt? People out here acting like this is the great financial crisis they’ve missed 12 years of alpha just waiting for. This is the least leveraged market participant in real estate lol. They flipped a few (thousand) houses maybe at an overall profit, made some of their developer and overleveraged home owner friends happy, and found an excuse to fire a bunch of their workforce! This game is obvious
- deleted 5y ago
  [deleted]
msapaydin 5y ago
Extremely well written article for someone who would like to dig deeper into forecasting, actually giving someone ne valuable links about the prophet tool that i used and recommended. I agree with the author that it is strange that the job posting only mentions the prophet tool, which is rather like a very basic and one particular library that i usually recommend to undergrads for their senior project.
pjdemers 5y ago
Very likely this was not a real job post, but rather the statutory post required for a visa application (H1-B, etc). Zillow HR looked at the current employee's background for a set of skills that is rarely found in one person, to keep the number of qualified outside applicants at zero. "Must be a a senior person with recent hands on experience in this novice tool". What they didn't say is, their exist employee used Prophet in a class unrelated to work.
- dav43 5y ago
  Similarly in Singapore - a lot of jobs require that, even if a direct candidate is ready to be hired, the job is advertised for a month (government mandate). As a result job ads are poor quality and extremely specific to get around visa constraints.
- rossdavidh 5y ago
  This is an excellent point, and whether or not it happened here it is often what happens, not only in developer jobs.
sushsjsuauahab 5y ago
My opinion is that we are not at the top of the housing market. People are realizing that it is not fun to live in dense areas. Local markets may ebb and flow, but desirable houses will rise. Phoenix is particularly dense and the house prices are cheap, which is why I believe Zillow sold them all-- they realized their actual profits weren't as good as what they projected.
- IIAOPSW 5y ago
  >People are realizing that it is not fun to live in dense areas. Sounds like you're assuming your personal preferences are the natural ones which "people" surely have by default.
  - sushsjsuauahab 5y ago
    While I respect your opinion, I would argue that if we were to look at the data, it would show some level of shift in people's preferences for where they want to live.
evrydayhustling 5y ago
> The main reason why competitive market price movements tend to be stochastic and I(1), and tend to exhibit little highly profitable predictable seasonality, is because if that wasn’t true, then people would be able to make free money and markets would no longer be weak-form efficient. Oh no! Someone invoked weak-form efficiency! I guess every market maker should shut down now? OP is trying to criticize the hypothetical Prophet-trader because Prophet seems to rely on trivial seasonalities for forecasting. But he is ignoring that the bare minimum configuration for Prophet requires several injections of domain expertise (or at least bias): - Prophet predicts event frequencies, not prices/events. So presumably it must be used in conjunction with another price model. - The Prophet user must acquire and group data for their forecast -- which is itself a form of locale-sensitive regularization, and is often >50% of the challenge of calibrating a predictive model. - Prophet users configure market sizes, change points and event streams ("holidays" are a special case, not the only example) based on external data. Obviously none of these things saved Zillow, but they are all outside the conditions supported by weak-form efficiency, which gives you a good idea of how useless that concept is in practical trading. I have zero investment in Prophet and wouldn't use it for a trading model, but it's annoying to see Data Twitter's culture of assuming others are dumb in order to quote grad school stats classes getting rebroadcast elsewhere.
kristjansson 5y ago
It's instructive to recall that the original use case for Prophet at FB was 'good enough' forecasts for a large number of time series. Both the modeling choices and interface are optimized for that point in design space.
lordnacho 5y ago
I see some decent points in the post, but they are mostly about modelling. What I'd like to say is even though you have a good model, that doesn't tell you what to do. If you know it's going to rain next week, should you go corner the umbrella market? If you are going down that route, there are a bunch of non-weather things that you'll need to know. Perhaps where to find a cheap source of umbrellas, likely places where people will be the most susceptible to buying, and so on. Basically domain expertise. The major issue you will have is whether your own presence causes your model to be inapplicable. If a guy sees you at the umbrella factory, maybe he will decide that's a signal that the weather will rain next week? Maybe people will just stay home in that case? The main thing I see with Zillow is a lack of due diligence. There seems to have been an idea that you can just have some data people, pour in data, and out comes money. In reality there's a lot of legwork in any business, and you pay for experience with money. In this case also your job.
- Guest42 5y ago
  Another ignored issue I see is that people choose the modeling methodology they use, the scoring methodology, and the data that goes into the model. The reactions act as though the model has a life of its own and that some unlikely error led to these results. Purchased data/models can frequently be wrong or unhelpful to one’s modeling objective. Finance also has a number of risks vs rewards with every decision.
microprediction 5y ago
Hi I'm the author of the Prophet article https://www.microprediction.com/blog/prophet https://www.microprediction.com/blog/prophet which for better or worse is why many people have recently questioned Prophet as a time-series method. (There are better, and certainly more formal critiques before and after - I referenced several). I just hope those who push back on your article read all the way to the last paragraph. I'll repeat it below. Very well put. I'm happy to answer technical questions about Prophet from anyone here but again, this is somewhat beside the point, which is... The requirement that people come to your company knowing how to use piss easy baby tools is an extremely dumb and lazy hiring practice. It is also, unfortunately, a common practice in data science job postings. The aggregate effect of this practice being widespread is that talented people with unusual backgrounds get gatekept out of good paying jobs that they’d be exceptional at. Making fun of the job posting and using Prophet has been compared to gatekeeping. To be clear, the Prophet prerequisite is an actual form of gatekeeping being undertaken by a major company that has actual material impacts on people’s careers. The job post excludes people not based on aptitude, but based on whether they have previous experience and familiarity with a tool they could be introduced to and then master in under 15 minutes. A tweet making fun of the job posting is not gatekeeping. Get over it, LinkedIn clout chasers I would add that since posting my own less-well worded version of this astonishment I have received numerous DM's from people at large companies who are aghast at the way Prophet is a favorite of management. So whether or not this was a problem at Zillow beyond, say, 2015, it might well be the case elsewhere.
- teitoklien 5y ago
  Your blogs are awesome ! I check on them every few weeks to see, What you wrote next :D
- krallistic 5y ago
  If claims to have good knowledge in TS, but had not played around with Prophet (which is dead simple, takes a few hours - or 15 minutes for you - to see some problems), how interested/knowledgeable are they in TS? (I assume that they look for candidates knowledgeable in that field, and just playing around with prophet dosnt make you knowledgeable) Especially in TS is saw many PhDs, really focussed on one specific method, not looking left and right and neither are pragmatic about their choices. Which is essential to solve anything in real life. Filtering candidates out, based on criteria like "played around with different stuff" is a good indicator for open-minded people IMHO.
alpineidyll3 5y ago
A good prediction is one which is made BEFORE the outcome. All of these armchair data scientists had the opportunity to short Zillow BEFORE it was obvious that zoffer was losing money. No one did? The best outcome which could flow from this is a sort of adverse peer review in data science based on asset trades. To be honest, the zillow data team should have sought this themselves, the community could have debugged them.
- letsgodaledo 5y ago
  I considered shorting when the Offers program was announced. Thankfully I didn't because I'd still be upside down on that bet.
haroldbolls 5y ago
Like with most market making operations, your main risk is adverse selection. I strongly suspect that this was what killed the zillow operation. Suppose zillow is on average right with their house price prediction, but in 50% of cases they offer 10% too much and in 50% of cases they offer 10% too little. In their backtest this will show as a fair price prediction. However in reality, they will mostly end up buying the houses where they overpay 10%. Only their bad quotes are being hit so they keep consistently overpaying for properties.
- dannykwells 5y ago
  I like this point, since it does resonate with what much of the anecdotal evidence was, i.e., Zillow offering too much for houses around the country. The high offers would be the ones accepted (and memorable), meaning Zillow would tend to over pay for houses (leading to the rut they are in).
xrd 5y ago
It's normal to assert that open source generates millions or even billions of dollars that isn't captured. In this case, it seems like an open source project destroyed billions of dollars. That's a new idea for me.
- vintermann 5y ago
  Well, it made money for someone, just not the ones who used the product.
- TheCoelacanth 5y ago
  The library didn't destroy any value. The morons who thought that they would be able to gain a competitive advantage in a highly competitive market just by applying a publicly-available library did.
stopagephobia 5y ago
I don't know, maybe, or only I liked it better when there was so much new houses being built that this wouldn't of happened. Some of us People of Age remember growing up in houses built in post-war baby boom buildings, so there was enough of a housing increase that people weren't paying so inflated prices for houses. Maybe this is what we get when interest rates are so low and so little money is needed for down payments, I know a bunch of people who are started getting into real estate in the last year because even though it's lots of money banks make it easy if you have credit.
- rgbrgb 5y ago
  > this is what we get when interest rates are so low I think it's what we get when the voting base is homeowners who want to protect property values with restrictive building policy. Edit: To expound a bit... one has to ask why, given that there is a huge and growing market of buyers and would-be buyers if they could afford it, the market is not producing supply like you describe for baby boomers. My anecdotal understanding of this is that there are just a ton of boomers who don't want to build anymore, who want to "protect the character" of their neighborhoods, and would prefer that new would-be-buyers look elsewhere.
  - poulsbohemian 5y ago
    You are absolutely on the right path, IMHO... in my market we have a lot of land, but the zoning of it is such that it is arduous to build. Likewise, what can be built profitably is not necessarily affordable for buyer. I see gaps at every price point - the first time buyer has to either buy a poorly built new construction that they can barely afford, or they have to buy older construction where they may not have the equity or capital to property maintain. At the top of market, IE: luxury, builders are hesitant to build without a ready buyer because of the massive outlay, especially knowing that those buyers are very selective. Middle of the market you end up with properties that smell of builders trying to push up price and perceived amenities while being very careful to not push up their costs. So you end up with nice houses on terrible lots, or you end up with cheap construction with a varnish of granite and stainless steel appliances to make it look better than it is.
mcastillon 5y ago
I'm surprised that nothing here has been written about human in the loop. All time series models are flawed in some capacity, and are going to have difficulties in the current market given how it won't necessarily match historical seasonalities. Just having more human safeguards to flag wacky outputs seems to be a more reasonable explanation than not handling Prophet's shortcomings properly. Zillow would have likely faced similar issues with other time series packages as well
- mcastillon 5y ago
  Reading some more, it does look like Zillow was in fact leveraging humans in the loop https://venturebeat.com/2019/12/11/why-explainable-ai-is-indispensable-to-zillows-business/ https://venturebeat.com/2019/12/11/why-explainable-ai-is-ind.... What seems like a likely explanation is that the proper balance between machine and human judgement was not calibrated to this current market. "Part of the challenge of Zillow Offers’ human-in-the-loop system — any such system, really — is finding the balance between humans and machines. “In order to optimize this human-in-the-loop system, we’d like to figure out when the human is best, when an assistive situation is best, and […] when a machine is best,” said Fagnan."
say_it_as_it_is 5y ago
There isn't a company or individual who has models that can accurately predict the future of housing prices. There are only charlatans who claim they can achieve the impossible. This guy can do no better, but he'll do his best to convince capitalists that he's got the right stuff.
gradient_desc 5y ago
I’m surprised I haven’t seen this rumor yet https://twitter.com/sh_reya/status/1457094567111528455?s=21 https://twitter.com/sh_reya/status/1457094567111528455?s=21
gradient_desc 5y ago
Blind is suggesting another reason this went wrong - misaligned incentives and overriding the predicted prices https://twitter.com/sh_reya/status/1457094567111528455?s=21 https://twitter.com/sh_reya/status/1457094567111528455?s=21
a_square_peg 5y ago
I use Prophet quite a bit because it allows me to add exogenous variables (temperature, wind, solar etc.) easily and assess their impact with reasonable outcome for electricity load analysis. One thing that baffles me is when people try to use it to predict weather itself.

213 points by zorked 5y ago

trhoad 5y ago

Imagine being this angry about a job listing mentioning a Python library...

extr 5y ago
He's not angry. He's responding to people who are angry he flippantly made fun of the job posting.
- trhoad 5y ago
  > The requirement that people come to your company knowing how to use piss easy baby tools is an extremely dumb and lazy hiring practice. Sounds angry to me!
  - brodouevencode 5y ago
    Nah he's mocking the crap out of them.
  - rmah 5y ago
    He sounded at least a bit put out to me too
- windsignaling 5y ago
  Worse. He sounds "fake angry" and spent what looks like an inordinate amount of time writing a crappy article to express his fake outrage. His understanding of time series seems OK at best.

evergrande 5y ago

Is it possible that Zillow artificially inflated the housing market? They bought up bunch of houses while having the most popular pricing tool? That's a strong incentive to predict ever increasing prices. Plus it whipped up a frenzy and site traffic. Dangerous incentives all around. You can see how this could be self reinforcing with systems that all want the numbers to go up.

brodouevencode 5y ago
Zillow is just an aggregate typically MLS data. Sure it can probably have some influence via it's Zestimate product, but that's not what failed. Anecdotally 1/4 of the folks in my neighborhood currently have a for sale sign in their yard because of the price increases. I'm not sure where this upward pressure is coming from, but I fear another 2008.
- gpapilion 5y ago
  The economy itself has been good the last 12 months in coastal cities. The upward pressure is coming from those who have money to spend upgrading the life. I don’t fear another 2008 since the mortgage market hasn’t gone as crazy. I do think we’ll see a correction, but it’s more likely to be a slow down and increase in time to sell homes.
- poulsbohemian 5y ago
  > Anecdotally 1/4 of the folks in my neighborhood currently have a for sale sign in their yard because of the price increases. If I might offer some thoughts as a real estate agent... so we are seeing some of this in my market as well. In particular, we're seeing houses that are sitting on market longer than they have the rest of the past two years. This is causing many people in my market to prognosticate that the market is slowing down / we're in a bubble / this is the top, etc. Instead the problem is that there are sellers who see dollar signs and are listing their homes much higher than the market can actually bear. It's not that there aren't buyers or that the cost of capital has increased, they have just gotten greedy as seller. So - I don't know what market you are in, but my guess would be that many of those opportunistic sellers will be disappointed and that the market will self correct - not that it will crash, but simply self-correct.
  - brodouevencode 5y ago
    That makes sense. The ones I do know that are selling - they're not tied to this area by family/jobs/other commitments and they probably see an easy cash grab because of that. One family was in their house for 9 months then sold it for $80k, or ~25%, more than what they paid for it (according to Zillow). Honestly not a bad payday but how long can something like that last?

rossdavidh 5y ago

While there are some decent points in this post, it ignores another possibility for why Zillow Offers got shut down. The first explanation, when it was just being paused, was that there was a shortage of labor and materials to do the renovating. Probably true, but not the real reason. It is entirely possible that the second explanation, that it was too hard to model accurately the price 6 months out, is also true but not the real reason. The real reason might be something the Zillow CEO doesn't want to say out loud. Like, we are near a top in house prices, and it looks like it might fall a lot, or for a long time, or both, before we get back to reasonable house prices that have a chance of going up again. Zillow does not want to say "real estate is about to bust, as an investment strategy", because they are still tied to the real estate market. But, flawed as their analysis might have been, they might have been able to see enough to know that no matter how good their algorithm could be made, they're not going to be able to buy low and sell high if the prices keep falling. Now, maybe Zillow is wrong again, and housing is not in for a rough ride in the very near future. But maybe they're not wrong, and shutting down Zillow Offers (rather than fixing it) will look like a quite prudent move in a year's time.

sokoloff 5y ago
Rising interest rates will be a drag on the market. It’s clear those will eventually come, but maybe not for five years. Pundits have called 23 (or more) of the last 2 housing market drops.
dsizzle 5y ago
The post seems more like an explanation for how Zillow they lost a bunch of money going long housing in a real estate boom than why Zillow is shutting down Zillow Offers.
anonAndOn 5y ago
The only time I've seen house flipping work at scale is 2010-2012 when those with huge piles of cash could buy foreclosures in a few heavily affected metros, do nothing for 3-6 months then put them back on the market for a tidy profit. After that brief window closed, you either had to become a landlord for a couple years or employ a crew of handymen on staff to fix all the surprises that come with being a homeowner. You're either a landlord or a builder if you want to actually make money outside of a recession.

tqi 5y ago

This was pretty stupid straw-man. I doubt anyone who has used Prophet is under any illusions as to what it is doing under the hood (after all, the only input is a single time series dataframe...). Companies use it to do quarterly/yearly goal setting and anomaly detection, not deploying it to production to power product features. This whole thing has the tone and substance of a giant "i am very smart" faff. The author also seems annoyed that people get paid 200k for such little "technical" skill, to which I would ask why he cares?

lainga 5y ago
>to which I would ask why he cares? Do you think he didn't think enough about the reasons he stated in TFA for why he cares? Or they're not good reasons? He says, from the employer's perspective, "$200k a year can attract people who actually know what they’re doing. Maybe a math or econ PhD. Maybe a Microsoft Excel pros with 10 years of finance sector experience"; from the prospective employee's, "The aggregate effect of this practice being widespread is that talented people with unusual backgrounds get gatekept out of good paying jobs that they’d be exceptional at. ... The job post excludes people not based on aptitude, but based on whether they have previous experience and familiarity with a tool they could be introduced to and then master in under 15 minutes."
- tqi 5y ago
  I think reason he gives about gatekeeping is only weakly related to the bulk of the article. The point is that job listings shouldn't be arbitrarily specific is an obvious one that most already agree with, and was a thin excuse to show off his deep knowledge of modeling jargon.
zwaps 5y ago
If you read the article further it morphs from "you gotta have a better time series tool bro" to an article about "if you don't know what adverse selection is, no amount of highly-paid machine learning engineers will help you". This, I think, it a more valid point (and it was not only made by the author). Or, let's say, it is at least a fun implication: When companies like Amazon and Google hire hundreds of Econ PhDs to design platforms, market ops and auctions, it IS amusing to imagine that Zillow thought they could just "Machine Learn" their way through it and they fall flat on their noses.

vmception 5y ago

BTFD. Zillow issued $4bn corporate bonds that they only have to pay 2% on annually while they pay it back over 10 years. This is way below their annual revenue which was already $2.7bn annually. The bonds were for the expansion of Zillow Offers, but based on their prior revenue. And now, they are trying to sell a bundle of homes for $2.8bn all at once “at a loss”, with no information about the percentage of loss. -5%? -25%? Does anyone else see that it doesn't matter even if it was 100% loss which it clearly isnt? People out here acting like this is the great financial crisis they’ve missed 12 years of alpha just waiting for. This is the least leveraged market participant in real estate lol. They flipped a few (thousand) houses maybe at an overall profit, made some of their developer and overleveraged home owner friends happy, and found an excuse to fire a bunch of their workforce! This game is obvious

deleted 5y ago
[deleted]

msapaydin 5y ago

Extremely well written article for someone who would like to dig deeper into forecasting, actually giving someone ne valuable links about the prophet tool that i used and recommended. I agree with the author that it is strange that the job posting only mentions the prophet tool, which is rather like a very basic and one particular library that i usually recommend to undergrads for their senior project.

pjdemers 5y ago

Very likely this was not a real job post, but rather the statutory post required for a visa application (H1-B, etc). Zillow HR looked at the current employee's background for a set of skills that is rarely found in one person, to keep the number of qualified outside applicants at zero. "Must be a a senior person with recent hands on experience in this novice tool". What they didn't say is, their exist employee used Prophet in a class unrelated to work.

dav43 5y ago
Similarly in Singapore - a lot of jobs require that, even if a direct candidate is ready to be hired, the job is advertised for a month (government mandate). As a result job ads are poor quality and extremely specific to get around visa constraints.
rossdavidh 5y ago
This is an excellent point, and whether or not it happened here it is often what happens, not only in developer jobs.

sushsjsuauahab 5y ago

My opinion is that we are not at the top of the housing market. People are realizing that it is not fun to live in dense areas. Local markets may ebb and flow, but desirable houses will rise. Phoenix is particularly dense and the house prices are cheap, which is why I believe Zillow sold them all-- they realized their actual profits weren't as good as what they projected.

IIAOPSW 5y ago
>People are realizing that it is not fun to live in dense areas. Sounds like you're assuming your personal preferences are the natural ones which "people" surely have by default.
- sushsjsuauahab 5y ago
  While I respect your opinion, I would argue that if we were to look at the data, it would show some level of shift in people's preferences for where they want to live.

evrydayhustling 5y ago

> The main reason why competitive market price movements tend to be stochastic and I(1), and tend to exhibit little highly profitable predictable seasonality, is because if that wasn’t true, then people would be able to make free money and markets would no longer be weak-form efficient. Oh no! Someone invoked weak-form efficiency! I guess every market maker should shut down now? OP is trying to criticize the hypothetical Prophet-trader because Prophet seems to rely on trivial seasonalities for forecasting. But he is ignoring that the bare minimum configuration for Prophet requires several injections of domain expertise (or at least bias): - Prophet predicts event frequencies, not prices/events. So presumably it must be used in conjunction with another price model. - The Prophet user must acquire and group data for their forecast -- which is itself a form of locale-sensitive regularization, and is often >50% of the challenge of calibrating a predictive model. - Prophet users configure market sizes, change points and event streams ("holidays" are a special case, not the only example) based on external data. Obviously none of these things saved Zillow, but they are all outside the conditions supported by weak-form efficiency, which gives you a good idea of how useless that concept is in practical trading. I have zero investment in Prophet and wouldn't use it for a trading model, but it's annoying to see Data Twitter's culture of assuming others are dumb in order to quote grad school stats classes getting rebroadcast elsewhere.

kristjansson 5y ago

It's instructive to recall that the original use case for Prophet at FB was 'good enough' forecasts for a large number of time series. Both the modeling choices and interface are optimized for that point in design space.

lordnacho 5y ago

I see some decent points in the post, but they are mostly about modelling. What I'd like to say is even though you have a good model, that doesn't tell you what to do. If you know it's going to rain next week, should you go corner the umbrella market? If you are going down that route, there are a bunch of non-weather things that you'll need to know. Perhaps where to find a cheap source of umbrellas, likely places where people will be the most susceptible to buying, and so on. Basically domain expertise. The major issue you will have is whether your own presence causes your model to be inapplicable. If a guy sees you at the umbrella factory, maybe he will decide that's a signal that the weather will rain next week? Maybe people will just stay home in that case? The main thing I see with Zillow is a lack of due diligence. There seems to have been an idea that you can just have some data people, pour in data, and out comes money. In reality there's a lot of legwork in any business, and you pay for experience with money. In this case also your job.

Guest42 5y ago
Another ignored issue I see is that people choose the modeling methodology they use, the scoring methodology, and the data that goes into the model. The reactions act as though the model has a life of its own and that some unlikely error led to these results. Purchased data/models can frequently be wrong or unhelpful to one’s modeling objective. Finance also has a number of risks vs rewards with every decision.

microprediction 5y ago

Hi I'm the author of the Prophet article https://www.microprediction.com/blog/prophet https://www.microprediction.com/blog/prophet which for better or worse is why many people have recently questioned Prophet as a time-series method. (There are better, and certainly more formal critiques before and after - I referenced several). I just hope those who push back on your article read all the way to the last paragraph. I'll repeat it below. Very well put. I'm happy to answer technical questions about Prophet from anyone here but again, this is somewhat beside the point, which is... The requirement that people come to your company knowing how to use piss easy baby tools is an extremely dumb and lazy hiring practice. It is also, unfortunately, a common practice in data science job postings. The aggregate effect of this practice being widespread is that talented people with unusual backgrounds get gatekept out of good paying jobs that they’d be exceptional at. Making fun of the job posting and using Prophet has been compared to gatekeeping. To be clear, the Prophet prerequisite is an actual form of gatekeeping being undertaken by a major company that has actual material impacts on people’s careers. The job post excludes people not based on aptitude, but based on whether they have previous experience and familiarity with a tool they could be introduced to and then master in under 15 minutes. A tweet making fun of the job posting is not gatekeeping. Get over it, LinkedIn clout chasers I would add that since posting my own less-well worded version of this astonishment I have received numerous DM's from people at large companies who are aghast at the way Prophet is a favorite of management. So whether or not this was a problem at Zillow beyond, say, 2015, it might well be the case elsewhere.

teitoklien 5y ago
Your blogs are awesome ! I check on them every few weeks to see, What you wrote next :D
krallistic 5y ago
If claims to have good knowledge in TS, but had not played around with Prophet (which is dead simple, takes a few hours - or 15 minutes for you - to see some problems), how interested/knowledgeable are they in TS? (I assume that they look for candidates knowledgeable in that field, and just playing around with prophet dosnt make you knowledgeable) Especially in TS is saw many PhDs, really focussed on one specific method, not looking left and right and neither are pragmatic about their choices. Which is essential to solve anything in real life. Filtering candidates out, based on criteria like "played around with different stuff" is a good indicator for open-minded people IMHO.

alpineidyll3 5y ago

A good prediction is one which is made BEFORE the outcome. All of these armchair data scientists had the opportunity to short Zillow BEFORE it was obvious that zoffer was losing money. No one did? The best outcome which could flow from this is a sort of adverse peer review in data science based on asset trades. To be honest, the zillow data team should have sought this themselves, the community could have debugged them.

letsgodaledo 5y ago
I considered shorting when the Offers program was announced. Thankfully I didn't because I'd still be upside down on that bet.

haroldbolls 5y ago

Like with most market making operations, your main risk is adverse selection. I strongly suspect that this was what killed the zillow operation. Suppose zillow is on average right with their house price prediction, but in 50% of cases they offer 10% too much and in 50% of cases they offer 10% too little. In their backtest this will show as a fair price prediction. However in reality, they will mostly end up buying the houses where they overpay 10%. Only their bad quotes are being hit so they keep consistently overpaying for properties.

dannykwells 5y ago
I like this point, since it does resonate with what much of the anecdotal evidence was, i.e., Zillow offering too much for houses around the country. The high offers would be the ones accepted (and memorable), meaning Zillow would tend to over pay for houses (leading to the rut they are in).

xrd 5y ago

It's normal to assert that open source generates millions or even billions of dollars that isn't captured. In this case, it seems like an open source project destroyed billions of dollars. That's a new idea for me.

vintermann 5y ago
Well, it made money for someone, just not the ones who used the product.
TheCoelacanth 5y ago
The library didn't destroy any value. The morons who thought that they would be able to gain a competitive advantage in a highly competitive market just by applying a publicly-available library did.

stopagephobia 5y ago

I don't know, maybe, or only I liked it better when there was so much new houses being built that this wouldn't of happened. Some of us People of Age remember growing up in houses built in post-war baby boom buildings, so there was enough of a housing increase that people weren't paying so inflated prices for houses. Maybe this is what we get when interest rates are so low and so little money is needed for down payments, I know a bunch of people who are started getting into real estate in the last year because even though it's lots of money banks make it easy if you have credit.

rgbrgb 5y ago
> this is what we get when interest rates are so low I think it's what we get when the voting base is homeowners who want to protect property values with restrictive building policy. Edit: To expound a bit... one has to ask why, given that there is a huge and growing market of buyers and would-be buyers if they could afford it, the market is not producing supply like you describe for baby boomers. My anecdotal understanding of this is that there are just a ton of boomers who don't want to build anymore, who want to "protect the character" of their neighborhoods, and would prefer that new would-be-buyers look elsewhere.
- poulsbohemian 5y ago
  You are absolutely on the right path, IMHO... in my market we have a lot of land, but the zoning of it is such that it is arduous to build. Likewise, what can be built profitably is not necessarily affordable for buyer. I see gaps at every price point - the first time buyer has to either buy a poorly built new construction that they can barely afford, or they have to buy older construction where they may not have the equity or capital to property maintain. At the top of market, IE: luxury, builders are hesitant to build without a ready buyer because of the massive outlay, especially knowing that those buyers are very selective. Middle of the market you end up with properties that smell of builders trying to push up price and perceived amenities while being very careful to not push up their costs. So you end up with nice houses on terrible lots, or you end up with cheap construction with a varnish of granite and stainless steel appliances to make it look better than it is.

mcastillon 5y ago

I'm surprised that nothing here has been written about human in the loop. All time series models are flawed in some capacity, and are going to have difficulties in the current market given how it won't necessarily match historical seasonalities. Just having more human safeguards to flag wacky outputs seems to be a more reasonable explanation than not handling Prophet's shortcomings properly. Zillow would have likely faced similar issues with other time series packages as well

mcastillon 5y ago
Reading some more, it does look like Zillow was in fact leveraging humans in the loop https://venturebeat.com/2019/12/11/why-explainable-ai-is-indispensable-to-zillows-business/ https://venturebeat.com/2019/12/11/why-explainable-ai-is-ind.... What seems like a likely explanation is that the proper balance between machine and human judgement was not calibrated to this current market. "Part of the challenge of Zillow Offers’ human-in-the-loop system — any such system, really — is finding the balance between humans and machines. “In order to optimize this human-in-the-loop system, we’d like to figure out when the human is best, when an assistive situation is best, and […] when a machine is best,” said Fagnan."

say_it_as_it_is 5y ago

There isn't a company or individual who has models that can accurately predict the future of housing prices. There are only charlatans who claim they can achieve the impossible. This guy can do no better, but he'll do his best to convince capitalists that he's got the right stuff.

gradient_desc 5y ago

I’m surprised I haven’t seen this rumor yet https://twitter.com/sh_reya/status/1457094567111528455?s=21 https://twitter.com/sh_reya/status/1457094567111528455?s=21

gradient_desc 5y ago

Blind is suggesting another reason this went wrong - misaligned incentives and overriding the predicted prices https://twitter.com/sh_reya/status/1457094567111528455?s=21 https://twitter.com/sh_reya/status/1457094567111528455?s=21

a_square_peg 5y ago

I use Prophet quite a bit because it allows me to add exogenous variables (temperature, wind, solar etc.) easily and assess their impact with reasonable outcome for electricity load analysis. One thing that baffles me is when people try to use it to predict weather itself.