5 ms·
> What almost certainly happened was they validated their model extensively, but overlooked the angle relevant to this use case. To me, this is a systemic orga
by ethbr0 5y ago
> What almost certainly happened was they validated their model extensively, but overlooked the angle relevant to this use case.
To me, this is a systemic organizational problem in ML, rather than a model problem. Or essentially, pre-Agile development all over again.
Stop me if this sequence sounds familiar: (1) attempt to capture all the details in a nuanced domain that's foreign to you, from people who are experts but not expert developers, (2) develop code from that spec, (3) deliver code to the experts as a black box that they can't opine on, (4) cross fingers and run in production.
Which isn't to say this is the way everyone does it, but is to say it will be the way the majority does it, just as the majority of development is still waterfall under the hood.
But the mismatch is a mismatch in understanding, not in correctness. The experts can't offer meaningful feedback to the technological black box, and developers don't have enough time to spec a lifetime of subject matter experience.
- ItsMonkk 5y agoRelevant xkcd: Here to Help[0]. If you don't have the minimum skills on the team, you're going to fail in ways that you don't see coming, the unknown unknowns. To spend billions when a former Zillow employee says that they have no finance people on the team, is just, drastically, wow. I would much sooner believe he is mistaken then believe that is what actually happened. [0]: https://xkcd.com/1831 https://xkcd.com/1831
- RandomLensman 5y agoNot sure how they could really validate their models given that "market making" in houses is probably really thin - if at all. Finding alpha in financial markets is just tough (real assets or not), and real estate also needs a lot of capital, which makes "waiting it out" tough
- JackFr 5y agoI can make a market by quoting a bid and ask. To run a market making business I have to know how to hedge my net position and manage my cost of carry. I find it crazy hard to believe that they just went long a few billion in houses just based on a model of price and failed to hedge vs. the secular market and didn’t model time to sale. But if you believe the popular press maybe that’s what they did.
- RandomLensman 5y agoAre there efficient hedges avaible on a few B of RE? I thought the derivatives space is pretty thin there these days, but not my area of expertise.
- pas 5y agooptions on anything that's correlated with it should do it. (basically if "RE is up" so should builders, suppliers, big corporate lenders, mortgage processors -- so if they want to hedge against RE going down they could have bought options that pay when those go down)
- JackFr 5y agoI don’t know of anything exchange traded or terribly liquid, but if I called Goldman and said I wanted a one or two billion dollar swap on the Case-Schiller Home Price index I’m sure they could work up a quote for me.
- evrydayhustling 5y agoYou are right about this anti-pattern, but it's not the way most ML models are operated - especially in trading operations. The following principles are well known: - enlist domain expertise - validate incrementally - expect adverse selection - expect failures to generalise Many, many production ML teams operate in this framework. The fact that Zillow either cut corners or hit a bigger snag suggests there is a more interesting story we haven't heard yet, not that they should have consulted OP or a few hn readers.
- ethbr0 5y agoTrading models are a very specific and exceptional (in multiple senses) subset of ML use. A given business probably doesn't employ more than a few people of that class and rigorousness: not least, because those people have the option of making a lot more money at a trading firm. So yes, all businesses should do all those things. And yes, the best run places do. But it feels like there's going to be a decade or two where employee supply & expertise lags corporate demand, with bad results.