4 ms·

Effectively, every "knob" your model has is a major point against the model, but because people don't really get that (it's another "Linda is a bank teller and

by cwillu 6y ago

Effectively, every "knob" your model has is a major point against the model, but because people don't really get that (it's another "Linda is a bank teller and is active in the feminist movement" instance) they tend to overcomplicate their models compared to the amount of validation they've performed on the model. An alternative to putting nearly all the weight on a single complicated model created ex nihilo is to have some reasonable "zero information" prior distribution over models (weighted by complexity) where, because rules-of-thumb are very simple models, those simple models dominate. In fact, the alternative offered is itself a simple rule-of-thumb approach to this, basically saying that the more complicated models can be almost entirely neglected, and so you can really just do an even split over rules-of-thumb and still outperform any model that naïvely tries to “know best” by accounting for everything up front. In short, it's really fucking hard to properly validate that a model applies to reality, and until you appreciate just how hard it is, you're much better off radically biased towards simplicity.

phkahler 6y ago
Optimized systems are fragile. Probably true of models. The most profound class I had in college was thermodynamics, where all the complexity of physical systems is lumped into a few parameters. It seems crude but those parameters include all the energy and can be used to make very useful calculations.
- cwillu 6y ago
  Is this a fair restatement: “Models (and systems) that accurately reflect reality with a high fidelity due in part to their accurate assignment of values to a larger number of parameters are fragile”? I agree that that's true, but my point (and I think part of what the article is saying) is that most “optimized” systems/models are not in fact doing that. In which case, the point I'd take (if I read you correctly) is that even once you _have_ managed to accurately set the parameters (which is harder to do and validate than it looks), you still have a fragile model/system. Alternatively [edited to add]: Define a model as a system that is attempting to match another system in behaviour. And so now you have two problems: ① a complicated model is itself fragile (by analogy to the system it's modelling) ② a complicated system has more ways to diverge from the system it's trying to model Hmm, in which case it's kinda the same thing after all, except one is about making the model in the first place, and the other is maintaining it, maybe?
- cwillu 6y ago
  Interesting point re: thermodynamics though. It occurs to me that there's a sort of “learned helplessness” trap you can fall into from the thermodynamic base case: you can fail to separate _anything_ from placebo if your model fails to include enough of the real levers in the modelled system vs the levers it has that don't exist in that system. You can turn any signal into noise if you slice it up too much; that's basically half of the replication crisis. Studies on anti-depressants are an interesting example, for instance.
renewiltord 6y ago
It’s helpful to add a regularization term to your model so that as you’re building it, you penalize high parameter models against low parameter models. It’s commonly done in ML contexts but the idea is sound. A similar idea in genetic algorithms is to penalize the length of the “genome” being optimized. All sort of related to the bias variance trade off, overfitting, and the fact that with sufficient order of polynomial you can “perfectly” fit any number of points to create the worst approximator.
- montecarl 6y ago
  In case anyone is curious, the most aggressive form of regularization is sometimes called L0 regularization or sparse regression. Here, there is a penalty term for every additional parameter used in the model. This allows you to define a cost associated with model complexity (i.e. if an improvement to the fit is not greater than some value, the number of parameters is not increased). Solving for the best set of parameters can be tricky and thus LASSO (or L1 regularization) is often a good substitute.
bopbeepboop 6y ago
“In the ballpark” It’s not really new, to say that you should use simple heuristics to bound fancy math. Physics does it all the time — eg, unless you’re standing next to a black hole, your Newtonian and relativistic calculations should nearly agree. The problems with simple metrics arise when you have complex distributions that are poorly represented by mean, eg the shell of a sphere. If you design a product to appeal “center of sphere”, then it’ll be mediocre to everyone and you end up being out competed on every front by people who design slightly more “biased” products. I spend a lot of my time explaining that to executives.
smabie 6y ago
This applies to trading perfectly as well. It's rare that the true math studs are the rainmakers (though it does happen), and instead the street smart players who just try to reasonable end up dominating. By "street smart", I don't mean people who don't do math though, I'm just talking about people who aren't math olympiads or don't have PhDs in theoretical mathematical fields. 80% models take 20% of the work and are usually more robust during market dislocations and highly resistant to noise and uncertainty compared to very mathematical approaches. Really good point about verifying a model as well. Once your model is complex enough, it becomes highly probably that much of the model is actually hurting you, but you will have an incredibly hard time determining which part is hurting you. With a simple model, things become a lot easier and as such, the chance of success becomes much higher. Many people have this misconception that sophisticated players like Citadel or Jane Street are using these complex non-linear models to generate profit. This couldn't be further from the truth: linear models dominate the space at the moment and are the most valuable tool you have. Of course, linear regression isn't going to do a good job at identifying cat pictures or translating text, but that's because those tasks have relatively high signal to noise ratios. The cleaner the data, the less random the data, the better complex models perform. On the other hand with any kind of financial data, the signal is incredibly weak and a complex non-linear model will most likely get confused by the random patterns in the data.

by cwillu 6y ago

phkahler 6y ago

Optimized systems are fragile. Probably true of models. The most profound class I had in college was thermodynamics, where all the complexity of physical systems is lumped into a few parameters. It seems crude but those parameters include all the energy and can be used to make very useful calculations.

cwillu 6y ago
Is this a fair restatement: “Models (and systems) that accurately reflect reality with a high fidelity due in part to their accurate assignment of values to a larger number of parameters are fragile”? I agree that that's true, but my point (and I think part of what the article is saying) is that most “optimized” systems/models are not in fact doing that. In which case, the point I'd take (if I read you correctly) is that even once you _have_ managed to accurately set the parameters (which is harder to do and validate than it looks), you still have a fragile model/system. Alternatively [edited to add]: Define a model as a system that is attempting to match another system in behaviour. And so now you have two problems: ① a complicated model is itself fragile (by analogy to the system it's modelling) ② a complicated system has more ways to diverge from the system it's trying to model Hmm, in which case it's kinda the same thing after all, except one is about making the model in the first place, and the other is maintaining it, maybe?
cwillu 6y ago
Interesting point re: thermodynamics though. It occurs to me that there's a sort of “learned helplessness” trap you can fall into from the thermodynamic base case: you can fail to separate _anything_ from placebo if your model fails to include enough of the real levers in the modelled system vs the levers it has that don't exist in that system. You can turn any signal into noise if you slice it up too much; that's basically half of the replication crisis. Studies on anti-depressants are an interesting example, for instance.

renewiltord 6y ago

It’s helpful to add a regularization term to your model so that as you’re building it, you penalize high parameter models against low parameter models. It’s commonly done in ML contexts but the idea is sound. A similar idea in genetic algorithms is to penalize the length of the “genome” being optimized. All sort of related to the bias variance trade off, overfitting, and the fact that with sufficient order of polynomial you can “perfectly” fit any number of points to create the worst approximator.

montecarl 6y ago
In case anyone is curious, the most aggressive form of regularization is sometimes called L0 regularization or sparse regression. Here, there is a penalty term for every additional parameter used in the model. This allows you to define a cost associated with model complexity (i.e. if an improvement to the fit is not greater than some value, the number of parameters is not increased). Solving for the best set of parameters can be tricky and thus LASSO (or L1 regularization) is often a good substitute.

bopbeepboop 6y ago

“In the ballpark” It’s not really new, to say that you should use simple heuristics to bound fancy math. Physics does it all the time — eg, unless you’re standing next to a black hole, your Newtonian and relativistic calculations should nearly agree. The problems with simple metrics arise when you have complex distributions that are poorly represented by mean, eg the shell of a sphere. If you design a product to appeal “center of sphere”, then it’ll be mediocre to everyone and you end up being out competed on every front by people who design slightly more “biased” products. I spend a lot of my time explaining that to executives.

smabie 6y ago

This applies to trading perfectly as well. It's rare that the true math studs are the rainmakers (though it does happen), and instead the street smart players who just try to reasonable end up dominating. By "street smart", I don't mean people who don't do math though, I'm just talking about people who aren't math olympiads or don't have PhDs in theoretical mathematical fields. 80% models take 20% of the work and are usually more robust during market dislocations and highly resistant to noise and uncertainty compared to very mathematical approaches. Really good point about verifying a model as well. Once your model is complex enough, it becomes highly probably that much of the model is actually hurting you, but you will have an incredibly hard time determining which part is hurting you. With a simple model, things become a lot easier and as such, the chance of success becomes much higher. Many people have this misconception that sophisticated players like Citadel or Jane Street are using these complex non-linear models to generate profit. This couldn't be further from the truth: linear models dominate the space at the moment and are the most valuable tool you have. Of course, linear regression isn't going to do a good job at identifying cat pictures or translating text, but that's because those tasks have relatively high signal to noise ratios. The cleaner the data, the less random the data, the better complex models perform. On the other hand with any kind of financial data, the signal is incredibly weak and a complex non-linear model will most likely get confused by the random patterns in the data.