5 ms·

Using predictive models for policy is not new, in fact it was the standard approach long before more inferential models, and the famed Lucas critique precisely

by zwaps 7y ago

Using predictive models for policy is not new, in fact it was the standard approach long before more inferential models, and the famed Lucas critique precisely targets a primitive approach similar to what you are proposing. The issue is the following: In economics, one is interested in an underlying parameter of a complex equilibrium system (or, if you wish, a non-equilibrium complex system of multi-agentic behavior). This may be, for example, some pricing parameter for a given firm - say - how your sold units react to setting a price. Economics faces two basic issues: First, any predictive model (like a NN or simple regression) that takes price as an input, will not correctly estimate the sensitivity of revenue to price. It is actually usually the case, that the inference is reversed. A model where price is input, and sold units or revenue is output (or vice-versa) will predict (you can check that using pretty much any dataset of prices and outputs) that higher prices lead to higher outputs, because that is the association in the data. Of course we know that in truth, prices and outputs are co-determined. They are simultaneous phenomena, and regressing one on the other is not sufficient to "causally identify" the correct effect. This is independent of how sophisticated your model is otherwise. Fitting a better non-linear representation does not help. The solution is of course to reduce down these "endogenous" phenomena to their basic ingredients. Say you have cost data, and some demand parameters. Then, using a regression model (or NN) to predict the vector of endogenous outcome variables will work, and roughly give you the right inference. Then, as a firm, you are able to use these (more) exogenous predictive variables to find your correct pricing. This is not new, pops up everywhere in social science, is the basis of a gigantic literature called econometrics, and really has nothing to do with how you do the prediction. The only thing that NN add are better predictions (better fitting) and the ability to deal with more data. As this inferential problem shows, using more (and more fine-grained) data is indeed crucial to predicting what a firm should do. BUT, it is crucial to understand and reason about the underlying causality FIRST, because otherwise even the most sophisticated statistical approach will simply give you wrong results. Secondly, the counterfactual data for economic issues is usually very scarce. The approach taken by machine learning is problematic, not only because of potentially wrong inference, but also because two points in time may simply not be based on comparable data-generating processes. In fact, this is exactly the blindness that led to people missing the financial crisis. Of course, with enough data, and long enough samples, one should expect to be become pretty good at predicting economic outcomes. But experience has shown that in economics, these data are simply too scarce. The unobserved variation between two quarters, two years, two countries, two firms (etc.) is simply very large and has fat tails. This leads to spontaneous breakdowns of such predicitive models. Taking these two issues together, we see that better non-linear function approximation is not the solution to our problems. Instead, it is a methodological improvement that must be used in conjunction with what we have learned about causality. Indeed the literature moves into a different direction. Good economic science nowadays means to identify effects via natural experiments and other exogenous shifts that can plausibly show causality. Of course such experiments are more rare, and more difficult, the larger the scale becomes. Which is why Macroeconomics is arguably the "worst science" in economics, while things like auctions and microstructure of markets are actually surprisingly good science (nowadays). Doors are wide open for ML techniques, but really only to the point that they are useful in operationalizing more and better data. Anyone trying to understand economic phenomena needs to be keenly aware of how inference can be done, which requires an understanding (or an approach to) - that is, a theory - of the underlying mechanisms.

zwaps 7y ago
Just to reiterate the practical issue here: We, as people, are just exceedingly bad in having AND putting the right data at the right place in any model. It's really not the models fault, and the contribution of ML is marginal in that regard. Whether it is subsidies of farmers, education, tax reduction, minimum wage, austerity measures... history is full of deliciously wrong predictions and policy measures. Almost all of them can be reduced to the simple fact that the DPG is not stable when varying the policy, and that simple fact is due to people being deliberatively reactive. In other words, you are missing data. Data about human behavior that is simply not observed, because it didn't happen, or because it happens inside people! And then, no matter how well you fit your conditional expectation (or other moment, or whatever you fit), the errors are simply not predictable. We miss the counterfactual data, AND we aren't even smart enough to use all the data that we have. The less we theorize, the less we use prior logic, the more we run into these paradoxes where our policy does the exact opposite of what we intended. This is pretty much the only real constant you can find in the last 100 years of social science. It is therefore entirely correct that social science focuses more and more on causality - and where it can be identified. Yes, it is much harder, and the opportunities to do it correctly are scarce, but necessary. In this, trusting in more data and AI is precisely the wrong approach.
westurner 7y ago
Yes, some combination of variables/features grouped and connected with operators that correlate to an optima (some of which are parameters we can specify) that occurs immediately or after a period of lag during which other variables of the given complex system are dangerously assumed to remain constant. > In fact, this is exactly the blindness that led to people missing the financial crisis ML was not necessary to recognize the yield curve inversion as a strongly predictive signal correlating to subsequent contraction. An NN can certainly learn to predict according to the presence or magnitude of a yield curve inversion and which combinations of other features. - [ ] Exercise: Learning this and other predictive signals by cherry-picking data and hand-optimizing features may be an extremely appropriate exercise. "This field is different because it's nonlinear, very complex, there are unquantified and/or uncollected human factors, and temporal" Maybe we're not in agreement about whether AI and ML can do causal inference just as well if not better than humans manipulating symbols with human cognition and physical world intuition. The time is nigh! In general, while skepticism and caution are appropriate, many fields suffer from a degree of hubris which prevents them from truly embracing stronger AI in their problem domain. (A human person cannot mutate symbol trees and validate with shuffled and split test data all night long) > Anyone trying to understand economic phenomena needs to be keenly aware of how inference can be done, which requires an understanding (or an approach to) - that is, a theory - of the underlying mechanisms. I read this as "must be biased by the literature and willing to disregard an unacceptable error term"; but also caution against rationalizing blind findings which can easily be rationalized as logical due to any number of cognitive biases. Compared to AI, we're not too rigorous about inductive or deductive inference; we simply store generalizations about human behavior and predict according to syntheses of activations in our human NNs. If you're suggesting that the information theory that underlies AI and ML is insufficient to learn what we humans have learned in a few hundred years of observing and attempting to optimize, I must disagree (regardless of the hardness or softness of the given complex field). Beyond a few combinations/scenarios, our puny little brains are no match for our department's new willing AI scientist.

by zwaps 7y ago

zwaps 7y ago

Just to reiterate the practical issue here: We, as people, are just exceedingly bad in having AND putting the right data at the right place in any model. It's really not the models fault, and the contribution of ML is marginal in that regard. Whether it is subsidies of farmers, education, tax reduction, minimum wage, austerity measures... history is full of deliciously wrong predictions and policy measures. Almost all of them can be reduced to the simple fact that the DPG is not stable when varying the policy, and that simple fact is due to people being deliberatively reactive. In other words, you are missing data. Data about human behavior that is simply not observed, because it didn't happen, or because it happens inside people! And then, no matter how well you fit your conditional expectation (or other moment, or whatever you fit), the errors are simply not predictable. We miss the counterfactual data, AND we aren't even smart enough to use all the data that we have. The less we theorize, the less we use prior logic, the more we run into these paradoxes where our policy does the exact opposite of what we intended. This is pretty much the only real constant you can find in the last 100 years of social science. It is therefore entirely correct that social science focuses more and more on causality - and where it can be identified. Yes, it is much harder, and the opportunities to do it correctly are scarce, but necessary. In this, trusting in more data and AI is precisely the wrong approach.

westurner 7y ago

Yes, some combination of variables/features grouped and connected with operators that correlate to an optima (some of which are parameters we can specify) that occurs immediately or after a period of lag during which other variables of the given complex system are dangerously assumed to remain constant. > In fact, this is exactly the blindness that led to people missing the financial crisis ML was not necessary to recognize the yield curve inversion as a strongly predictive signal correlating to subsequent contraction. An NN can certainly learn to predict according to the presence or magnitude of a yield curve inversion and which combinations of other features. - [ ] Exercise: Learning this and other predictive signals by cherry-picking data and hand-optimizing features may be an extremely appropriate exercise. "This field is different because it's nonlinear, very complex, there are unquantified and/or uncollected human factors, and temporal" Maybe we're not in agreement about whether AI and ML can do causal inference just as well if not better than humans manipulating symbols with human cognition and physical world intuition. The time is nigh! In general, while skepticism and caution are appropriate, many fields suffer from a degree of hubris which prevents them from truly embracing stronger AI in their problem domain. (A human person cannot mutate symbol trees and validate with shuffled and split test data all night long) > Anyone trying to understand economic phenomena needs to be keenly aware of how inference can be done, which requires an understanding (or an approach to) - that is, a theory - of the underlying mechanisms. I read this as "must be biased by the literature and willing to disregard an unacceptable error term"; but also caution against rationalizing blind findings which can easily be rationalized as logical due to any number of cognitive biases. Compared to AI, we're not too rigorous about inductive or deductive inference; we simply store generalizations about human behavior and predict according to syntheses of activations in our human NNs. If you're suggesting that the information theory that underlies AI and ML is insufficient to learn what we humans have learned in a few hundred years of observing and attempting to optimize, I must disagree (regardless of the hardness or softness of the given complex field). Beyond a few combinations/scenarios, our puny little brains are no match for our department's new willing AI scientist.