6 ms·

The difference big data/data science/empirical study and 'classical' economics (by which I'd include any system of economics that seeks to explain human behavio

The difference big data/data science/empirical study and 'classical' economics (by which I'd include any system of economics that seeks to explain human behavior via an underlying metatheory) is that a primarily empirical approach obscures the necessary underlying theory present in any experiment where you're trying to fit data to a curve. For example, when you run a science experiment and you plot the data, you may find that you're looking at a line. While this is an interesting finding, it has zero predictive value for anything other than the exact situation you've collected data for. In order to formulate scientific law, you first must (a) believe that such a thing exists and (b) have some theory as to what shape the curve ought to fit. For example, a naive look at physics using an 'empirical' approach might incorrectly conclude that force is mass times acceleration. While moderately useful for many problems, this offers little predictive power in the general case. In order to actually formulate a law that can be of predictive value, you have to first consider various other laws and axioms (such as the constant speed of light for force), at which point -- by deduction, without any need of empirism -- you determine that this is wrong, and you need another kind of equation to fit your data to. I don't know if the simplistic demand curves drawn in the original text book are correct or not. However, at least those are based on a particular set of assumptions that can be validated or not. The kind of empiricism put forth by Mr Chetty does not offer this at all. All this is to say that, while data is useful for validation, it is not useful for prediction. The last thing we need is a black-box machine learning model to make major economic decisions off of. What we do need is proper models that are then validated, which don't necessarily need 'big data.'

westurner 7y ago
> All this is to say that, while data is useful for validation, it is not useful for prediction. The last thing we need is a black-box machine learning model to make major economic decisions off of. What we do need is proper models that are then validated, which don't necessarily need 'big data.' Hand-wavy theory - predicated upon physical-world models of equillibrium which are themselves classical and incomplete - without validation is preferable to empirical models? Please. Estimating the predictive power of some LaTeX equations is a different task than measuring error of a trained model. If the model does not fit all of the big data, the error term is higher; regardless of whether the model was pulled out of a hat in front of a captive audience or deduced though inference from actual data fed through an unbiased analysis pipeline. If the 'black-box predictive model' has lower error for all available data, the task is then to reverse the model! Not to argue for unvalidated theory. Here are a few discussions regarding validating economic models, some excellent open econometric lectures (as notebooks that are unfortunately not in an easily-testable programmatic form), the lack of responsible validation, and some tools and datasets that may be useful for validating hand-wavy classical economic theories: "When does the concept of equilibrium work in economics?" https://news.ycombinator.com/item?id=19214650 https://news.ycombinator.com/item?id=19214650 > "Lectures in Quantitative Economics as Python and Julia Notebooks" https://news.ycombinator.com/item?id=19083479 https://news.ycombinator.com/item?id=19083479 (data sources (pandas-datareader, pandaSDMX), tools, latex2sympy) That's just an equation in a PDF. (edit) Here's another useful thread: "Ask HN: Data analysis workflow?" https://news.ycombinator.com/item?id=18798244 https://news.ycombinator.com/item?id=18798244
- tathougies 7y ago
  > Hand-wavy theory - predicated upon physical-world models of equillibrium which are themselves classical and incomplete - without validation is preferable to empirical models? Please. My friend, you are strawmanning. I said, > What we do need is proper models that are then validated, which don't necessarily need 'big data.' Which agrees with you. I said we need both and, not one or the other. > If the model does not fit all of the big data, the error term is higher; regardless of whether the model was pulled out of a hat in front of a captive audience or deduced though inference from actual data fed through an unbiased analysis pipeline. Big data without a model is still only valid for the scenario the data were collected in > If the 'black-box predictive model' has lower error for all available data, the task is then to reverse the model! Not to argue for unvalidated theory. Certainly, but we should simultaneously recognize that any model so conceived is still only valid in the situations the data were collected in, which makes the not necessarily useful for the future. You could turn such an equation into an economic philosophy, but you'd have to do a lot more, non metric, work.
  - westurner 7y ago
    How can you possibly be arguing that we should not be testing models with all available data? All models are limited by the data they're trained from; regardless of whether they are derived through rigorous, standardized, unbiased analysis or though laudable divine inspiration. From https://news.ycombinator.com/item?id=19084622 https://news.ycombinator.com/item?id=19084622 : > pandas-datareader can pull data from e.g. FRED, Eurostat, Quandl, World Bank: https://pandas-datareader.readthedocs.io/en/latest/remote_data.html https://pandas-datareader.readthedocs.io/en/latest/remote_da... > pandaSDMX can pull SDMX data from e.g. ECB, Eurostat, ILO, IMF, OECD, UNSD, UNESCO, World Bank; with requests-cache for caching data requests: https://pandasdmx.readthedocs.io/en/latest/#supported-data-providers https://pandasdmx.readthedocs.io/en/latest/#supported-data-p...
    - AnthonyMouse 7y ago
      > All models are limited by the data they're trained from; regardless of whether they are derived through rigorous, standardized, unbiased analysis or though laudable divine inspiration. Some of the data we have isn't training data. Purely data-driven models tend to be ensnared by Goodhart's law. For example, suppose we're issuing 30-year term loans and we have some data that shows that people with things like country club memberships and foie gras on their credit card statements have a higher tendency not to miss payments. So we use that information to make our determination. But people are aware we're doing this and the same data is externally available, so now people start to waste resources on extravagant luxuries in order to qualify for a loan or a low interest rate, and that only makes it more likely that they ultimately default. However, that consequence doesn't become part of the data set until years have passed and the defaults actually occur, and in the meantime we're using flawed reasoning to issue loans. When we finally figure that out after ten years, the new data we use then will have some fresh different criteria for people to game, because the data is always from the past rather than the future. We've already seen the kind of damage this can do. Politicians see data that college-educated people are better off so subsidize college loans, only to discover that the signal from having a degree that caused it to result in such gainful employment is diluted as it becomes more common, and subsidizing loans results in price inflation, and making a degree a prerequisite for jobs that shouldn't require it creates incentives for degree mills that pump out credentials but not real education. To get out of this we have to consider not only what people have done in the past but how they are likely to respond to a given policy change, for which we have no historical data prior to when the policy is enacted, and so we need to make those predictions based on logic in addition to data or we go astray.
treydey 7y ago
This is exactly what the Breiman paper ("Statistical Modeling: The Two Cultures") discusses. If you haven't read it already, I would say it's worth a read.
seirl 7y ago
> machine learning model to make major decisions off of I have basically no experience with ML, but from what I know I'm having difficulty understanding how it's different from OLS with constructed regressors. Can anyone explain?

westurner 7y ago

> All this is to say that, while data is useful for validation, it is not useful for prediction. The last thing we need is a black-box machine learning model to make major economic decisions off of. What we do need is proper models that are then validated, which don't necessarily need 'big data.' Hand-wavy theory - predicated upon physical-world models of equillibrium which are themselves classical and incomplete - without validation is preferable to empirical models? Please. Estimating the predictive power of some LaTeX equations is a different task than measuring error of a trained model. If the model does not fit all of the big data, the error term is higher; regardless of whether the model was pulled out of a hat in front of a captive audience or deduced though inference from actual data fed through an unbiased analysis pipeline. If the 'black-box predictive model' has lower error for all available data, the task is then to reverse the model! Not to argue for unvalidated theory. Here are a few discussions regarding validating economic models, some excellent open econometric lectures (as notebooks that are unfortunately not in an easily-testable programmatic form), the lack of responsible validation, and some tools and datasets that may be useful for validating hand-wavy classical economic theories: "When does the concept of equilibrium work in economics?" https://news.ycombinator.com/item?id=19214650 https://news.ycombinator.com/item?id=19214650 > "Lectures in Quantitative Economics as Python and Julia Notebooks" https://news.ycombinator.com/item?id=19083479 https://news.ycombinator.com/item?id=19083479 (data sources (pandas-datareader, pandaSDMX), tools, latex2sympy) That's just an equation in a PDF. (edit) Here's another useful thread: "Ask HN: Data analysis workflow?" https://news.ycombinator.com/item?id=18798244 https://news.ycombinator.com/item?id=18798244

tathougies 7y ago
> Hand-wavy theory - predicated upon physical-world models of equillibrium which are themselves classical and incomplete - without validation is preferable to empirical models? Please. My friend, you are strawmanning. I said, > What we do need is proper models that are then validated, which don't necessarily need 'big data.' Which agrees with you. I said we need both and, not one or the other. > If the model does not fit all of the big data, the error term is higher; regardless of whether the model was pulled out of a hat in front of a captive audience or deduced though inference from actual data fed through an unbiased analysis pipeline. Big data without a model is still only valid for the scenario the data were collected in > If the 'black-box predictive model' has lower error for all available data, the task is then to reverse the model! Not to argue for unvalidated theory. Certainly, but we should simultaneously recognize that any model so conceived is still only valid in the situations the data were collected in, which makes the not necessarily useful for the future. You could turn such an equation into an economic philosophy, but you'd have to do a lot more, non metric, work.
- westurner 7y ago
  How can you possibly be arguing that we should not be testing models with all available data? All models are limited by the data they're trained from; regardless of whether they are derived through rigorous, standardized, unbiased analysis or though laudable divine inspiration. From https://news.ycombinator.com/item?id=19084622 https://news.ycombinator.com/item?id=19084622 : > pandas-datareader can pull data from e.g. FRED, Eurostat, Quandl, World Bank: https://pandas-datareader.readthedocs.io/en/latest/remote_data.html https://pandas-datareader.readthedocs.io/en/latest/remote_da... > pandaSDMX can pull SDMX data from e.g. ECB, Eurostat, ILO, IMF, OECD, UNSD, UNESCO, World Bank; with requests-cache for caching data requests: https://pandasdmx.readthedocs.io/en/latest/#supported-data-providers https://pandasdmx.readthedocs.io/en/latest/#supported-data-p...
  - AnthonyMouse 7y ago
    > All models are limited by the data they're trained from; regardless of whether they are derived through rigorous, standardized, unbiased analysis or though laudable divine inspiration. Some of the data we have isn't training data. Purely data-driven models tend to be ensnared by Goodhart's law. For example, suppose we're issuing 30-year term loans and we have some data that shows that people with things like country club memberships and foie gras on their credit card statements have a higher tendency not to miss payments. So we use that information to make our determination. But people are aware we're doing this and the same data is externally available, so now people start to waste resources on extravagant luxuries in order to qualify for a loan or a low interest rate, and that only makes it more likely that they ultimately default. However, that consequence doesn't become part of the data set until years have passed and the defaults actually occur, and in the meantime we're using flawed reasoning to issue loans. When we finally figure that out after ten years, the new data we use then will have some fresh different criteria for people to game, because the data is always from the past rather than the future. We've already seen the kind of damage this can do. Politicians see data that college-educated people are better off so subsidize college loans, only to discover that the signal from having a degree that caused it to result in such gainful employment is diluted as it becomes more common, and subsidizing loans results in price inflation, and making a degree a prerequisite for jobs that shouldn't require it creates incentives for degree mills that pump out credentials but not real education. To get out of this we have to consider not only what people have done in the past but how they are likely to respond to a given policy change, for which we have no historical data prior to when the policy is enacted, and so we need to make those predictions based on logic in addition to data or we go astray.

treydey 7y ago

This is exactly what the Breiman paper ("Statistical Modeling: The Two Cultures") discusses. If you haven't read it already, I would say it's worth a read.

seirl 7y ago

> machine learning model to make major decisions off of I have basically no experience with ML, but from what I know I'm having difficulty understanding how it's different from OLS with constructed regressors. Can anyone explain?