4 ms·

Every time you want to evaluate results from social research, you need to ask yourself two questions: 1. Are the results internally valid? That is, did the aut

Every time you want to evaluate results from social research, you need to ask yourself two questions: 1. Are the results internally valid? That is, did the authors reasonably control for factors other than the treatment (in this case, UBI) that could affect the observed results? Turn out, that this is actually difficult to do outside of randomized control trials. Quasi-experiments and natural experiments all lack random treatment assignment. This leads to biased estimates and wrong conclusions. 2. Are the results externally valid? That is, to whom do the results apply? Most UBI experiments are done in contexts that are different than the US (if you are interested in what would happened if UBI becomes law in the US). Local factors could impact the way that people react to UBI. Because these factors do not apply to the US, the study results do not readily translate to the US. For example, the article cites this NBER working paper[1] as proof that UBI has positive impacts on unemployment. The article uses a difference-in-differences model with a systemic control group to get to a causal estimate of receiving dividends from the Alaska Permanent Fund. This boils down to compare changes in employment between Alaska and states that had similar employment trends during the pre-treatment period. Their analysis seem to be sound and they conduct multiple robustness checks. As far as external validity, how similar is Alaska to the rest of the US? Would results from that setting translate to California, or Mississippi, or New York? [1]: http://www.nber.org/papers/w24312.pdf http://www.nber.org/papers/w24312.pdf

aaron-lebo 8y ago
Yes, this is unfortunately pretty lazy research used to push an agenda. Related to that, the main claim: the 40% reduction in crime (no link, why?) is in Namibia. Is there anything about Namibia which makes it different from a developed economy like the US? I know it comes off as pedantic to complain about such stuff, but there's gotta be higher standards for evidence, otherwise you don't really trust your own argument.
- xkjkls 8y ago
  I wouldn't necessarily call it "lazy research". It's imperfect, but there aren't exactly that many natural experiments in UBI that provide you great data. Sometimes you have to piece together insights from imperfect data.
  - aaron-lebo 8y ago
    It's not the research itself that is lazy, it's the way results with very limited scope are used by people to push an agenda. I guarantee you the authors are aware of the limits of their research. They aren't the ones making overarching claims. I challenge you to look up how social scientists themselves (and not one stop issue sites) feel about UBI. You can't bother to provide a link when you are citing a statistic? That's just lazy. In graduate school they train you like an attack dog to find this stuff, so it's pedantic, but that really doesn't bother anyone? Just a link supporting your main claim - too much to ask? The association with academia and "stats" without the same standards is weird.
- pacbard 8y ago
  The reference chain for the Namibia experiment is an Huffington post article[1] and a report by Bignam[2]. It looks like "Otjivero-Omitara was selected for its manageable size, accessibility, and poverty situation" (p.19). Ask yourself, could any of those characteristics impact the reduction in crime regardless of UBI? Also, I wasn't able to find a control (or comparison) group for those results. The time series that they provide seem to indicate that there are positive economic trends (see the reduction in poverty reported at page 49) in Namibia during the study's time frame (2007-2009). If these trends are not accounted for (e.g., using a difference-in-differences approach), you risk to conflate outside factors with UBI. Another issue is attrition in the study. Their 40% result only takes in consideration the people who were present in the village for the 2-year study. Now, we could imagine that unemployed people might have migrated out of the village, mechanically pushing crime down thus biasing the UBI results. (see page 56 and 71) The moral of the story is that social research is hard, especially research that wants to be causal. I would be skeptic of all click-baity results unless backed up by strong evidence. [1]: https://www.huffingtonpost.com/scott-santens/universal-basic-income-wi_b_8354072.html https://www.huffingtonpost.com/scott-santens/universal-basic... [2]: http://www.bignam.org/Publications/BIG_Assessment_report_08b.pdf http://www.bignam.org/Publications/BIG_Assessment_report_08b...
  - aaron-lebo 8y ago
    Thanks, I did come across the Huffington Post link but the title itself wasn't obviously about Namibia so I missed it. I appreciate you breaking this down. I've published nothing but I've worked with those who have and I've seen just how difficult this kind of research truly is. There's so many confounding factors, but they try and usually do a good job. It frustrates me when these very complex issue are intentionally distorted. Reality and policy don't give a damn about anyone's beliefs, so it's important that we understand the holes in studies like this. Thanks again.
jaggederest 8y ago
From an armchair perspective, there are significant ways in which Alaska is behind the mainland US, so it's encouraging to see that the APF distributions provide a positive impact. I would suspect that the results would be broadly applicable to underdeveloped and rural areas, so of the three examples, there are certainly areas in rural north California, rural Mississippi, and New York (probably upstate) that would benefit. I think extending it to apply to urban or highly developed areas would be pretty questionable, though, I'd guess Manhattan is not going to behave similarly enough to draw any conclusions.
MAXPOOL 8y ago
I'm not an economist, but when I once asked examples of good econometric methods to study especially related to synthetic control. I was given just that Marinescu's Alaska paper you are referring to. They needed a counterfactual for exactly one state. They create "synthetic Alaska" elsewhere in in the United states to make the comparison valid. Even if the researchers did not intent it that way, the selection methodology selected areas in the continental US that have geographic featurs similar to Alaska. It's really fine paper and good methodology. Another fine point about basic income in the paper is that it makes difference between tradeable businesses (can be are exported) and non-tradable businesses (stays local). Employment rates in non-tradeable business don't change. By contrast part-time work did increase, and employment fell in companies that sold goods outside Alaska (this has implications for wider applicability I guess).
- pacbard 8y ago
  I think that the original synthetic control paper is this paper by Abadie and colleagues[1]. Figure 1 and 2 give you the intuition behind the need of using the a synthetic control rather than compare California to the rest of the US. You can notice figure 1 that California’s sales of tobacco products was decreasing before the prop 99 and that this trend is different than the rest of the US. This violates the required parallel trends assumption that is required to have an unbiased estimate in a DD model. Abadie et al. reweight the results from the other states to create a synthetic comparison group. This gives them a synthetic California sample that is similar to pre-prop 99 California. You can see that in figure 2. They claim that any deviation from this synthetic group is due to the effects of prop 99 on tobacco sales (Do you believe them? This paper’s conclusion rests on your answer to this question.). Bonus: Figure 3 shows the normalized results. I always appreciate Econ papers that put their results in a graph. Tables are hard to read sometimes. [1]: https://economics.mit.edu/files/11859 https://economics.mit.edu/files/11859
xienze 8y ago
To your points, I'm sure communism comes out looking quite incredible in a limited-time trial with a small population. It's just that dang problem with scaling it up.
131012 8y ago
About point 1: Since we are dealing here with humans, this criteria is almost impossible to satisfy. You cannot enroll people in a control trial if you know this has a chance to make their lives worst. > This leads to biased estimates and wrong conclusions. It does not necessarily leads to 'wrong' conclusions. In social sciences, you have to deal with a truth level that will never yield 100% certainty.
- aaron-lebo 8y ago
  The real issue is that the "true" confidence levels in the social sciences are nowhere near that. We can hack together models and fit them to make an argument look compelling, but most of the stuff we "know" to be objectively true is descriptive, not prescriptive. I think any good social scientist regarding UBI will tell you that we honestly don't have any fucking clue what would happen because the differences between Namibia, Finland, etc are so vast and we have so little data. These studies are important parts of building up that dataset, but to make policy considerations (or even something like an assumption in social science) is much, much more likely to provide a "wrong" or "bad" choice, than after we've got better data. If you really were to dig into any given study, you can in many cases find a lot of assumptions which we have to accept to even attempt a study. We all tend to be pretty bad at measuring truth levels, that's why the criticism is necessary.

aaron-lebo 8y ago

Yes, this is unfortunately pretty lazy research used to push an agenda. Related to that, the main claim: the 40% reduction in crime (no link, why?) is in Namibia. Is there anything about Namibia which makes it different from a developed economy like the US? I know it comes off as pedantic to complain about such stuff, but there's gotta be higher standards for evidence, otherwise you don't really trust your own argument.

xkjkls 8y ago
I wouldn't necessarily call it "lazy research". It's imperfect, but there aren't exactly that many natural experiments in UBI that provide you great data. Sometimes you have to piece together insights from imperfect data.
- aaron-lebo 8y ago
  It's not the research itself that is lazy, it's the way results with very limited scope are used by people to push an agenda. I guarantee you the authors are aware of the limits of their research. They aren't the ones making overarching claims. I challenge you to look up how social scientists themselves (and not one stop issue sites) feel about UBI. You can't bother to provide a link when you are citing a statistic? That's just lazy. In graduate school they train you like an attack dog to find this stuff, so it's pedantic, but that really doesn't bother anyone? Just a link supporting your main claim - too much to ask? The association with academia and "stats" without the same standards is weird.
pacbard 8y ago
The reference chain for the Namibia experiment is an Huffington post article[1] and a report by Bignam[2]. It looks like "Otjivero-Omitara was selected for its manageable size, accessibility, and poverty situation" (p.19). Ask yourself, could any of those characteristics impact the reduction in crime regardless of UBI? Also, I wasn't able to find a control (or comparison) group for those results. The time series that they provide seem to indicate that there are positive economic trends (see the reduction in poverty reported at page 49) in Namibia during the study's time frame (2007-2009). If these trends are not accounted for (e.g., using a difference-in-differences approach), you risk to conflate outside factors with UBI. Another issue is attrition in the study. Their 40% result only takes in consideration the people who were present in the village for the 2-year study. Now, we could imagine that unemployed people might have migrated out of the village, mechanically pushing crime down thus biasing the UBI results. (see page 56 and 71) The moral of the story is that social research is hard, especially research that wants to be causal. I would be skeptic of all click-baity results unless backed up by strong evidence. [1]: https://www.huffingtonpost.com/scott-santens/universal-basic-income-wi_b_8354072.html https://www.huffingtonpost.com/scott-santens/universal-basic... [2]: http://www.bignam.org/Publications/BIG_Assessment_report_08b.pdf http://www.bignam.org/Publications/BIG_Assessment_report_08b...
- aaron-lebo 8y ago
  Thanks, I did come across the Huffington Post link but the title itself wasn't obviously about Namibia so I missed it. I appreciate you breaking this down. I've published nothing but I've worked with those who have and I've seen just how difficult this kind of research truly is. There's so many confounding factors, but they try and usually do a good job. It frustrates me when these very complex issue are intentionally distorted. Reality and policy don't give a damn about anyone's beliefs, so it's important that we understand the holes in studies like this. Thanks again.

jaggederest 8y ago

From an armchair perspective, there are significant ways in which Alaska is behind the mainland US, so it's encouraging to see that the APF distributions provide a positive impact. I would suspect that the results would be broadly applicable to underdeveloped and rural areas, so of the three examples, there are certainly areas in rural north California, rural Mississippi, and New York (probably upstate) that would benefit. I think extending it to apply to urban or highly developed areas would be pretty questionable, though, I'd guess Manhattan is not going to behave similarly enough to draw any conclusions.

MAXPOOL 8y ago

I'm not an economist, but when I once asked examples of good econometric methods to study especially related to synthetic control. I was given just that Marinescu's Alaska paper you are referring to. They needed a counterfactual for exactly one state. They create "synthetic Alaska" elsewhere in in the United states to make the comparison valid. Even if the researchers did not intent it that way, the selection methodology selected areas in the continental US that have geographic featurs similar to Alaska. It's really fine paper and good methodology. Another fine point about basic income in the paper is that it makes difference between tradeable businesses (can be are exported) and non-tradable businesses (stays local). Employment rates in non-tradeable business don't change. By contrast part-time work did increase, and employment fell in companies that sold goods outside Alaska (this has implications for wider applicability I guess).

pacbard 8y ago
I think that the original synthetic control paper is this paper by Abadie and colleagues[1]. Figure 1 and 2 give you the intuition behind the need of using the a synthetic control rather than compare California to the rest of the US. You can notice figure 1 that California’s sales of tobacco products was decreasing before the prop 99 and that this trend is different than the rest of the US. This violates the required parallel trends assumption that is required to have an unbiased estimate in a DD model. Abadie et al. reweight the results from the other states to create a synthetic comparison group. This gives them a synthetic California sample that is similar to pre-prop 99 California. You can see that in figure 2. They claim that any deviation from this synthetic group is due to the effects of prop 99 on tobacco sales (Do you believe them? This paper’s conclusion rests on your answer to this question.). Bonus: Figure 3 shows the normalized results. I always appreciate Econ papers that put their results in a graph. Tables are hard to read sometimes. [1]: https://economics.mit.edu/files/11859 https://economics.mit.edu/files/11859

xienze 8y ago

To your points, I'm sure communism comes out looking quite incredible in a limited-time trial with a small population. It's just that dang problem with scaling it up.

131012 8y ago

About point 1: Since we are dealing here with humans, this criteria is almost impossible to satisfy. You cannot enroll people in a control trial if you know this has a chance to make their lives worst. > This leads to biased estimates and wrong conclusions. It does not necessarily leads to 'wrong' conclusions. In social sciences, you have to deal with a truth level that will never yield 100% certainty.

aaron-lebo 8y ago
The real issue is that the "true" confidence levels in the social sciences are nowhere near that. We can hack together models and fit them to make an argument look compelling, but most of the stuff we "know" to be objectively true is descriptive, not prescriptive. I think any good social scientist regarding UBI will tell you that we honestly don't have any fucking clue what would happen because the differences between Namibia, Finland, etc are so vast and we have so little data. These studies are important parts of building up that dataset, but to make policy considerations (or even something like an assumption in social science) is much, much more likely to provide a "wrong" or "bad" choice, than after we've got better data. If you really were to dig into any given study, you can in many cases find a lot of assumptions which we have to accept to even attempt a study. We all tend to be pretty bad at measuring truth levels, that's why the criticism is necessary.