4 ms·

I'm one of the creators of the quiz! Note that the projects were not chosen based on their results, and certainly not because the results are counterintuitive.

I'm one of the creators of the quiz! Note that the projects were not chosen based on their results, and certainly not because the results are counterintuitive. Rather they are the ten best understood social programs we could find. You can read about the findings of our research into whether people can guess what works and what doesn't ahead of time: http://www.vox.com/2015/8/13/9148123/quiz-which-programs-work http://www.vox.com/2015/8/13/9148123/quiz-which-programs-wor...

NateLawson 11y ago
Hey, this is really neat. It reminds me of the lesswrong.com community, where the goal is to figure out ways of offsetting cognitive biases. One thing I've always wanted to see is a mandatory A/B test built into any new legislation. That is, funding for a study after X years of the new law or program being in place would have to be part of the law. It could even including a threshold of the desired controlled effect of the law or else it is immediately revoked at that point. This way we'd force people to argue not about 2nd order issues ("I think mandatory sentencing is good" vs. "No, I think it's bad") when they could agree on first order issues ("We all want to see a 10% decrease in robberies within 3 years.") This way a bad program would automatically expire if it didn't meet its goals, and the goals would have to be open enough that people could throw out bad laws ("Goal is to incarcerate 20% of the population" wouldn't pass, no matter what your political leaning).
- asgard1024 11y ago
  > One thing I've always wanted to see is a mandatory A/B test built into any new legislation This is what I find great about US and EU (I am from Czech Republic), that in a federal system, you can test the legislation in one state first, and then see the results and this lets the other states to perhaps adopt or avoid that approach. Unfortunately, strong moneyed interests break that model by lobbying strongly in Washington and Brussels from the start, which I think is a bad trend, but I have no idea how to prevent it. The effect of having more cultural diversity is very hard to quantify economically. (There is also other side of the coin though, which is too much state competition can lead to polarization and war.) Although sometimes there are things where you have to be careful about this approach - for example, if one state engages in lower taxes or pro-export policies, it may gain, but the same policy enacted by everybody will make everyone lose.
  - Retric 11y ago
    There are a lot of things which work better as federal programs vs state / local programs. Homeless populations spring to mind, if some city's treat there homeless population poorly they can 'export' the problem to another city / state. Where a state with overly generous benefits may see migration. Education is another with states that export there most educated population having little incentive to subsidize their higher education system. Which likely has lead to the dismantling of state funded schools as people became more mobile.
    - asgard1024 11y ago
      > Which likely has lead to the dismantling of state funded schools I disagree this is a necessary outcome. It may as well happen that backward states will take notice and improve their higher education system in various ways. But I think what you are saying is a fair point and I actually mentioned it. What I would add to my comment is the common solution to this is called "principle of subsidiarity" and it's used EU and Switzerland (which is another example of a federation that often tests different law variations in smaller scale first).
- nulltype 11y ago
  What if robberies decreased by 20% for some unrelated reason and the legislation increased robberies by 10%? It would still look like a 10% decrease overall.
  - Asbostos 11y ago
    I would think the law would have to be designed to be compatible with an experiment. Such as being phased in at different times in different places. You'd really have to design the experiment and set objective criteria for pass/fail before passing the law otherwise there'll be all kinds of interested groups distorting it.
- jhundal 11y ago
  Not sure how many folks are aware of this, but the ACA included funding for an 'Innovation Center' to run experiments on potential approaches to improve medical outcomes and cut costs: http://www.npr.org/sections/health-shots/2015/01/14/377192626/early-test-of-an-obamacare-experiment-posts-little-progress http://www.npr.org/sections/health-shots/2015/01/14/37719262... I think that the A/B tests you propose would be great, though there will be plenty of ideologues still finding excuses not to accept the results. Also, hi Nate! You probably don't remember, but you interviewed me sometime around December 2011.
  - tomjen3 11y ago
    We shouldn't experiment with sick people, at least with their informed and voluntary consent.
    - jhundal 11y ago
      I certainly agree for trials for treatments for specific ailments. But insurers, hospital administrators, and others have always been making decisions that balance clinical outcomes and cost of care without patients knowing such a decision has been made. The experiments I refer to are similar to those choices and typically the funding will go testing choices that haven't been made because they are more costly in the short run.
    - frabcus 11y ago
      It is unethical NOT to A/B test, because you may be doing the worst thing for all patients. You're right that it should be generally known this is done.
- susiecambria 11y ago
  I would like to quote your comment in a blog post I am doing about the test. With your permission, would quote the last two paragraphs - as an example of how the test results show change is needed and voila, here's a way to change things. Gr8 idea btw. A twist on sunsetting.
- michaelt 11y ago
  For certain types of legislation the A/B test itself would create a perverse incentive for opponents of the legislation to make the negative effects occur to get the legislation revoked. For example, if you said "we're going to put twice as many cops on patrol, but we'll revoke this if crime rates don't decrease" then a rational criminal would commit more crime, in order to get the law revoked. Or if you said "we're going to tax carbon emissions, but we'll revoke the law if domestic oil production drops by more than 20%" you've just invited oil companies to shut down their wells and wait you out.
  - JamesBarney 11y ago
    It is not in the criminals best interest to commit more crime. One criminal is probably only responsible for .0001% of the crime rate in any given city. So the incentive to commit more crime to revoke the law is infinitesimal. The disincentive created by the increased risk of incarceration would completely overwhelm the incentive to create more crime. This isn't true when the effects are measured in an industry with only several major actors. For example you decide to test out net neutrality to see what effects it has on broadband cost. So you create a bill that says "Net Neutrality unless broadband costs go up". Then Comcast could just increase their prices to deactivate the bill. But situations like these would be easy to stop and the exception.
dang 11y ago
But don't read the article before taking the quiz. It gives away one of the best answers.
- tomjen3 11y ago
  I don't think so, in fact that one was obvious, and unlike most of the others, believable.
ikeboy 11y ago
Maybe you can link to that at the end of the quiz?
robertwiblin 11y ago
If you liked this you may well also like our quiz on how to have the most social impact with your career: https://80000hours.org/career-guide/career-recommender/ https://80000hours.org/career-guide/career-recommender/
deleted 11y ago
[deleted]
bko 11y ago
I scored a 2/10. It's very interesting how this quiz plays so well into your political and personal biases. What I take away is reinforcing my bias that intentions and consequences are separate and that we should not conflate the two. However, I imagine many others see this and think "X is bad but Y is good. Let's do more of X and less of Y"
maho 11y ago
I enjoyed the quiz, but I am not a big fan of the positive_effect/no_effect/negative_effect classification, because it hides the error of the measurement. Instead, I propose to use a slider, like: > "In your estimate, how does the second response program influence the rate of repeat offenses?" > Underneath, a slider on a one-dimensional axis, with appropriate labels. For example, on the leftmost side: "50% increase in repeat offenses", and on the rightmost side "50% decrease in repeat offenses". > Clicking the "reveal" button superimposes the measured effect and its errorbar (this is important!) onto the slider. That way everyone can see if the measured effect is relevant, or if the measurement error is so large that the study might have missed a relevant effect. The downside of a slider is that the slider range will influence ("prime") the responses. But I doubt that it would influence the direction of the guess, so you could still extract useful information out of the reponses.
- BenjaminTodd 11y ago
  That would be better, though quite a bit more effort! Usually these studies only have enough resolution to say "probably worked", "no measurable effect", "probably negative", and even then are often wrong.

NateLawson 11y ago

Hey, this is really neat. It reminds me of the lesswrong.com community, where the goal is to figure out ways of offsetting cognitive biases. One thing I've always wanted to see is a mandatory A/B test built into any new legislation. That is, funding for a study after X years of the new law or program being in place would have to be part of the law. It could even including a threshold of the desired controlled effect of the law or else it is immediately revoked at that point. This way we'd force people to argue not about 2nd order issues ("I think mandatory sentencing is good" vs. "No, I think it's bad") when they could agree on first order issues ("We all want to see a 10% decrease in robberies within 3 years.") This way a bad program would automatically expire if it didn't meet its goals, and the goals would have to be open enough that people could throw out bad laws ("Goal is to incarcerate 20% of the population" wouldn't pass, no matter what your political leaning).

asgard1024 11y ago
> One thing I've always wanted to see is a mandatory A/B test built into any new legislation This is what I find great about US and EU (I am from Czech Republic), that in a federal system, you can test the legislation in one state first, and then see the results and this lets the other states to perhaps adopt or avoid that approach. Unfortunately, strong moneyed interests break that model by lobbying strongly in Washington and Brussels from the start, which I think is a bad trend, but I have no idea how to prevent it. The effect of having more cultural diversity is very hard to quantify economically. (There is also other side of the coin though, which is too much state competition can lead to polarization and war.) Although sometimes there are things where you have to be careful about this approach - for example, if one state engages in lower taxes or pro-export policies, it may gain, but the same policy enacted by everybody will make everyone lose.
- Retric 11y ago
  There are a lot of things which work better as federal programs vs state / local programs. Homeless populations spring to mind, if some city's treat there homeless population poorly they can 'export' the problem to another city / state. Where a state with overly generous benefits may see migration. Education is another with states that export there most educated population having little incentive to subsidize their higher education system. Which likely has lead to the dismantling of state funded schools as people became more mobile.
  - asgard1024 11y ago
    > Which likely has lead to the dismantling of state funded schools I disagree this is a necessary outcome. It may as well happen that backward states will take notice and improve their higher education system in various ways. But I think what you are saying is a fair point and I actually mentioned it. What I would add to my comment is the common solution to this is called "principle of subsidiarity" and it's used EU and Switzerland (which is another example of a federation that often tests different law variations in smaller scale first).
nulltype 11y ago
What if robberies decreased by 20% for some unrelated reason and the legislation increased robberies by 10%? It would still look like a 10% decrease overall.
- Asbostos 11y ago
  I would think the law would have to be designed to be compatible with an experiment. Such as being phased in at different times in different places. You'd really have to design the experiment and set objective criteria for pass/fail before passing the law otherwise there'll be all kinds of interested groups distorting it.
jhundal 11y ago
Not sure how many folks are aware of this, but the ACA included funding for an 'Innovation Center' to run experiments on potential approaches to improve medical outcomes and cut costs: http://www.npr.org/sections/health-shots/2015/01/14/377192626/early-test-of-an-obamacare-experiment-posts-little-progress http://www.npr.org/sections/health-shots/2015/01/14/37719262... I think that the A/B tests you propose would be great, though there will be plenty of ideologues still finding excuses not to accept the results. Also, hi Nate! You probably don't remember, but you interviewed me sometime around December 2011.
- tomjen3 11y ago
  We shouldn't experiment with sick people, at least with their informed and voluntary consent.
  - jhundal 11y ago
    I certainly agree for trials for treatments for specific ailments. But insurers, hospital administrators, and others have always been making decisions that balance clinical outcomes and cost of care without patients knowing such a decision has been made. The experiments I refer to are similar to those choices and typically the funding will go testing choices that haven't been made because they are more costly in the short run.
  - frabcus 11y ago
    It is unethical NOT to A/B test, because you may be doing the worst thing for all patients. You're right that it should be generally known this is done.
susiecambria 11y ago
I would like to quote your comment in a blog post I am doing about the test. With your permission, would quote the last two paragraphs - as an example of how the test results show change is needed and voila, here's a way to change things. Gr8 idea btw. A twist on sunsetting.
michaelt 11y ago
For certain types of legislation the A/B test itself would create a perverse incentive for opponents of the legislation to make the negative effects occur to get the legislation revoked. For example, if you said "we're going to put twice as many cops on patrol, but we'll revoke this if crime rates don't decrease" then a rational criminal would commit more crime, in order to get the law revoked. Or if you said "we're going to tax carbon emissions, but we'll revoke the law if domestic oil production drops by more than 20%" you've just invited oil companies to shut down their wells and wait you out.
- JamesBarney 11y ago
  It is not in the criminals best interest to commit more crime. One criminal is probably only responsible for .0001% of the crime rate in any given city. So the incentive to commit more crime to revoke the law is infinitesimal. The disincentive created by the increased risk of incarceration would completely overwhelm the incentive to create more crime. This isn't true when the effects are measured in an industry with only several major actors. For example you decide to test out net neutrality to see what effects it has on broadband cost. So you create a bill that says "Net Neutrality unless broadband costs go up". Then Comcast could just increase their prices to deactivate the bill. But situations like these would be easy to stop and the exception.

dang 11y ago

But don't read the article before taking the quiz. It gives away one of the best answers.

tomjen3 11y ago
I don't think so, in fact that one was obvious, and unlike most of the others, believable.

ikeboy 11y ago

Maybe you can link to that at the end of the quiz?

robertwiblin 11y ago

If you liked this you may well also like our quiz on how to have the most social impact with your career: https://80000hours.org/career-guide/career-recommender/ https://80000hours.org/career-guide/career-recommender/

deleted 11y ago

[deleted]

bko 11y ago

I scored a 2/10. It's very interesting how this quiz plays so well into your political and personal biases. What I take away is reinforcing my bias that intentions and consequences are separate and that we should not conflate the two. However, I imagine many others see this and think "X is bad but Y is good. Let's do more of X and less of Y"

maho 11y ago

I enjoyed the quiz, but I am not a big fan of the positive_effect/no_effect/negative_effect classification, because it hides the error of the measurement. Instead, I propose to use a slider, like: > "In your estimate, how does the second response program influence the rate of repeat offenses?" > Underneath, a slider on a one-dimensional axis, with appropriate labels. For example, on the leftmost side: "50% increase in repeat offenses", and on the rightmost side "50% decrease in repeat offenses". > Clicking the "reveal" button superimposes the measured effect and its errorbar (this is important!) onto the slider. That way everyone can see if the measured effect is relevant, or if the measurement error is so large that the study might have missed a relevant effect. The downside of a slider is that the slider range will influence ("prime") the responses. But I doubt that it would influence the direction of the guess, so you could still extract useful information out of the reponses.

BenjaminTodd 11y ago
That would be better, though quite a bit more effort! Usually these studies only have enough resolution to say "probably worked", "no measurable effect", "probably negative", and even then are often wrong.